I am working on a big set of lte devices.
We need to monitor the quality of the LTE interference and quality over time.
I have seen the next doc:
GPS-tracking using HTTP POST - RouterOS - MikroTik Documentation
And I am considering to publish a similar for the LTE db.
The options are POST or the rest api scrapping.
I was thinking about using some kind of json documents storage like mongo.
Are there any interested in such an example?
Can't you use SNMP for that ?
And log to like Opensearch
@patrikg I need the data to be readable by software.
Currently what I am doing, I am dumping the lte monitor using the rest api.
It's json and it's the best format as far as I can tell to store the data.
I am thinking about using mongo.
I don't need it to be "searchable" in the fastest method.
I just need to store the data of 1k + LTE devices for 365 days in intervals of 10-2 minutes.
Later on, I will retrieve the data to understand the changes per device over time and season of the year.
JSON was never the best format for storing the data. It's only the most convenient format when some kind of PHP is used to visualize that data (and then you need some kind of object storage on top of it to make searches possible, such as mongoDB or elastic search or some such). And you're talking about millions of data points, storing milions of files is not exactly trivial for any of file systems.
One of best (if not the best) means of storing scalar data is SQL (which comes with strong search/filter/aggregation capabilities out of the box) ... and then it's possible to build an API (if it doesn't exist already) to read/store it to/from JSON if that's the most convenient way. There are at least two very decent open source possibilities (PostgreSQL and MariaDB... in order of capabilities but inversely so in order of complexity). Many of modern developers dismiss SQL just because it's "old school"...
There is always The Dude. See
and the data goes to SQLite, which is exportable to CSV using dude.exe
@mkx The LTE data is not "static" in it's format, and due to this there are two options to my understanding.
Or we will include all available fields in the sql table or we would use a json doc.
I probably do not understand how to use SQL ie MariaDB or PostgreSQL to store these documents which from what I have seen do not have a static format.
There is always the option to store JSON in PostgreSQL but I prefer to use PostgreSQL only if I do not have any other option.
@Amm0 The Dude doesn't scale enough even on beefy hardware for this function.
I have tried to use The Dude for simple linux VMs and it just couldn't work good enough.
Eventually I have used nagios and when this worked I wrote a customized service for specific tasks which the generic ones couldn't handle with the same CPU+RAM+DISK.
Currently I am using a combination of a custom service combined with zabbix.
Even Prometheus had some issues and missing features for generic use on more then modest hardware.
I guess my question is what's different about LTE data that's giving you trouble? It sounded like you want one-off solution for LTE. If you already I have an NMS, I'm confused at the question...
e.g. SNMP poling works for the common metrics like RSRQ/RSRP/etc., which should pretty easy to wire up as any other SNMP things your monitoring. With the caveat to note from my Dude probe that the SNMP OID for the LTE interface can vary depending on device, so you need the know the index of the LTE interface
If the question is formats... One option is record the raw data as NDJSON (JSON where each line is independent record) or Apache Parquet or even CSV, and then use something like DuckDB to process any desired visualization. Or use NDJSON/Parquet/CSV to store collected records, then sweep them into a "big" database as separate process, as the two-step process so you seperate the monitoring from the storage.
@Amm0 I haven't slept for too many hours now and my head giving me trouble when trying to understand the written words.
I will try to respond later on.
SNMP is a very good protocol, however, it's UDP based and in this scenario we need retry and couple other things which only TCP might give us.
It's not a my bottom line yet. but, this is where I am at now.
Fair enough. Sometime MikroTiks give me a headache too ;).
From an efficiency POV, using the RouterOS native API may work better than REST API. Since has a /listen and a small Python/JS/etc script can process a continuous stream of /interface/lte/monitor data. And you can insert the collected data into whatever you like. I suspect some AI could write the script to do the RouterOS native API.
But using REST API with LTE is a bit tricky since you have to using duration so that it collects the various monitor items before returning them. Using { ..., "once": true} will return some random JSON elements since the various internal polling of modem does not always collect data on the same schedule... so the once may return irregular things. So { ..., "duration": "10s"} may provide better results.
1000 devices × 2 min ≈ 260M rows/year — that's squarely TimescaleDB territory (Postgres extension, so you keep SQL and every existing tool). Three things that matter at that scale: (1) hypertable partitioned by time, (2) native compression after 7 days — LTE metrics compress 10-20x because consecutive samples barely change, (3) a retention policy so you never think about it again. On the "LTE data has variable formatting" worry: promote the ~8 stable fields (rsrp, rsrq, sinr, cell-id, operator) to real typed columns and keep a jsonb column for the rest. You get indexed queries on what you actually chart, and nothing is lost. Continuous aggregates give you the 1-year dashboards without touching raw rows.
One gotcha from polling MikroTik at scale over the native API (8728) rather than REST: if a command leaves a !trap/!done unconsumed, the socket desyncs and every later response is shifted by one — you get plausible-looking wrong values instead of an error. Always drain to !done before reusing the connection. Cost me a while to find.
I guess I've been lost in what the actual question is here. The choice of DB is not different from choices like what hardware or OS you're going to use. The answer is "it depends".
And the premise that there is no existing monitoring system and no existing DBs in use in your org, as LTE/5G data is someway no different than another variable you might want to monitor in a network.
But on the theoretical,
Perhaps, at 1000 * 2min * ~5-10 data points. But begs the question do you really care to know if 2 years ago there was one reading of RSRP of -120 for a two minute period. And nearly all existing monitoring solution have some solution to roll or summarize older tracked data, and also have even more refined schemes for queuing monitored data than a generic DB where you'll be left figuring out the JOINs and table schemas.
So if you really wanted a DB solution designed for LTE/5G monitoring... you'd want both time-series and geospatial queries... since ideally your DB know about towers, can could join cell-id/etc with the tower at the point in time the RSRP/RSRQ/etc are collected. And likely more dimensions like band# and allocated channel sizing...
On the Native API...
I said "From an efficiency POV", not a complexity POV... Since you imply per device polling is only every 2 minutes, if all your doing is connecting every two minutes using Native API and disconnecting the socket, the complexity may not be worth it. Where the Native API shines is:
/listen method works similar to how WinBox updates the various UI elements, so custom code can mimic the same updatingtag= and your client parses the tags back out to match the requests as the data comes back ... but the is more complex than mere needing to wait for a !done (or !trap).So for simple polling the RSRP/etc...native api may be more trouble than it worth, since all you save is the JSON wrapping (and does not actually save you're own code having to parse the API "sentences", instead of a common JSON parser)
@monitorwisp Specifically I do not feel comfortable with Postgres.
Native API is nice, but, for me it's much simpler to use the rest API since I have the infrastructure to use it easily.
I need to do some homework!!
I am thinking about trying to do something with Prometheus since I have more experience with it until now.
So the options for my homework are:
SQL is great to my eyes.
It's just that I have a range of LTE modems on the devices (5+) and the output is not consistent from what I have seen.
Since I have access to data from about 100 device now I will try to see what is the best approach that seems to me and later on I will try to update about it.
SQLite?
Fair enough — if you're not comfortable with Postgres, don't use it. A stack you can't debug at 3am is worse than a theoretically better one.
For your constraints (REST collection, variable schema, write-heavy, read-rarely, 365 days), two options that aren't Postgres:
ClickHouse — columnar, handles JSON natively, and 260M rows is genuinely small for it. Closest thing to "just works" for this shape of data.
NDJSON or Parquet + DuckDB, as Amm0 suggested. You're not querying live, you're doing retrospective seasonal analysis — that's exactly what DuckDB is for, and there's no server to operate.
On SQLite: it'll hold the rows fine, but 1000 devices writing concurrently is where it hurts — single writer, so you'd be fighting lock contention rather than storage.
One thing worth flagging that isn't about the database at all: 1000 devices every 2 minutes over REST is ~8 requests/second sustained, each one a fresh HTTPS handshake against a router CPU. In my experience that's where these setups actually fall over — not on storage. Whatever you pick, decouple collection from writing with a queue, so one slow router doesn't stall the pipeline.
Why 1000 devices writing to one database? Shouldn't monitoring tool get data and write it to database? Do you assume that each device writes independently?
That looks promising: https://duckdb.org/2026/08/18/reconciling-json
Can't resist chipping in: MongoDB has a timeseries feature since version 5.0. Really nice for this sort of data: the data points are automatically bucket-ized, so indexes don't explode, and a columnar storage engine is used.
This is what I'd choose, though all of the others mentioned here are totally fine as well.