I am working on a big set of lte devices.
We need to monitor the quality of the LTE interference and quality over time.
I have seen the next doc:
GPS-tracking using HTTP POST - RouterOS - MikroTik Documentation
And I am considering to publish a similar for the LTE db.
The options are POST or the rest api scrapping.
I was thinking about using some kind of json documents storage like mongo.
Are there any interested in such an example?
Can't you use SNMP for that ?
And log to like Opensearch
@patrikg I need the data to be readable by software.
Currently what I am doing, I am dumping the lte monitor using the rest api.
It's json and it's the best format as far as I can tell to store the data.
I am thinking about using mongo.
I don't need it to be "searchable" in the fastest method.
I just need to store the data of 1k + LTE devices for 365 days in intervals of 10-2 minutes.
Later on, I will retrieve the data to understand the changes per device over time and season of the year.
JSON was never the best format for storing the data. It's only the most convenient format when some kind of PHP is used to visualize that data (and then you need some kind of object storage on top of it to make searches possible, such as mongoDB or elastic search or some such). And you're talking about millions of data points, storing milions of files is not exactly trivial for any of file systems.
One of best (if not the best) means of storing scalar data is SQL (which comes with strong search/filter/aggregation capabilities out of the box) ... and then it's possible to build an API (if it doesn't exist already) to read/store it to/from JSON if that's the most convenient way. There are at least two very decent open source possibilities (PostgreSQL and MariaDB... in order of capabilities but inversely so in order of complexity). Many of modern developers dismiss SQL just because it's "old school"...
There is always The Dude. See
and the data goes to SQLite, which is exportable to CSV using dude.exe
@mkx The LTE data is not "static" in it's format, and due to this there are two options to my understanding.
Or we will include all available fields in the sql table or we would use a json doc.
I probably do not understand how to use SQL ie MariaDB or PostgreSQL to store these documents which from what I have seen do not have a static format.
There is always the option to store JSON in PostgreSQL but I prefer to use PostgreSQL only if I do not have any other option.
@Amm0 The Dude doesn't scale enough even on beefy hardware for this function.
I have tried to use The Dude for simple linux VMs and it just couldn't work good enough.
Eventually I have used nagios and when this worked I wrote a customized service for specific tasks which the generic ones couldn't handle with the same CPU+RAM+DISK.
Currently I am using a combination of a custom service combined with zabbix.
Even Prometheus had some issues and missing features for generic use on more then modest hardware.
I guess my question is what's different about LTE data that's giving you trouble? It sounded like you want one-off solution for LTE. If you already I have an NMS, I'm confused at the question...
e.g. SNMP poling works for the common metrics like RSRQ/RSRP/etc., which should pretty easy to wire up as any other SNMP things your monitoring. With the caveat to note from my Dude probe that the SNMP OID for the LTE interface can vary depending on device, so you need the know the index of the LTE interface
If the question is formats... One option is record the raw data as NDJSON (JSON where each line is independent record) or Apache Parquet or even CSV, and then use something like DuckDB to process any desired visualization. Or use NDJSON/Parquet/CSV to store collected records, then sweep them into a "big" database as separate process, as the two-step process so you seperate the monitoring from the storage.
@Amm0 I haven't slept for too many hours now and my head giving me trouble when trying to understand the written words.
I will try to respond later on.
SNMP is a very good protocol, however, it's UDP based and in this scenario we need retry and couple other things which only TCP might give us.
It's not a my bottom line yet. but, this is where I am at now.
Fair enough. Sometime MikroTiks give me a headache too ;).
From an efficiency POV, using the RouterOS native API may work better than REST API. Since has a /listen and a small Python/JS/etc script can process a continuous stream of /interface/lte/monitor data. And you can insert the collected data into whatever you like. I suspect some AI could write the script to do the RouterOS native API.
But using REST API with LTE is a bit tricky since you have to using duration so that it collects the various monitor items before returning them. Using { ..., "once": true} will return some random JSON elements since the various internal polling of modem does not always collect data on the same schedule... so the once may return irregular things. So { ..., "duration": "10s"} may provide better results.