I am trying to edit my routers firewall address-list via the REST API.
So far I have managed to get everything running via PUT & DELETE but those methods are rather slow for adding and removing 180k entries.
Can anyone shine a light on how to use the POST command for that?
I am struggeling at Postman as either the router is not accepting it due to “invalid JSON” or “missing =list=”.
Unfortunately I was only able to find how to add a single address at the forum, but not tons of them.
If there is an intend to add some entries and then after time remove them… You can set an expiration when adding the entry, that save having to delete them.
To use POST, it’s still a two step operations. First, you still need to get the list of id from GET (*or POST …/address-list/print). You can optimize this part by adding a “.proplist” in JSON request with only “.id”. And, then to remove them, you provide the list of .id in the POST …/remove. NOW… the tricky part is POST wants a comma-separated list of .id in its JSON — not a JSON array of .id. And it’s this latter step to get a “CSV” to provide as String type to POST’s .id field.
So I suspect you’re trying to provide an “array of .id” in your call to POST /rest/ip/firewall/address-list/remove — when it wants a comma-separated string of .id. The reason for this is POST is more a “wrapper” over the lower-level API & that deals in strings. Now it likely should do that for you IMO, but does not.
Note: .id is a string with comma separated values.
I actually cannot say if providing all .id in a POST …/remove is that much quicker than a loop over the simple DELETE. But there is also another consideration when using POST to remove — it is NOT transactional. So if you provide an already deleted .id or bad value, it will stop processing the rest of the list, and return a 404 at that point. Anything already delete is deleted, anything after the missing .id would not be deleted.
I think you’re likely better off using the built in TTL mechanism vs ANY scripting — if you can. Maybe one POST with a 180K .id is quicker, but even that I’m not necessarily sure & provided a string list of 180K records may have different problems for REST API. So… if you need remove, a loop over DELETE give you feedback if anything was not removed, even if it takes more time.
And to be clear, .query does NOT work on a /remove. So this DOES NOT WORK:
curl … https://…/ip/firewall/address-list/remove —json ‘{“.query”: “list=mylistname”}’
But that’s the command you may be trying to find. And it doesn’t not work like that.
Thanks @Amm0 for your suggestions.
In general deleting works pretty okay for me.
My intention was more into uploading a list to the router via the API and wait until he finished with the logic.
So far I was able to to get it running at a proper speed but this comes at some costs:
Running a list with native ROS script language costs around 30 % CPU on my CCR2004.
Compared to that with a more or less equal speed at adding entries via the REST API with PUT it maxes out to ~90 % CPU utilization.
Not sure if this is OK or if I am torturing my device.
What specific commands are you running? In general you can model them with REST POST.
The main difference is when using REST is each request is a new connection, so >100K you hitting router with that many TCP connections, and it’s having to do a lot of more string parsing than CLI would.
The other approach be to put your add/remove commands into is a /system/script via , then run it. So you could use REST to PUT a script, then POST /system/script/run as alternative.
Every new IP is being written to an array and every IP which is no longer on the new list (= to be deleted) is being written to another array.
Looping through the array to add new IPs:
POST https://router/rest/ip/firewall/address-list/add with a JSON body including {“address”: ip, “list”: list_name}.
So one request per new IP.
Cost ~90 % CPU
Initial I tested a script found on Github (https://github.com/multiduplikator/mikrotik_blocklist) which also does the job but only on the router itself via the scripting language.
It loads all IPs into a global array and comparse it locally on ROS. Then it adds or deletes IPs.
This script consumes around 30 % CPU but takes sometimes long.
My intention was to do the comparing on a dedicated device (VM, Raspberry PI, WSL,…) to do the heavy job and then simple tells ROS “add this IPs and delete those”.
Well at the end of the day, processing a lot of records is going to use the CPU. The only other thing to do if you want to “optimize” is use the native API. Since you’re using Python, Mikrotik has a “library” (well a class, with an example main) here: https://help.mikrotik.com/docs/spaces/ROS/pages/47579209/Python3+Example.
I don’t necessarily think you’re going to see a night-and-day difference… But likely at least some over REST since API will skip the REST JSON to the native API =list=mylist string, since you’re Python will do that. Also with API the 100K requests can be on same connection/socket, so you’ll save open/closing 100K sockets by using API, over REST. More work however.
Thanks but I wanted to explicit use the REST API as GET, POST, and so on are common operations and are easier to understand compared to a dedicated API.
I am just curious why there is such a big difference between local script execution with Mikrotiks scripting language vs a REST API (~ 60 % CPU difference).
Is there any place where I may share my scripts so other may or may not use them?
With these large lists, I can only guess at the why script would be fast or slow. But with REST API each call has to both get authenticated and JSON is even more string parsing. That’s overhead that both API and scripting don’t have, since auth is done once.
Some speculation… With some RSC script with 100K list of /ip/firewall/address-list/add’s that runs from /system/script will be parsed once from script, than executed, with auth being aligning the UNIX user id. If you want to get really dorky, scripting, config and CLI all all get converted some “s-expression like thing” that scripting typeof call “(code)”:
So I view the (evl (())) things as the lowest level before any user thing hits RouterOS C code. And when you “run” a script it’s first converted into a very long (evl (()())) as the first phase, and that’s whole thing is handed off to just run. NOW… when you call REST… track your TCP, check /users each time, convert the JSON into the API “string sentences”…and API likely has same/similar string process to get a similar IL like (evl ()()). Basically with REST, you’ll add 100K auth calls, and 100K*2 (200K) extra string transformations when starting at REST.. API get you to 1 auth call, and 100K extra string transformation. CLI would also have a similar amount string transformation (once, so 100K), except it’s only parsing it once, not per request (or API sentence) so there is no thread context switches when using /system/script.
Now… the corollary here… you may not want to drive the CPU to 100%… so may want to throttle anyway to avoid the CPU hit that could impact routing.
In terms of posting your code, feel free to do it here. Or, just start a new topic like “Scripting address-list with Python with example”, or whatever, if you want a more organized presentation of it.
There is an aggregated list of IPs and IP ranges which may be blocked due to “bad” traffic: https://github.com/stamparm/ipsum
My intention was to import the list and keep it up to date in an easy manner
@An5teifo Generally calling API requests as replacement for ROS script which performs many config updates is bad idea, especially using many API requests in loops. Difference in CPU load is because there is much more overhead on API request vs local command, CPU is used for firewall, networking, API service processing - parsing JSON metadata, executing local command, process it and return response.
In you case you can create hybrid solution like:
fetch current address list with API request and compare it against new list
generate ROS script file with commands for adding and deleting IPs into address list by difference from 1st step
upload script file into ROS (for eg. using SFTP) and perform POST /import API request for uploaded script. It is also possible to execute ROS script automatically after upload without calling API, then script file must be named in format “[someting].auto.rsc”
My current workflow is to download lists on a device, aggregate them into a single big file which is being a RouterOS array and upload it to my Git repo.
On my routers I am using an adopted script from here https://github.com/multiduplikator/mikrotik_blocklist (2c - fancy list stuff).
This works so far okay.
My original intention was to do the heavy job of comparing IPs from two different lists rather on a utility device instead of my router.
I understand, hybrid solution will work in this case, comparing lists and generating new one can be done outside ROS (CPU load will not be on ROS device), but importing it should be done with ROS script with only add and remove address list commands in it to avoid heavy CPU load on ROS device by performing many API requests.
Yes, because there will no be additional logic in ROS script for processing lists to create aggregated list, just commands for adding / removing address list items.
@optio: This solutions is the most performant one!
My “mikrotik util” VM will download the existing list via REST API from the device, parse the IP adresses & ranges, compares it with my new list and simply create an add or remove command.
The command list is being pushed to my git repository from where my router will download and execute it.
