I am observing frequent yet not 100% repeatable cases where a console command aimed at removing all items from a given list (remove [find] to be exact) gets stuck while executing. The following is true:
Happens to different collections of items, e.g. /ip/dhcp-server remove [find], /interface/bridge remove [find] and so on.
Happens both when the target collection of items is non-empty and empty.
Happens both in the script as well as a standalone call in the terminal window, but seems like it happens more often when running from a script.
Happens in about 20% of calls.
Sometimes never completes, sometimes takes very long time to complete, but still finishes, sometimes takes long time and then exits with a timeout message.
RouterOS 7.14.3 (most current stable version)
Hence the questions:
Why such a trivial call would get stuck, especially in non-deterministic manner? Is there a way to prevent it from getting stuck by performing some preparatory actions?
Is there another, more reliable way to remove all items from a given collection that would not cause the execution to get stuck?
Update 1:
Upgraded to recently released 7.15.1, same problem, same place.
Happens on multiple CRS354-48P-4S+2Q+ switches (MIPSBE-based, I own 11 of them)
Never happens on CCR2004-1G-12S+2XS and CCR2216-1G-12XS-12XQ routers (both ARM64-based)
Can be reproduced even without remove command, it is [find] that gets stuck (for example, :put [find])
print gets stuck too when called from the script, similarly to :put [find]
If the script gets stuck and then interrupted, running it again always results in getting stuck again
Happens specifically to /interface/bridge and /ip/dhcp-server collections
This is gotta be some kind of race-condition-based bug in MIPSBE version of RouterOS. Can anybody from MikroTik debug this?
Update 2 (solution): I solved the problem, here is TLDR (full solution in the thread below)
Only happens on MIPS-based switches
Happens when you are trying to delete a bridge configuration that uses interface lists to specify ports
Removing interface list containing a lot of ports (like 48 for example) from the bridge port list just prior to attempting to remove the bridge itself causes an unexplained spike in CPU load to 100% that lasts for several minutes after apparent completion of the operation (which will effectively hang subsequent attempt to remove the bridge itself)
The way to avoid this problem is to remove the bridge from the bridge list first, and only then dismantle that bridge’s ports by removing interface list from the bridge port list
Basically shut down the bridge first, and then take care of its ports, do not manipulate ports on active bridge
I doubt it is because of error while removing dynamic objects because it will stop on error, not stuck or go in timeout. Could be device performance issue or storage corruption.
Regarding removing dynamic objects and error handling with retry (but it doubt that’s the case here), it can be solved with simple script function:
:local retryCmdOnError do={
:if (retryCnt = null) do={ :set $retryCnt 1 }
:onerror err in={ $cmd } do={
:if ($retryCnt < $maxRetry) do={
:return [$retryCmdOnError retryCmdOnError=$retryCmdOnError cmd=$cmd retryCnt=($retryCnt + 1) maxRetry=$maxRetry]
} else={ :error $err }
}
}
# edit cmd function with command(s) for error handling
:local cmd do={ <some_cmd> }
# adjust maxRetry argument if needed to max amount of retries on error
$retryCmdOnError retryCmdOnError=$retryCmdOnError cmd=$cmd maxRetry=5
I am only trying to clear simple static lists, so no, it is not related to lists where items can disappear on their own in the middle of command execution like described in the post you are referring to.
Furthermore, in cases where the list can contain dynamic or built-in items, I am taking additional precautions like:
/interface/bridge/port remove [find where !dynamic]
/interface/list remove [find where !builtin]
However, where it does gets stuck are simple lists where only static, previously manually added items can be present:
When it gets stuck, it shows 1, then never shows 2.
Other observations:
If I interrupt (Ctrl-C) stuck script, then run it again, it still gets stuck in the same place every time.
If I interrupt stuck script, then execute :put [find] as a standalone command, then run the script again, it does not get stuck.
I own 11 identical switches and it gets stuck on all of them in the same place (/interface/bridge and /ip/dhcp-server), so no, it is not a hardware problem of a specific switch.
Is this happening in /system/script or via the CLI.
If it’s the CLI, a local is gone upon the next prompt - so it’s nil [:nothing]. You’d need some curly braces at CLI:
/interface/bridge/vlans { :local x [find]; remove $x }
You may also want to qualify the “find” as things can be weird in scripting sometimes. So if you mean /interface/bridge/port/find … use the full form to see if it works then.
The hang is just weird. If same script works on ARM, does sound like a bug.
Using a :local in a :import script without it being inside {} might be allowed/okay, but still not my recommendation. Does doing just a plain:
:put [/interface/bridge/find]
at CLI also hang?
The only thing to try is add some condition to the find, like adding as [find name~“*”] which match all names. Perhaps it needs some filter - it shouldn’t, but that might avoid the hang (or at least be another data point for a bug report).
One reason I could think of is that it is related to the storage, not the architecture.
The CCR2004 has a lot of storage and so it never runs out, but some other devices have only 16MB of flash and it can easily fill up especially with RouterOS v7.
Remember that when you remove something, it isn’t really freeing up space in the storage.
The config is stored in a set of database-like tables (not really a database, more like what ancient MS-DOS programs used, where each table is a separate set of data and index files).
When you repeatedly add and delete config items, the size of this “database” is growing all the time, and at some point there may be issues.
Maybe you encounter an issue because you regularly run scripts that cleanup and re-create config items?
Unfortunately there is no command to “compact the database”, as there usually is for environments like this.
(which would take all active records and put them in the smallest possible file)
Thank you for your insight. One possible counterargument to this theory is that if the cause was the lack of space, then I would expect a consistent, deterministic behavior - once you run out of space, it always should malfunction. Yet what I observe is that the script gets stuck only sometimes, not all the time. However, I will now test this theory by performing a complete device wipe (storage format) using NetInstall in order to see if after the wipe the problem would be gone for a while. Will report on results.
Yes, it does occasionally, but in my observation it happens much less frequently. Roughly speaking, the script hangs in about 20-40% of runs any day of the week, but a plain CLI single-liner hung at most twice in the last month that I was trying to debug this problem.
Yes, gets stuck too:
:put "Resetting device configuration"
# Clear all the lists
:put " Clearing bridge port list"
/interface/bridge/port
:local ids [find where !dynamic]
remove $ids
:put " Clearing bridge list"
/interface/bridge
:put " 1"
:local ids [/interface/bridge/find where name~"*"]
:put " 2"
remove $ids
:put " 3"
Results in:
Resetting device configuration
Clearing bridge port list
Clearing bridge list
1
interrupted
(here “interrupted” line is a result of me pressing Ctrl-C after waiting for a long time)
OK, I think I am getting somewhere. At least I am now able to reproduce the problem deterministically.
Steps to reproduce (for your convenience I have automated most of these steps in the script at the end of the message):
Need to use MIPS-based CRS354-48P-4S+2Q+ switch (you won’t reproduce the problem on ARM64-based routers)
Connect to the device via MGMT Ethernet port (ether49) using WinBox
Create custom interface list and add all 48 Ethernet ports to that list as list members (this can be automated in a script). Do not include ether49 to this list unless you want to suffer from your WinBox connection being reset in the next step.
Create a bridge and add previously created interface list to its ports list (this can be automated in a script)
Open /System/Resources GUI dialog to monitor CPU load (no, WinBox dashboard CPU load gadget does not faithfully report true CPU load, so do use /System/Resource GUI dialog!)
Ensure that CPU load in GUI dialog is not 100%. If it is, do not proceed until it settles down.
Run a script that does the following in one shot: first removes all non-dynamic items from /interface/bridge/port and then removes all items from /interface/bridge:
The script will get stuck in its second part (removing all items from /interface/bridge). To be more exact, it will get stuck just finding all such items.
Observe CPU load - it will now be 100%
Interrupt the script by pressing Ctrl-C
Observe CPU load - it will still be 100% for a few minutes.
Without waiting for CPU load to drop down, run the script again - it will get stuck again in exact same place - attempting to read a list of /interface/bridge/. It will be getting stuck every time you keep running it while the CPU load is still 100%. Interrupt the script by pressing Ctrl-C.
Go get some tea/coffee and wait for the CPU load to drop down from 100% back to normal values. It will take several minutes.
Run the script again - this time it won’t get stuck. The difference this time - it has nothing to delete in /interface/bridge/port anymore in the first portion of the script, so the CPU remains unloaded and proceeds to the second step where it does not get stuck anymore.
Conclusion:
It takes a combination of the following for the script to get stuck:
Loading the CPU with removing one non-dynamic list-type item from /interface/bridge/port, which in turn causes an automatic removal of 48 dynamic items.
And then immediately proceeding to an /interface/bridge [find] call which gets stuck.
Important caveats:
Notice that if /interface/bridge/port is already empty, there will be no spike in CPU load and the script will not get stuck
Notice also that when /interface/bridge/port is not empty, control returns immediately and successfully from removal of the /interface/bridge/port and corresponding GUI window shows immediate and successful disappearance of all items from this list (both static and dynamic), yet the CPU remains overloaded for a while after this operation and can’t correctly handle subsequent operations.
As a matter of fact, you don’t even need a script to reproduce a CPU overload - you can simply remove one non-dynamic list-type item from /interface/bridge/port in GUI and watch the CPU go into 100% overload for the next two minutes for no particular reason (while all items, both dynamic and static are immediately gone from the list).
Here is a unified script that will reproduce both setup and the problem (it won’t get stuck first time you are running it because it first cleans up and then sets up, so run it twice).
At this point I have done everything I could to deterministically reproduce the problem, I think now it is time for someone from MikroTik to fix it as soon as Līgo is over.
Well, learn something new. I didn’t know …/bridge/ports could be an interface-list.
The combo of MIPSBE (which I’d imagine Mikrotik tests less these days) & “dynamic” bridge ports (which I wouldn’t say is common, albeit seemingly useful) does let itself to a bug someplace…
You should open a case at help.mikrotik.com, attach a supout.rif from the CRS. You may have already done this.. but you can also try a netinstall – @pe1chl is right, “stuff” left in filesystem from by past upgrades might be messing this up.
OK, as I was promising, I have tried a complete wipe using NetInstall - observed the software reporting “formatting a harddrive” and then installing the latest RouterOS version (7.15.1). Then ran the script I have listed in the previous message - it still gets stuck right from the first attempt. So no, it is not a problem of “leftovers” in the filesystem. It is something that fundamentally overloads the CPU every single time.
Wow, I might have just found a workaround to this problem:
Turns out if you do removal in opposite, counterintuitive order (bridge first, then bridge ports), nothing gets stuck and the CPU does not go into two minutes of overload.
Even if you do not do any scripting and merely using WinBox GUI, this equally applies to you as well. So the rule is: When removing the bridge configuration made with interface list, first remove the bridge itself, and only then clean up the bridge ports assignment. If you remove bridge port assignment first while the bridge is still active, this will cause about two minutes of CPU doing who knows what at 100% load factor after seemingly successfully completing the operation.
Huge thanks to @Amm0 and @pe1chl for pointing me in several productive directions!
It may be that there is some form of locking that is not working completely right.
Of course you can consider it a bug and it would be good to report it in the ticket system.
When the first command causes CPU loading that continues past the end of command execution, at least the software should wait for completion of that before it starts the next command.
You would still be surprised that it takes long, but at least it would not really hang.
That being said, remember that those processors have quite low performance. The switch itself can do wirespeed forwarding because the hardware is doing it, but the CPU is not really up to par to handle such large (48) amounts of items.
The designers probably assume that you would not perform actions like this on a regular basis.
And probably you should not. I can imagine that you run into this when you have a couple of those switches and want to change the configuration on all of them to conform to some new ideas, and make a clever script to do it, but in normal operation it is not a good idea to do this.
(i.e. you should not plan a setup where you need to do this every day, for example)
Yes, indeed I am writing a configuration script that is never run during normal operation of the switch as I fully understand that the MIPS CPUs in switches are not meant for any real-time processing. That being said, even configuration scripts can not afford to get stuck forever, as it renders them useless.
You are also correct in saying that if certain action takes long time, it should not return control earlier than the action is completed so that at least we know where we are in the process.
At the same time, since I found a way to avoid this horrible “submarine” delay by simply swapping the order of operations, it is clear to me that much faster speed for this given operation is achievable with the same CPU in principle. My current working theory is that when I remove ports from the active bridge (which causes horrible “submarine” delay), the switch CPU is somehow re-enumerating all the remaining active ports after deleting each of 48 dynamic entries, which causes quadratically progressing delay (48+47+46+…+1 =(n^2+n)/2). When I reverse the order and remove the bridge first, the bridge becomes inactive and does not re-process each remaining port during removal of each port. So while my workaround solves the problem, there is clearly some underlying deficit that causes this unnecessary workload. And even if this workload if architecturally unavoidable while the bridge is active (and therefore we should always remove the bridge first), it certainly should not do it asynchronously by quickly exiting and then doing something in the background for 2 minutes, it should do it synchronously by not returning control until everything is done.