hi im using a script to get mac addr from username in usernamager and put it to bridge nat like this
:log info ("Script Started Please wait....");
:foreach macaddr in=[/tool user-manager user find] do={
:local mac [/tool user-manager user get $macaddr username];
:local rule [/interface bridge nat find where src-mac-address="$mac/FF:FF:FF:FF:FF:FF"]
:if ($rule = "") do={
:execute "/interface bridge nat add chain=dstnat in-bridge=1-Bridge-Neotel src-mac-address=$mac/FF:FF:FF:FF:FF:FF action=accept";
:log info ("Rule for mac-address $mac does not exist. Adding new Bridge nat Rule for mac-address $mac");
}
}
:local ruleid [/interface bridge nat find where src-mac-address="00:00:00:00:00:00/00:00:00:00:00:00"];
:execute "/interface bridge nat move $ruleid";
:log info ("Moving Block rule at the end");
:log info ("Adding Rules to Bridge Finished");
the script was working fine until users build up. Now with 1200 users it takes 20min to finish.
Is there a way to make it faster?
:log info ("Script Started Please wait....");
/interface bridge nat
:foreach user in=[:tool user-manager user print as-value] do={
:local mac ($user->"username");
:if ([find where src-mac-address="$mac/FF:FF:FF:FF:FF:FF"] = "") do={
add chain=dstnat in-bridge=1-Bridge-Neotel src-mac-address="$mac/FF:FF:FF:FF:FF:FF" action=accept;
:log info ("Rule for mac-address $mac does not exist. Adding new Bridge nat Rule for mac-address $mac");
}
}
move [find where src-mac-address="00:00:00:00:00:00/00:00:00:00:00:00"];
:log info ("Moving Block rule at the end");
:log info ("Adding Rules to Bridge Finished");
(removing “:execute” calls in favor of directly calling the command will definitely help, as will eliminating variables that are only used once; I believe getting the usernames with a single “print as-value” is more efficient than multiple “get” calls, but I haven’t actually done measurements, so I’ll very much love it if you could share the difference)
Huh… weird. Could be that the last move is creating some additional penalty, due to the whole list being reordered.
So maybe try to place-before the items, i.e.
:log info ("Script Started Please wait....");
/interface bridge nat
:local blockRule [find where src-mac-address="00:00:00:00:00:00/00:00:00:00:00:00"];
:foreach user in=[:tool user-manager user print as-value] do={
:local mac ($user->"username");
:if ([find where src-mac-address="$mac/FF:FF:FF:FF:FF:FF"] = "") do={
add chain=dstnat in-bridge=1-Bridge-Neotel src-mac-address="$mac/FF:FF:FF:FF:FF:FF" action=accept place-before=$blockRule;
:log info ("Rule for mac-address $mac does not exist. Adding new Bridge nat Rule for mac-address $mac");
}
}
:log info ("Moving Block rule at the end");
:log info ("Adding Rules to Bridge Finished");
Or… Where do the biggest time gaps occur (when you look at the log)? Do many “Rule for mac address” items appear? What about if you remove all logs (except the first and last one)?
Last but not least… Maybe you can readjust the router so that you don’t need the script at all? You could set up all users to be assigned to a certain address list, and then via “/ip firewall mangle”, assign a packet mark. The bridge would just have one rule about any packet with the packet mark.
(I haven’t tested this, but I see no reason it wouldn’t work)
The slowing down happens when comparing user-manager username with nat rules.
I tested how it goes.
I deleted all the rules in /interface bridge nat.
Activated the script.
The first 400 rules are applied in 30 sec. after that it gradually slows down, it takes 2 min to get to 700 another 5min to get to 900 and another 10 min to get to 1130.
After all rules are applied the next time i activate the script no new rules are added because usermanager is in sync with nat rules but it still takes 20min to finish just comparing.
If i add new user to usermanager than when the script goes again scheduled every 20min adds only one rule but it takes 20min to add it.
The problem is that i have another 10k users that are trying to get ip via dhcp and i dont need them there.
If i remove all the rules and allow all the users mikrotik cpu usage goes up by 10-20% i dont want that. This way stays stable at 0-2%
This is not really a problem right now but when i get to 2000 users script will need 1hour to finish. It will be faster to add rules by hand but not all the customers have mikrotik access other then userman.
Maybe Mikrotik devs can see if they can speed up the process.
A. Such serach as you do is, I assume, linear one so you (read router) are doing O(N*N) cost serches…the worst ones… 49 990 000 comparisons…wow…big WOW
B. Each rule “eats” memory. Is your router “big enough” for such userbase ?
OK, maybe search before adding, and add the “matches” to an array, adding the rules afterwards, so:
:log info ("Script Started Please wait....");
/interface bridge nat
:local macsToAdd ({});
:foreach user in=[:tool user-manager user print as-value] do={
:local mac ($user->"username");
:if ([find where src-mac-address="$mac/FF:FF:FF:FF:FF:FF"] = "") do={
:set macsToAdd ($macsToAdd, $mac);
:log info ("Rule for mac-address $mac does not exist.");
}
}
:local blockRule [find where src-mac-address="00:00:00:00:00:00/00:00:00:00:00:00"];
:foreach mac in=$macsToAdd do={
add chain=dstnat in-bridge=1-Bridge-Neotel src-mac-address="$mac/FF:FF:FF:FF:FF:FF" action=accept place-before=$blockRule;
:log info ("Adding new Bridge nat Rule for mac-address $mac");
}
:log info ("Adding Rules to Bridge Finished");
This would consume as much RAM as you have users to add (plus whatever RAM the previous version needed), but since the search is always a constant, the total time should be less.
Right. Because the second time around, you’re searching through your old rules, plus all previously added rules.
OK, now that we know for sure this is the biggest penalty, here’s a new idea to improve the second time around too - search one time for all rules that are the user specific rules, and search through the user manager users for any ones that are NOT those users, and add those unconditionally.
i.e.
:log info ("Script Started Please wait....");
/interface bridge nat
:local userRegex "^(00:00:00:00:00:00";
:foreach existingNatRule in=[print as-value where chain=dstnat and in-bridge=1-Bridge-Neotel and action=accept] do={
:set userRegex ($userRegex . "|" . [:pick ($existingNatRule->"src-mac-address")] 0 [:find ($existingNatRule->"src-mac-address") "/"]);
}
:set userRegex ($userRegex . ")\$");
:local blockRule [find where src-mac-address="00:00:00:00:00:00/00:00:00:00:00:00"];
:foreach user in=[:tool user-manager user print as-value where !(username~$userRegex)] do={
add chain=dstnat in-bridge=1-Bridge-Neotel src-mac-address=(($user->"username") . "/FF:FF:FF:FF:FF:FF") action=accept place-before=$blockRule;
:log info ("Adding new Bridge nat Rule for mac-address $mac");
}
:log info ("Adding Rules to Bridge Finished");
(the “00:00:00:00:00:00” MAC is added to keep the regex valid, while still not allowing it to match an empty string or a weird username)
IMHO “place-before” makes long searches in rule adding process and I am almost sure that this particular search is linear one.
Therefore first script run takes 5 minutes and the second one takes 20 mins. Why ? Script starts with N bridge rules and ends with 2N.
As cost is O(NN) so you ends with O( 2N * 2N ) = 4* O (N*N) … voilà … it explains jump from 5 to 20 mins.
How to speed up process ? Just change add rule to:
It does not matter if the new rule is added at the end of the list, just before the $block-rule. It could be inserted as the first one.
I do knot know if ROS moves rules in memory to insert the new one or just changes some pointers in rules tables.
If it moves rules, then the speed will be also as bad as in current solution. If not, then you should have much, much better result.
i fixed it again it was missing “]” [:pick ($existingNatRule->“src-mac-address”)
There is a problem again this way adds the rules every time. Takes 5 sec but every run it add duplicates
That’s weird… The regex should’ve prevented that very thing. Maybe it’s not matching what I’d think it’s matching.
But OK, now that you’ve mentioned duplicates, you’ve given me a new idea - add everyone with a specific comment (generating rules that are duplicate in effect, but different thanks to the comment), and then remove everyone with a certain other comment (the previous set of users). Afterwards, you can change the first comment to the other comment, thus making the second time around targets the previously added users.
So
:log info ("Script Started Please wait....");
/interface bridge nat
:local newUsers ({});
:foreach user in=[:tool user-manager user print as-value] do={
:set newUsers ($newUsers, [add chain=dstnat in-bridge=1-Bridge-Neotel src-mac-address=(($user->"username") . "/FF:FF:FF:FF:FF:FF") action=accept place-before=$blockRule comment="NEW-User"]);
:log info ("Adding new Bridge nat Rule for mac-address $mac");
}
remove [find where comment="User"];
set $newUsers comment="User";
:log info ("Adding Rules to Bridge Finished");
Note that as BartoszP mentioned, each rule eats memory… In this case, you’d need double of however much RAM all user rules take, since a duplicate is created for all of them, not just for the new users. Sure, they get removed afterwards, but the router needs to be able to hold them in for a moment, before the old ones get removed. You could remove the old rules first to avoid that memory penalty, but this means 5 seconds during which your users would be disconnected.
