Community discussions

MUM Europe 2020
 
kevinds
Member Candidate
Member Candidate
Topic Author
Posts: 108
Joined: Wed Jan 14, 2015 8:41 am

Modify Raw Rule 'add src to address list'

Fri Jul 20, 2018 3:28 am

Alright.. I have a Raw rule using the 'add src to address list' action.

I want to add the /24 of an IP that meets the rule, not just the single /32

Is there a way to do this? If there are multiple ways, what is the best way?
add action=add-src-to-address-list address-list=Drop-IP-List address-list-timeout=1d chain=prerouting dst-address-list=Honeypot-IPs
So any IP trying to communicate with my honeypot IPs, I want to drop the source's entire /24.

I will work on rules to drop larger than /24 if multiple 24's are listed from a /16, but that is in the future..
 
kevinds
Member Candidate
Member Candidate
Topic Author
Posts: 108
Joined: Wed Jan 14, 2015 8:41 am

Re: Modify Raw Rule 'add src to address list'

Fri Jul 20, 2018 8:35 am

I think I need to do this with a script?

Run a script that does this every 5 seconds or so?

Get the "Drop-IP-List"  print where list="Drop-IP-List"

For each address in the list,
:local Address
:local CIDRnetmask 255.255.255.0;
:put ($Address&$CIDRnetmask);

add list=Drop-These-Networks address=Result from above

Delete Address from Drop-IP-List or Just go to the next in the list
Repeat
I hope there is a better way, otherwise I will start figuring out how to do this script tonight
 
sindy
Forum Guru
Forum Guru
Posts: 4220
Joined: Mon Dec 04, 2017 9:19 pm

Re: Modify Raw Rule 'add src to address list'

Fri Jul 20, 2018 10:39 am

No hope here, you'll have to schedule the script. Just don't forget to add the /x after setting the tail of the address to 0s by bitwise anding it with the mask.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
kevinds
Member Candidate
Member Candidate
Topic Author
Posts: 108
Joined: Wed Jan 14, 2015 8:41 am

Re: Modify Raw Rule 'add src to address list'

Fri Jul 20, 2018 1:41 pm

No hope here, you'll have to schedule the script. Just don't forget to add the /x after setting the tail of the address to 0s by bitwise anding it with the mask.
That is what this part was for.. :put ($Address&$CIDRnetmask); It should be the approperate /24.

As for how to make this flow into a script.. Just starting to scratch the surface.. lol As the rule is now, it is annoying to see 90% of a /24 as separate rules, hoping to make this a lot more efficient and effective. It is a fairly effective way to avoid port-scan triggers though, scanning each IP/port with a different source IP..
 
sindy
Forum Guru
Forum Guru
Posts: 4220
Joined: Mon Dec 04, 2017 9:19 pm

Re: Modify Raw Rule 'add src to address list'

Fri Jul 20, 2018 3:18 pm

Well, ($ipAddress&$cidrMask) suggests that the actual expression would be like (4.3.2.1&255.255.255.0) which yields 4.3.2.0, but what I had in mind was that you have to place 4.3.2.0/24 to the address-list as e.g. within a /21 network, 4.3.2.0 is not an address of a network but an ordinary host address.

But on the other hand, the address-list code is clever itself:
[me@MyTik] > ip firewall address-list print
Flags: X - disabled, D - dynamic
 #   LIST                            ADDRESS                                              CREATION-TIME        TIMEOUT
 0   my-test                         192.168.1.3                                          jul/20/2018 13:55:53
[me@MyTik] > ip firewall address-list set [find list=my-test address=192.168.1.3] address=192.168.1.3/24
[me@MyTik] > ip firewall address-list print
Flags: X - disabled, D - dynamic
 #   LIST                            ADDRESS                                              CREATION-TIME        TIMEOUT
 0   my-test                         192.168.1.0/24                                       jul/20/2018 13:55:53
So it is enough to specify the mask length and the address-list code zeroes the bits of the prefix which exceed the mask length automatically.

Therefore, the code only needs to find list items whose address values don't contain the / character and add the /24 to them:
[me@MyTik] > ip firewall address-list print
Flags: X - disabled, D - dynamic
 #   LIST                            ADDRESS                                              CREATION-TIME        TIMEOUT
 0   my-test                         192.168.1.17                                         jul/20/2018 13:55:53
 1   my-test                         192.168.5.43                                         jul/20/2018 14:07:55
[me@MyTik] > :foreach id in=[ip firewall address-list find list=my-test address~"^[0-9\\.]*\$"] do={local ipAddr [/ip firewall address-list get $id address];/ip firewall address-list set $id address=($ipAddr."/24")}
[me@MyTik] > ip firewall address-list print
Flags: X - disabled, D - dynamic
 #   LIST                            ADDRESS                                              CREATION-TIME        TIMEOUT
 0   my-test                         192.168.1.0/24                                       jul/20/2018 13:55:53
 1   my-test                         192.168.5.0/24                                       jul/20/2018 14:07:55
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
kevinds
Member Candidate
Member Candidate
Topic Author
Posts: 108
Joined: Wed Jan 14, 2015 8:41 am

Re: Modify Raw Rule 'add src to address list'

Sat Jul 21, 2018 9:10 pm

Well, ($ipAddress&$cidrMask) suggests that the actual expression would be like (4.3.2.1&255.255.255.0) which yields 4.3.2.0, but what I had in mind was that you have to place 4.3.2.0/24 to the address-list as e.g. within a /21 network, 4.3.2.0 is not an address of a network but an ordinary host address.

But on the other hand, the address-list code is clever itself:
[me@MyTik] > ip firewall address-list print
Flags: X - disabled, D - dynamic
 #   LIST                            ADDRESS                                              CREATION-TIME        TIMEOUT
 0   my-test                         192.168.1.3                                          jul/20/2018 13:55:53
[me@MyTik] > ip firewall address-list set [find list=my-test address=192.168.1.3] address=192.168.1.3/24
[me@MyTik] > ip firewall address-list print
Flags: X - disabled, D - dynamic
 #   LIST                            ADDRESS                                              CREATION-TIME        TIMEOUT
 0   my-test                         192.168.1.0/24                                       jul/20/2018 13:55:53
So it is enough to specify the mask length and the address-list code zeroes the bits of the prefix which exceed the mask length automatically.

Therefore, the code only needs to find list items whose address values don't contain the / character and add the /24 to them:
[me@MyTik] > ip firewall address-list print
Flags: X - disabled, D - dynamic
 #   LIST                            ADDRESS                                              CREATION-TIME        TIMEOUT
 0   my-test                         192.168.1.17                                         jul/20/2018 13:55:53
 1   my-test                         192.168.5.43                                         jul/20/2018 14:07:55
[me@MyTik] > :foreach id in=[ip firewall address-list find list=my-test address~"^[0-9\\.]*\$"] do={local ipAddr [/ip firewall address-list get $id address];/ip firewall address-list set $id address=($ipAddr."/24")}
[me@MyTik] > ip firewall address-list print
Flags: X - disabled, D - dynamic
 #   LIST                            ADDRESS                                              CREATION-TIME        TIMEOUT
 0   my-test                         192.168.1.0/24                                       jul/20/2018 13:55:53
 1   my-test                         192.168.5.0/24                                       jul/20/2018 14:07:55
Interesting.. When adding IPs to lists in Winbox.. It will not accept 192.168.1.1/24 "Couldn't add New Firewall Address List - 192.168.1.1/24 is not a valid dns name (6).

This is why I was looking for a way to do it the other way. Cool that CLI will take them.

However
:foreach id in=[ip firewall address-list find list=my-test address~"^[0-9\\.]*\$"] do={local ipAddr [/ip firewall address-list get $id address];/ip firewall address-list set $id address=($ipAddr."/24")}
This command doesn't seem to do anything though

If I re-create my-test with your IPs it does though.. Change the list to what I am using, and no-go..

This is a huge help though. I'll play with this and hopefully figure it out.

Does everything need to start with lower-case when doing RouterOS scripting? Thought I read that somewhere? If that is the case, I should just need to re-name my lists.. :)

That fixed it.. My lists have to be lower-case (or just start with lowercase?).

Thank you @sindy !
Last edited by kevinds on Sun Jul 22, 2018 3:34 am, edited 2 times in total.
 
kevinds
Member Candidate
Member Candidate
Topic Author
Posts: 108
Joined: Wed Jan 14, 2015 8:41 am

Re: Modify Raw Rule 'add src to address list'

Sun Jul 22, 2018 2:43 am

Thank you again. This has helped a lot.

I am running that line/command every 5 seconds.. It gets stuck though if in that time my system adds two IPs from the /24.. When it tries to add the second... It is getting 'Already exists' and then stops.

I tried adding onerror {} to the end but I think it is just repeating the process with the same IP each time, rather than moving on..

Is there a way to either skip or remove the IP when it encounters an error?
ip firewall address-list set $id address=($ipAddr."/24")
On error
ip firewall address-list remove $id
Or would I need to check my-list to see if $ipAddr."/24" already exists first?
 
sindy
Forum Guru
Forum Guru
Posts: 4220
Joined: Mon Dec 04, 2017 9:19 pm

Re: Modify Raw Rule 'add src to address list'

Sun Jul 22, 2018 11:07 am

The correct use of on-error is rather peculiar (although logical, you want the on-error to be targeted to a specific part of the script) so maybe the issue is only that.

I would first do my best to make the on-error work because a check whether the result of the /32 to /24 address conversion exists in the address-list becomes more complex to do if you plan on more mask lengths than just the /24 one. So you would either have to maintain a table of mask length to bitmap translations, or you would have to insert an address into an auxiliary address list and use the result of the automatic prefix adjustment (and remove the address from the auxiliary address-list at once of course). See the difference:
[me@MyTik] > local myIP 192.168.1.7/24 ; put $myIP
192.168.1.7/24

vs.
[me@MyTik] > ip firewall address-list add list=aux address=192.168.1.7/24 ; put [/ip firewall address-list get [find list=aux] address]
192.168.1.0/24
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
kevinds
Member Candidate
Member Candidate
Topic Author
Posts: 108
Joined: Wed Jan 14, 2015 8:41 am

Re: Modify Raw Rule 'add src to address list'

Tue Jul 24, 2018 3:23 am

The correct use of on-error is rather peculiar (although logical, you want the on-error to be targeted to a specific part of the script) so maybe the issue is only that.

I would first do my best to make the on-error work because a check whether the result of the /32 to /24 address conversion exists in the address-list becomes more complex to do if you plan on more mask lengths than just the /24 one. So you would either have to maintain a table of mask length to bitmap translations, or you would have to insert an address into an auxiliary address list and use the result of the automatic prefix adjustment (and remove the address from the auxiliary address-list at once of course). See the difference:
RouterOS' address-lists don't have an issue with the same IP with a different mask.. Only has issue with the same IP with the same mask. Also has no issue with over-lapping either.

240.0.0.0/24
240.0.0.0/22
240.0.0.0/16

Can be in the same list without an issue.
[kevinds@MikroTik] > /ip firewall address-list print where list=class-e 
Flags: X - disabled, D - dynamic 
 #   LIST                      ADDRESS                                       CREATION-TIME        TIMEOUT             
 0   class-e                   240.0.0.0/24                                  jul/23/2018 17:25:01
 1   class-e                   240.0.0.0/22                                  jul/23/2018 17:25:07
 2   class-e                   240.0.0.0/16                                  jul/23/2018 17:25:10
[kevinds@MikroTik] > 
I am having an issue with getting the syntax correct for on-error/onerror though

https://wiki.mikrotik.com/wiki/Manual:S ... ime_errors

The example shows "on-error={" after the } from the 'do' operation.. So I try "/24")} on-error= and the 'o' stays red.. If I push ? on the console it gives me. counter, do, and in... Everything seems balanced
:foreach id in=[ip firewall address-list find list=my-list address~"^[0-9\\.]*\$"] do={local ipAddr [/ip firewall address-list get $id address];/ip firewall address-list set $id address=($ipAddr."/24")}
I think I am missing something really simple.. But I can't find it. This page has been a great help too http://mikrotik.net.pl/wiki/Scripty
 
sindy
Forum Guru
Forum Guru
Posts: 4220
Joined: Mon Dec 04, 2017 9:19 pm

Re: Modify Raw Rule 'add src to address list'  [SOLVED]

Tue Jul 24, 2018 9:51 am

Yes, you do... as the link I gave shows, the on-error is just a part of a larger scheme. You take just a part of the script which you expect might fail, and for that part you specify the on-error handling. So it is
:do {some script} on-error={some error handling}

And you have to bear in mind that in this case, the do is not a parameter of another command (like in :foreach counter=id in=value-list do={foreach body}) but it is a command itself, so there must be a space, not =, between the do and the first { following it.

So the whole thing looks like below:
:foreach id in=[ip firewall address-list find list=my-list address~"^[0-9\\.]*\$"] do={local ipAddr [/ip firewall address-list get $id address];do {/ip firewall address-list set $id address=($ipAddr."/24")} on-error={log info message="duplicate address found"}}
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
kevinds
Member Candidate
Member Candidate
Topic Author
Posts: 108
Joined: Wed Jan 14, 2015 8:41 am

Re: Modify Raw Rule 'add src to address list'

Tue Jul 24, 2018 10:49 am

Yes, you do... as the link I gave shows, the on-error is just a part of a larger scheme. You take just a part of the script which you expect might fail, and for that part you specify the on-error handling. So it is
:do {some script} on-error={some error handling}

And you have to bear in mind that in this case, the do is not a parameter of another command (like in :foreach counter=id in=value-list do={foreach body}) but it is a command itself, so there must be a space, not =, between the do and the first { following it.

So the whole thing looks like below:
:foreach id in=[ip firewall address-list find list=my-list address~"^[0-9\\.]*\$"] do={local ipAddr [/ip firewall address-list get $id address];do {/ip firewall address-list set $id address=($ipAddr."/24")} on-error={log info message="duplicate address found"}}
do {/ip firewal address-list

Without the 'do {' the on-error doesn't work.. That is why it didn't work. On the plus side my 'on-error' action was correct.

This is what I have set in 'scheduler' and working great so far.
:foreach id in=[ip firewall address-list find list=ip-scanners-list address~"^[0-9\\.]*\$"] do={local ipAddr [/ip firewall address-list get $id address];do {/ip firewal address-list set $id address=($ipAddr."/24")} on-error={/ip firewall address-list remove $id}}
As for impact.. My 'hits' went from 40,000 IPs in 12 hours to ~3000, so a lot less RAM used. So far so good. Put in a few /19s and /20s to summarize the /24s further, but not many yet.

Now to wait for users to complain they can't access stuff.. :)

Thank you again.
 
User avatar
che
Frequent Visitor
Frequent Visitor
Posts: 94
Joined: Fri Oct 07, 2005 1:04 pm

Re: Modify Raw Rule 'add src to address list'

Tue Jul 24, 2018 12:04 pm

I expanded a bit on your idea.

- Script takes addresses in your dynamic list
- Converts it to clean /24 network address and adds whole statement to new blacklist

:foreach addr in=[/ip firewall address-list find list=dynamic_list address~"^[0-9\\.]*\$"] do={
  :local ipAddr [/ip firewall address-list get $addr address];
  :local dotPos "0";
  :local dot ".";
  :local endString ".0/24";
  :local octet1;
  :local octet2;
  :local octet3;
  :local threeOctets;
  :set dotPos [:find $ipAddr "."];
  :set octet1 [:pick $ipAddr 0 $dotPos];
  :set dotPos ($dotPos+1);
  :set octet2 [:pick $ipAddr $dotPos [:find $ipAddr "." $dotPos]];
  :set dotPos ([:find $ipAddr "." $dotPos]+1);
  :set octet3 [:pick $ipAddr $dotPos [:find $ipAddr "." $dotPos]];
  :set threeOctets ("$octet1$dot$octet2$dot$octet3$endString");
  do { /ip firewall address-list add list=blacklist_clean address=$threeOctets} on-error={} }
}

I would advise for you to set timeout on your dynamic list, so you optimize this thing a bit. Timeout should be somewhat higher than your script interval, just in case.
 
kevinds
Member Candidate
Member Candidate
Topic Author
Posts: 108
Joined: Wed Jan 14, 2015 8:41 am

Re: Modify Raw Rule 'add src to address list'

Tue Jul 24, 2018 12:24 pm

I expanded a bit on your idea.

- Script takes addresses in your dynamic list
- Converts it to clean /24 network address and adds whole statement to new blacklist

:foreach addr in=[/ip firewall address-list find list=dynamic_list address~"^[0-9\\.]*\$"] do={
  :local ipAddr [/ip firewall address-list get $addr address];
  :local dotPos "0";
  :local dot ".";
  :local endString ".0/24";
  :local octet1;
  :local octet2;
  :local octet3;
  :local threeOctets;
  :set dotPos [:find $ipAddr "."];
  :set octet1 [:pick $ipAddr 0 $dotPos];
  :set dotPos ($dotPos+1);
  :set octet2 [:pick $ipAddr $dotPos [:find $ipAddr "." $dotPos]];
  :set dotPos ([:find $ipAddr "." $dotPos]+1);
  :set octet3 [:pick $ipAddr $dotPos [:find $ipAddr "." $dotPos]];
  :set threeOctets ("$octet1$dot$octet2$dot$octet3$endString");
  do { /ip firewall address-list add list=blacklist_clean address=$threeOctets} on-error={} }
}

I would advise for you to set timeout on your dynamic list, so you optimize this thing a bit. Timeout should be somewhat higher than your script interval, just in case.
Cool, will this then parse the new list faster or more efficiently because it will be a smaller list? Or does RouterOS take the same amount of time to get the (lets say 5) entries in the dynamic_list out of the 30,000 address-list entries as it does for looking for the 5 entries without a / in a much longer list? (Hopefully that makes sense) I'm not sure how to measure this to confirm.

The CPU usage is 25%+ higher on my RB750Gr3 when the script in my post is running.
 
User avatar
che
Frequent Visitor
Frequent Visitor
Posts: 94
Joined: Fri Oct 07, 2005 1:04 pm

Re: Modify Raw Rule 'add src to address list'

Wed Jul 25, 2018 12:05 am

I'm not sure how fast hashing methods in RouterOS are, never benchmarked it. My initial idea is to have less entries in the list and (in theory) faster rule processing, because my assumption is, as you stated: less address list entries > faster firewall. Using this method you will only have a network address, not separate addresses with /24 sufix that belong to the same network.

You mentioned /16 in the future - I had that in mind as well because part of this code could be used for creating those bigger prefixes.
 
kevinds
Member Candidate
Member Candidate
Topic Author
Posts: 108
Joined: Wed Jan 14, 2015 8:41 am

Re: Modify Raw Rule 'add src to address list'

Wed Jul 25, 2018 1:19 am

I'm not sure how fast hashing methods in RouterOS are, never benchmarked it. My initial idea is to have less entries in the list and (in theory) faster rule processing, because my assumption is, as you stated: less address list entries > faster firewall. Using this method you will only have a network address, not separate addresses with /24 sufix that belong to the same network.

You mentioned /16 in the future - I had that in mind as well because part of this code could be used for creating those bigger prefixes.
The script modifies the existing entries, changes them. It doesn't leave the individual IPs, except when the script broke because the /24 already existed. The on-error part corrected that.

I'm concerned about the efficiency of the script and CPU usage.. Other than that, seems to be working well. Your edit doesn't remove the individual IP, resulting in 2x the address lists and memory usage. Why I was asking about the time parsing the lists for IPs without the /.
 
User avatar
che
Frequent Visitor
Frequent Visitor
Posts: 94
Joined: Fri Oct 07, 2005 1:04 pm

Re: Modify Raw Rule 'add src to address list'

Wed Jul 25, 2018 3:43 am

My script doesn't deal with deleting entries because I think that is a waste of router's resources. I suggested you set timeout on automatic list, so entries disapear on their own. You are free to see zero value in my approach, I had fun contemplating the solution.
 
sindy
Forum Guru
Forum Guru
Posts: 4220
Joined: Mon Dec 04, 2017 9:19 pm

Re: Modify Raw Rule 'add src to address list'

Wed Jul 25, 2018 8:20 am

I have myself suggested someone somewhere else here to move items from one list populated by the action=add-src-to-address-list rule to another list referred to by the action=drop rule for just the same reason, i.e. not to search among thousands of items over and over again. But on the other hand, that's exactly what the firewall does with each packet checked by any rule referring to an address-list, so it simply cannot be that much resource-intensive, so if for the find in scripting/configuration the same task is implemented separately in a less efficient way, this would be the way to spot that. On yet another hand, in most cases address-lists are consulted only for initial packets of each connection, so the fact that their processing is more resource-intensive than processsing of mere prefixes doesn't affect overall throughput much. And if it is a DDoS, the download bandwidth of your uplink will often be exhausted before the power of your CPU.

I was reluctant to suggest two lists in your case because I was suspecting you to want also the individual address to be blocked at once and not only after they get expanded to subnets and migrated to the target list, but after a bit of sleep, it has appeared in my head that you can simply have two action=drop rules, one per each of the two lists. So the find suspected to be resource-intensive would be applied only on the small source address-list, and the conflict (addition of the same subnet a second time) would happen on the target address-list, meaning that on-error={nothing} would be sufficient to resolve it. Again, the conflict must first be spotted, which involves an address-list search, but I sincerely believe that it is using the runtime method if two methods exist. This theory is supported by the fact that, as you've noticed, x.x.x.0/24 and x.x.x.0/25 can coexist on the list, which suggests that the conflict is only detected once the hash of the address being added is found to be identical with an existing one.

Regarding limiting the lifetime of items added to the source list - deleting the item from the source list "manually" once you already have obtained its id should not be a big deal from the point of view of the CPU, but adding the address with a short timeout seems nevertheless "cheaper". But there is a hidden caveat - the address actually survives on the list still 5 seconds after its timeout has expired! So you'd either handle some addresses twice (which is much more expensive than removing them manually) or you would miss them, depending on the periodicity of running the script.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
kevinds
Member Candidate
Member Candidate
Topic Author
Posts: 108
Joined: Wed Jan 14, 2015 8:41 am

Re: Modify Raw Rule 'add src to address list'

Sat Jul 28, 2018 3:19 am

My script doesn't deal with deleting entries because I think that is a waste of router's resources. I suggested you set timeout on automatic list, so entries disapear on their own. You are free to see zero value in my approach, I had fun contemplating the solution.
I never saw zero value.. :) I didn't consider setting the timeout in seconds.. Also I'm still not sure if I want the blacklist_clean non-dynamic or not.

But yes, I'm still trying to figure out how to test/time/benchmark if one method is faster than the other. I suspect it depends on how it processes or saves the lists in memory.

For stats, 50,000 hits, 2,000,000 blocked

Who is online

Users browsing this forum: bukanbondan, zallace and 96 guests