Remove address from address-list

Cartman · Tue Sep 18, 2018 2:48 pm

Hello world !!!

I would like to improve my firewall scripts to keep the address list a bit shorter:
Currently I have four lists to create a blacklist for ssh:
ssh_stage1/2/3 and ssh_blacklist.
If an ssh connection is established and there is no entry, yet, the address will be put in stage1 else it is put into the
next higher list. It works, so far.

Now I would like to remove the address from the lower list when an adress ist added to avoid multiple entries for
the same IP.
Best way would be in the firewall rule, but I did not find a setting there.
Another possibility would be a script running every minute scanning the address lists for every IP and keeping only
the entry in the highest one.
This I just do not get to work.
Maybe someone has a hint for me.

Problem is: I have 800 - 1000 tries per hour to log in via ssh. Address list are getting quite big and I want to shorten
them a bit.

TIA
Wayne

rendezz · Tue Sep 18, 2018 3:46 pm

Can you send us the relevant code/config snippet on how you are doing that, and lets see what can be done with that.

Cartman · Tue Sep 18, 2018 4:49 pm

Currently I am trying

/ip firewall address-list find list=
"ssh_stage2" remove [/ip firewall address-list find list="ssh_stage1"] ;

Stage1 has 858 entries, stage2 489

CPU goes to 100% and stays there without anything happening.

Another try :

:foreach i in=[/ip firewall address-list find name="ssh_stage2"] do=(
/ip firewall address-list find name="ssh_stage1" remove $i
)

Mikrotik scripting is a really cryptic thing.
Does not consume as much CPU, but has no effect.

rendezz · Wed Sep 19, 2018 11:33 am

:foreach i in=[/ip firewall address-list find name="ssh_stage2"] do=(
/ip firewall address-list find name="ssh_stage1" remove $i
)

The above is very broken in terms of syntax and just about everything.

Try

:foreach i in=[/ip firewall address-list find list="ssh_stage2"] do={
    :local address [/ip firewall address-list get $i address]
    :log info "Removing $address from ssh_stage1, found in ssh_stage2"
    /ip firewall address-list remove [find list="ssh_stage1" address="$address"]
}

Few things to note, I would remove the logging, it will slow things down. Also, find is a very nasty function, probably linear in terms of matching, certainly when number of entries get high there will be serious CPU usage because in the above code you are running find against ssh_stage1 through 858 entries 489 times, its bound to get ugly. An easier method might be to remove from ssh_stage1 as you add it to ssh_stage2 if you have control over that process. thats 1 find and 1 insert per record found.

Cartman · Wed Sep 19, 2018 12:34 pm

Thank you rendezz.

Like I said, MikroTik scripting looks a bit like rocket science to me.
Google and playing around sent me in the same direction as your script, but
still did not get it to work.

Your script just clears the ssh_stage1 list.

/ip firewall address-list remove [find list="ssh_stage1" address="$address"]

I think the "find" just takes the list value and does not care about the second search term.
Maybe it should look like

/ip firewall address-list remove [find list="ssh_stage1" [find address="$address"]]

But this is again beyond my knowlege.

Second thing is, that the CPU gets locked at 100%, as you already mentioned.

An easier method might be to remove from ssh_stage1 as you add it to ssh_stage2

That might be what I am looking for, but I do not see a chance to put a script in the "Action" of the
firewall rule. It would be just a rename of the "list" value from stage1 to stage2.

Finally it seems like the bot gave up for now. after ~3500 entries, the rate went down to one attempt per minute
which can be handled by the device. Hope it stays like this, until I have found a solution.

Thanks again.

rendezz · Wed Sep 19, 2018 12:53 pm

Broke my own code when I added the logging component. Seems pre storing the address in $address causes the behaviour you saw. Put is straight in find and it works

:foreach i in=[/ip firewall address-list find list="ssh_stage2"] do={
    :local address [/ip firewall address-list get $i address]
    :log info "Removing $address from ssh_stage1, found in ssh_stage2"
    /ip firewall address-list remove [find list="ssh_stage1" address=[get $i address] ]
}

(You can remove both the :local and :log lines above, they are superfluous

My output is

[admin@client1-cpt.test] > /ip firewall address-list print where list=ssh_stage1 or list=ssh_stage2
Flags: X - disabled, D - dynamic 
 #   LIST                                          ADDRESS                                              
 0   ssh_stage2                                    2.2.2.2                                              
 1   ssh_stage2                                    3.3.3.3                                              
 2   ssh_stage1                                    1.1.1.1                                              
 3   ssh_stage1                                    2.2.2.2                                              
 4   ssh_stage1                                    3.3.3.3                                              
 5   ssh_stage1                                    4.4.4.4                                              
[admin@client1-cpt.test] > :foreach i in=[/ip firewall address-list find list="ssh_stage2"] do={
{...     :local address [/ip firewall address-list get $i address]                              
{...     :log info "Removing $address from ssh_stage1, found in ssh_stage2"
{...     /ip firewall address-list remove [find list="ssh_stage1" address=[get $i address] ]
{... }                                                                                      
[admin@client1-cpt.test] > /ip firewall address-list print where list=ssh_stage1 or list=ssh_stage2 
Flags: X - disabled, D - dynamic 
 #   LIST                                          ADDRESS                                              
 0   ssh_stage2                                    2.2.2.2                                              
 1   ssh_stage2                                    3.3.3.3                                              
 2   ssh_stage1                                    1.1.1.1                                              
 3   ssh_stage1                                    4.4.4.4

With regard to your firewall rule. Add the IPs to a pre-processing list, then run a script which reads in the elements of the pre-processing list and make modifications (single remove ssh_stage1, single insert ssh_stage2, and remove from processing list), it should be much less CPU intensive (I haven't done the maths, but it should be)

Remove address from address-list

Remove address from address-list

Re: Remove address from address-list

Re: Remove address from address-list

Re: Remove address from address-list

Re: Remove address from address-list

Re: Remove address from address-list

Who is online