[Feature request] Address List extension

vasilevkirill · Sat Apr 06, 2019 1:37 am

Hi,
I would be interested to see such functionality in the address lists.

Code: Select all

/ip firewall address-list add list=hosts address=192.168.0.0/16 wildcard=255.255.10-20.255

Only networks will be third octet is in the range of 10-20

Code: Select all

/ip firewall address-list add list=hosts address=192.168.0.0/16 wildcard=255.255.0.10

third octet 0-255 and fourth octet 10

Code: Select all

/ip firewall address-list add list=hosts address=192.168.0.0/16 wildcard=255.255.255.10,15,20,30-35

fourth octet 10 or 15 or 20 or in range 30-35

Code: Select all

/ip firewall address-list add list=GOOGLE address=AS15169

Dynamic address list, address from AS number

Code: Select all


[admin@MT-AP-KIRILL-ROOM] /ip firewall address-list> pr

Flags: X - disabled, D - dynamic 

 #   LIST                               ADDRESS                                                 CREATION-TIME        TIMEOUT             

 0   GOOGLE                       AS15169                                                    apr/06/2019 01:07:35

 1 D ;;; AS15169

     GOOGLE                        103.21.184.0/22                                         apr/06/2019 01:27:57

 2 D ;;; AS15169

     GOOGLE                        103.227.68.0/22                                         apr/06/2019 01:27:57

 3 D ;;; AS15169

     GOOGLE                        103.240.192.0/22                                         apr/06/2019 01:27:57

.......

Code: Select all

/ip firewall address-list add list=mysite.com address=*.mysite.com

wildcard or asterisk in all sub domain.

I know that you cannot get a list of sub domains, but if the router is a DNS server, it can find all such domains that are in the cache.

Jotne · Sat Apr 06, 2019 7:23 pm

I would like a command that list all the address list and if possible the number of entries in them.

neutronlaser · Sun Apr 07, 2019 3:18 am

How could we use this:

whois -h whois.radb.net -- '-i origin AS15169' | grep ^route

Which gets every IP address range Google uses

Into a Mikrotik address list?

nostromog · Sun Apr 07, 2019 10:57 am

EDIT: Change sort to sort -u so that no full duplicates remain.

How could we use this:

whois -h whois.radb.net -- '-i origin AS15169' | grep ^route

Which gets every IP address range Google uses

Into a Mikrotik address list?

Those two give separate raw prefix lists, one for IPv4 and another for IPv6:

whois -h whois.radb.net -- '-i origin AS15169' | grep ^route: | awk '{print $2}'
whois -h whois.radb.net -- '-i origin AS15169' | grep ^route6: | awk '{print $2}'

Splitting into ipv4 and ipv6 is needed due to the strange way RouterOS deals with IPv6.

Now this will turn them into .rsc files ready to import:

(echo "/ip firewall address-list {"; for prefix in $(whois -h whois.radb.net -- '-i origin AS15169' | grep ^route: | awk '{print $2}'| sort -u); do echo "add list=goog address=$prefix timeout=1d"; done; echo "}") > goog4.rsc
(echo "/ipv6 firewall address-list {"; for prefix in $(whois -h whois.radb.net -- '-i origin AS15169' | grep ^route6: | awk '{print $2}'| sort -u); do echo "add list=goog address=$prefix timeout=1d"; done; echo "}") >goog6.rsc
$ ls -l goog*
-rw-r--r-- 1 user user  345227 apr  7 08:25 goog4.rsc
-rw-r--r-- 1 user user   11439 apr  7 08:24 goog6.rsc
$ wc -l goog*.rsc
  7146 goog4.rsc
   225 goog6.rsc
  7371 total
#...
$ scp goog6.rsc 192.168.88.1:
$ ssh 192.168.88.1 "/import goog6.rsc"
$ ssh 192.168.88.1 "/ipv6 firewall address-list print where list=goog"
Flags: X - disabled, D - dynamic 
 #   LIST                 ADDRESS                                              
 0 D goo                  2001:1900:2292::/48                                  
 1 D goo                  2001:4860::/32                                       
 2 D goo                  2401:fa00::/32                                       
 3 D goo                  2401:fa00::/42                                       
 4 D goo                  2401:fa00:4::/48                                     
 5 D goo                  2404:6800::/32                                       
 6 D goo                  2404:6800:4001::/48        
 (...)

Note that I only tested the ipv6 one, it is way smaller. Also, I put a timeout so that the lists will go on RAM.

Note also that the IPv6 list will need deduplication. It contains separate sub-prefixes of a given prefix as can be seen in my example. For instance it contains 2401:fa00::/32, 2401:fa00::/42 and 2401:fa00:4::/48, all included in the first.

neutronlaser · Sun Apr 07, 2019 7:10 pm

That's awesome. It is a good start to making a script that could for example let Google or Facebook in a Walled Garden list or perhaps QoS rule or blocking. I wish I knew how to deduplicate it.
It would be great as an online script generator. I tested it and it seemed an effective way to block Facebook in my firewall.

nostromog · Mon Apr 08, 2019 12:27 pm

I wish I knew how to deduplicate it.

When I tried ipv4 it was failing due to a duplicate, but changing sort -> sort -u makes it load. I edited the post. Removing entries that fall "inside"other entries, though, is a non-trivial programming problem.

neutronlaser · Mon Apr 08, 2019 2:34 pm

That's awesome. The extra efficiency of deduplication of overlapping ranges isn't important enough in my case.

Here is how to export Facebook to a single address list:

(echo "/ip firewall address-list {"; for prefix in $(whois -h whois.radb.net -- '-i origin AS63293' | grep ^route: | awk '{print $2}'| sort -u); do echo "add list=facebook address=$prefix timeout=1d"; done; echo "}") > facebook.rsc
(echo "/ip firewall address-list {"; for prefix in $(whois -h whois.radb.net -- '-i origin AS54115' | grep ^route: | awk '{print $2}'| sort -u); do echo "add list=facebook address=$prefix timeout=1d"; done; echo "}") > facebook.rsc
(echo "/ip firewall address-list {"; for prefix in $(whois -h whois.radb.net -- '-i origin AS32934' | grep ^route: | awk '{print $2}'| sort -u); do echo "add list=facebook address=$prefix timeout=1d"; done; echo "}") > facebook.rsc

msatter · Mon Apr 08, 2019 2:43 pm

That's awesome. It is a good start to making a script that could for example let Google or Facebook in a Walled Garden list or perhaps QoS rule or blocking. I wish I knew how to deduplicate it.
It would be great as an online script generator. I tested it and it seemed an effective way to block Facebook in my firewall.

If you look on bgp.he.net then you see for Facebook huge IPv6 ranges that they own and so easily block-able:

2a03:2880::/32
2a03:2887:ff34::/48
2401:db00::/32
2620:0:1c00::/40
2803:6080::/32

https://bgp.he.net/search?search%5Bsear ... mit=Search

neutronlaser · Mon Apr 08, 2019 2:54 pm

I don't use IPv6 but great information for those who do.

Simono · Tue Apr 09, 2019 1:48 pm

Now I don't have Linux (HDD failure). How do this on router or Windows PC?

Sent from my phone by Tapatalk

neutronlaser · Tue Apr 09, 2019 2:03 pm

Cygwin with package for whois (not sure which one it is in, maybe bindutils)

Simono · Tue Apr 09, 2019 2:17 pm

Thanks, I will try

Sent from my phone by Tapatalk

[Feature request] Address List extension

[Feature request] Address List extension

Re: [Feature request] Address List extension

Re: [Feature request] Address List extension

Re: [Feature request] Address List extension

Re: [Feature request] Address List extension

Re: [Feature request] Address List extension

Re: [Feature request] Address List extension

Re: [Feature request] Address List extension

Re: [Feature request] Address List extension

Re: [Feature request] Address List extension

Re: [Feature request] Address List extension

Re: [Feature request] Address List extension

Who is online