Importing IP List from file

RouterOS Scripting
1

Hello.

I’m struggling with getting to run script that I found to import IP addresses from file and adding them to my address list.
Idea is to fetch file with all official country IP addresses from Web page (file name is “local.net”) and add them to list.
Fetching script is working fine, bet when I run the script to add IP to list I get only first number added to my address list.
Screenshot 2020-09-17 at 11.27.53.png
Also, is there “4096 byte read/write limit” workaround?

Maybe there is some scripting guru that can help me?

Script that I am using is:

  :if ( [/file get [/file find name=local.net] size] > 0 ) do={
    /ip firewall address-list remove [/ip firewall address-list find list=Test]
    :local content [/file get [/file find name=local.net] contents] ;
    :local contentLen [ :len $content ] ;
    :put ( "=>" . $content . "<=" )
    :local lineEnd 0
    :local line ""
    :local lastEnd [ :tonum 0 ]


    :do {
      :set line [ :pick $content $lastEnd [ :find $content "\r\n" $lastEnd ] ]
      :set lineEnd [:find $content "\n" $lastEnd ] ;
      :put "lineEnd=$lineEnd< - lastEnd=$lastEnd - line=$line<"
    :if ( [ :len $content ] = 0 ) do={
      :put "UNIX"
      :set line [ :pick $content $lastEnd [ :find $content "\r\n" $lastEnd ] ]
       :set lineEnd [:find $content "\n" $lastEnd ] ;
    }
      :set lastEnd ( $lineEnd + 1 ) ;

      :if ( [:pick $line 0 1] != "\n" ) do={
      :put ">$line<"
        :if ( [:len $line ] > 0 ) do={ /ip firewall address-list add list=Test address=$line }
      }
    } while ($lineEnd < $contentLen)
  }
2

Please post local.net file example.

3

You should prepare your list before import.
Read this topic: http://forum.mikrotik.com/t/blacklist-filter-update-script/89817/385

4

Try these 5 lines.
just add your own TLD’s, it will add an address list for each TLD. Run it again to update them. You will need to create a firewall rule to drop the TLD lists. Beware the lists are huge (who woulda guessed) and if you load them into memory they will take it all, so select your TLD’s well and don’t go for too many, or at least monitor your memory level.

All the best.

foreach i in={ "NL"; "CN"; "RU"; "DE"; "UA"} do={
/tool fetch url="http://www.iwik.org/ipcountry/mikrotik/$i" dst-path=/DISK/downloads/$i;
/ip firewall address-list remove [/ip firewall address-list find list=$i];
/import file-name=/DISK/downloads/$i;
/file remove /DISK/downloads/$i;
}
5

Reading pure IP adresses is possible up to 64KB large files.

http://forum.mikrotik.com/t/address-lists-downloader-dshield-spamhaus-drop-edrop-etc/133640/1

I am on the moment busy to create backup/restore for adresslists present in the router and it will export a .RSC file that smaller than the normal export.

6

I recall this … not perfect but it could help
http://forum.mikrotik.com/t/how-to-opitimize-list-of-ip4-addresses/129916/1

7

When I run this script I get the following error - expected end of command (line 2 column 86)

8

There is a bug in folder part of the script. Folder not needed so this does work fine.

foreach i in={ "NL"; "CN"; "RU"; "DE"; "UA"} do={
	/tool fetch url="http://www.iwik.org/ipcountry/mikrotik/$i" dst-path=$i
	/ip firewall address-list remove [/ip firewall address-list find list=$i]
	/import file-name=$i
	/file remove $i
}
9

Awesome thanks

10

I consider a true security breach import any file downloaded from 3rd party site with /import file-name=$scriptName command.

If some hacker or the author itself put some commands inside the downloaded file, can do anything he want with the RouterBOARD…

11

True, one should always be very vigilant when obtaining some stuff from a 3e party.
A blunt “import” and just executing some imported stuff are not best practices for sure.
As these are block-lists, you could argue that IF the list was maliciously modified and for example legit systems added you could also potentially “disrupt” legit traffic.

I have to say I also do this for this specific list with TLD’s.

So in fact the best thing is to sort of combine existing scripts.

foreach i in={ “CN”; “RU”; “UA”} do={
/tool fetch url=“http://www.iwik.org/ipcountry/mikrotik/$i” dst-path=$i
/ip firewall address-list remove [/ip firewall address-list find list=$i]
/import file-name=$i
/file remove $i
}

and then below a typical script used for importing others (eg. here a TOR-exit nodes list, similar concept)
The difference is that the IP’s are without any CIDR notation at the end (eg /24 /22 or something) and offcourse do not contain all other routerOS commands, but the regex should extraxt only IP info anyway.
So perhaps the logic of the script below needs to be slightly adapted to form a combined one. This method is safe as IP’s are placed and handled/parsed). Any fake command in the list would cause an error anyway.

→ No need to fetch them, as the above script has the iteration and downloads separate files for the different TLD’s (CN / RU / UA in this example) so $i could be used as the source.

{
/ip firewall address-list
:local update do={
:do {
:local result [/tool fetch url=$url as-value output=user]; :if ($result->“downloaded” != “63”) do={ :local data ($result->“data”)
:do { remove [find list=$blacklist comment!=“Optional”] } on-error={}

:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data “\n”]]~“^([0-2]{0,1}[0-9]{1,2}\.){3}[0-2]{0,1}[0-9]{1,2}(\/[0-3]{0,1}[0-9]{1,1}){0,1}”) do={
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) } on-error={}
}
:set data [:pick $data ([:find $data “\n”]+1) [:len $data]]
} ; :log warning “Imported address list < $blacklist> from file: $url”
} else={:log warning “Address list: <$blacklist>, downloaded file to big: $url” }
} on-error={:log warning “Address list <$blacklist> update failed”}
}
$update url=https://check.torproject.org/torbulkexitlist blacklist=“TOR-Nodes” delimiter=(“\n”)
}

12

This must also be considered:
http://forum.mikrotik.com/t/recommend-way-to-block-ads-with-mikrotik/128679/1

13

Anybody wants to give a go at this below ?
So basically these TLD’s come from the onsite website in this form for IPv4 and IPv6
Let’s start simple and only look at IPv4

/log info “Loading GR ipv4 address list”
/ip firewall address-list remove [/ip firewall address-list find list=GR]
/ip firewall address-list
:do { add address=2.84.0.0/14 list=GR } on-error={}
:do { add address=5.54.0.0/15 list=GR } on-error={}
:do { add address=5.144.192.0/18 list=GR } on-error={}
:do { add address=5.172.192.0/20 list=GR } on-error={}
:do { add address=5.203.0.0/16 list=GR } on-error={}
:do { add address=31.14.168.0/21 list=GR } on-error={}

I don’t have the impression that the logic below handles it well. It should throw away/ignore anything besides X.X.X.X/Y and then import it into a ACL.
Regex guru’s here ?

:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data “\n”]]~“^([0-2]{0,1}[0-9]{1,2}\.){3}[0-2]{0,1}[0-9]{1,2}(\/[0-3]{0,1}[0-9]{1,1}){0,1}”) do={
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) } on-error={}

14

Ehm…just search “ip regex”…
“Someone” has already wroten something… :laughing:
http://forum.mikrotik.com/t/address-lists-downloader-dshield-spamhaus-drop-edrop-etc/133640/88

remember than this are wroten for be put directly on script,
if tested on terminal you must add \ before the ?
if tested on regex101 must be removed \ before . and /

What you have post is my version that uses less CPU because it approximates the value, with or without zero, but it is quite reliable.
0.0.0.0/0 to 299.299.299.299/39
000.000.000.000/00 to 299.299.299.299/39

This is the most precise version I have made, but use more CPU, really not soo much:

((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\/(3[0-2]|[0-2]?[0-9])

This match exactly IP-prefix (on the link on this post are present alternative “IP only” and “IP with optional prefix”)
0.0.0.0/0 to 255.255.255.255/32
with or without the 0 like 008.000.000.000/08

15

Doesn’t seem to do anything. I get the message in my logs (like with other updates)

Imported address list < GR-TLD> from file: https://www.iwik.org/ipcountry/mikrotik/GR

But the ACL is not created on my RB3011
I’ve tried the code below, I’ve omitted the “^” because we will not see any match at the beginning of a string anyway.
Tried it on regex101.com but I had to modify it in order to capture/match an IP like 10.20.10.0/12

((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9]).){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])/(3[0-2]|[012]?[0-9])

{
/ip firewall address-list
:local update do={
:do {
:local result [/tool fetch url=$url as-value output=user]; :if ($result->“downloaded” != “63”) do={ :local data ($result->“data”)
:do { remove [find list=$blacklist comment!=“Optional”] } on-error={}

:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data “\n”]]~“((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])****/(3[0-2]|[012]?[0-9])”) do={
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) } on-error={}
}
:set data [:pick $data ([:find $data “\n”]+1) [:len $data]]
} ; :log warning “Imported address list < $blacklist> from file: $url”
} else={:log warning “Address list: <$blacklist>, downloaded file to big: $url” }
} on-error={:log warning “Address list <$blacklist> update failed”}
}
$update url=https://www.iwik.org/ipcountry/mikrotik/GR blacklist=“GR-TLD” delimiter=(“\n”)
}

Not there yet :wink:

16

I add a warning for be clear, also on other post, thanks for make me notice that!!! :laughing:

remember than this are wroten for be put directly on script,
if tested on terminal you must add \ before the ?
if tested on regex101 must be removed \ before . and /

On regex101 you must remove MikroTik required \ on terminal, but you must not remove the \ on . and on / because must be literal . and / , not any character…

from
((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])</big>.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])</big>/(3[0-2]|[0-2]?[0-9])to
((25[0-5]|(2[0-4]|[01]</big>?[0-9]</big>?)[0-9])\.){3}(25[0-5]|(2[0-4]|[01]</big>?[0-9]</big>?)[0-9])\/(3[0-2]|[0-2]</big>?[0-9])and to
((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9]).){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])/(3[0-2]|[0-2]?[0-9])

test_regex.png
17

Hmm, don’t get it. It creates no ACL at all, yet it reports no issues ;-(
So the only way to troubleshoot is execute on CLI and see what comes on the screen ?
But if tested on CLI, then regex-code differs slightly.

{
/ip firewall address-list
:local update do={
:do {
:local result [/tool fetch url=$url as-value output=user]; :if ($result->“downloaded” != “63”) do={ :local data ($result->“data”)
:do { remove [find list=$blacklist comment!=“Optional”] } on-error={}

:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data “\n”]]~“((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\/(3[0-2]|[0-2]?[0-9])”) do={
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]])} on-error={}
}
:set data [:pick $data ([:find $data “\n”]+1) [:len $data]]
} ; :log warning “Imported address list < $blacklist> from file: $url”
} else={:log warning “Address list: <$blacklist>, downloaded file to big: $url” }
} on-error={:log warning “Address list <$blacklist> update failed”}
}
$update url=https://www.iwik.org/ipcountry/mikrotik/GR blacklist=“GRTLD” delimiter=(“\n”)
}

18

First of all you must split on three parts because the IP is not at the start and not at the end, like other lists…
Space replaced with “·”
:do·{·add·address=|2.84.0.0/14|·list=GR·}·on-error={}left delimiter:
address=
right delimiter:
·list=
including space

Example for CLI, I use global just for the test, on script can/must be used local.

# test string
:global teststr ":do { add address=2.84.0.0/14 list=GR } on-error={}\r\n:do { add address=5.54.0.0/15 list=GR } on-error={}"

# remove head and tail, used + 8 because is the character lenght of "address="
# instead of a fixed value can be set also, for example, with [:len $rightstrdelimiter] on script
# notice the needed space before list
:global testip [:pick $teststr ([:find $teststr "address=" -1] + 8) [:find $teststr " list=" -1]]

# now I chech against regexp if is valid IP-prefix
:if ($testip~"((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\/(3[0-2]|[0-2]\?[0-9])") do={ \
    :put "$testip is a IP-prefix"
} else={:put "$testip is NOT a IP-prefix"}

Doing again adding last “right” position value (on example 29), give next IP, without split the file on lines and without modify everytime the data value.

:global testip [:pick $teststr ([:find $teststr "address=" 29] + 8) [:find $teststr " list=" 29]]
:if ($testip~"((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\/(3[0-2]|[0-2]\?[0-9])") do={ \
    :put "$testip is a IP-prefix"
} else={:put "$testip is NOT a IP-prefix"}
19

Thx for the feedback, I’ll play with that and try to get it working.
I was under the impression that perhaps the parsing was done a bit like “sed” (streaming editor) in Linux/Unix so it would scan the line (until the delimeter \n) and capture the regex logic in there without
the need to split it yourself etc.

20

if on mikrotik the find can accept the regexp as search string, the code are more easy to do…