Community discussions

MikroTik App
 
nbctcp
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Tue Sep 16, 2014 7:32 pm

Script to convert ip to address-list

Thu Mar 03, 2016 1:49 pm

Hi,

Let say I have 7k ips in this text
http://torstatus.blutmagie.de/ip_list_a ... st_ALL.csv

I want to convert all those ips into address list using script
something like this
:do {/ip firewall address-list add address=$entry list=list-TOR}

I am following this method but the script can't create MY-IP-LIST address list
http://wiki.mikrotik.com/wiki/Using_Fet ... ress_Lists

Anyone have working script with RouterOS 6.34.2

tq
 
User avatar
pukkita
Trainer
Trainer
Posts: 3051
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: Script to convert ip to address-list

Thu Mar 03, 2016 6:05 pm

Not sure if this is a one-time need or maybe you are seeking for a way to automate this; if it needs to be the script way, check for the CSV having suitable end of line characters, where is it failing?

If this is a one time need: a hack or workaround without resorting to scripting would be opening that csv file on a text editor then using find/replace to put

/ip firewall address-list add address=

by searching for the beggining of line, so that text is placed before the IP, and then

list=YOURLIST

by searching for end of line.

Another way would be by using a text editor with block or column mode editing capabilities, like SublimeText or notepad++ for example (see https://notepad-plus-plus.org/features/ ... iting.html).

Both by the way have plugins for ROS syntax coloring.

The idea is to end with the CLI commands on the text file; that way you can rename it to .rsc, upload it to the router then do an import on the .rsc.

Following this idea, and as you may be reaching maximum file size for file processing using scripting, you could also generate the .rsc file containing all the CLI to fill the list by coding an smal cgi in bash or php, then just fetching the file from your web server...
 
nbctcp
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Tue Sep 16, 2014 7:32 pm

Re: Script to convert ip to address-list

Sat Mar 05, 2016 1:30 am

I am trying to avoid to use other server to process it.
It can be done by following this method
http://robert.penz.name/983/filter-traf ... -routeros/
I prefer everything using buil-in script in Mikrotik
I think if older RouterOS can do that, newer one will be more capable doing that
 
jo2jo
Forum Guru
Forum Guru
Posts: 1003
Joined: Fri May 26, 2006 1:25 am

Re: Script to convert ip to address-list

Sun Dec 01, 2019 1:15 am

Hi,

Let say I have 7k ips in this text
http://torstatus.blutmagie.de/ip_list_a ... st_ALL.csv

I want to convert all those ips into address list using script
something like this
:do {/ip firewall address-list add address=$entry list=list-TOR}

I am following this method but the script can't create MY-IP-LIST address list
http://wiki.mikrotik.com/wiki/Using_Fet ... ress_Lists

Anyone have working script with RouterOS 6.34.2

tq

hi, yes that script from the wiki is old and does not work on 6.x , however if you make changes to the do / while loop, it will work (move the :while up), change it like this:
       :while ($lineEnd < $contentLen) do={
             :set lineEnd [:find $content "\n" $lastEnd ] ;
             :set line [:pick $content $lastEnd $lineEnd] ;
             :set lastEnd ( $lineEnd + 1 ) ;
       #If the line doesn't start with a hash then process and add to the list
             :if ( [:pick $line 0 1] != "#" ) do={
       :local entry [:pick $line 0 $lineEnd ]
           :if ( [:len $entry ] > 0 ) do={
               /ip firewall address-list add list="MY-IP-LIST"
            }
          }
       }
       }
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Script to convert ip to address-list

Tue Apr 06, 2021 5:51 pm

Anyone that can help me out why this does not work ?
Basically I combined some posting in order to make a script that should work, but it only cleans/empties my "DNS" address-list.
I've download the link below containing a bunch of DOH/DOT public servers that I want to convert into an ACL.

https://github.com/oneoffdallas/dohserv ... iplist.txt

Basically I took the (non-working on RouterOS 6.x) script from the WIKI and added the correction that is suggested to make it work again in this post.
But still, list remains empty basically.
The "iplist.txt" contains some line starting with # which should be ignored.
It also contains some "blank" lines, I wonder if that is ok.

Below a small piece of the list.
-----------------
# DNS over HTTPS (DoH) server IP list
# This is a list of IPs which correspond to publicly available DoH providers
# It can be used to firewall these IP addresses
# This list was generated by running `dig +short HOSTNAME A`
# Twitter: @oneoffdallas
# Homepage: https://github.com/oneoffdallas/dohservers
# Raw data: https://raw.githubusercontent.com/oneof ... iplist.txt
#
# Added: 14 Oct 2019
# Last modified: 05 Apr 2021

# Optionally include Cloudflare's main addresses (Google and Quad9 included below)
# 1.1.1.1
# 1.0.0.1
#
# security.cloudflare-dns.com - see IPs further down this list
# 1.1.1.2
# 1.0.0.2
#
# family.cloudflare-dns.com
# 1.1.1.3
# 1.0.0.3

# dns.cloudflare.com
# (optional because cdnjs.cloudflare.com uses the same IPs)
# 104.16.132.229
# 104.16.133.229

# NOTES
# Quad9 is also using Port 5053 for DoH: (last checked 11 Jan 2020)
# https://www.quad9.net/doh-quad9-dns-ser ... nformation
#
# dnscrypt.ca is using Port 453 for DoH (instead of 443)

168.235.81.167
176.56.236.175
176.103.130.131
176.103.130.130
176.103.130.132
176.103.130.134
37.252.185.229
206.189.215.75
-------------------------------

So the script itself :

## Generic IP address list input
## Based on a script written by Sam Norris, ChangeIP.com 2008
## Edited by Andrew Cox, AccessPlus.com.au 2008
:if ( [/file get [/file find name=iplist.txt] size] > 0 ) do={
# Remove exisiting addresses from the current Address list
/ip firewall address-list remove [/ip firewall address-list find list=DNS]
:global content [/file get [/file find name=iplist.txt] contents] ;
:global contentLen [ :len $content ] ;
:global lineEnd 0;
:global line "";
:global lastEnd 0;
:while ($lineEnd < $contentLen) do={
:set lineEnd [:find $content "\n" $lastEnd ] ;
:set line [:pick $content $lastEnd $lineEnd] ;
:set lastEnd ( $lineEnd + 1 ) ;
#If the line doesn't start with a hash then process and add to the list
:if ( [:pick $line 0 1] != "#" ) do={
:local entry [:pick $line 0 $lineEnd ]
:if ( [:len $entry ] > 0 ) do={
/ip firewall address-list add list="DNS"
}
}
}
}
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Script to convert ip to address-list

Tue Apr 06, 2021 6:15 pm

As long the file is smaller than 64KB you could use the script written by Shumkov:

viewtopic.php?f=9&t=152632&p=758435
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Script to convert ip to address-list

Tue Apr 06, 2021 7:13 pm

As long the file is smaller than 64KB you could use the script written by Shumkov:

viewtopic.php?f=9&t=152632&p=758435
List is only 5Kb big, but the script does not work, just throws an error in the log as the script is instructed to do.
Looking at the original script, the "list" that mostly resembled the one I try to import is the one below

$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")

################################################################
# abuse.ch SSLBL Botnet C2 IP Blacklist (IPs only) #
# Last updated: 2021-04-04 15:14:45 UTC #
# #
# Terms Of Use: https://sslbl.abuse.ch/blacklist/ #
# For questions please contact sslbl [at] abuse.ch #
################################################################
#
# DstIP
45.141.37.7
193.233.78.102
79.134.225.23
91.200.41.42
140.82.57.172
23.95.0.100
92.223.90.242
193.142.58.181
141.164.36.203
207.32.219.41
45.144.225.107
88.80.186.210
3.138.180.119
45.129.137.247
46.243.221.41
18.224.135.48
45.77.122.108
23.105.131.172


The URL I'm using contains some blank lines too, but I would think the script (looking at the regex) really looks for dotted IP constructs and ignored everything else, including "#" characters etc
And before starting the script i do have a list called "DNS" and it does not remove it, even when I change the script slightly like

ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
remove [find list=DNS]

I would expect it to simply deleted list "DNS" but it remains in place.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10183
Joined: Mon Jun 08, 2015 12:09 pm

Re: Script to convert ip to address-list

Tue Apr 06, 2021 9:25 pm

Of course for this particular purpose you could also make a DNS-based address list. Either using the DNS names from that list or by hosting some domain and loading it with the proper addresses for some name like doh-servers.example.com
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Script to convert ip to address-list

Wed Apr 07, 2021 1:43 am

I have tested it with this version and it works. Run it in terminal and see if it works. If it does not work in script then you have to set the rights. (ftp, read, write, policy, test, password) It could be to much rights but it works for me.
# Written by Shumkov
# Adapted by blacklister
# 20201025
{
/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}

   :while ([:len $data]!=0) do={
      :if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) } on-error={}
     }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}
$update url=https://raw.githubusercontent.com/oneoffdallas/dohservers/master/iplist.txt blacklist="RougeDNS" delimiter=("\n") 
}
It is a really nice bit of code and very sturdy so assumed it work also for this and it does? As long the files are not longer then 63KB this is just the code to use.

This will only accept IP addresses that are at the beginning of the line(RegEx):
"^[0-9]
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Script to convert ip to address-list

Wed Apr 07, 2021 11:02 am

Hi,
Made a copy-paste and now it works indeed! I've set the permissions exactly like you mentioned in your example.

Weird, but I'm happy it works now as I was pulling the last strands of hair from my skull !

Thx!
 
pe1chl
Forum Guru
Forum Guru
Posts: 10183
Joined: Mon Jun 08, 2015 12:09 pm

Re: Script to convert ip to address-list

Wed Apr 07, 2021 11:04 am

I think I would add the addresses to the list with some huge timeout so they are not written to flash...
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Script to convert ip to address-list

Wed Apr 07, 2021 11:28 am

I think I would add the addresses to the list with some huge timeout so they are not written to flash...
This list probably is not so "dynamic" compared to others. So 1 update per day (or even per week) should be OK.
I'm going to check IF there are some hits against the counters anyway.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Script to convert ip to address-list

Wed Apr 07, 2021 3:04 pm

It is more that they are not stored in the flash, so that they are also not put in any backup files.

The version that I had my router, which was inactive, used a foreach instead of a while and I got some extra grey hairs before finding that.

Some how it is burned-in in my brain array-->foreach. Here only one field in the array is used so foreach is only one loop.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Script to convert ip to address-list

Wed Apr 07, 2021 4:27 pm

Nog een versie die de commentaren in het bestand meeneemt en achter de betreffende IP adressen zet...

Oops in English.

Underneath a version that also gets the comments from the file and put them in the address list as comments
# Written by Shumkov
# Adapted by blacklister
# 20210407
{
/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}

   :while ([:len $data]!=0) do={
      :if ([:pick $data 0 1] = "#") do={ :set $Comment [:pick $data 2 [:find $data "\n"]] } 
      :if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
      :do {add list=$blacklist comment=$Comment address=([:pick $data 0 [:find $data $delimiter]].$cidr)} on-error={}
     }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}
$update url=https://raw.githubusercontent.com/oneoffdallas/dohservers/master/iplist.txt blacklist="RougeDNS" delimiter=("\n") 
}
 
pe1chl
Forum Guru
Forum Guru
Posts: 10183
Joined: Mon Jun 08, 2015 12:09 pm

Re: Script to convert ip to address-list

Wed Apr 07, 2021 4:34 pm

I think I would add the addresses to the list with some huge timeout so they are not written to flash...
This list probably is not so "dynamic" compared to others. So 1 update per day (or even per week) should be OK.
I'm going to check IF there are some hits against the counters anyway.
Unfortunately RouterOS does not support counters per address list entry. Plain Linux does, at least in the current version (the first version of "ipset" did not support that).
It would be nice when RouterOS address list had these counters, so you can see how active such entries are. Now you can only count per firewall rule, i.e. for all list items together.
 
tramil
just joined
Posts: 13
Joined: Sun Mar 15, 2020 2:04 am

Re: Script to convert ip to address-list

Tue Oct 18, 2022 11:36 am

I started using this script but don't understand this bit:
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr)
Where does $cidr come from?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Script to convert ip to address-list

Tue Oct 18, 2022 11:41 am

if you provide the cidr parameter to that function, will be used
 
tramil
just joined
Posts: 13
Joined: Sun Mar 15, 2020 2:04 am

Re: Script to convert ip to address-list

Tue Oct 18, 2022 12:12 pm

Ok, got it. It's an optional argument, not used in the example given.
Thanks

Who is online

Users browsing this forum: ko00000000001 and 24 guests