Hi,
Let say I have 7k ips in this text
http://torstatus.blutmagie.de/ip_list_all.php/Tor_ip_list_ALL.csv
I want to convert all those ips into address list using script
something like this
:do {/ip firewall address-list add address=$entry list=list-TOR}
I am following this method but the script can’t create MY-IP-LIST address list
http://wiki.mikrotik.com/wiki/Using_Fetch_and_Scripting_to_add_IP_Address_Lists
Anyone have working script with RouterOS 6.34.2
tq
Not sure if this is a one-time need or maybe you are seeking for a way to automate this; if it needs to be the script way, check for the CSV having suitable end of line characters, where is it failing?
If this is a one time need: a hack or workaround without resorting to scripting would be opening that csv file on a text editor then using find/replace to put
/ip firewall address-list add address=
by searching for the beggining of line, so that text is placed before the IP, and then
list=YOURLIST
by searching for end of line.
Another way would be by using a text editor with block or column mode editing capabilities, like SublimeText or notepad++ for example (see https://notepad-plus-plus.org/features/column-mode-editing.html).
Both by the way have plugins for ROS syntax coloring.
The idea is to end with the CLI commands on the text file; that way you can rename it to .rsc, upload it to the router then do an import on the .rsc.
Following this idea, and as you may be reaching maximum file size for file processing using scripting, you could also generate the .rsc file containing all the CLI to fill the list by coding an smal cgi in bash or php, then just fetching the file from your web server…
I am trying to avoid to use other server to process it.
It can be done by following this method
http://robert.penz.name/983/filter-traffic-from-and-to-tor-ip-addresses-automatically-with-mikrotik-routeros/
I prefer everything using buil-in script in Mikrotik
I think if older RouterOS can do that, newer one will be more capable doing that
hi, yes that script from the wiki is old and does not work on 6.x , however if you make changes to the do / while loop, it will work (move the :while up), change it like this:
:while ($lineEnd < $contentLen) do={
:set lineEnd [:find $content "\n" $lastEnd ] ;
:set line [:pick $content $lastEnd $lineEnd] ;
:set lastEnd ( $lineEnd + 1 ) ;
#If the line doesn't start with a hash then process and add to the list
:if ( [:pick $line 0 1] != "#" ) do={
:local entry [:pick $line 0 $lineEnd ]
:if ( [:len $entry ] > 0 ) do={
/ip firewall address-list add list="MY-IP-LIST"
}
}
}
}
Anyone that can help me out why this does not work ?
Basically I combined some posting in order to make a script that should work, but it only cleans/empties my "DNS" address-list.
I've download the link below containing a bunch of DOH/DOT public servers that I want to convert into an ACL.
Basically I took the (non-working on RouterOS 6.x) script from the WIKI and added the correction that is suggested to make it work again in this post.
But still, list remains empty basically.
The "iplist.txt" contains some line starting with # which should be ignored.
It also contains some "blank" lines, I wonder if that is ok.
dig +short HOSTNAME ASo the script itself :
:if ( [/file get [/file find name=iplist.txt] size] > 0 ) do={
/ip firewall address-list remove [/ip firewall address-list find list=DNS]
:global content [/file get [/file find name=iplist.txt] contents] ;
:global contentLen [ :len $content ] ;
:global lineEnd 0;
:global line "";
:global lastEnd 0;
:while ($lineEnd < $contentLen) do={
:set lineEnd [:find $content "\n" $lastEnd ] ;
:set line [:pick $content $lastEnd $lineEnd] ;
:set lastEnd ( $lineEnd + 1 ) ;
#If the line doesn't start with a hash then process and add to the list
:if ( [:pick $line 0 1] != "#" ) do={
:local entry [:pick $line 0 $lineEnd ]
:if ( [:len $entry ] > 0 ) do={
/ip firewall address-list add list="DNS"
}
}
}
}
As long the file is smaller than 64KB you could use the script written by Shumkov:
http://forum.mikrotik.com/t/address-lists-downloader-dshield-spamhaus-drop-edrop-etc/133640/5
List is only 5Kb big, but the script does not work, just throws an error in the log as the script is instructed to do.
Looking at the original script, the "list" that mostly resembled the one I try to import is the one below
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
################################################################
################################################################
45.141.37.7
193.233.78.102
79.134.225.23
91.200.41.42
140.82.57.172
23.95.0.100
92.223.90.242
193.142.58.181
141.164.36.203
207.32.219.41
45.144.225.107
88.80.186.210
3.138.180.119
45.129.137.247
46.243.221.41
18.224.135.48
45.77.122.108
23.105.131.172
The URL I'm using contains some blank lines too, but I would think the script (looking at the regex) really looks for dotted IP constructs and ignored everything else, including "#" characters etc
And before starting the script i do have a list called "DNS" and it does not remove it, even when I change the script slightly like
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
remove [find list=DNS]
I would expect it to simply deleted list "DNS" but it remains in place.
Of course for this particular purpose you could also make a DNS-based address list. Either using the DNS names from that list or by hosting some domain and loading it with the proper addresses for some name like doh-servers.example.com
I have tested it with this version and it works. Run it in terminal and see if it works. If it does not work in script then you have to set the rights. (ftp, read, write, policy, test, password) It could be to much rights but it works for me.
# Written by Shumkov
# Adapted by blacklister
# 20201025
{
/ip firewall address-list
:local update do={
:do {
:local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
:do { remove [find list=$blacklist] } on-error={}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) } on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
} ; :log warning "Imported address list < $blacklist> from file: $url"
} else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
} on-error={:log warning "Address list <$blacklist> update failed"}
}
$update url=https://raw.githubusercontent.com/oneoffdallas/dohservers/master/iplist.txt blacklist="RougeDNS" delimiter=("\n")
}
It is a really nice bit of code and very sturdy so assumed it work also for this and it does? As long the files are not longer then 63KB this is just the code to use.
This will only accept IP addresses that are at the beginning of the line(RegEx):
"^[0-9]
Hi,
Made a copy-paste and now it works indeed! I’ve set the permissions exactly like you mentioned in your example.
Weird, but I’m happy it works now as I was pulling the last strands of hair from my skull !
Thx!
I think I would add the addresses to the list with some huge timeout so they are not written to flash…
This list probably is not so “dynamic” compared to others. So 1 update per day (or even per week) should be OK.
I’m going to check IF there are some hits against the counters anyway.
It is more that they are not stored in the flash, so that they are also not put in any backup files.
The version that I had my router, which was inactive, used a foreach instead of a while and I got some extra grey hairs before finding that.
Some how it is burned-in in my brain array–>foreach. Here only one field in the array is used so foreach is only one loop.
Nog een versie die de commentaren in het bestand meeneemt en achter de betreffende IP adressen zet…
Oops in English.
Underneath a version that also gets the comments from the file and put them in the address list as comments
# Written by Shumkov
# Adapted by blacklister
# 20210407
{
/ip firewall address-list
:local update do={
:do {
:local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
:do { remove [find list=$blacklist] } on-error={}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 1] = "#") do={ :set $Comment [:pick $data 2 [:find $data "\n"]] }
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:do {add list=$blacklist comment=$Comment address=([:pick $data 0 [:find $data $delimiter]].$cidr)} on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
} ; :log warning "Imported address list < $blacklist> from file: $url"
} else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
} on-error={:log warning "Address list <$blacklist> update failed"}
}
$update url=https://raw.githubusercontent.com/oneoffdallas/dohservers/master/iplist.txt blacklist="RougeDNS" delimiter=("\n")
}
Unfortunately RouterOS does not support counters per address list entry. Plain Linux does, at least in the current version (the first version of “ipset” did not support that).
It would be nice when RouterOS address list had these counters, so you can see how active such entries are. Now you can only count per firewall rule, i.e. for all list items together.
I started using this script but don’t understand this bit:
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr)
Where does $cidr come from?
if you provide the cidr parameter to that function, will be used
Ok, got it. It’s an optional argument, not used in the example given.
Thanks