This is some documentation I put together on using a Python script I wrote to parse DShield’s block list and import into the MikroTik automatically on schedule. I put some serious time into this and hope it helps some of you out. Please let me know what you think.
http://infaland.weebly.com/infablog/mikrotik-automating-dshield-block-list
Best Regards,
Infatuas
Great job. Have you considered updating the script to process the top 100 list instead?
The topip list is so simple to parse you may as well do it with scripting alone.
Here’s one code that I haven’t actually tested, but should probably work.
/system scheduler add name=DShield_top interval=1d on-event={
:local addressListName "DShield_TOP";
:local addressListTimeout 1d;
/tool fetch keep-result=yes url="http://feeds.dshield.org/topips.txt";
#To ensure the file is readable
:delay 2s;
:local contents [/file get "topips.txt" contents];
:while ($contents != "") do={
:local ip [:pick $contents 0 [:find $contents " "]];
/ip firewall address-list add address=$ip list=$addressListName timeout=$addressListTimeout;
:set contents [:pick $contents ([:find $contents "\n"]+1)];
};
};
boen
thanks for the script. I did create a python version that parses BOTH lists into a single file so that it can be imported by Mikrotik automatically if anyone wants it.
import urllib.request
import re
dshield_url = “http://feeds.dshield.org/topips.txt”
dshield_raw_text = urllib.request.urlopen(dshield_url)
dshield_string = str(dshield_raw_text.read())
ipv4_pattern = r’\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}’
iplist = re.findall( ipv4_pattern, dshield_string )
dshield_rsc = “.\DShield.rsc”
dshield_clean_text = open(dshield_rsc, “w”)
dshield_clean_text.write(“/ip firewall address-list\n”)
for ip in iplist:
dshield_clean_text.write(“add list=DShield address=” + (ip) + “/32\n”)
dshield_url = “http://feeds.dshield.org/block.txt”
dshield_raw_text = urllib.request.urlopen(dshield_url)
dshield_string = str(dshield_raw_text.read())
ipv4_pattern = r’\d{1,3}.\d{1,3}.\d{1,3}.0’
iplist = re.findall( ipv4_pattern, dshield_string )
for ip in iplist:
dshield_clean_text.write(“add list=DShield address=” + (ip) + “/24\n”)
dshield_clean_text.close()
Another great list to process : http://rules.emergingthreats.net/blockrules/compromised-ips.txt
Be sure to use:
ipv4_pattern = r’\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}’
for processing this list in python.
Brian, the script you wrote to parse both works when writing to the output file but if you import into the mikrotik it probably wont import both the /32 and the /24 because in most cases the /32 addresses reside within the /24 subnets. The current OS only seems to take the /32 subnets on import instead of the /24 when they are on the same import list. I personally will continue to use the /24.
Boen,
I agree that it can be done on the router itself but I assume you are not considering needing to make a change to the script and manually changing it on 500 different devices. I prefer centralized scripting in that case.
In that case, using the API is a better approach. Instead of asking each router to “pull” in changes by setting a script on them, you “push” the changes to them (no script inside necessary).
What’s even better is that you don’t really need to check whether the first value is an IP address (or subnet), since the API protocol ensures that the value will not “leak” into other arguments. Worst case scenario is that the “add” command would fail - no harm done.
Using PHP f.e.
<?php
use PEAR2\Net\RouterOS;
require_once 'PEAR2_Net_RouterOS-1.0.0b5.phar';
//<config>
$url = 'http://feeds.dshield.org/topips.txt';
$listName = 'DShield_TOP';
$listTimeout = '1d';
$routers = array(
array('192.168.88.1', 'admin', 'password'),
array('192.168.88.2', 'admin', 'password'),
//etc.
array('192.168.88.254', 'admin', 'password')
);
//</config>
$addRequest = new RouterOS\Request('/ip firewall address-list add');
$addRequest
->setArgument('list', $listName)
->setArgument('timeout', $listTimeout);
foreach ($routers as $router) {
$client = new RouterOS\Client($router[0], $router[1], $router[2]);
foreach (file($url, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES) as $line) {
$linePieces = preg_split('/\s+/', $line);
if ($linePieces[0][0] === '#') {//Line is a comment; skip it
continue;
}
$ip = $linePieces[0];
$domain = isset($linePieces[1]) ? $linePieces[1] : null;
$addRequest
->setArgument('address', $ip)
->setArgument('comment', $domain);
$client->sendSync($addRequest);
}
}
The same could be done analogously for Python too, or other languages.
(P.S. The script above is completely agnostic with regards to which one of all above URLs is used. Just switch it up, or make an array with all URLs and loop it to get all IPs of all lists)
I love your adaptation but I’m not able to get it to work with the HPB list
$url = ‘http://www.dshield.org/hpb.html?key=oiUTq74ue5KvKQXfZYxsXw==’;
$listName = ‘DShield_HPB’;
Any ideas? No errors but no DSHIELD_HPB address lists added after its processed.
I found the script imports both the /32 and /24 lists without issue. If you look at the lists, most of the /32 are NOT in the /24 subnet lists! If even one or two are the Mikrotik will treat them as two separate entries. I have the block/top lists processed in one python script as DSHIELD_BLOCK list. I have the emergingthreats processed in another script as EmergingThreats address list. I’ve removed the ‘comment’ line because its not needed when using timeout=2h
A duplicate entry in the import will cause RouterOS to stop and not process further down the list (v6.16)
ie
/ip firewall address-list
add list=DShield address=122.226.73.131/32 timeout=2h
add list=DShield address=85.13.160.11/32 timeout=2h
add list=DShield address=50.23.113.146/32 timeout=2h
add list=DShield address=50.23.113.146/32 timeout=2h
add list=DShield address=122.225.109.212/32 timeout=2h
add list=DShield address=176.31.251.200/32 timeout=2h
add list=DShield address=218.2.0.137/32 timeout=2h
add list=DShield address=122.226.73.131/32 timeout=2h
It will import 50.23.113.146 and stop processing the import. Bug in RouterOS?
Using /import file-name=dshield.rsc; to import list
Well, THAT new list is a little more complicated to parse, so no wonder.
Replace
$ip = $linePieces[0];
$domain = isset($linePieces[1]) ? $linePieces[1] : null;
with
if ($linePieces[0] === 'Start') {//Line is a header; skip it
continue;
}
if (count($linePieces) >= 4) {//network-zerofilled, broadcast-zerofilled, mask, attacks(, name(, country(, email)?)?)?
//Remove zerofill
$ipPieces = explode('.', $linePieces[0]);
for ($i=0, $c=count($ipPieces); $i<$c; ++$i) {
$ipPieces[$i] = ltrim($ipPieces[$i], '0');
if ('' === $ipPieces[$i]) {
$ipPieces[$i] = '0';
}
}
$ip = implode('.', $ipPieces) . '/' . $linePieces[2];
$domain = implode(' ; ', array_slice($linePieces, 3));//Use all meta data as the comment
} else {//subnet(, dns)?
$ip = $linePieces[0];
$domain = isset($linePieces[1]) ? $linePieces[1] : null;
}
Gave it a try but it still doesn’t add any lists. no error. Thanks for trying
I forgot to restore the octet to “0” when removing the zeroes in “000”. I’ve corrected the above. Try it now.
(If that too fails, I’ll do something which I hadn’t actually done yet: Test the code)
It works great! Populates the router firewall with DSHIELD_HPB addresses on a 24 hour timeout. Scheduled it to run once a day so it keeps the list fresh and have firewall blocking any traffic from DSHIELD_HPB. Very nice! Thank you!
I apologize for revisiting what seems to be a older thread. I’ve imported this script, but it doesn’t seem to create the address list. Any suggestions?