add name="Startup updateBlacklist" on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test start-time=startup
/system scheduler
add interval=1d name=UpdateBlackListDaily on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=jan/01/1970 start-time=05:00:00
add name=UpdateBlackListOnReboot on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-time=startup
/ip firewall raw
add action=drop chain=prerouting comment=\
"Drop connections from Blacklisted addresses" src-address-list=\
dynamicBlacklist
add action=drop chain=prerouting comment=\
"Drop connections to Blacklisted addresses" dst-address-list=\
dynamicBlacklist
Are you sure because I see twice the same script run command.
2. The run command differed in both schedules (run updateBlacklist) VS (run blacklistUpdate)
Here is what I ended up with and works as expected. List reloads about 30 seconds after reboot.
Code: Select all/system scheduler add interval=1d name=UpdateBlackListDaily on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=jan/01/1970 start-time=05:00:00 add name=UpdateBlackListOnReboot on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-time=startup
# Script will now download IP blacklists
/tool fetch url="http://www.securelan.eu/mikrotik/torexitnodes.rsc" mode=http;
:log info "Downloaded torexitnodes.rsc from SecureLAN.eu";
/tool fetch url="http://www.securelan.eu/mikrotik/openbl.rsc" mode=http;
:log info "Downloaded openbl.rsc from SecureLAN.eu";
/tool fetch url="http://www.securelan.eu/mikrotik/spamhaus.rsc" mode=http;
:log info "Downloaded spamhaus.rsc from SecureLAN.eu";
/tool fetch url="http://www.securelan.eu/mikrotik/dshield.rsc" mode=http;
:log info "Downloaded dshield.rsc from SecureLAN.eu";
/tool fetch url="http://www.securelan.eu/mikrotik/malc0de.rsc" mode=http;
:log info "Downloaded malc0de.rsc from SecureLAN.eu";
# Script will now replace old blacklists with the new ones
/ip firewall address-list remove [find where comment="TorExitNodes"]
/import file-name=torexitnodes.rsc;
:log info "TorExitNodes records updated successfully.";
/ip firewall address-list remove [find where comment="DShield"]
/import file-name=dshield.rsc;
:log info "DShield records updated successfully.";
/ip firewall address-list remove [find where comment="SpamHaus"]
/import file-name=spamhaus.rsc;
:log info "SpamHaus records updated successfully.";
/ip firewall address-list remove [find where comment="OpenBL"]
/import file-name=openbl.rsc;
:log info "OpenBL records updated successfully.";
/ip firewall address-list remove [find where comment="malc0de"]
/import file-name=malc0de.rsc;
:log info "Malc0de records updated successfully.";
:log info "All blacklist records were updated successfully.";Does your list contain also TOR network exit nodes? If not, you can probably add it.I've gone ahead and started publishing my dynamic filter list for RouterOS 6.x. My server generates the list each night after collecting data on all known botnets, C&C server, and spammers. Currently the list runs about 3k entries, so it may not work well on low end routers. Here is the script to update the list, as well as my personal firewall rules. As always, adjust them to fit your needs.
for me, in enterprise environment, tor should not be allowed. Check my post earlier. Could you compare my lists with yours? Probably there's something to make better...on both. So if you can convert the list and put it in a DNS, then one record/domain name will supply all IP addresses in one go.Name: microsoft.com
Addresses: 23.100.122.175
23.96.52.53
191.239.213.197
104.40.211.35
104.43.195.251
Good to see the grow from 2700 to 4000 clients in the last seven months.Just hit 4000 active routers using the BlackList.
Notable users are T-Mobile, using it on there Fixed LTE deployments. And even more so, several US Government sites have begun pulling the list.
If you insist in doing it via DNS then look into rbldnsd which is designed for exactly this purpose. You can feed it a list of IP's/hostnames and it can respond with whatever you want. RBL's used for mail etc commonly uses this method for their black/white or rep lists.I have an idea how to bring back the traffic generated by the Blacklist.
When I lookup sites I get sometimes a list of IP addresses back:
So if you can convert the list and put it in a DNS, then one record/domain name will supply all IP addresses in one go.Name: microsoft.com
Addresses: 23.100.122.175
23.96.52.53
191.239.213.197
104.40.211.35
104.43.195.251
You could to make weekday's list like monday.blacklist.xxx / tuesday.blacklist.xxx....sunday.blacklist.xxx
Give the DNS-record a lifetime of 24+1 hour and remove the that day when it is the next day is generated and is uploaded. In this way the your are sure that the cache DNS servers up the stream are cleaned to read in that weekday.blacklist.xxx when there is an request for it in the Internet.
When a weekday*7.backlist.xxx is in the cache of the DNS in the Mikrotik, you only need one line in the address list to be able to filter. I think that a script is useful to make a hard delete of the outdated weekday to make room for the new weekday list.
The DNS of the provider/supplier which the Mikrotik owner is using is handling the traffic now. You have each day a one time upload and the the DNS structure is distributing your list for you. Delays are common and because the used weekday was not present for the last 5 days so there should be direct request to the DNS.
This way of working I already use myself and I put the extra IP addresses in the host file on the machine where my DMSmasq is running. DNSmasq reads the host file and returns the list of IP addresses when the domainname is requested. In doing so I have only need one line to be able to filter more addresses in one go.
I don't know if this is possible or even legal to use the DNS in that way.....
updated: 12 February 2017
Distributed & cached which the cache will lower the amount of traffic needed.Using RBL's crossed my mind but then the amount of traffiic would be the same like it is with BGP.
When using DNS you will also have some traffic but the main part is distributed by external DNS severs as I see it.
# Import Intrus Managed Filter Lists
# (C)2016 David Joyce, Intrus Technologies
{
:log warning "Blacklist download will start in 30 seconds..."
:delay 30
:local model [/system resource get board-name]
:local version [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
:local scriptVer "2016.7.4a (Deantwo)"
:put "Script version: $scriptVer"
:log warning "Downloading current Blacklist for this model"
/tool fetch mode=https dst-path="/dynamic.rsc" \
url="https://mikrotikfilters.com/download.php\?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname&ver=$scriptVer"
:log warning "Disabling info logging..."
/system logging disable 0
:log warning "Removing expiring address-list entries..."
/ip firewall address-list remove [find list="dynamicBlacklist"]
:log warning "Importing current Blacklist..."
/import file-name=/dynamic.rsc
:log warning "Removing temp file..."
/file remove dynamic.rsc
:log warning "Blacklist Update Complete."
/system logging enable 0
}
Quotes from IntrusDaveWhen testing it I also found that the address-list entry timeout don't quite match up with what you say in the opening post.
It states that the address-list entries are dynamic with a 48 hour timeout, but the file I am getting shows them having a 24 hour timeout. Shouldn't this at least be 25 hours to patch the possible hole between updates, or be changed back to 48 hours?
My server collects the banned IP's 24/7 and publish the list at 3am PST.
That means that the ip/ subnet is or has been serving malware for at least 12 hours. The list is automated and will remove the address once it has been clean for 24 hours.
I will not manually remove addresses.
Doesn't explain why the timeout of the dynamic address-list entries is only 24 hours when it is stated in the opening post that the timeout is 48 hours.Quotes from IntrusDaveWhen testing it I also found that the address-list entry timeout don't quite match up with what you say in the opening post.
It states that the address-list entries are dynamic with a 48 hour timeout, but the file I am getting shows them having a 24 hour timeout. Shouldn't this at least be 25 hours to patch the possible hole between updates, or be changed back to 48 hours?My server collects the banned IP's 24/7 and publish the list at 3am PST.That means that the ip/ subnet is or has been serving malware for at least 12 hours. The list is automated and will remove the address once it has been clean for 24 hours.
I will not manually remove addresses.
If nothing else the opening post just needs to be updated.The address-list entries are now Dynamic with a 48 hour timeout. This will cut the number of writes to NAND down dramatically.
/system script run UpdateBlacklistI was mostly asking because we have customer numbers and names as router identity, so I may be forced to not send you those if we start using your service.I use the identity to group the routers for stats and troubleshooting. Example; all of my routers ID's start with "Intrus :: " this allows me to sort them and quickly track down problems. While it's not currently required, it really is the only method that I have to keep track of how many routers are active daily. I do not use the serial number because I feel that is too invasive to request. I can not go by IP, because many are behind the same proxies. I could use the WAN MAC address, but I was betting that some would object to that too.
/system scheduler
add name="MyScheduler1" \
start-time=startup \
policy=read,write,test \
on-event=":delay 120\r\
\n/system script run \"MyScript1\""
Not 100% sure rather or not to add the "start-date=jan/01/1970" to the scheduler, since I haven't messed with them for a while. But the scheduler I posted does work, and I use a two minute delay before calling my scripts because I need to be sure that VPN tunnels are up.As for the schedule, you will have to play with it. It was originally setup back when the routers didn't store the date and time over a reboot, so on first boot the date and time was "1970-01-01 00:00:00". RouterOS seems to have some issues with startup scripts, and I haven't had time to work out what needs to be changed.
RouterOS seems to have some issues with startup scripts, and I haven't had time to work out what needs to be changed.
Found a little error in your provided example firewall.And, if you are interested, here are my filter rules:Code: Select all/ip firewall address-list add address=172.16.0.0/16 list=PrivateIPs add address=10.0.0.0/8 list=PrivateIPs add address=192.168.0.0/16 list=PrivateIPs
/ip firewall address-list
add address=10.0.0.0/8 list=PrivateIPs
add address=172.16.0.0/12 list=PrivateIPs
add address=192.168.0.0/16 list=PrivateIPs# Generated on Mon Mar 20 04:00:54 PDT 2017 by Intrus Technologies
/ip firewall address-list
add list=dynamicBlacklist address=1.10.16.0/20 timeout="1d 01:00:00" comment=Blacklisted
# Omited 5226 lines.
add list=dynamicBlacklist address=42.62.51.27 timeout="1d 01:00:00" comment=Blacklisted
add list=dynamicBlacklist address=42.83.80.0/22 timeout="1d 01mar/21 05:21:57 script,error script error: failure: already have such entry
mar/20 05:21:44 script,error script error: expected end of command (line 5586 column 70)
mar/19 05:21:56 script,error script error: expected end of command (line 5770 column 27)
mar/17 05:22:08 script,error script error: value of address expects range of ip addresses
mar/16 05:22:09 script,error script error: invalid time value for argument timeoutOk yeah, maybe I was a little hasty to my conclusion.You issue is that the router simply didn't complete the download. Today's download is 603k. If it's getting out off, you may want to see if your ISP is trying to proxy ssl connections.
[deantwo@router] > /tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=RB493&version=6.35 (stable)&memory=67108864&id=A-----:############&ver=2016.7.4a (Deantwo)"
status: finished
downloaded: 496KiB
total: 603KiB
duration: 3s
[deantwo@router] > /tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=RB493&version=6.35 (stable)&memory=67108864&id=A-----:############&ver=2016.7.4a (Deantwo)"
status: finished
downloaded: 336KiB
total: 603KiB
duration: 2s
[deantwo@router] > /tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=RB493&version=6.35 (stable)&memory=67108864&id=A-----:############&ver=2016.7.4a (Deantwo)"
status: finished
downloaded: 510KiB
total: 603KiB
duration: 2s
[deantwo@router] > /tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=RB493&version=6.35 (stable)&memory=67108864&id=A-----:############&ver=2016.7.4a (Deantwo)"
status: finished
downloaded: 460KiB
total: 603KiB
duration: 3s+-----+--------------------+
| QTY | model |
+-----+--------------------+
| 721 | RB951G-2HnD |
| 548 | RB2011UiAS-2HnD |
| 374 | RB2011UiAS |
| 309 | hAP+ac |
| 298 | RB951Ui-2HnD |
| 182 | RB751G-2HnD |
| 178 | CCR1016-12G |
| 174 | SXT+Lite5 |
| 166 | CCR1009-8G-1S-1S+ |
| 159 | RB3011UiAS |
| 148 | hAP+lite |
| 114 | RB850Gx2 |
| 112 | RB450G |
| 102 | RB750GL |
| 94 | RB750 |
| 82 | hEX |
| 81 | CCR1036-12G-4S |
| 78 | RB1100AHx2 |
| 68 | hAP+ac+lite |
| 65 | RB2011UAS |
| 64 | SXT+LTE |
| 54 | CRS109-8G-1S-2HnD |
| 53 | CHR |
| 52 | x86 |
| 47 | RB493G |
| 45 | hEX+lite |
| 40 | mAP |
| 40 | hAP |
| 30 | CCR1009-8G-1S |
| 30 | RB912UAG-2HPnD |
| 28 | RB912UAG-5HPnD |
| 25 | RB+Groove+5Hn |
| 22 | mAP+lite |
| 21 | CCR1036-8G-2S+ |
| 20 | CRS125-24G-1S |
| 18 | RB2011UAS-2HnD |
| 17 | RB751U-2HnD |
| 16 | RB2011L |
| 15 | RB2011iL |
| 12 | RB750UP |
| 8 | CCR1016-12S-1S+ |
| 6 | RB1100 |
| 6 | RB1200 |
| 6 | RB951-2n |
| 5 | CRS125-24G-1S-2HnD |
| 4 | RB1100AH |
| 4 | RB750G |
| 4 | RB2011iLS |
| 4 | RB433 |
| 2 | OmniTIK+5+ac |
| 2 | CRS226-24G-2S+ |
| 2 | RB1100Hx2 |
| 2 | hEX+PoE |
| 2 | hEX+PoE+lite |
| 2 | %24model |
| 2 | CCR1009-7G-1C |
| 2 | CCR1009-7G-1C-1S+ |
| 2 | RB2011LS |
| 1 | RB+SXT+5HnD |
| 1 | RB433AH |
| 1 | RB800 |
| 1 | GrooveA+52 |
| 1 | CCR1072-1G-8S+ |
| 1 | PowerBOX |
| 1 | RB750r2 |
| 1 | SXT+Lite5+ac |
| 1 | RB333 |
| 1 | 911+Lite5+dual |
| 1 | RB1100AH2X |
| 1 | RB1000 |
| 1 | RB911G-5HPnD |
| 1 | RB+OmniTIK+U-5HnD |
| 1 | RB493 |
| 1 | RB450 |
| 1 | BaseBox+5 |
| 1 | wAP+ac |
| 1 | RB600 |
| 1 | |
+-----+--------------------+
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1 connection-nat-state=!dstnat
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2 connection-nat-state=!dstnatI dont know why, but it finnally works in my CCR1072, thanks for your help!Unfortunately, I don't know how to help you with this. I don't see any errors in my server logs. I can only assume that you are getting ssl errors. You should be able to manually install the scripts from the first post.
/system script run updateBlacklist;I am confused by this about using RAW. Using the filters for incoming traffic in the RAW part not as efficient?The filters are intended to be used as incoming filters, not outgoing. If you change your rules to only block new connections coming in on the WAN interface, all should be good. I don't recommend using the list with the RAW filters.
By blocking incoming on the WAN and new connections, you prevent the attacks, but you do not block new outbound connections.
Thanks for your reply, I will try pasting it via Notepad later this week.Every time that I have seen a 400 Error, it is because the Copy/Paste didn't work. Something is the script is wrong... Maybe it has extra formatting, or maybe invalid characters. Make sure the OS that you are using supports UTF-8. Try copying and pasting the script to Notepad, and then copying and pasting into WinBox.
chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist
chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist
chain=output action=drop log=yes log-prefix="" src-address-list=dynamicBlacklist
chain=output action=drop log=yes log-prefix="" dst-address-list=dynamicBlacklist
should it like this ? and what about order of the rules ? is that correct ?Yes, You can create an address list with addresses that you never want blocked, then add an accept rule above the drop rules.
8 chain=prerouting action=accept log=no log-prefix="" src-address-list=Exceptions dst-address-list=Exceptions
9 chain=output action=accept log=no log-prefix="" src-address-list=Exceptions dst-address-list=Exceptions
10;;; BlackList
chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist
11 ;;; BlackList
chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist
12 ;;; BlackList
chain=output action=drop log=yes log-prefix="" src-address-list=dynamicBlacklist
13;;; BlackList
chain=output action=drop log=yes log-prefix="" dst-address-list=dynamicBlacklist
Can you send me the script you are using?That is the same unit I use for writing my scripts. I have just over 500 of them pulling the list every morning. The error you posted is almost always a simple format or encoding error.
:local model [/system resource get board-name]
:local version [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
:local scriptVer 2016.7.4a
"https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname&ver=$scriptVer";
Like this?Can you give me an update URL without or with preset variables?
...
I think it is going wrong with the URL containing (maybe unknown) variables.
/tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=750&version=6.39&memory=33554432&id=mk13139&ver=DeanHelp";should it like this ? and what about order of the rules ? is that correct ?Yes, You can create an address list with addresses that you never want blocked, then add an accept rule above the drop rules.
ThanksCode: Select all8 chain=prerouting action=accept log=no log-prefix="" src-address-list=Exceptions dst-address-list=Exceptions 9 chain=output action=accept log=no log-prefix="" src-address-list=Exceptions dst-address-list=Exceptions 10;;; BlackList chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist 11 ;;; BlackList chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist 12 ;;; BlackList chain=output action=drop log=yes log-prefix="" src-address-list=dynamicBlacklist 13;;; BlackList chain=output action=drop log=yes log-prefix="" dst-address-list=dynamicBlacklist
Yes exactly!Can you give me an update URL without or with preset variables?
...
I think it is going wrong with the URL containing (maybe unknown) variables.Like that?Code: Select all/tool fetch mode=https dst-path="/dynamic.rsc" \ url="https://mikrotikfilters.com/download.php\?get=dynamic&model=750&version=6.39&memory=33554432&id=mk13139&ver=DeanHelp";
/tool fetch mode=https dst-path="/dynamic.rsc" \
url="https://mikrotikfilters.com/download.php?get=dynamic&model=RB2011UAS-2HnD&version=6.38.5+(stable)&memory=128.0MiB&id=MikroTik+router&ver=2016.7.4a";Out of curiosity, what does your router say to the following if you paste it in the terminal?I think it is going wrong with the URL containing (maybe unknown) variables.
:put [/system resource get board-name]
:put [/system resource get version]
:put [/system resource get total-memory]
:put [/system identity get name]# Import Intrus Managed Filter Lists
# ©2016-2017 David Joyce, Intrus Technologies
:log warning "Blacklist update in 30 seconds";
# :delay 10
:local model [/system resource get board-name]
:local version [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
:local scriptVer 2017.5.2b
:local name ""
:local ver ""
:for i from=0 to=([:len $uname] - 1) do={
:local char [:pick $uname $i]
:if ($char = " ") do={ :set $char "%20" }
:set name ($name . $char)
}
:for i from=0 to=([:len $version] - 1) do={
:local char [:pick $version $i]
:if ($char = " ") do={
:set $char "%20"
}
:set ver ($ver . $char)
}
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/dynamic.rsc" \
url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer";
:log warning "Disabling info logging...";
/system logging disable 0
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
do={ /ip firewall address-list remove $i } }
:log warning "Importing current Blacklist...";
/import file-name=/dynamic.rsc
:log warning "Removing temp file...";
/file remove dynamic.rsc
:log warning "Blacklist Update Complete.";
/system logging enable 0add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1 connection-nat-state=!dstnat
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2 connection-nat-state=!dstnat8 ;;; Exceptions
chain=prerouting action=accept log=no log-prefix="" src-address-list=exceptions
9 ;;; Exceptions
chain=prerouting action=accept log=no log-prefix="" dst-address-list=exceptions
10 ;;; Exceptions
chain=output action=accept log=no log-prefix="" src-address-list=exceptions
11 ;;; Exceptions
chain=output action=accept log=no log-prefix="" dst-address-list=exceptions
12 ;;; BlackList
chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist
13 ;;; BlackList
chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist
14 ;;; BlackList
chain=output action=drop log=no log-prefix="blcklist src" src-address-list=dynamicBlacklist
15 ;;; BlackList
chain=output action=drop log=no log-prefix="blcklist dst" dst-address-list=dynamicBlacklist
16 chain=prerouting action=drop dst-port=22,23,8290,8291 log=no log-prefix="" protocol=tcp src-address-list=!secure
17 chain=prerouting action=drop dst-port=22,23,8290,8291 log=no log-prefix="" protocol=udp src-address-list=!secure
18 chain=output action=drop dst-port=22,23,8290,8291 log=no log-prefix="" protocol=tcp src-address-list=!secure
19 chain=output action=drop dst-port=22,23,8290,8291 log=no log-prefix="" protocol=udp src-address-list=!secure
20 chain=prerouting action=drop in-interface=wan dst-port=53 log=no log-prefix="" protocol=tcp
21 chain=prerouting action=drop in-interface=wan dst-port=53 log=no log-prefix="" protocol=udp
Thanks! I will check it out when I'm on location.Give this a try...
Code: Select all# Import Intrus Managed Filter Lists # ©2016-2017 David Joyce, Intrus Technologies :log warning "Blacklist update in 30 seconds"; # :delay 10 :local model [/system resource get board-name] :local version [/system resource get version] :local memory [/system resource get total-memory] :local uname [/system identity get name] :local scriptVer 2017.5.2b :local name "" :local ver "" :for i from=0 to=([:len $uname] - 1) do={ :local char [:pick $uname $i] :if ($char = " ") do={ :set $char "%20" } :set name ($name . $char) } :for i from=0 to=([:len $version] - 1) do={ :local char [:pick $version $i] :if ($char = " ") do={ :set $char "%20" } :set ver ($ver . $char) } :log warning "Downloading current Blacklist for this model"; /tool fetch mode=https dst-path="/dynamic.rsc" \ url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer"; :log warning "Disabling info logging..."; /system logging disable 0 :log warning "Removing expiring address-list entries..."; :foreach i in=[/ip firewall address-list find ] \ do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \ do={ /ip firewall address-list remove $i } } :log warning "Importing current Blacklist..."; /import file-name=/dynamic.rsc :log warning "Removing temp file..."; /file remove dynamic.rsc :log warning "Blacklist Update Complete."; /system logging enable 0
Actually, Im glad to inform you today that the current release has added a new patch for greatly improved import speed for the importing of static dns entries, one thing you will notice is that, the cpu usage is no longer at 100% during import and the import process is much faster. I will be doing some benchmarks of RouterOS before and after the patch to demonstrate the difference, and it is a remarkable improvement indeed.it's very possible to do that, but I would need to see what the impact on the routers would be. I'm not a big fan of the built-in DNS as it is and I'm not sure how well it would hold up with several thousand hostnames added to it.
/system scheduler
add interval=1d name=UpdateBlackList on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive \
start-date=jan/01/1970 start-time=05:00:0
/system scheduler
add interval=00:00:00 name=UpdateBlackList on-event="/system script run blacklistUpdate" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive \
start-date=jan/01/1970 start-time=00:00:0/ip firewall filter
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=dynamicBlacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=dynamicBlacklistTwo schedulers can have the same name, it is weird to have though.1. Why there are 2 schedules? And if there are 2 it can't have the same name as in your example.Code: Select all/system scheduler add interval=1d name=UpdateBlackList on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive \ start-date=jan/01/1970 start-time=05:00:0 /system scheduler add interval=00:00:00 name=UpdateBlackList on-event="/system script run blacklistUpdate" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive \ start-date=jan/01/1970 start-time=00:00:0
The "Attacks" chain is a custom chain, take a look at the jump rules further down.2. What kind of chain is this "Attacks" ? It should be input or forward chain, am I right ?Code: Select all/ip firewall filter add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=dynamicBlacklist add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=dynamicBlacklist
/ip firewall filter
#...
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
#...
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
# Import Intrus Managed Filter Lists
# (C)2016 David Joyce, Intrus Technologies
:log warning "Blacklist update in 30 seconds";
# :delay 10
:local model [/system resource get board-name]
:local version [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
:local scriptVer 2017.5.2a
:local name ""
:local ver ""
:for i from=0 to=([:len $uname] - 1) do={
:local char [:pick $uname $i]
:if ($char = " ") do={ :set $char "%20" }
:set name ($name . $char)
}
:for i from=0 to=([:len $version] - 1) do={
:local char [:pick $version $i]
:if ($char = " ") do={
:set $char "%20"
}
:set ver ($ver . $char)
}
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/usb1/dynamic.rsc" \
url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer";
:log warning "Disabling info logging...";
/system logging disable 0
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
do={ /ip firewall address-list remove $i } }
:log warning "Importing current Blacklist...";
/import file-name=/usb1/dynamic.rsc
:log warning "Removing temp file...";
/file remove usb1/dynamic.rsc
:log warning "Blacklist Update Complete.";
/system logging enable 0# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies
##### Update your path, is you are using a USB Flash or other storage
:global datapath "disk1/dynamic.rsc"
###### DO NOT EDIT BELOW THIS POINT ######
##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10
##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different
##### routers behind a NAT router. Please do not remove it.
:local model [/system resource get board-name]
:local version [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
:local softid [/system license get software-id]
:local scriptVer 2017.5.30c
##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""
:for i from=0 to=([:len $uname] - 1) do={
:local char [:pick $uname $i]
:if ($char = " ") do={ :set $char "%20" }
:set name ($name . $char)
}
:for i from=0 to=([:len $version] - 1) do={
:local char [:pick $version $i]
:if ($char = " ") do={
:set $char "%20"
}
:set ver ($ver . $char)
}
#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="$datapath" \
url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";
##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0
##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
do={ /ip firewall address-list remove $i } }
##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="$datapath"
##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove [find name=$datapath]
##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies
##### Update your path, is you are using a USB Flash or other storage
:global datapath "usb1/"
:global datafile "dynamic.rsc"
###### DO NOT EDIT BELOW THIS POINT ######
##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10
##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different
##### routers behind a NAT router. Please do not remove it.
:local model [/system resource get board-name]
:local version [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
:local softid [/system license get software-id]
:local scriptVer 2017.5.30b
##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""
:for i from=0 to=([:len $uname] - 1) do={
:local char [:pick $uname $i]
:if ($char = " ") do={ :set $char "%20" }
:set name ($name . $char)
}
:for i from=0 to=([:len $version] - 1) do={
:local char [:pick $version $i]
:if ($char = " ") do={
:set $char "%20"
}
:set ver ($ver . $char)
}
#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/$datapath$datafile" \
url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";
##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0
##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
do={ /ip firewall address-list remove $i } }
##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="/$datapath$datafile"
##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove "$datapath$datafile"
##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies
##### Update your path, is you are using a USB Flash or other storage
:global datapath "disk-8G/"
:global datafile "dynamic.rsc"
###### DO NOT EDIT BELOW THIS POINT ######
##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10
##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different
##### routers behind a NAT router. Please do not remove it.
:local model [/system resource get board-name]
:local version [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
:local softid [/system license get software-id]
:local scriptVer 2017.5.30b
##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""
:for i from=0 to=([:len $uname] - 1) do={
:local char [:pick $uname $i]
:if ($char = " ") do={ :set $char "%20" }
:set name ($name . $char)
}
:for i from=0 to=([:len $version] - 1) do={
:local char [:pick $version $i]
:if ($char = " ") do={
:set $char "%20"
}
:set ver ($ver . $char)
}
#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/$datapath$datafile" \
url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";
##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0
##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
do={ /ip firewall address-list remove $i } }
##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="/$datapath$datafile"
##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove "$datapath$datafile"
##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies
##### Update your path, is you are using a USB Flash or other storage
:global datapath "disk1/dynamic.rsc"
###### DO NOT EDIT BELOW THIS POINT ######
##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10
##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different
##### routers behind a NAT router. Please do not remove it.
:local model [/system resource get board-name]
:local version [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
:local softid [/system license get software-id]
:local scriptVer 2017.5.30c
##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""
:for i from=0 to=([:len $uname] - 1) do={
:local char [:pick $uname $i]
:if ($char = " ") do={ :set $char "%20" }
:set name ($name . $char)
}
:for i from=0 to=([:len $version] - 1) do={
:local char [:pick $version $i]
:if ($char = " ") do={
:set $char "%20"
}
:set ver ($ver . $char)
}
#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="$datapath" \
url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";
##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0
##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
do={ /ip firewall address-list remove $i } }
##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="$datapath"
##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove [find name=$datapath]
##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid"[admin@TaylorMikrotik] >> /import updateBlacklist.rsc;
syntax error (line 62 column 11)All fields are required. Please update your script.can you post the /system license print ?I am on a RB951Ui-2HnD
CCR1009-7G-1C-1S+
CCR1009-8G-1S-1S+
CCR1016-12G
CCR1036-12G-4S
CHR
CRS109-8G-1S-2HnD
CRS125-24G-1S
CRS125-24G-1S-2HnD
hAP+ac
hAP+ac+lite
hEX
RB2011UAS-2HnD
RB2011UiAS
RB2011UiAS-2HnD
RB3011UiAS
RB450G
RB951G-2HnD
RB951Ui-2HnD
x86I found the line 62 error and corrected it. delete the items you have, and reinstall. it should be good to go.
syntax error (line 62 column 11)[/code]
[admin@TaylorMikrotik] >> /system license print
software-id: 15LP-6RVD
nlevel: 4
features:
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies
##### Update your path, is you are using a USB Flash or other storage
:global datapath "disk1/dynamic.rsc"
###### DO NOT EDIT BELOW THIS POINT ######
##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10
##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different
##### routers behind a NAT router. Please do not remove it.
:local model [/system resource get board-name]
:local version [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
:local softid [/system license get software-id]
:local scriptVer 2017.5.30c
##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""
:for i from=0 to=([:len $uname] - 1) do={
:local char [:pick $uname $i]
:if ($char = " ") do={ :set $char "%20" }
:set name ($name . $char)
}
:for i from=0 to=([:len $version] - 1) do={
:local char [:pick $version $i]
:if ($char = " ") do={
:set $char "%20"
}
:set ver ($ver . $char)
}
#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="$datapath" \
url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";
##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0
##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
do={ /ip firewall address-list remove $i } }
##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="$datapath"
##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove [find name=$datapath]
##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0I was just starting on a page that shows each type and number of routers that pulls the list.Awesome! Thanks for still doing this. Now that you got more stats, you should create some public pages cause i love me some random statistics!
What are the last two octets of the public IP?Not sure why its not working all of a sudden. I updated the script a few days ago and was working as of yesterday... Now when the script runs, it says its downloading the blacklist but nothing else happens.
I'm not sure what you are asking here. You are always welcome to contact a site and ask them to fix any issues. The subnet will be removed from the list automatically once whatever issue they were having is fixed. Many times it's that they are hosting a botnet that they do not even know about. Other times it may be that they are serving viruses in ads. AWS and Google Compute have both been blocked several times because they refuse to take down a virtual host that is being used to attack other networks.I noticed today when I started Firefox that I were getting hits on the blacklist. I followed the IP and found that it lead to hackademix.net and secure.informaction.com and looking on the site it was probably an plug-in was generating the hits and that was No-script. I use this plug-in for years and I allow or disallow the default running of scripts filtered on the domain the are served by.
Plugin homepage: https://noscript.net/
Name: secure.informaction.com
Addresses: 69.195.158.194
69.195.158.198
69.195.158.197
69.195.158.195
69.195.158.196
I understand how the blacklist is build and that it based on bad traffic and if there is a problem of a domain being misused then I can contact them to ask to look if they are hacked in any way?
Found it and I don't know why I did not see it before: the block is: 69.195.158.0/24 in the dynamicblacklist
Dave you could just escape the "?". That would allow it to be run in the terminal without issue, and it will make no difference for non-terminal running.That would mean that you need the current script. It's available in the first post.Hello Dave,
The script has the ?, when pasted in terminal it disappears.
The log only has an entry of-
script error: expected command name (line 1 column 1)
The downloaded dynamic.rsc only has one line-
All fields are required. Please update your script.
"...\?..."[admin@redacted_name.com] > /system script run updateBlacklist;
status: failed
failure: cannot open file
[admin@redacted_name.com] /system resource> print
uptime: 4d4h2m55s
version: 6.39.2 (stable)
build-time: Mar/09/2017 11:32:49
free-memory: 1712.6MiB
total-memory: 1956.2MiB
cpu: tilegx
cpu-count: 9
cpu-frequency: 1200MHz
cpu-load: 0%
free-hdd-space: 78.5MiB
total-hdd-space: 128.0MiB
architecture-name: tile
board-name: CCR1009-8G-1S-1S+
platform: MikroTik
[admin@redacted_name.com] /system resource> /system license print
software-id: 8RW2-IFMS
nlevel: 6
features:
Sweet, glad it fixed it for you.Doing the copy and paste from post #1 worked. Still not sure why it stopped working. Thank you!
:foreach i in=[/ip firewall address-list find ] \
do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
do={ /ip firewall address-list remove $i } }
/ip firewall address-list remove [find list="dynamicBlacklist"]
root@search:~# curl --header "Accept-Encoding: gzip,deflate,sdch" -I https://mikrotikfilters.com/updateBlacklist.rsc
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 21 Jun 2017 08:11:29 GMT
Content-Type: application/octet-stream
Content-Length: 4141
Last-Modified: Thu, 01 Jun 2017 04:22:22 GMT
Connection: keep-alive
Keep-Alive: timeout=2
Accept-Ranges: bytes................./dynamic.txt is Compressed
Uncompressed Page Size: 1817.7 KB
Compressed Page Size: 77.8 KB
Savings: 95.7%[admin@MikroTik] > /tool fetch mode=http url=https://xxxx.xx/index.html
status: finished
downloaded: 0KiBC-z pause]
total: 0KiB
duration: 1s[admin@MikroTik] > /tool fetch mode=http url=https://xxxx.xx/dynamic.rsc
status: finished
downloaded: 1817KiB pause]
total: 1817KiB
duration: 1s[admin@MikroTik] > /tool fetch mode=http url=https://xxxxx.xx/files/xxxxxxx.pdf
status: finished
downloaded: 71KiB-z pause]
total: 71KiB
duration: 1sthe C-z means "Control-Z to Pause", not compressed-zipI think that C-z in "0KiBC-z" stands for Compression gzip so it is there and now it is how to get that working for the .RSC
# CHAIN ACTION BYTES PACKETS
0 D ;;; special dummy rule to show fasttrack counters
prerouting passthrough 205 064 681 238 851
1 ;;; Attack from Intrus blocklist
prerouting drop 8 846 206
2 ;;; Attack from sbl malc0de
prerouting drop 0 0
3 ;;; Attack from sbl dshield
prerouting drop 52 1
4 ;;; Attack from sbl blocklist.de
prerouting drop 3 309 42
5 ;;; Attack from sbl spamhaus
prerouting drop 0 0
Hahahaha I know and on the moment I noticed that it was not funny because a lot of time went in. This is the part of my posting about it and what I put above it:the C-z means "Control-Z to Pause", not compressed-zipI think that C-z in "0KiBC-z" stands for Compression gzip so it is there and now it is how to get that working for the .RSC
I see different data when downloading html or the dynamic.rsc when I test it on my own server:
Darn the whole bit below is obsolete because the things I though I could deduct, is bases on not cleared characters by RouterOS. The result is written on the same line as the line shown during transfer "-- [Q quit|D dump|C-z pause]" so I was mislead by what it seemed to state and I was looking for........GRRRRRRRRRRRRRRRR
Code: Select all
then consider using both =) first quickly remove for recent versions, then slow cleanup for older ones if necessaryThe loop is to deal with older versions of RouterOS that would only remove the first item it found when using Find.
I'll do that for the next release.then consider using both =) first quickly remove for recent versions, then slow cleanup for older ones if necessary
##### Update your path, is you are using a USB Flash or other storagejun/23/2017 10:50:44 system,error,critical router was rebooted without proper shutdown
jun/23/2017 10:50:44 system,error,critical kernel failure in previous boot
jun/23/2017 10:50:44 system,error,critical out of memory condition was detectedjun/23/2017 11:29:13 system,error,critical router was rebooted without proper shut
down by watchdog timer
jun/23/2017 11:42:31 system,error,critical router was rebooted without proper shut
down by watchdog timer# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies
##### DO NOT EDIT THE LINES BELOW ######
:local path "";
:local filename "dynamic.rsc"
##### Update your path, to where you have your storage
##### Examples: "disk1/" or "usb/" and the default is the temporary storage
#:local path "usb/"
:local path "disk1/"
:global datapath "$path$filename";
:delay 5;
##### Disable the log (We don't need 20k lines of adds and removes in the log
/system logging disable 0
##### Import the downloaded blacklist
:log warning "Importing saved file $datapath as dynamicBlacklist...";
:if ([:len [/file find name="$datapath"]] > 0) do={/import file-name="$datapath"};
:if ([:len [/file find name="$datapath"]] = 0) do={/system script run updateBlacklist};
##### Turn the logging back on
/system logging enable 0
:log warning "dynamicBlacklist $datapath imported.";so that logging is enabled again./system logging enable 0
Thanks , however this has first have to be agreed, because Dave has also to change the original updateBlaclist so that dynamic.rsc is not erased after import. There can be a problem when the file is always President on devices with not much free space.Thanks!! Very good ...! i'll test in my RB450G
Connection Timout on that would imply that your IP may be blocked to start with.it didn't work for me (CCR1016-12G)
error :
/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https;
status: failed
failure: connection timeout
because that is the default path of a USB or SATA drive. If the driver does not exist, it simply creates that path. This way the USB is used if it's there.By the way, why is the default path "disk1/dynamic.rsc"?
I don't have any 32M units myself, but the blacklist stats show that 8 of them are currently pulling the list. It looks like it was a bad weekend for botnets as the list grew to 21,000 items. it may simply be too much for the smallest of routers.Anyway, fun fun. I hadn't tried this before:My poor little RB750 doesn't seem to like it either way.Code: Select alljun/23/2017 10:50:44 system,error,critical router was rebooted without proper shutdown jun/23/2017 10:50:44 system,error,critical kernel failure in previous boot jun/23/2017 10:50:44 system,error,critical out of memory condition was detectedCode: Select alljun/23/2017 11:29:13 system,error,critical router was rebooted without proper shut down by watchdog timer jun/23/2017 11:42:31 system,error,critical router was rebooted without proper shut down by watchdog timer
Done.David, please consider including blocklist.de's block list. I've been using both your blocklist and the one from squidblacklist.org for a little bit and so far the only major difference is from blocklist.de. If you add that then I can drop squidblacklist.org.
##### Find the "dynamicBlacklist" entries and remove them
:while ([/ip firewall address-list find list="dynamicBlacklist"] != "") do={ /ip firewall address-list remove [find list="dynamicBlacklist"]};##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
/ip firewall address-list remove [find list="dynamicBlacklist"]
##### Remove again - Some older RouterOS versions wont catch them all with the above line.
:foreach i in=[/ip firewall address-list find ] \
do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
do={ /ip firewall address-list remove $i } }:log error "Blacklist is updated at 10:00:00 UTC. Please update only once per day."
:log error "You have updated 7 times is the last 24 hours."
:log error "You will be able to update again in 24 hours."
:for i from=1 to=3 step=1 do={
:beep frequency=550 length=494ms;
:delay 494ms;
:beep frequency=400 length=494ms;
:delay 494ms;
}
##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0
#### Get size of the downloaded file
:local fileSize [/file get [ find where name=$datapath] value-name=size];
##### Find the "dynamicBlacklist" entries and remove them
:if ($fileSize > 1000) do={:log warning "Removing expiring address-list entries..."} else={:log error "Using the old Blacklist. Look for info about this error in the log underneath."};
:if ($fileSize > 1000) do={:while ([/ip firewall address-list find list="dynamicBlacklist"] != "") do={ /ip firewall address-list remove [find list="dynamicBlacklist"]}};
##### Import the downloaded blacklist
:if ($fileSize > 1000) do={:log warning "Importing current Blacklist..."};
/import file-name="$datapath";
##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove [find name=$datapath]
##### Turn the logging back on
:if ($fileSize > 1000) do={:log warning "Blacklist Update Complete."};
/system logging enable 0Thank you!Done.David, please consider including blocklist.de's block list. I've been using both your blocklist and the one from squidblacklist.org for a little bit and so far the only major difference is from blocklist.de. If you add that then I can drop squidblacklist.org.
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies
##### DO NOT EDIT THE LINES BELOW ######
:local path "";
:local filename "dynamic.rsc"
##### Update your path, to where you have your storage
##### Examples: "disk1/" or "usb/" and the default is the temporary storage
#:local path "usb/"
:local path "disk1/"
:global datapath "$path$filename";
##### Delay for 10 seconds to allow the WAN to come online after a reboot
:delay 10;
##### Disable the log (We don't need 20k lines of adds and removes in the log
/system logging disable 0
# Declaring and filling the date1 and date2 variable for calculating the time difference
:global globalDaysDiff
:local time [/system clock get time];
:local date [/system clock get date];
:global date2 ("$date" . " " . "$time");
:global date1 [/file get [ find where name=$datapath] value-name=creation-time];
# This script calculates difference between two dates
/system script run diffDate
##### Import the downloaded blacklist
:log warning "Importing saved file $datapath as dynamicBlacklist...";
:if ([:len [/file find name="$datapath"]] > 0) do={:if ($globalDaysDiff != 0) do={:log error "dynamicBlacklist $datapath to old for fast import."} else={/import file-name="$datapath"}};
# Download Blacklist if there is no dynamic.rsc present
:if ([:len [/file find name="$datapath"]] = 0) do={/system script run updateBlacklist};
##### Turn the logging back on
/system logging enable 0
:if ([:len [/file find name="$datapath"]] != 0) do={:log warning "dynamicBlacklist $datapath imported."} else={:log error "Nothing happened and no protection by dynamicBlacklist provided!"};
### calculate diff between two dates - yoan tanguy 2017
# format: :global date1 "jan/05/2017 10:00:00";:global date2 "may/15/2018 12:30:00";/system script run diffDate
# expected date format : month/day/year hours:minutes:seconds (ex: mar/14/2017 09:13:54)
:global date1
:global date2
# date to array format :
# m a r / 1 4 / 2 0 1 7 0 9 : 1 3 : 5 4
# 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
:local date1month [:pick $date1 0 3]
:local date1day [:pick $date1 4 6]
:local date1year [:pick $date1 7 11]
:local date1hours [:pick $date1 12 14]
:local date1minutes [:pick $date1 15 17]
:local date1seconds [:pick $date1 18 20]
:local date2month [:pick $date2 0 3]
:local date2day [:pick $date2 4 6]
:local date2year [:pick $date2 7 11]
:local date2hours [:pick $date2 12 14]
:local date2minutes [:pick $date2 15 17]
:local date2seconds [:pick $date2 18 20]
# month to decimal converter - https://forum.mikrotik.com/viewtopic.php?t=58674
:local months ("jan","feb","mar","apr","may","jun","jul","aug","sep","oct","nov","dec");
:set date1month ([:find $months $date1month -1 ] + 1)
:set date2month ([:find $months $date2month -1 ] + 1)
:global globalDiff
:local yearDiff ($date2year - $date1year)
:local monthDiff ($date2month - $date1month)
:local dayDiff ($date2day - $date1day)
:local hoursDiff ($date2hours - $date1hours)
:local minutesDiff ($date2minutes - $date1minutes)
:local secondsDiff ($date2seconds - $date1seconds)
# handle diff by converting in seconds, avoid negative hours/minutes/seconds (ex: jan/01/1970 09:00:00, jan/02/1970 08:00:00 must give 0 days 23:00:00 and not 1 days 0-1:00:00)
# 1 days 23:30:10
# 1*24*60*60 + 23*60*60 + 30*60 + 10
# ($dayDiff * 24*60*60) + ($hoursDiff * 60*60) + ($minutesDiff *60) + $secondsDiff
# ($dayDiff * 86400) + ($hoursDiff * 3600) + ($minutesDiff *60) + $secondsDiff
:local secondsGlobalDiff
:set secondsGlobalDiff (($dayDiff * 86400) + ($hoursDiff * 3600) + ($minutesDiff *60) + $secondsDiff)
:set dayDiff ($secondsGlobalDiff / 86400)
:set secondsGlobalDiff ($secondsGlobalDiff - ($dayDiff * 86400))
:set hoursDiff ($secondsGlobalDiff / 3600)
:set secondsGlobalDiff ($secondsGlobalDiff - ($hoursDiff * 3600))
:set minutesDiff ($secondsGlobalDiff / 60)
:set secondsGlobalDiff ($secondsGlobalDiff - ($minutesDiff * 60))
:set secondsDiff $secondsGlobalDiff
# check if date1 is older than date2 to avoid errors in calculation
if ($yearDiff < 0) do={
:return "error : date1 should be older that date2 (year check), exiting"
} else={
if ($yearDiff = 0) do={
if ($monthDiff <0) do={
:return "error : date1 should be older that date2 (month check), exiting"
} else={
if ($monthDiff = 0) do={
if ($dayDiff < 0) do={
:return "error : date1 should be older that date2 (day check), exiting"
} else={
if ($dayDiff = 0) do={
if ($hoursDiff < 0) do={
:return "error : date1 should be older that date2 (hours check), exiting"
} else={
if ($hoursDiff = 0) do={
if ($minutesDiff < 0) do={
:return "error : date1 should be older that date2 (minutes check), exiting"
} else={
if ($minutesDiff = 0) do={
if ($secondsDiff < 0) do={
:return "error : date1 should be older that date2 (seconds check), exiting"
}
}
}
}
}
}
}
}
}
}
}
# check if leap years - https://wiki.mikrotik.com/wiki/AutomatedBilling/MonthEndScript
:local isYear1Leap 0
:local isYear2Leap 0
if ((($date1year / 4) * 4) = $date1year) do={
:set isYear1Leap 1
}
if ((($date2year / 4) * 4) = $date2year) do={
:set isYear2Leap 1
}
# find the right amount of days between 2 months
:local daysInEachMonth ("31","28","31","30","31","30","31","31","30","31","30","31");
:local daysInEachMonthLeapYear ("31","29","31","30","31","30","31","31","30","31","30","31");
:local totalDaysBetweenMonths
# same year; yearDiff = 0 so year1 = year2
if ($yearDiff = 0 and $monthDiff >= 1) do={
if ($isYear1Leap = 0) do={
for month from=($date1month - 1) to=($date2month - 1) step=1 do={
:set totalDaysBetweenMonths ($totalDaysBetweenMonths + [:pick $daysInEachMonth $month])
}
}
if ($isYear1Leap = 1) do={
for month from=($date1month - 1) to=(($date2month - 1) - 1) step=1 do={
:set totalDaysBetweenMonths ($totalDaysBetweenMonths + [:pick $daysInEachMonthLeapYear $month])
}
}
}
# different year, make concatenation of daysInEachMonth arrays first
:local daysInEachMonthConcatenatedYears
if ($yearDiff >= 1) do={
for year from=$date1year to=$date2year step=1 do={
# if leap year, concatenate the right daysInEachMonth array
if ((($year / 4) * 4) = $year) do={
:set daysInEachMonthConcatenatedYears ($daysInEachMonthConcatenatedYears, $daysInEachMonthLeapYear)
} else={
:set daysInEachMonthConcatenatedYears ($daysInEachMonthConcatenatedYears, $daysInEachMonth)
}
}
# must add years count
for month from=($date1month - 1) to=(($date2month - 1) + (($yearDiff * 12) - 1)) step=1 do={
:set totalDaysBetweenMonths ($totalDaysBetweenMonths + [:pick $daysInEachMonthConcatenatedYears $month])
}
}
:global globalDaysDiff ($totalDaysBetweenMonths + $dayDiff)
# add leading zeros if necessary
:if ($hoursDiff < 10) do={
:set hoursDiff ("0" . $hoursDiff)
}
:if ($minutesDiff < 10) do={
:set minutesDiff ("0" . $minutesDiff)
}
:if ($secondsDiff < 10) do={
:set secondsDiff ("0" . $secondsDiff)
}
:local d "d"
:set globalDiff "$globalDaysDiff$d$hoursDiff:$minutesDiff:$secondsDiff"
:put $globalDiffI think you need to check that you have a reliable date in the first place. It can be a while between boot up and acquiring the current date and time. I would not count on a simple delay being enough, I would sanity check the date.Hi Dave,
So I have updated the start-up schedule so that dynamic.rsc files older than one day are not imported from flash/disk1/usb and the normal updateBlacklist script is run.
I second that. If I've learned anything about RouterOS, it's that you can NOT trust the date and time at boot. I have several routers that take up to 20 minutes before the time is synced correctly.I think you need to check that you have a reliable date in the first place. It can be a while between boot up and acquiring the current date and time. I would not count on a simple delay being enough, I would sanity check the date.
Source: https://wiki.mikrotik.com/wiki/Setup_local_NTP_serversSince v6.16 the current time is saved in the system configuration on reboot and on clock adjustment and is used to set the initial time after reboot.
Benefits:
Router doesn't need direct access to internet and public NTP servers
Allow control of a primary source of clock for your router on only two main routers (primary and secondary)
It can reduce traffic and the load of some public NTP servers by local time caching
:if ([:len [/file find name="flash"]] != 0) do={set datapath "dynamic.rsc"};
:if ([:len [/file find name="disk1"]] != 0) do={set datapath "disk1/dynamic.rsc"};
:if ([:len [/file find name="disk2"]] != 0) do={set datapath "disk2/dynamic.rsc"};
:if ([:len [/file find name="disk3"]] != 0) do={set datapath "disk3/dynamic.rsc"};
:if ([:len [/file find name="usb"]] != 0) do={set datapath "usb/dynamic.rsc"};
:log info "Default location for Blacklist is: $datapath";:if ([:len [/file find name="flash"]] != 0) do={:if ([/system resource get free-hdd-space] > 3000000) do={set datapath "dynamic.rsc"}};
:if ([:len [/file find name="disk1"]] != 0) do={:if ([/disk get [ find where name="disk1"] value-name=free] > 3000000) do={set datapath "disk1/dynamic.rsc"}};
:if ([:len [/file find name="disk2"]] != 0) do={:if ([/disk get [ find where name="disk2"] value-name=free] > 3000000) do={set datapath "disk2/dynamic.rsc"}};
:if ([:len [/file find name="disk3"]] != 0) do={:if ([/disk get [ find where name="disk3"] value-name=free] > 3000000) do={set datapath "disk3/dynamic.rsc"}};
:if ([:len [/file find name="usb"]] != 0) do={:if ([/disk get [ find where name="usb"] value-name=free] > 3000000) do={set datapath "usb/dynamic.rsc"}};
:log info "Default save locationwith 3MB free for Blacklist is: $datapath";#### Share how many packets are blocked by the Blacklist on your device
:local filterdownBlacklist "0";
:local rawdownBlacklist "0";
:local filterupBlacklist "0";
:local rawupBlacklist "0";
##### downstream
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0) do={set filterdownBlacklist [/ip firewall filter get [ find src-address-list="dynamicBlacklist"] packets]} else={set filterdownBlacklist "0"};
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0) do={/ip firewall filter reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0) do={set rawdownBlacklist [/ip firewall raw get [ find src-address-list="dynamicBlacklist"] packets]} else={set rawdownBlacklist "0"};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0) do={/ip firewall raw reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
##### upstream
:if ([:len [/ip firewall filter find dst-address-list="dynamicBlacklist"]] != 0) do={set filterupBlacklist [/ip firewall filter get [ find dst-address-list="dynamicBlacklist"] packets]} else={set filterupBlacklist "0"};
:if ([:len [/ip firewall filter find dst-address-list="dynamicBlacklist"]] != 0) do={/ip firewall filter reset-counters numbers=[find dst-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find dst-address-list="dynamicBlacklist"]] != 0) do={set rawupBlacklist [/ip firewall raw get [ find dst-address-list="dynamicBlacklist"] packets]} else={set rawupBlacklist "0"};
:if ([:len [/ip firewall raw find dst-address-list="dynamicBlacklist"]] != 0) do={/ip firewall raw reset-counters numbers=[find dst-address-list="dynamicBlacklist"]};
#### Begin download of current blacklist
:log warning "Downloading current Blacklist for model $model $ver";
/tool fetch mode=https dst-path="$datapath" \
url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid&filterdown=$filterdownBlacklist&rawdown=$rawdownBlacklist&filterup=$filterupBlacklist&rawup=$rawupBlacklist";
Too much workwere there thoughts about BGP feed?..
Thanks for your great work! I had to make a minor correction to version 2017.7.1d, and propose a modification to give more info to the person who is checking the log.The new backend and script are live. Make sure you read the comments and select the correct script for your router.
*** DO NOT SELECT THE LARGE LIST FOR ROUTERS WITH LESS THAN 20M FREE DISK OR LESS THAN 256M Memory! ***
Recommendation:
Routers with 32M~128M memory - "small" list
Routers with 256M~512M memory - "medium" list
Routers with 1G memory and up - "large" list
#### Select your list size ####
#### large - 10 to 20 Megabyte download - 100k+ entries - intended for firewalls protecting internet servers
#### medium - 2 to 5 Megabyte download - 40k+ entries - intended for corporate networks
#### small - 200 to 700 Kilobyte download - 2k+ entries - intended for networks with no open ports
#### Begin download of current blacklist
:log warning "Downloading current $listSize sized Blacklist for this model";
/tool fetch mode=https dst-path="$datapath" \
url="https://mikrotikfilters.com/download.ph ... id=$softid";
:local fileSize [/file get [ find where name=$datapath] value-name=size];
Hi, interesting scripting ...Collecting how many packets are blocked by the Blacklist:
After collecting the numbers, each packets counter in Filters and RAW are reset to zero. In this way you won't get double countings on the next update of the Blacklist.Code: Select all#### Share how many packets are blocked by the Blacklist on your device :local filterBlacklist "0"; :local rawBlacklist "0"; :if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0) do={set filterBlacklist [/ip firewall filter get [ find src-address-list="dynamicBlacklist"] packets]} else={set filterBlacklist "0"}; :if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0) do={/ip firewall filter reset-counters numbers=[find src-address-list="dynamicBlacklist"]}; :if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0) do={set rawBlacklist [/ip firewall raw get [ find src-address-list="dynamicBlacklist"] packets]} else={set rawBlacklist "0"}; :if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0) do={/ip firewall raw reset-counters numbers=[find src-address-list="dynamicBlacklist"]}; #### Begin download of current blacklist :log warning "Downloading current Blacklist for model $model $ver"; /tool fetch mode=https dst-path="$datapath" \ url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid&filter=$filterBlacklist&raw=$rawBlacklist";
#### Share how many packets are blocked by the Blacklist on your device
:local filterBlacklist "0";
:local rawBlacklist "0";
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0) do={set filterBlacklist [/ip firewall filter get [ find src-address-list="dynamicBlacklist"] packets]} else={set filterBlacklist "0"};
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0) do={/ip firewall filter reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0) do={set rawBlacklist [/ip firewall raw get [ find src-address-list="dynamicBlacklist"] packets]} else={set rawBlacklist "0"};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0) do={/ip firewall raw reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
:log warning "Count filterBlacklist=$filterBlacklist rawBlacklist=$rawBlacklist";Try:Hi, interesting scripting ...
I tried it as a separate script in the following way :BUT the counters are NOT reset and the log displays zeroes ...Code: Select all:log warning "Count filterBlacklist=$filterBlacklist rawBlacklist=$rawBlacklist";
any suggestions ?
yes, scripting in the Mikrotik is a PITA. I have that experienced that enough in the last week.:log warning "Count filterBlacklist= $filterBlacklist rawBlacklist= $rawBlacklist";
Count filterBlacklist=0 rawBlacklist=30This only for private use on the moment and if you only want to know the score remove the reset lines. When Dave is ready for more statistics then he can implement it.tnx,
scripting can be a pain, sometimes it just does not work ...
it worksCode: Select allCount filterBlacklist=0 rawBlacklist=30
:local timeOffset [/system clock get value-name=gmt-offset];##### Read the save statistics
/import file blacklist.rsc
:global statsFilterBlacklist;
:global statsRAWBlacklist
##### Get current time and set filename to keep statistics
:local date [/system clock get date];
:local time [/system clock get time];
:local filename "blacklist.rsc";
##### Collect and reset packet counters
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0) do={set filterdownBlacklist [/ip firewall filter get [ find src-address-list="dynamicBlacklist"] packets]} else={set filterdownBlacklist "0"};
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0) do={/ip firewall filter reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0) do={set rawdownBlacklist [/ip firewall raw get [ find src-address-list="dynamicBlacklist"] packets]} else={set rawdownBlacklist "0"};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0) do={/ip firewall raw reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
#### Build new stats string
:local newStatsFilterBlacklist "$statsFilterBlacklist" . " " . "$filterdownBlacklist";
:local newStatsRAWBlacklist "$statsRAWBlacklist" . " " . "$rawdownBlacklist";
:local newStatDate ("$date" . " " . "$time");
:local writeString ":global $lastStatDate;" . ":global statsFilterBlacklist $newStatsFilterBlacklist;" . " " . ":global statsRAWBlacklist "$StatsRAWBlacklist";
/file set $filename content=$writeString;The file supplies the last sample date and time and maybe a the gmt-offset can sync the data with other available data already in the database.:global lastStatDate "jul/02/2017 15:49:19"; :global statsFilterBlacklist "1 2 3 4 5 6 7 8 9"; :global statsRAWBlacklist "0 9 8 7 6 5 4 3 2 1";
# https://forum.mikrotik.com/viewtopic.php?f=9&t=98804
# Import Intrus Managed Filter Lists
# CUSTOMIZED by jgro, different globals, do not simply replace with update from Intrus
# © 2016-2017 David Joyce, Intrus Technologies
##### Update your path, is you are using a USB Flash or other storage
##### Examples:
##### "disk1/dynamic.rsc" or "usb/dynamic.rsc" or "dynamic.rsc"
:global intrusPath "disk1/dl/dynamic.rsc"
#### Select your list size ####
#### large - 10 to 20 Megabyte download - 100k+ entries - intended for firewalls protecting internet servers
#### medium - 2 to 5 Megabyte download - 40k+ entries - intended for corporate networks
#### small - 200 to 700 Kilobyte download - 2k+ entries - intended for networks with no open ports
:local listSize "medium"
###### DO NOT EDIT BELOW THIS POINT ######
##### Delay for 10 seconds to allow the WAN to come online after a reboot
#:log warning "Blacklist update in 10 seconds";
#:delay 10
##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different
##### routers behind a NAT router. Please do not remove it.
:local model [/system resource get board-name]
:local version [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
:local softid [/system license get software-id]
:if ($model = "CHR") do={
:local temp [/system license get system-id]
:for i from=0 to=([:len $temp] - 1) do={
:local char [:pick $temp $i]
:if ($char = "/") do={ :set $char "-" }
:set softid ($softid . $char)
}
}
:if ($model !="CHR") do={
:global softid [/system license get software-id]
}
:local scriptVer 2017.7.1d
##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""
:for i from=0 to=([:len $uname] - 1) do={
:local char [:pick $uname $i]
:if ($char = " ") do={ :set $char "%20" }
:set name ($name . $char)
}
:for i from=0 to=([:len $version] - 1) do={
:local char [:pick $version $i]
:if ($char = " ") do={
:set $char "%20"
}
:set ver ($ver . $char)
}
#### Begin download of current blacklist
:local fileSize
:log warning "Downloading current Intrus dynamicBlacklist for this model";
:do {
:do {
/tool fetch mode=https dst-path="$intrusPath" \
url="https://mikrotikfilters.com/download.php?get=$listSize&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";
:set fileSize [/file get [ find where name=$intrusPath] value-name=size];
:if ($fileSize < 500) do={
:log error "IntrusBL download is too small"
:error "IntrusBL download is too small"
}
} on-error={
:log error "FAILED to download Intrus dynamicBlacklist"
/system script run "play-alert-sound"
}
:if ($fileSize > 500) do={
/system script run import-intrus-block-list
}
} on-error {
:log error "FAILED to update Intrus dynamicBlacklist";
}##### Update your path, is you are using a USB Flash or other storage
##### Examples:
##### "disk1/dynamic.rsc" or "usb/dynamic.rsc" or "dynamic.rsc"
:global intrusPath
:log warning "Starting import of Intrus dynamicBlocklist"
# intrusPath set by code that does the fetch
# set fallback in case it is unset
:if ("x$intrusPath " = "x") do={
:set intrusPath "disk1/dl/dynamic.rsc"
:log warning "Importing dynamicBlacklist from fallback location: $intrusPath"
}
:if ([/file find name=$intrusPath ] = "") do= {
:error "FAILED: Importing dynamicBlacklist: file not found: $intrusPath "
}
##### Disable the log (We don't need 20k lines of adds and removes in the log)
:log warning "Disabling info logging while loading dynamicBlacklist...";
:log info "Disabling info logging while loading dynamicBlacklist...";
/system logging disable 0
##### Find the "dynamicBlacklist" entries and remove them
:local status "failed"
:local fileSize [/file get [ find where name=$intrusPath] value-name=size];
:if ($fileSize > 500) do={
:log warning "Removing expiring address-list entries...";
/ip firewall address-list remove [find list="dynamicBlacklist"]
##### Import the downloaded blacklist
:log warning "Importing downloaded dynamicBlacklist from $intrusPath ";
do {
/import $intrusPath
:set status "success"
} on-error {
:log warning "FAILED to import $intrusPath "
}
####### Find and remove the downloaded file
###:log warning "Removing dynamicBlacklist temp file...";
###/file remove [find name=$intrusPath ]
} else= { :log warning "Intrus blacklist file $intrusPath too small ($fileSize), aborting" }
##### Turn the logging back on
/system logging enable 0
:log warning "info logging enabled"
:log info "info logging enabled";
:if ($status = "success") do={
:log warning "Intrus dynamicBlacklist Update Complete.";
} else={
:error "FAILED to update Intrus dynamicBlacklist"
}:log warning "Playing alert sound"
:for i from=1 to=3 step=1 do={
:beep frequency=550 length=494ms;
:delay 494ms;
:beep frequency=400 length=494ms;
:delay 494ms;
}#### Select your list size ####
#### large - 10 to 20 Megabyte download - 100k+ entries - intended for firewalls protecting internet servers
#### megium - 2 to 5 Megabyte download - 40k+ entries - intended for corporate networks
#### small - 200 to 700 Kilobyte download - 2k+ entries - intended for networks with no open ports
:local listSize "small"
Small mistake.. its normalSmall typo, megium ...Code: Select all#### Select your list size #### #### large - 10 to 20 Megabyte download - 100k+ entries - intended for firewalls protecting internet servers #### megium - 2 to 5 Megabyte download - 40k+ entries - intended for corporate networks #### small - 200 to 700 Kilobyte download - 2k+ entries - intended for networks with no open ports :local listSize "small"
I can not change the format, as there are still several hundred units that have never (and likely will never) update the script. The first version of the script removed the entries based on the comment (RouterOS was unable to remove by list name at the time) So removing the comments would stop them from working. Versions over the last year remove based on the list name. Again, many have never and will never update.I went through different options how to reduce traffic and the quick and easy one is removing the comment in the medium and large file and that gives a reduction in traffic of over 20% assuming that the users of the medium and large file know what that addresslist is named dynamicBlacklist stands for...you can shorten also that name "dynamicBlacklist" and saves an other 5 to 10 percent.
With more than 80% of the routers pulling the list only having a MIPS CPU, passing only the IPs in CIDR format would cause 100% for more than 10 minutes. (up to 30 minutes in some of my testing). During this time, the router would experience dramatic pocket loss. It also complicates the script. Same reason I won't do BGP - it's just far too complicated for most to setup.Next thought is to only supply the addresses itself and that would shrink the size of the medium file from 4.1MB to 729KB but then we have to split it up in more than 177 files due to 4096 bytes String limit present in RouterOS.
It's not a problem. The only issue is with the CHR. The CHR license often has a "/" in it, which needs to be replaced or encoded.I had a quick peek at 2017.7.3f and I have to admit that I am a bit lost on it.
Update: Before the v [ScriptVer] would undergo a cleaning of spaces which are replaced by %20 for use in the URL which is not not more done. I have still the word (testing) in my version string with a space in front.
@IntrusDave, Your primary post still says your list is updated only once a day and I was still under the impression pulling it more than 4 times a day will result in being banned. Please update your recommendation and limit if needed.please keep in mind that with all the chaos in the world now, the list is regenerated every 4 hours. I don't recommend holding on to an older list for more than 8 hours. Also, I have no bandwidth caps so I have no issue with people downloading several times a day - But I don't want it abused and pulled every 5 minutes. My router does limit the connection speed to 100mbps, so no one can saturate the full gigabit WAN.
I corrected my typo. Also changed the one global to local. (It was global on my dev unit because another script was using it too)
Fair enough. So I can do my own investigation, would you please post (and keep updated) the block lists you are including? Of course, you do not need to disclose anything proprietary, but where you are using public lists, it would help to know.Unfortunately it's not possible to tell the source of the block. The lists are generated from 12 different high profile blocklists, as well as a network of over 200 routers.
That is completely fair and understandable and I thank you again for providing this free service. I have tried to contribute to your effort with more efficient code and better error handling in the same spirit.I've stated here many time before - this list and the script are built for my own routers that I manage.... If a recommendation helps my clients or myself, I will likely implement it. If I see no benefit, it will not likely be added.
#/system logging enable [find topics="info"]; and disable logging: #/system logging disable [find topics="info"];I'll include that in the next update.I have a request. I was testing with a more informative disabling and enabling from the log entries and when I did not disable and enable again as normal is done on an import I did not get only the normal logging but not the huge numbers of the removals and adds to the list in the log. I was very nice to see that other services were still logging during import.
I had defined ! firewall on info, removed that again and applied but still no removal or add logging by script.Code: Select all#/system logging enable [find topics="info"]; and disable logging: #/system logging disable [find topics="info"];
I see still the blocking by the dynamicBlaclist so the firewall is making log entries.
I'm not sure if that was directed at me, but in case it was, I want to say I was never asking you to include a change log. What I wanted was for you to keep up-to-date whatever is true about the current system, things like when it is generated and how often people can and cannot download it, etc.New script updated. I'm not including a change log on in the first post.
Thanks for the changelog and that save al lot of scrolling the the code in posting one to see what has changed.I'll include that in the next update.I have a request. I was testing with a more informative disabling and enabling from the log entries and when I did not disable and enable again as normal is done on an import I did not get only the normal logging but not the huge numbers of the removals and adds to the list in the log. I was very nice to see that other services were still logging during import.
I had defined ! firewall on info, removed that again and applied but still no removal or add logging by script.Code: Select all#/system logging enable [find topics="info"]; and disable logging: #/system logging disable [find topics="info"];
I see still the blocking by the dynamicBlaclist so the firewall is making log entries.
Glad we are all laughing.ROFL oops. I fixed it. Should have been NOW not NOT
#instead of: /system logging set numbers=0 topics=info;
:local logTopics [/system logging get number=0 value-name=topics]
/system logging set number=0 topics=info,!firewall,!system
#...
/system logging set number=0 topics=$logTopic
:execute "sub-script-remove"; :delay 10; :execute "sub-script-import"/ip firewall address-list remove [find list="dynamicBlacklist"];/import file-name="disk1/dynamic.rsc";I am really puzzled why I don't any logging of the adding of the addresses in my Mikrotik while I don't disable/enable logging in any way.Glad we are all laughing.ROFL oops. I fixed it. Should have been NOW not NOT
Here is a "best practices" tweak: save and restore log state rather than reset it
For my setup, I had to include "!system" to the first setting because sometimes the adds and removes show up there instead of under firewall.
/system logging> print
Flags: X - disabled, I - invalid, * - default
# TOPICS ACTION
0 * info memory
1 * error memory
2 * warning memory
3 * critical echo:local logTopics [/system logging get [find topics="info"] value-name=topics]
/system logging set number=0 topics=info,!firewall,!system
.
.
/system logging set [find topics="info"] topics=$logTopic;###### DO NOT EDIT BELOW THIS POINT ######
##### Delay for 5 seconds to allow the WAN to come online after a reboot
##### You can change this if you need more or less time. Loading the list
##### on reboot will not work without this delay.
:local d 0;
:put "Delaying $d seconds to allow WAN to stabilize.";
:log warning "Blacklist update in $d seconds";
:delay $d;
Thank you. Corrected.In your posted code, you have the delay set to 0. It's fine in the hosted code at https://mikrotikfilters.com/updateBlacklist.rsc
I'm going to test with the various devices I have. I may just include code to make a choice between one-at-a-time and both-at-once.Great new that you are going to take the next step, to have better control of and more flexible way of initiating updates by means of DNS.
I have managed this morning to not have any need any more for smaller files now I can remove and import the dynamicBlacklist at same moment. This reduces the exposure during renewing the Blacklist. This may work for me and similar devices but older equipment can have problems with doing two things at the same time.
I want to share what I noticed today. After erasing the Blacklist the memory is still reserved in RouterOS and not given back to the pool. Importing the Blacklist will reuse that reserved memory so no loss there of space. After the next start the pool will be back to it original size.
I propose to prepare import script as follows:
/ip firewall address-list
:do { add address=X.X.X.X list=blackmail timeout=25h } on-error={set [find where address=X.X.X.X] timeout=25h}
:do { add address=Y.Y.Y.Y list=blackmail timeout=25h } on-error={set [find where address=Y.Y.Y.Y] timeout=25h}
/ip firewall address-list
:do { add address=X.X.X.X list=blackmail timeout=25h } on-error={set [find where address=X.X.X.X list=blackmail ] timeout=25h}
:do { add address=Y.Y.Y.Y list=blackmail timeout=25h } on-error={set [find where address=Y.Y.Y.Y list=blackmail ] timeout=25h}
/ip firewall address-list
# Update timeouts of addresses from old list as they are on the current so they stay and just need new timeout
set [find where address=X.X.X.X list=blackmail ] timeout=25h
set [find where address=Y.Y.Y.Y list=blackmail ] timeout=25h
....
# add new address
add address=Z.Z.Z.Z list=blackmail timeout=25h
...
:do {/ip fi ad add address=101.231.46.34 list=blackmail ti=25h} on-error={set [fi wh address=101.231.46.34 list=blackmail] ti=25h}add li=dynamicBlacklist ad=1.0.128.0/17 ti="1d"add l=IDDBL a=1.0.128.0/17 t=25hYou can slim it even more and be backwards compatible with this:Change fromto IDDBL=IntrusDaveDynamicBlackListCode: Select alladd li=dynamicBlacklist ad=1.0.128.0/17 ti="1d"saves statistically 27% of size but breaks current filters as list name changesCode: Select alladd l=IDDBL a=1.0.128.0/17 t=25h
a l=dynamicBlacklist a=xxx.xxx.xxx.xxx/xx t=1da l=IDDBL a=1.0.128.0/17 t=25h
I like this. going to see how much it slows things down.
for you:
..[CUT]..
# Turn the logging back on
:if (\$blDebug = 1) do={ \$log t=\"Enabling firewall info logging...\"; }
/system logging set numbers=0 topics=\$cl;
..[CUT]..
Okay, I give. Can you point me to a basic setup for BGP. I don't even know where to start.were there thoughts about BGP feed?..
I will say that the BGP method would be simpler to manage over a large distribution, and the implementation on the client side is brain-dead simple:
enable BGP (if not already using BGP) with any private ASN other than 64567. (or just use their real ASN if they're already running BGP).
in-filter=accept all -> action=set route type=blackhole
out-filter=discard all
enable strict RPF in IP options.
Except I woke up early to have an other look and going now to have breakfast but not after I want to share this and I had a look at your latest version of the Bartozp/Dave way initiated by Chupaka of using the :do.# Medium Blacklist Generated on Sa=t Jul 8 02:00:16 PDT 2017 by Intrus Technologies
:global blSerial 60
:global blDate 1499504416
:local i do={/ip f a a l=dynamicBlacklist t=25h a=$a }
$i a=1.0.128.0/17
$i a=1.1.128.0/17
$i a=1.2.128.0/17
$i a=1.4.128.0/17
$i a=1.9.69.35/32
.
.
.# remove entries removed by diff
/ip firewall address-list
:do { remove [find where address=192.168.1.0/24 list=dynamicBlacklist]}
:do { remove [find where address=192.168.2.0/24 list=dynamicBlacklist]}
.....
# update list with new one entries
/ip firewall address-list
add address=192.168.3.1 list=dynamicBlacklist ti=25h
add address=192.168.3.1 list=dynamicBlacklist ti=25h
.....
:foreach i in=[ find where list=dynamicBlacklist ] do={set $i ti=25h }add address=20170709 list=dynamicBlacklistTimeSstampFullTable
add address=20170709 list=dynamicBlacklistTimeStampDaily ti=25h
:set ts :put [/ip firewall address-list get [/ip firewall address-list find where list=dynamicBlacklistTimeSstampFullTable] address]]
:do {/import file-name=$ts.rsc }
/ip firewall filter set src-address-list=intrusBL [find where src-address-list="dynamicBlacklist"]
/ip firewall raw set src-address-list=intrusBL [find where src-address-list="dynamicBlacklist"]
/ip firewall filter set dst-address-list=intrusBL [find where dst-address-list="dynamicBlacklist"]
/ip firewall raw set dst-address-list=intrusBL [find where dst-address-list="dynamicBlacklist"]A few things to note - the implementation is different depending on whether the client and server peer using eBGP or iBGP.Can you post an export rsc to give me a basic BGP setup to drop incoming packets from 10.252.0.7/32?
I'm hoping you can give me a starting point so I can understand how this works.
/ip settings
set rp-filter=strict
/routing bgp instance
set default as=65530 router-id=10.10.10.10
/routing bgp peer
add in-filter=BlackholeDestination multihop=yes name=BlacklistServer1 out-filter=NoRoutes \
remote-address=192.0.2.100 remote-as=65000 ttl=default
/routing filter
add action=accept chain=BlackholeDestination set-type=blackhole
add action=discard chain=NoRoutes/routing bgp instance
set default as=65000 router-id=192.0.2.100 redistribute-static=yes
/routing bgp peer
add in-filter=NoRoutes multihop=yes name=Client1 out-filter=OnlyRedistributeBlackholes \
remote-address=10.10.10.10 remote-as=65530 ttl=default
/routing filter
add action=discard chain=NoRoutes
add action=accept bgp-communities=65000:666 chain=OnlyBlackholes
add action=discard chain=OnlyBlackholeschain=prerouting action=drop in-interface=pppoe-out1 dst-port=!25,80,443,554 log=yes log-prefix="TCP hacker" protocol=tcp tcp-flags=syn,!fin,!rst,!ack,!urg,!ece,!cwr chain=prerouting action=drop in-interface=pppoe-out1 src-port="" dst-port=!53,546,547,5060-5070,7078-7098 port="" log=yes log-prefix="hacker drop" protocol=udpchain=prerouting action=drop log=no log-prefix="" src-address-list=intrusBL Hi Amt, have a look at an posting by Dave and put the whitelist above blacklist linesHi all,
Im using this list and sometimes my ip addreses comes with list. what should i do ? and further more some customers cant see their cameras when their ip comes in blacklist they cant connect to their system.
Im using only this rule at raw table to drop;one of my customer has an internet at another country and his ip also in black list and he can not accses his system. I would like to learn that, with this rule in raw table Im thinking I only block incoming from these src address list but i cant ping any of them also. should i select an in-interface in here ?Code: Select allchain=prerouting action=drop log=no log-prefix="" src-address-list=intrusBL
Thanks
chain=prerouting action=drop log=no log-prefix="" src-address-list=intrusBLUse a filter drop instead of a raw drop.Hi msatter,
thank you very much for your quick answer.
I solve the problem as here but I wonder that when i add my ip block in here like 123.123.32.0/22, is this not make problem to me ? because when i add rule to accept for my ip blocks, blacklisted ip's can attack to my ip range if I true.
further more I wonder that also why i cant ping any of this blacklisted ip. if i disable that rulethere is no problem. when i enable rule ping stop. Im trying to drop blacklist ip to can acsses to me. but i want to acsses to them. I need it cause some of my customers have some VPN, DVR and they cant acces to them.Code: Select allchain=prerouting action=drop log=no log-prefix="" src-address-list=intrusBL
Thanks.
chain=Filter action=drop in-interface=wan0 connection-state=new src-address-list=intrusBL log=no log-prefix=""You should be very very careful in allowing clients to inject blackhole information into a publicly distributed list. One malicious actor could very easily black list tons of legitimate addresses, either by directly advertising addresses into the master list (if he controls a subscribed client), or by sending spoofed packets to a client that will trigger the spoofed source's IP into an automatic blacklist.I have question on BGP. Is it possible to have clients to sent back IP addresses of attacking addresses up to the main BGP and that after certain threshold the address will be merged into that BGP. An client is only allowed to sent once in the 24h the same address to that you get a balanced threshold for an address to be blacklisted.
:put [find address=x.xx.172.2];
*154c53
:do { /ip firewall address-list add timeout="25h" list=intrusBL address=x.x.172.2} on-error={set *154c53 timeout=25h};
How do I call directly the .id this because I think that on the moment of the error the .id is filled and so the set can use the .id (index) directly to change the timeout.151 D intrusBL x.xx.172.2 jul/13/2017 11:52:58 1d59m50s
:local blListName "intrusBL";
:global blScriptVersion "2.0.1";
:local cc $blCount;
:local bn [ $urlEncode t=[/system resource get board-name ]];
:local rv [ $urlEncode t=[/system resource get version ]];
:local tm [ /system resource get total-memory ];
:local cl [ /system logging get number=0 value-name=topics ]
:local bs [ :resolve server=$blDnsHost server-port=$blDnsPort domain-name=127.0.0.3 ]
:local sv $blScriptVersion:if ($blDebug = 1) do={
:put "System ID: $si";
:put "Board Name: $bn";
:put "RouterOS Version: $rv";
:put "Total Memory: $tm";
:put "Script Version: $sv";
:log info "System ID: $si";
:log info "Board Name: $bn";
:log info "RouterOS Version: $rv";
:log info "Total Memory: $tm";
:log info "Script Version: $blScriptVersion";