I’m not sure what you are asking here. You are always welcome to contact a site and ask them to fix any issues. The subnet will be removed from the list automatically once whatever issue they were having is fixed. Many times it’s that they are hosting a botnet that they do not even know about. Other times it may be that they are serving viruses in ads. AWS and Google Compute have both been blocked several times because they refuse to take down a virtual host that is being used to attack other networks.
Hi Dave,
I use your dynamic blacklist and it suits well.
For some reason a subnet from which I really need to use 1 address from appeared in one of the lists loaded into dynamicBlacklist …
It would be a great help if I could whitelist a subnet inside the script somehow …
Keep up the great work
Eddie
Whitelisting is accomplished by creating a new address-list and a new filter rule.
- Create an address list - say.. “Whitelist” and add the IP addresses that you need never be blocked.
- create a new filter “Accept” rule, using the src-address-list you created.
- place the new Whitelist Accept rule ABOVE the blacklist Drop rule.
There is no need to modify the script, and this can not be done on the server side.
Please keep in mind that it’s always better to understand why the IP/Subnet ended up on the blacklist and attempt to get that corrected first. I have seen several networks penetrated because an admin whitelisted an address that was serving malware, instead of contacting that site/service and getting the issue resolved.
Hi Dave,
thanks for your response. It was a bit complex but I did manage to add the subnet in a whitelist and that works for now…
I am aware of the problems that might cause such a whitelist. In this case the subnet is from a local provider with many customers fighting spam …
sometimes one of them gets blacklisted for that reason and sometimes the entire subnet is.
To do make sure that 2 of the servers within my responsibility are not causing troubles I need to have access to them.
For now I whitelisted those 2 and that does the job.
regards,
Eddie
Hi Dave,
Dear,
It’s not working on my RB600.
It used to work for a long time…
Any idea how come?
Thanks,
Patrick
The script was updated last week to work with the new backend servers. You can find the update in the first post of this thread
Thanks to someone setting up 50 routers to download every 2 minutes, the server is now blocking any router that downloads more than 4 times in a 24 hour period.
Is the blockage permanent or will it be lifted after a certain time? I can imagine that one that is testing has to reboot a few times in a few hours. So when testing one should switch off the startup reload until all is stable again.
SHouldn’t be an issue for most. The server will flag routers that get excessive and throttle them to 4 download in a 24 hour period.
And if the devil was at work with my Mikrotik and made it crashed and luckily I managed to switch off the startupscript. I had already had three strikes so I also should disable the normal update, for the time being.
Maybe it is possible to keep the file on the disk (when not using flash) and delete it on the next regular reload of 24 hours. After the first import it would have to be renamed with the time, of the first import in the name.
The script looks on a restart or regular reload if the file is older dan 23 hours and then it would get a new one. If the file is younger than 23hours the script will reload the file form the disk.
You can then still throttle addresses that reload more than three times with a blank router. If they have to setup routers then they also should copy the current file to the disk on each router.
AAAARGH lost some sleep by trying to find out the way how to convert date+time so that I could subtract those and have the difference in time. No I did not manage but manage to go to sleep after staying up much to long.
In the morning my mind started to seek for a solution and I had different ideas but none of the would solve this. Then I got a great idea to just make a different script just for only the start-up. All pieces fell in place and no calculation and string delidding needed and just let nature do its work and follow the natural flow.
The main script updateBlacklist is changed so that dynamic.rsc file is not deleted after importing. The new startup startupBlacklist is the current updateBlacklist stripped of all download and statistical parts.
In updateBlacklist I commented out the removal of the dynamic.rsc file after importing and it will overwritten by the new dynamic.rsc file when the daily update is run. This is the changed code part form updateBlacklist
Hope that you like this adaptation and so also give your server bit of rest because restarted Mikrotik devices will not bother it when just restarting and only knock on the door for the real daily update.
So I am going now to eat my breakfast and enjoy my Sunday which is also today a sunny day.
Hi DAVE
yesterday , i change new script ,
it’s working in my RB450G & RB750GL,
but in my CCR1009-8G-1S-1S+ …,
just show Message “Downloading current Blacklist for this model”,
but can’t download anything…
i use the same script,…
How can i to deal with this problem?
Sorry,English not my mother Language…
Dave you could just escape the “?”. That would allow it to be run in the terminal without issue, and it will make no difference for non-terminal running.
I mentioned it before, here: http://forum.mikrotik.com/t/radius-server-not-working-in-2-8-11/127/1
"...\?..."
A bit of saving on traffic you could save about 20% of traffic by not adding the “comment” part on every dynamicblacklist line.
I don’t know if RouterOS can handle deflated traffic when downloading. However there is xz/LZMA used when a firmware update is applied. So if on saving a file with the .gz extension then it could be automatically be extracted and then your dynamic.rsc.gz would 20 times smaller and just 50KB instead of 970KB.
Update: all the .NPK files are zipped and extracted in some way when installed. Looking into the system.npk I find the program “unexpak” but I can’t see what it is doing. When I look in \lib\ I see the library libz.so and if I am not wrong that is a compress/decompress code.
The only thing I found Mikrotik talking about compression stated that due to limitations of the size of the flash not allowing a compression tool to the users.
So maybe Mikrotik could give us the option to export compressed and then to normal extension like .RSC and .BACKUP add .GZ and automatically decompress files with .GZ when read.
My updates stopped working so I went to investigate and found when running the update script (v2017.5.31f) from the command line:
[admin@redacted_name.com] > /system script run updateBlacklist;
status: failed
failure: cannot open file
I’ve installed the script via the download method described in the first post so I don’t believe it’s a copy/paste issue. Maybe I accessed the download more than four times during testing one day? Are there any other reasons to get that failure message?
Here’s my basic info:
[admin@redacted_name.com] /system resource> print
uptime: 4d4h2m55s
version: 6.39.2 (stable)
build-time: Mar/09/2017 11:32:49
free-memory: 1712.6MiB
total-memory: 1956.2MiB
cpu: tilegx
cpu-count: 9
cpu-frequency: 1200MHz
cpu-load: 0%
free-hdd-space: 78.5MiB
total-hdd-space: 128.0MiB
architecture-name: tile
board-name: CCR1009-8G-1S-1S+
platform: MikroTik
[admin@redacted_name.com] /system resource> /system license print
software-id: 8RW2-IFMS
nlevel: 6
features:
try a copy/paste from the first post. Not sure what the issue is, the server isn’t reporting any issues.
Doing the copy and paste from post #1 worked. Still not sure why it stopped working. Thank you!
Sweet, glad it fixed it for you.
Thank you for this, David!
Curious why you use a loop:
:foreach i in=[/ip firewall address-list find ] \
do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
do={ /ip firewall address-list remove $i } }
instead of
/ip firewall address-list remove [find list="dynamicBlacklist"]
On my system, the loop takes 88 seconds while the 1-liner takes 32 seconds
How does your blacklist compare with the one being provided by squidblacklist.org (which just combines spamhaus drop, edrop, dshield, malc0de, blocklist.de )?
That is indeed an easier and faster way. Dave wrote about that earlier about using those lists: http://forum.mikrotik.com/t/blacklist-filter-update-script/89817/43
I tried the change in code and the removal of the addresses went from 37 seconds down to 20 seconds and the total time is now 48 seconds and before it was 66 seconds. So the save time when using almost 20.000 IP addresses is around 17 seconds.