Security concern - IP's on list not always blocked

RouterOS General
1

We have a dynamic address list of approximately 25,000 IP’s with a time on list of 45 minutes. The list is dynamically generated by a certain type of traffic offense (i.e. failed login attempt). We have a rule that blocks connections from the IP’s on this list. We have witnessed the list does not always block the IP’s on the list. We tested this by adding a static entry to the list and then performing a simple ping from the blocked IP. Eventually we were able to successfully ping and reach the target host that is protected form the IP’s on the list. Our config is an Intel PC dual P3 1.4Ghz 2GB RAM average 3% CPU utilization running 3.11. Should we expect a list of this size to not always block the entries because of the sheer number of IP’s on the list? We’re just trying to set our expectations appropriately. Thank you, Jeff

2

How are you blocking the addresses, if you’re only blocking tcp or udp traffic from those addresses then a ping (icmp packet) would still get through.

3

hi, here’s the rule:

add action=drop chain=forward comment=
“Drop incoming connections from IPs in Blacklist” disabled=no
dst-address=1.2.3.4 dst-port=23 in-interface=ether1 out-interface=
ether2 protocol=tcp src-address-list=Blacklist

pretty basic. we’ve tried it with ports and with no ports.

4

jtuttle, if you can quickly reproduce this issue and can give MikroTik access to your router to install debug package, please contact support@mikrotik.com
because I’ve installed it, but I face address-list issue not as frequent as I’d like to for debug purposes =)

5

jtuttle, your not blocking ALL traffic using that rule.

However, my coworker has seen this as well on 2.9.51 and 3.xxx.

I do have a address-list of over 80,000 and it seems to function fine. I am using it not to block but to NAT so its not a security issue, but Ive seen how it could be a REAL problem if using it to protect your network. This needs to be addressed…

6

As omega-00 said, you’re not blocking all traffic..
Try this

add action=drop chain=forward comment=\
    "Drop incoming connections from IPs in Blacklist" disabled=no \
    dst-address=1.2.3.4 in-interface=ether1 out-interface=\
    ether2 src-address-list=Blacklist
7

We’re not really trying to block all communication with this rule, the idea is to prevent positively identified spammers from pounding on our email servers excessively. Perhaps some background on the whole thing is needed.

We have two border email servers running Alligate (http://www.alligate.com) that handle incoming message traffic for all of the domains that we host. The first thing they do when a host connects to them is to check the incoming IP address against publicly maintained blacklists. If that check comes back clean they may perform some tarpitting for violations such as the sending server not providing a subdomain (i.e. identifying themselves as somedomain.com instead of mailhost.somedomain.com). Once any tarpitting has been performed the Alligate servers validate recipient addresses against our main mailserver. After Alligate actually accepts a message it passes it along to our Symantec servers to be scanned for viruses and spam, then they deliver to our mailserver.

The Alligate servers dramatically decreased the load placed on our other email systems, but even with their functionality at work we were still dealing with a lot of overhead due to thousands of known spammers constantly attempting to deliver email. We decided to start doing packet inspection, watching SMTP communication for messages that indicate the incoming IP is somebody that we don’t want to allow in repeatedly and put them on a dynamic address list on the Mikrotik to be dropped before they even talk to our mailservers. One of the rules to accomplish that trick is this:

;;; Blacklist IPs causing 550 errors from Alligate for 4 hours. chain=forward action=add-dst-to-address-list protocol=tcp src-address=192.168.1.3 address-list=SMTP_Blacklist address-list-timeout=4h src-port=25  content=550 vproc13.arainc.com refused - see http://www.mxrate.com/lookup/refused.asp?ipaddress=

There are, of course, other filter rules in place that perform the same function of placing undesirable IPs in the address list named “SMTP_Blacklist.” This part of puzzle seems to work correctly, SMTP_Blacklist tends to have between 10K to 20K entries at any given time. Based upon that list my current SMTP blocking rule is intended to prevent the members of the list from opening an SMTP connection to any of the servers behind this firewall. I started specifying “new” connections because cutting off established connections mid-conversation had bad effects on my mailservers:
ether1 is my public interface

;;; Drop **NEW** SMTP connections from IPs in SMTP_Blacklist chain=forward action=drop connection-state=new protocol=tcp src-address-list=SMTP_Blacklist in-interface=ether1 dst-port=25

This configuration mostly works, but not completely. The blocking rule prevents a huge number of connections per second from reaching my mailservers, but I am absolutely positive that things are randomly getting through that shouldn’t. I noticed the problem when inspecting the logs on my Alligate servers and have verified that it is an issue with both RouterOS v3.11 and v3.15 running on this x86 system.

I rebooted the Mikrotik this morning about 9AM, clearing out the dynamic address lists. The the Alligate logs recorded the following entries for one small sample of the thousands of spamming IPs that constantly try to connect. Notice the timestamps in green:

That reflects 5 individual hosts, possibly all belonging to the same known spammer since they’re all within 212.117.189.0/27. As expected, the filters on the Mikrotik saw those messages go past and correctly added the spamming IPs to the SMTP_Blacklist. Everything appeared to be almost perfect for an hour, there were only two sessions that any of those hosts managed to open that shouldn’t have been allowed since the timeout on the list is set to 4 hours. I’ve marked the IP in those sessions in blue.

I have set up the following rule before AND after my SMTP block so that I can watch how many packets those hosts are getting past the firewall in real time:

;;; LOG communications with 212.117.189.0/24 chain=forward action=log src-address=212.117.189.0/24 in-interface=ether1 log-prefix="SMTP"

The packet counters on the logging rules showed that those hosts were ATTEMPTING to connect to our servers, but mostly failing to get past the blocking rule. This changed dramatically during the hour of 10AM, most obviously in a 5 minute period when we got almost 50 new connections from IPs that were still on the SMTP_Blacklist from the previous hour.

Here is the evidence from my logs, matching IPs are color coded to help make the duplicates easier to identify:

  • 2008-10-28 14:01:00:0641 47183 10:19:27.609 - (878690) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 14:01:00:0641 47213 10:19:28.234 - (878693) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 13:58:52:0838 43932 10:19:28.259 - (807250) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 14:01:00:0641 47232 10:19:28.859 - (878697) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 13:58:52:0838 43943 10:19:28.884 - (807253) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 14:01:00:0641 47256 10:19:29.469 - (878699) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 13:58:52:0838 43961 10:19:29.509 - (807257) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 13:58:52:0838 43978 10:19:30.119 - (807261) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 13:58:52:0838 43987 10:19:30.759 - (807263) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 14:01:00:0703 54359 10:23:00.375 - (879362) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 14:01:00:0703 54393 10:23:00.984 - (879366) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 13:58:52:0900 51005 10:23:01.009 - (807864) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 14:01:00:0703 54423 10:23:01.609 - (879369) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 13:58:52:0900 51015 10:23:01.634 - (807866) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 14:01:00:0703 54455 10:23:02.234 - (879371) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 13:58:52:0900 51026 10:23:02.259 - (807869) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 14:01:00:0703 54500 10:23:02.859 - (879375) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 13:58:52:0900 51043 10:23:02.869 - (807870) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 14:01:00:0703 54554 10:23:03.484 - (879378) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 13:58:52:0900 51057 10:23:03.494 - (807872) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 13:58:52:0900 51073 10:23:04.134 - (807875) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 14:01:00:0703 54588 10:23:04.188 - (879381) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 14:01:00:0703 54605 10:23:04.813 - (879383) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 13:58:52:0900 51108 10:23:04.838 - (807880) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 14:01:00:0703 54633 10:23:05.438 - (879384) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 13:58:52:0900 51129 10:23:05.447 - (807881) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 14:01:00:0703 54676 10:23:06.063 - (879388) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 13:58:52:0900 51150 10:23:06.072 - (807884) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 14:01:00:0703 54715 10:23:06.688 - (879391) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 13:58:52:0900 51164 10:23:06.697 - (807886) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 14:01:00:0703 54752 10:23:07.297 - (879393) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 13:58:52:0900 51176 10:23:07.322 - (807888) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 14:01:00:0703 54770 10:23:07.922 - (879394) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 13:58:52:0900 51185 10:23:07.947 - (807889) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 14:01:00:0703 54813 10:23:08.531 - (879395) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 13:58:52:0900 51194 10:23:08.556 - (807891) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 14:01:00:0703 54842 10:23:09.172 - (879398) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 13:58:52:0900 51237 10:23:09.181 - (807892) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 14:01:00:0703 54873 10:23:09.797 - (879402) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 13:58:52:0900 51251 10:23:09.806 - (807893) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 14:01:00:0703 54900 10:23:10.422 - (879405) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 13:58:52:0900 51256 10:23:10.431 - (807894) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 14:01:00:0703 54922 10:23:11.047 - (879407) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 13:58:52:0900 51271 10:23:11.056 - (807896) Cmd sent: 550 ipaddress=212.117.189.20
    2008-10-28 14:01:00:0703 54987 10:23:11.656 - (879413) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 13:58:52:0900 51280 10:23:11.697 - (807898) Cmd sent: 550 ipaddress=212.117.189.14
    2008-10-28 13:58:52:0900 51303 10:23:12.306 - (807900) Cmd sent: 550 ipaddress=212.117.189.18
    2008-10-28 13:58:53:0681 130344 10:54:46.572 - (815510) Cmd sent: 550 ipaddress=212.117.189.19
    2008-10-28 13:58:53:0728 135464 10:57:55.838 - (815989) Cmd sent: 550 ipaddress=212.117.189.21

This is a very small example of the problem, but it should be clear now. Even when I try to block ALL new traffic from hosts in the SMTP_Blacklist (instead of only blocking TCP sessions destined for port 25) the filter sometimes goes through spurts of allowing traffic in that it shouldn’t. This is extremely troubling, it makes me wonder what other sessions might be getting through even though the configuration specifies that they should be blocked.

8

I can confirm this as well. I have seen this at least 4 times in the past week now. Only with 3.x however, I might have seen this on 2.9.51 but not positive. Which version are you running?

We need to get this info to support@mikrotik and have it troubleshot… it’s not very secure if the address-lists are ignoring entries.

Sam

9

I’re working with support to troubleshoot the problem, they instaled debug package on my router… after I write e-mail to support saying that I have next ‘bad’ item in address list, they logged in my router, do something (>1 Mb incoming traffic =) ), but still no response what they have seen…