Management high CPU on lots of Mikrotiks today - DDoS??

RouterOS General
1

We monitor over 100 Mikrotik routers - all of which are independent to each other and have a fairly basic config.

We lock down all the management ports (WinBOX, SSH) with only IPs allowed from our network.

We then monitor all our estate with LibreNMS (SNMP Monitoring server)

What is strange is we have noticed about 50% of our Mikrotik estate all suddenly show the same high CPU behaviour at the same time.

example screenshots of independent Mikrotik routers


As you can see a high percentage of these routers all for some reason have jumped to 40% CPU all at the same time, even though these devices are not connected or aware of each other. They simply are acting as small business routers with a FTTC/Broadband Modem attached.

When i look at the /tool profile on the higher CPU devices the culprit appears to be ‘management’
Screenshot 2018-11-19 at 12.56.45.png
I havea feeling this is some type of DDoS type attempt against Mikrotik devices as there is no explanation for this.

Please can others go and check their Mikrotiks and see if they see similar behaviour that happened around 1-2am UK time today.

Example of our config is here

/interface bridge
add name=bridge1
/interface ethernet
set [ find default-name=ether1 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=ether2 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=ether3 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=ether4 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
set [ find default-name=ether5 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 name=pppoe-out1 password=Password use-peer-dns=yes user=c@xxx.co.uk
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-Ce disabled=no frequency=2442 mode=ap-bridge ssid=“Bezza WiFi” wps-mode=disabled
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa-psk,wpa2-psk eap-methods=“” group-ciphers=tkip,aes-ccm mode=dynamic-keys supplicant-identity=MikroTik unicast-ciphers=tkip,aes-ccm wpa-pre-shared-key=password! wpa2-pre-shared-key=password
/ip pool
add name=dhcp_pool0 ranges=192.168.77.10-192.168.77.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=bridge1 lease-time=1d name=dhcp1
/system logging action
set 0 memory-lines=1
set 1 memory-lines=1 target=memory
set 2 memory-lines=1 target=memory
/interface bridge port
add bridge=bridge1 interface=ether2
add bridge=bridge1 interface=ether3
add bridge=bridge1 interface=ether4
add bridge=bridge1 interface=ether5
add bridge=bridge1 interface=wlan1
/ip address
add address=192.168.77.1/24 interface=bridge1 network=192.168.77.0
/ip cloud
set ddns-enabled=yes
/ip dhcp-server config
set store-leases-disk=1d
/ip dhcp-server network
add address=192.168.77.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.77.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 name=router.lan
/ip firewall filter
add action=fasttrack-connection chain=input
add action=drop chain=input dst-port=53 in-interface=pppoe-out1 protocol=udp
add action=drop chain=input dst-port=53 in-interface=pppoe-out1 protocol=tcp
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-out1
/ip firewall service-port
set sip disabled=yes
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www address=192.168.77.0/24
set ssh address=192.168.77.0/24,185.109.104.0/22,77.76.73.254/32,192.168.77.0/24
set api disabled=yes
set winbox address=185.109.104.0/22,212.56.94.105/32,77.76.73.254/32,192.168.77.0/24
set api-ssl disabled=yes
/snmp
set enabled=yes location=“PR7 XXX” trap-target=185.109.106.4 trap-version=2
/system clock
set time-zone-name=Europe/London
/system identity
set name=“Customer : cpe”
/system routerboard settings
set silent-boot=no

Thanks

2

Ok, figured it out ..

it appears an IP 144.168.68.43 is doing a mass scan on UDP 161 (SNMP) to all Mikrotik devices constantly. Unsure what the reason its doing this is but this is the cause.

Also the IP owner has a suspect name if you whois them - Organization: ZERO DDOS LLC (ZDL) ??? zero DDOS! more like lots of DDOS

144.168.68.43

We have resolved this by a simple filter that blocks SNMP unless it comes from our monitoring server IP

3

You clearly don’t have a secure firewall if this traffic is able to reach your router. Rather than locking down only management services, you should block all unknown traffic by default.

As for the IP - you are actually being used to cause a DDoS against this company. Someone is spoofing the IP of this “zero DDoS” company, and your router is replying with large amounts of SNMP traffic. This is called a reflected amplification attack.

4

what firewall command would you recommend that blocks all unknown?

right now i am blocking SNMP to unknown sources unless its ours and this is resolving the issue.

Other than that, the firewall is fairly locked down other than this one service that appeared to be open

5

DNS is also open to the world!! Your firewall rules fast track anything going input then you have drop rules after this which will never work as you have already fast tracked the traffic.

steve@general:~$ dig forum.mikrotik.com @X.X.X.X

; <<>> DiG 9.10.3-P4-Ubuntu <<>> forum.mikrotik.com @X.X.X.X
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61779
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;forum.mikrotik.com.            IN      A

;; ANSWER SECTION:
forum.mikrotik.com.     1472    IN      A       159.148.147.205

;; Query time: 25 msec
;; SERVER: X.X.X.X#53(X.X.X.X)
;; WHEN: Mon Nov 19 14:15:11 GMT 2018
;; MSG SIZE  rcvd: 52
6

Probably 1 out of 50 have fast track enabled so i can address this one - this isnt our standard config with fast track enabled as we find we don’t really need fasttrack or see any performance gain

7

You probably wouldn’t with that implementation. You are only FT’ing the “input” traffic and not the “forward” with that rule. Once you apply it to the forward chain then things start to get a lot more interesting but it begs the question if you really “need” to?

Trying to help you here, with a firewall as “slim” as you have I probably wouldn’t be plastering client IP’s all over the web though.
You could try a firewall like this on each client router? I’ve tried to tailor it to you by putting in the “in-interface” and adding what looks like your MGMT networks to the “router-access” list (so you don’t get locked out). I would absolutely implore you to bench this first though (although I’m sure you would).

/ip firewall address-list
add address=185.109.104.0/22 list=router-access
add address=212.56.94.105/32 list=router-access
add address=77.76.73.254/32 list=router-access
/ip firewall filter
add action=accept chain=forward comment="ACCEPT established & related" connection-state=established,related in-interface=pppoe-out1
add action=accept chain=input comment="ACCEPT established & related" connection-state=established,related in-interface=pppoe-out1
add action=accept chain=forward comment="ACCEPT dst-nat connections" connection-nat-state=dst-nat in-interface=pppoe-out1
add action=accept chain=input comment="ACCEPT ICMP" in-interface=pppoe-out1 protocol=icmp
add action=accept chain=input comment="ACCEPT MGMT Network" in-interface=pppoe-out1 src-address-list=router-access
add action=drop chain=forward comment="Drop invalid" connection-state=invalid in-interface=pppoe-out1
add action=drop chain=input comment="DROP invalid" connection-state=invalid in-interface=pppoe-out1
add action=drop chain=input comment="DROP ALL" in-interface=pppoe-out1
add action=drop chain=forward comment="DROP ALL FROM WAN" in-interface=pppoe-out1
8

Same attack, started yesterday and stopped this morning by dropping. In 6 hours we drop a lot of traffic.
attackover161.jpg

9

Thanks for some feedback - i will look at making a few amendments to my base config generator to include some enhancements to the firewall.

I have remove the IPs / screenshots from the post above

Just removed my home routers firewall config with this to see how it works .. we specialise in VoIP so want to make sure this has no negative impact

10

I have just removed the IP’s from my DNS lookup post for you also (although your quoted post may also want the same treatment?).
There should be no negative effects on VOIP as it’s first job is to allow established and related connections as well as allowing NAT’d ones.
For VOIP I read a lot about disabling SIP-Helper or SIP-ALG which you can do like this;

/ip firewall service-port
set sip disabled=yes

Let the forum/me know how you get on!

11

Start with a reset to default configuration and then add what you need.

I have a rule that block all IP for 24 hours that tries any port that not are open to internet.
If you do try a non open port at my router, you have nothing here to do.

12

Yes , we always disable SIP helper - it causes no end of trouble for audio when its enabled. This isn’t unique to Mikrotik, pretty much all Firewalls cause problems. It begs the question why firewall manfactures think its a ‘helper’ when all it does it cause problems.

13

Not quite sure the firewall rules are working

see screenshots
Screenshot 2018-11-19 at 16.25.18.png
Screenshot 2018-11-19 at 16.25.03.png
if i add specific 161 UDP drop it will lower the CPU on management

14

this is the one that is allowing it to come in still
add action=accept chain=input comment=“ACCEPT established & related” connection-state=established,related in-interface=pppoe-out1

15

ok, figured it out, mikrotik needs a reboot to clear the connection table or to manually clear the connection table and then it doesn’t treat it as ‘established’

16

Glad you got it sorted.
Maybe just shuffle the “accept dst-nat” rule to number 3? You really want the rules with the most traffic towards the top so the packets are not delayed in being handled and est&rel will be the highest ones (in most applications).