Community discussions

MUM Europe 2020
 
draknor
just joined
Topic Author
Posts: 9
Joined: Fri Nov 27, 2015 8:08 pm

DNS pegging CPU to 100%?

Wed Apr 20, 2016 5:45 am

I had an issue today on my Routerboard 2011UAS-2HnD (Firmware 3.24), running 6.33.3.

Everything was working fine all day, then all of a sudden LAN clients started getting DNS timeout errors. I was remotely connected at the time (through the Routerboard) and never lost my connection, but I verified no local clients could do DNS lookups via the RB.

Then I fired up Winbox and saw that CPU was at 100%, and profiler revealed it was DNS pegging the CPU. I attempted to make DNS changes - change DNS servers, remove static entries, etc - but nothing seemed to stick.

Suspecting that I was part of a DNS attack, I added specific rules to block WAN-initiated DNS lookups; but the interface counters didn't show heavy traffic or packet volume. (And in fact I realized later that the default "drop" rule was already blocking WAN incoming DNS requests)

I attempted to take a supout.rif file - had to stop it manually, so I'm not sure if I got a complete file. I ended up rebooting the router, and when it came back up everything was fine.

I realize my several revisions behind, but I didn't see anything in any of the release notes that would address a DNS CPU issue.

Any thoughts what might have caused this??
 
kiaunel
Member Candidate
Member Candidate
Posts: 211
Joined: Mon Jul 21, 2014 7:59 pm
Location: Romania

Wed Apr 20, 2016 6:48 am

Make sure you deny DNS requests from outside. In filter rules make an input deny rule for TCP/UDP on port 53

Sent from my Lenovo K50a40 using Tapatalk
 
pe1chl
Forum Guru
Forum Guru
Posts: 5993
Joined: Mon Jun 08, 2015 12:09 pm

Re: DNS pegging CPU to 100%?

Wed Apr 20, 2016 3:22 pm

Is your internet on another interface than ether1?
Did you set it to PPPoE using some guide found on a random internet site instead of the
correct procedure?

In that case go to the IP->Firewall page, find the input rule that matches ether1-gateway incoming
interface and modify that to match your pppoe interface.
 
draknor
just joined
Topic Author
Posts: 9
Joined: Fri Nov 27, 2015 8:08 pm

Re:

Wed Apr 20, 2016 10:41 pm

Make sure you deny DNS requests from outside. In filter rules make an input deny rule for TCP/UDP on port 53

Sent from my Lenovo K50a40 using Tapatalk
I already have a default "drop" rule for all new connections incoming on eth1; I verified I could not do DNS lookups from the WAN side.
 
draknor
just joined
Topic Author
Posts: 9
Joined: Fri Nov 27, 2015 8:08 pm

Re: DNS pegging CPU to 100%?

Wed Apr 20, 2016 10:42 pm

Is your internet on another interface than ether1?
Did you set it to PPPoE using some guide found on a random internet site instead of the
correct procedure?

In that case go to the IP->Firewall page, find the input rule that matches ether1-gateway incoming
interface and modify that to match your pppoe interface.
No PPPoE; primary WAN is on eth1 and backup WAN on eth2, and I have default drop rules for both of those interfaces.
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: DNS pegging CPU to 100%?

Wed Apr 20, 2016 11:21 pm

Are you redirecting your LAN stations' dns requests to the Mikrotik?
(i.e. do you have a dstnat rule that matches udp port 53, action=redirect, or action=dst-nat to-address=Mikrotik's IP)

If you're doing this, and some of your LAN hosts are botnet-infected, then this could be the source of your problem as the local clients are pounding at your DNS service from the inside. It's possible....

The real test would be to try to perform a dns query on your Mikrotik's public IP address, but originating from out on the Internet somewhere.
http://openresolver.com/?ip=x.x.x.x

(change x.x.x.x to the public IP of your router)

I tested this just now on my own router and it is correct - if I block DNS, then the page takes a few moments and comes back with a green "no open resolver" result. If I allow DNS, then it quickly comes back with a red "open recursive resolver detected" message.
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
draknor
just joined
Topic Author
Posts: 9
Joined: Fri Nov 27, 2015 8:08 pm

Re: DNS pegging CPU to 100%?

Thu Apr 21, 2016 9:03 pm

Are you redirecting your LAN stations' dns requests to the Mikrotik?
(i.e. do you have a dstnat rule that matches udp port 53, action=redirect, or action=dst-nat to-address=Mikrotik's IP)
Not redirecting, but I do specify the Mikrotik as the primary DNS (via DHCP).
If you're doing this, and some of your LAN hosts are botnet-infected, then this could be the source of your problem as the local clients are pounding at your DNS service from the inside. It's possible....
Interesting theory - I wasn't checking the internal interface counters, just the external interfaces. But we do have antivirus + OpenDNS agents on most of the devices, so I'm hoping we're not botnet-infected!
The real test would be to try to perform a dns query on your Mikrotik's public IP address, but originating from out on the Internet somewhere.
http://openresolver.com/?ip=x.x.x.x
I verified that the default / standard rules are blocking external DNS requests.

In addition - rebooting the router eliminated the CPU pegging, so I have to imagine it was some kind of bug that got triggered, rather than an attack.
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: DNS pegging CPU to 100%?

Thu Apr 21, 2016 9:33 pm

I verified that the default / standard rules are blocking external DNS requests.
In addition - rebooting the router eliminated the CPU pegging, so I have to imagine it was some kind of bug that got triggered, rather than an attack.
Odd - but if you were seeing a big CPU spike and no traffic to go along with it, that seems like a reasonable analysis. Perhaps a supout to tech support is called for if it happens again.
Interesting theory - I wasn't checking the internal interface counters, just the external interfaces. But we do have antivirus + OpenDNS agents on most of the devices, so I'm hoping we're not botnet-infected!
I've been thinking that the way modern attack vectors work, it might be time to start thinking more about filtering what is allowed to go OUT. It's much more common now for computers to invite the malware in - like vampires. They're not relying on worms to hit your computer and exploit vulnerabilities. They send you emails with phishing links, and email with trojan installers, etc. That's been our biggest vector here. And we actually do filter our outbound traffic with an appliance (something like snort).

As for filtering outbound at home, I'm starting to look into implementing an RPZ dns server with some dynamic input from some RBL vendor such as Spamhaus's DBL.
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
draknor
just joined
Topic Author
Posts: 9
Joined: Fri Nov 27, 2015 8:08 pm

Re: DNS pegging CPU to 100%?

Thu Apr 21, 2016 10:20 pm

I've been thinking that the way modern attack vectors work, it might be time to start thinking more about filtering what is allowed to go OUT. It's much more common now for computers to invite the malware in - like vampires. They're not relying on worms to hit your computer and exploit vulnerabilities. They send you emails with phishing links, and email with trojan installers, etc. That's been our biggest vector here. And we actually do filter our outbound traffic with an appliance (something like snort).

As for filtering outbound at home, I'm starting to look into implementing an RPZ dns server with some dynamic input from some RBL vendor such as Spamhaus's DBL.
Yeah, our IT vendor/partner just rolled out OpenDNS to help mitigate this. I haven't done anything more sophisticated yet from firewall / UTM perspective, but definitely thinking about it!
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: DNS pegging CPU to 100%?

Thu Apr 21, 2016 10:56 pm

Spamhaus publishes the drop and edrop lists in raw text form on a web page.
I'm messing around right now with a script to download it, check it for changes, and insert the changes into ipset (the Linux command line tool used by the address-list firewall feature) so that my server can benefit from it. I may also make my auto-updater do double duty by putting them all into an RPZ.

I'd also like to include the Spamhaus XBL in that list, but I don't see anywhere they've published it for "roll-your-own" solutions - so it would require a subscription to the datafeed service, which isn't terribly expensive - around $300 per year... but while I'm just messin' around, I'd rather use a free source if it's trustworthy.

If I ran anything like wordpress or phpbb (any well-known web solution that tends to get hacked from time to time), I would make an htaccess rule that checks the remote host IP against the ZEN list, which is free for personal use over global DNS.
When given a spoon,
you should not cling to your fork.
The soup will get cold.

Who is online

Users browsing this forum: No registered users and 93 guests