Mikrotik router asking for ARP resolution of alot of IPs

[lish]

Hello All,

Sorry about the long post!

We have juniper router as core router and all our clients use Mikrotik routers. mix of mikroitk RBs/CCRs.
On friday last week, we had a situation where Mikrotik routers couldnt ping the GW. it started with few Mikrotiks and it kept increasing, that is more mikroitk routers started having on/off intermitency.
When we check on the juniper end, it shows that some mikrotik routers are asking for ARP resolution of alot IPs on the internet (on dieal case we expect mikroitik to ask for GW ARP resolution only and not other IPs). This was chocking the ARP policer on the juniper and leading to ARPs not replied in time and hence mikrotiks were not installing the default route. after some time service resumes and stays for a while then drops. What we found as temp solution is to set check-gateway option as ARP instead of ping for V6 routers and check-gateway option as none for V7 rotuers. this was an advise from a colleague who was facing similar issue as ours. this has realy helped restore traffic and somehow reduced the intermitency. this happened to multiple different ASNs that have juniper as core routers. it all started the same time last week. friday morning.. bit of wiered isssue. i have some few routers now that are still sending too many ARP requests for strange internet distinations.

below is one of the mikroitks asking for ARP requests for multiple IPs instead of just the Default route IP. help me shed some light please on what we could be doign wrong on the mikrotik end. or what can trigger the mikroitkt to this.

-CORE-ROUTER> monitor traffic interface ae0.2001 no-resolve |match arp    
verbose output suppressed, use <detail> or <extensive> for full protocol decode
Address resolution is OFF.
Listening on ae0.2001, capture size 96 bytes

17:37:12.010280  In arp who-has 36.139.54.251 tell 102.217.121.166
17:37:12.015058  In arp who-has 36.139.55.185 tell 102.217.121.166
17:37:12.015372  In arp who-has 209.182.232.249 tell 102.217.121.166
17:37:12.018153  In arp who-has 88.222.241.44 tell 102.217.121.166
17:37:12.023473  In arp who-has 109.233.92.44 tell 102.217.121.166
17:37:12.026060  In arp who-has 47.236.122.254 tell 102.217.121.166
17:37:12.029183  In arp who-has 43.136.47.117 tell 102.217.121.166
17:37:12.031751  In arp who-has 42.53.14.145 tell 102.217.121.166
17:37:12.032106  In arp who-has 36.139.55.47 tell 102.217.121.166
17:37:12.037065  In arp who-has 107.151.87.130 tell 102.217.121.166
17:37:12.041719  In arp who-has 109.233.92.62 tell 102.217.121.166
17:37:12.111414  In arp who-has 36.139.55.172 tell 102.217.121.166
17:37:12.184415  In arp who-has 36.139.55.149 tell 102.217.121.166
17:37:12.192381  In arp who-has 36.139.55.200 tell 102.217.121.166
17:37:12.198380  In arp who-has 36.139.55.160 tell 102.217.121.166
17:37:13.047559  In arp who-has 109.233.92.62 tell 102.217.121.166
17:37:13.047574  In arp who-has 107.151.87.130 tell 102.217.121.166
17:37:13.047583  In arp who-has 36.139.55.47 tell 102.217.121.166
17:37:13.047592  In arp who-has 36.139.54.251 tell 102.217.121.166
17:37:13.047601  In arp who-has 42.53.14.145 tell 102.217.121.166
17:37:13.047611  In arp who-has 88.222.241.44 tell 102.217.121.166
17:37:13.047620  In arp who-has 36.139.55.185 tell 102.217.121.166
17:37:13.047630  In arp who-has 43.136.47.117 tell 102.217.121.166
17:37:13.047639  In arp who-has 47.236.122.254 tell 102.217.121.166
17:37:13.047648  In arp who-has 209.182.232.249 tell 102.217.121.166
17:37:13.047657  In arp who-has 109.233.92.44 tell 102.217.121.166
17:37:13.127159  In arp who-has 36.139.55.172 tell 102.217.121.166
17:37:13.207277  In arp who-has 36.139.55.149 tell 102.217.121.166
17:37:13.207292  In arp who-has 36.139.55.160 tell 102.217.121.166
17:37:13.217241  In arp who-has 36.139.55.200 tell 102.217.121.166
17:37:14.087528  In arp who-has 109.233.92.44 tell 102.217.121.166
17:37:14.087543  In arp who-has 36.139.55.185 tell 102.217.121.166
17:37:14.087552  In arp who-has 88.222.241.44 tell 102.217.121.166
17:37:14.087561  In arp who-has 47.236.122.254 tell 102.217.121.166
17:37:14.087570  In arp who-has 109.233.92.62 tell 102.217.121.166
17:37:14.087579  In arp who-has 43.136.47.117 tell 102.217.121.166
17:37:14.087588  In arp who-has 42.53.14.145 tell 102.217.121.166
17:37:14.087597  In arp who-has 36.139.55.47 tell 102.217.121.166
17:37:14.087607  In arp who-has 209.182.232.249 tell 102.217.121.166
17:37:14.097172  In arp who-has 36.139.54.251 tell 102.217.121.166
17:37:14.097185  In arp who-has 107.151.87.130 tell 102.217.121.166
17:37:14.167245  In arp who-has 36.139.55.172 tell 102.217.121.166
17:37:14.247284  In arp who-has 36.139.55.200 tell 102.217.121.166
17:37:14.247297  In arp who-has 36.139.55.149 tell 102.217.121.166
17:37:14.247306  In arp who-has 36.139.55.160 tell 102.217.121.166
17:37:15.135019  In arp who-has 74.48.84.34 tell 102.217.121.166
17:37:15.137272  In arp who-has 109.233.92.166 tell 102.217.121.166
17:37:15.138768  In arp who-has 209.182.232.249 tell 102.217.121.166
17:37:15.140845  In arp who-has 47.236.122.254 tell 102.217.121.166
17:37:15.143318  In arp who-has 88.222.241.44 tell 102.217.121.166
17:37:15.150130  In arp who-has 109.233.92.44 tell 102.217.121.166
17:37:15.161307  In arp who-has 36.139.55.32 tell 102.217.121.166
17:37:15.171199  In arp who-has 107.151.87.130 tell 102.217.121.166
17:37:15.187252  In arp who-has 36.139.54.251 tell 102.217.121.166
17:37:15.188468  In arp who-has 42.53.14.145 tell 102.217.121.166
17:37:15.212846  In arp who-has 36.139.55.38 tell 102.217.121.166
17:37:15.218730  In arp who-has 116.202.221.159 tell 102.217.121.166
17:37:15.228235  In arp who-has 109.233.92.236 tell 102.217.121.166

[mkx]

To me it seems like Mikrotik was set with default route to use interface instead of gateway, i.e. add default gateway ether1 (where ether1 is interface used for upstream connection). It seems that there are a few tutorials floating around internet suggesting such approach.

[jaclaz]

Interesting possibility.
But how/why this started happening last friday on multiple devices?

[mkx]

Interesting possibility.
But how/why this started happening last friday on multiple devices?

@OP says that ARP policer kicked in … what I’m saying is that the same thing might be going on for ages but last friday cummulative internet usage of those with flawed setups exceeded ARP policer’s threshold.

[lish]

To me it seems like Mikrotik was set with default route to use interface instead of gateway, i.e. add default gateway ether1 (where ether1 is interface used for upstream connection). It seems that there are a few tutorials floating around internet suggesting such approach.

i have seen this response before several times to the question of ARPs appearing on WAN interface and we have followed the advise several times and avoided using interface as GW NHOP. i have attached PIC of how we define the default route.
We select the IP as NH for the default route but still have this ARP lists on WAN. sometimes as many as 4000 entries all with D entry. what could we doing wrong? one thing i find all our clients do with Their mikroitk is they delete all FW rules and leave router open to the internet. their version of securing the router is to change the WIBOX Port and disable all other IP services.. am not sure this really secures the router as FW rules does.

One question: what is the difference when i choose ARP instead of PING as check-gateway option? this seemed to have reduced the ARP trafffic overwhelming the juniper ARP policer hence traffic started behaving .

[lish]

Interesting possibility.
But how/why this started happening last friday on multiple devices?

Are you by any chance in a position to shed light on the diffrence between ARP or PING check gateway option ? which one has less ARP Noise? juniper policer seem to behave when ARP option is chosen.
why would PING option cause more noise? or its a wrong assumption i might be making?

[jaclaz]

I am sorry, I have no idea about that.
The (little) theory I understood is that ARP is a L2 protocol, so it should be limited to local network.
From this (and the linked to Linux article):
http://forum.mikrotik.com/t/arp-ping/150095/1
I understand that when you do a “normal” ping (at least on local network address) you are actually also updating the ARP table, whilst when you do an ARP ping, you do not (very counterintuitive).

My note was only about the issue starting concurrently on more than one device exactly at the same date/time, I mean it is entirely possible that one user has decided on friday morning to scan the whole internet so that one of your routers starts flooding the juniper upstream (or whatever) with requests, but more than one user?

Something must have changed in the configuration of some device, and as well it is improbable that at the same time more than one user changed their configuration and changed it in the same way or at least in a way that produces the same result.

Given your:

one thing i find all our clients do with Their mikroitk is they delete all FW rules and leave router open to the internet. their version of securing the router is to change the WIBOX Port and disable all other IP services

(if you want I have a spare qualifying adverb , “mindlessly” , that you can insert in at least a couple positions in the above sentence )
even if I am usually an optimist, I would consider the possibility that those routers have been attacked or compromised or however used for some kind of botnet activity.

[lish]

I understand that when you do a “normal” ping (at least on local network address) you are actually also updating the ARP table, whilst when you do an ARP ping, you do not (very counterintuitive).

Hmmm . This explains the less ARP noise then. the ARP ping dosnt lead to ARP table update meaning no need for ARP resolution.. … this makes sense as to how it leads less ARP Noise.

[smithg400]

I little late to the party here, but just wondered in you’ve installed ‘The Dude’ package on one of your routers.

I’ve had a problem with streams periodically losing packets for several seconds at a time and I’ve had lots of complaints from the rest of the family watching Netflix (or similar), or making video calls. The only odd thing I could see was that the router was sending out ARP requests at an enormous rate. Seems it goes back to me installing The Dude on my router and it was continuously sending out ARPs to discover what devices were on the network; now I’ve uninstalled The Dude everything is now.

I’m guessing that the large number of ARP requests were causing my (relatively) cheap TP-Link Easysmart switches to drop packets, though I haven’t (yet) fully confirmed why this is the case. Cutting the network down to router ↔ single switch ↔ linux PC and running iperf3 from the linux PC to a cloud hosted machine, there were lots of packets lost and the test paused for several seconds at a time if the switch was a TP-Link Easysmart switch. Swapping it for a dumb switch (also TP-Link) the problem was resolved. The Easysmart switch still works fine though when I disable / uninstall ‘The Dude’. So I expect my real problem is the Easysmart switches but for the moment not using ‘The Dude’ has stopped the moaning from the family!