Community discussions

MikroTik App
 
mwhitaker
just joined
Topic Author
Posts: 2
Joined: Wed Aug 24, 2022 6:04 pm

ARP entries building up

Fri Apr 28, 2023 4:27 pm

Hi Folks,

I've run into this on several sites where we have RouterOS 7 deployed. The symptoms are that the router doesn't remove stale ARP entries from the ARP table for devices that have long since disconnected from the network.

The symptoms are quite distinctive..... If I use the web interface to view the ARP table then I see many entries that have a valid IP, MAC and hostname, but a blank entry for interface (these are all devices that have connected in the past). This is in addition to currently connected devices that show a valid interface. If I use /ip/arp/print detail then I only see the currently connected devices, so I don't see any of the stale ones.

The existence of these stale entries prevents the DHCP server issuing leases when 'Conflict Detection' is turned on. Note that 'Add ARP for leases' is not turned on.

The problem has been seen on 7.7 and 7.8.
 
MartinW
just joined
Posts: 6
Joined: Mon May 15, 2023 2:01 pm

Re: ARP entries building up

Mon May 15, 2023 2:08 pm

Has anyone else seen this?

I can't really believe we're the only ones....
 
MartinW
just joined
Posts: 6
Joined: Mon May 15, 2023 2:01 pm

Re: ARP entries building up

Tue Jun 20, 2023 5:47 pm

Still seeing this on 7.9.2 (will let you know if 7.10 still has this issue)....
 
MartinW
just joined
Posts: 6
Joined: Mon May 15, 2023 2:01 pm

Re: ARP entries building up

Thu Jul 27, 2023 5:47 pm

Still seeing this on 7.10.2.

I haven't managed to re-create it on a minimal setup yet, but I'm wondering if it's related to setting arp=proxy-arp. Although this isn't set on the vlans that are experiencing this lack of ARP clearing, it is set on another interface on the system. That's the only thing I can think of that would make the config I'm running anything other than plain and simple. (note it's not a config issue in itself - all the v6 routers we have out there running the same config don't experience the issue).
 
pe1chl
Forum Guru
Forum Guru
Posts: 10183
Joined: Mon Jun 08, 2015 12:09 pm

Re: ARP entries building up

Thu Jul 27, 2023 6:04 pm

I noticed this on our router as well after upgrading to v7.
I worked around it by adding a scheduled script to remove the ARP entries during the night.
Indeed we are using local-proxy-arp so it may be related, although at another router without that I see many old ARP entries as well...
 
MartinW
just joined
Posts: 6
Joined: Mon May 15, 2023 2:01 pm

Re: ARP entries building up

Mon Aug 07, 2023 7:35 pm

I've now just seen it on a router without proxy-arp configured (in fact with a fairly simple config), so I think the proxy-arp connection is perhaps a red-herring....

I too have had to add a scheduled script to clear entries every night.... Not ideal, but it'll do for now.

I can't believe 100s of people aren't reporting it here.... I'm clearly going to have to spend a bit of time finding the minimum config I can reliably reproduce this with.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10183
Joined: Mon Jun 08, 2015 12:09 pm

Re: ARP entries building up

Mon Aug 07, 2023 8:32 pm

Well, unless you happen to be looking at it (I was graphing the number of ARP entries in MTRG and noticed the increase after upgrading to v7) you probably will never notice it and not see it as "a problem".
You can also enable "add arp entry for leases" in DHCP and the ARP table will remain full all the time... cutting down on the amount of ARP traffic.

What I find confusing (and have not explained yet) is that there sometimes are unconfirmed ARP entries for the addresses of the router itself.
 
MartinW
just joined
Posts: 6
Joined: Mon May 15, 2023 2:01 pm

Re: ARP entries building up

Mon Aug 07, 2023 10:36 pm

The thing is that it does cause a problem on sites where you have a high 'churn' of users. When conflict detection is turn on the DHCP server (which is always a good idea) it sends an ARP request for the IP it is about to hand out. If the entry already exists in the ARP table it doesn't even bother sending the ARP request - it just marks that IP as in-use and tries the next one....
 
User avatar
colinardo
just joined
Posts: 18
Joined: Sun Jan 08, 2017 9:02 pm

Re: ARP entries building up

Wed Aug 16, 2023 3:20 pm

Have the same issue on all my current RouterOS v7 systems wenn arp is "enabled". Also in version 7.11 this problem is still present. No cleanup occurs, also when using minimal config.
Setting the timeout value explicitly on the interface also does not change anything.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10183
Joined: Mon Jun 08, 2015 12:09 pm

Re: ARP entries building up

Wed Aug 16, 2023 5:13 pm

I think the timeout value has nothing to do with the amount of time an ARP entry remains cached in the router.
Rather it is the time that packets sent to an interface remain "queued" while the ARP entry is not yet in confirmed state (i.e. no reply received).
When data remains queued for longer than that set "timeout" and the ARP entry still is not confirmed, the data is thrown away (memory freed).
When the ARP reply is received before that timeout, the queued data is sent.

There maybe should be another parameter that specifies how long a confirmed ARP entry can remain in the table before being purged and required to be re-established. But it is not there, and I have seen the same issue in plain Linux systems with a recent kernel (lots of ARP entries for systems that are long gone from the network).

As written before, you can work around it by creating a scheduled script that removes the ARP entries.
You can start with "/ip arp remove [ find where !complete ]" and when that still does not remove as much as you like: "/ip arp remove [ find ]".
 
User avatar
TeWe
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Tue Sep 12, 2023 1:27 pm

Re: ARP entries building up

Tue Sep 12, 2023 2:25 pm

Same here - 7.11.2 and ARP entries not disappearing.
Very very basic config with no VLANs or 'Add ARP for leases'.
Occasionally having some 200+ users on a /24 network so I'd be nice if this worked as it should instead of running scheduled scripts do do the job...
 
pe1chl
Forum Guru
Forum Guru
Posts: 10183
Joined: Mon Jun 08, 2015 12:09 pm

Re: ARP entries building up

Tue Sep 12, 2023 4:58 pm

So, create the scheduled script. That works, and causes no further issues.
 
holvoetn
Forum Guru
Forum Guru
Posts: 5317
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: ARP entries building up

Tue Sep 12, 2023 5:14 pm

It has been for quite a while a scheduled script or netwatch was needed for DNS change of wireguard peer.
Eventually it has been solved so maybe give it some time ?
 
pe1chl
Forum Guru
Forum Guru
Posts: 10183
Joined: Mon Jun 08, 2015 12:09 pm

Re: ARP entries building up

Tue Sep 12, 2023 5:41 pm

As I wrote before, it is something that the new Linux kernel does. Probably there is a reason for that change, is more efficient or whatever.
 
User avatar
pcunite
Forum Guru
Forum Guru
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: ARP entries building up

Fri Sep 15, 2023 2:48 pm

Yes, I have this issue RoS 7.9.2
 
EdPa
MikroTik Support
MikroTik Support
Posts: 274
Joined: Fri Sep 15, 2017 10:05 am
Location: Riga
Contact:

Re: ARP entries building up

Fri Sep 15, 2023 3:27 pm

These are expected and documented results, see the max-neighbor-entries property under IP settings:

The ARP cache stores ARP entries, and if some of these entries are incomplete, they can stay in the cache for an indefinite period of time. This will only happen if the number of entries in the cache is less than one-fourth of the maximum number allowed. The reason for this is to prevent the unnecessary running of the garbage-collector when the ARP table is not close to being full.

If this is unwanted, it is possible to either manually clear the incomplete entries or decrease the max-neighbor-entries.
 
User avatar
TeWe
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Tue Sep 12, 2023 1:27 pm

Re: ARP entries building up

Fri Sep 15, 2023 3:30 pm

Perfect - thank you for the final clarification.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: ARP entries building up

Fri Sep 15, 2023 4:51 pm

The multithreaded address lookup algorithm has zero overhead for accessing the ARP table, sacrificing entry insertion/deletion performance. That makes perfect sense: the router adds an ARP entry once, then may access it a million times. The garbage collector locks the entire ARP table, completely halting address resolution. While it lasts only a split second, it still matters, considering gigabits of traffic. That is the reason why we keep disconnected hosts in the ARP table if the memory allows it (up to max-neighbor-entries / 4). Modifying the existing ARP entries has way less overhead than adding/removing them (modification locks only the record, not the entire table).

If you still want to purge incomplete/unreachable ARP entries, you can schedule the script:
/ip/arp/remove [find where !complete]
 
User avatar
TeWe
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Tue Sep 12, 2023 1:27 pm

Re: ARP entries building up

Fri Sep 15, 2023 4:58 pm

What you say makes perfect sense.
In my case I am running the scheduled script now with
/ip arp remove [ find where !complete ]
and I am super happy with it as it does exactly what I wanted.

Thank you pe1chl and raimondsp - keep up the good work!
 
beaudettejl42
just joined
Posts: 2
Joined: Wed Nov 22, 2023 7:18 pm

Re: ARP entries building up

Wed Nov 22, 2023 7:30 pm

I had a lot of double entries that were not complete; however, this is out of control I have a dozen or more, one with a dozen of its own entries showing complete. Is there not a built in arp flush somewhere with this change to 7. I understand the change might be good but we should have a manual flush. Any other scripts?
 
pe1chl
Forum Guru
Forum Guru
Posts: 10183
Joined: Mon Jun 08, 2015 12:09 pm

Re: ARP entries building up

Wed Nov 22, 2023 9:31 pm

When you have lots of incomplete entries there is something that is scanning your network.
Try to get rid of that. Sometimes "autodiscover" options in certain devices or drivers do that.
(e.g. to find a printer on the network)
 
beaudettejl42
just joined
Posts: 2
Joined: Wed Nov 22, 2023 7:18 pm

Re: ARP entries building up

Wed Nov 22, 2023 9:38 pm

" I had a lot of double entries that were not complete; however, this is out of control I have a dozen or more, one with a dozen of its own entries showing complete. Is there not a built in arp flush somewhere with this change to 7. I understand the change might be good but we should have a manual flush. Any other scripts?"

***My apologies, the ARP entries status was Complete, so the script to remove incomplete is not going to work for that. The in-complete duplicates were rather minimal.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10183
Joined: Mon Jun 08, 2015 12:09 pm

Re: ARP entries building up

Thu Nov 23, 2023 2:03 pm

When you have no special ARP entries you made yourself (static, published) but only entries that show up in the list as DC (with the occasional D) you can clear the entire table using:
/ip arp remove [ find ]
Then it will immediately re-build itself with what is minimally necessary at that moment.
 
anovojr
just joined
Posts: 10
Joined: Wed Nov 15, 2017 9:24 am
Location: Philippines
Contact:

Re: ARP entries building up

Thu Nov 23, 2023 3:15 pm

To deal with stale ARP entries, one workaround is scheduling a script to run regularly that clears out old ARP entries using the "/ip arp remove [find where interface=""]" command.

Who is online

Users browsing this forum: archemist, Bing [Bot], Google [Bot], tangent and 59 guests