Maybe this is a dumb question but I have to ask it.

ARP Timeout in IP Settings seem to be ignored. Right now it is at the default factory value (30sec). The reason I ask this is because I have the following situation:
- CCR1016-12G
- RouterOS 6.39 (stable)
- WAN on eth1, LAN on eth2 (srcnat, no firewall rules added yet)
- DHCS Server on eth2 192.168.0.x/24 (IP Pool limits it to 192.168.0.10-192.168.0.240 though) with some static leases defined
- eth2 interface is in ARP "enabled" mode, same is eth1
- DHCP has "Add ARP For Leases" enabled (not that it makes any difference in this case - I think)

Problem is that I had one DHCP client with static lease not getting an external connection (WAN1 so no Internet). After a little investigation, I found out that his static IP assigned with DCHP static lease was .154 but ARP table entry for that IP was with NULL MAC address (00:00:00...). Clearing the entry in in the ARP fixed it. But I went further and found the other threads about this issue (or feature -> *) arp - show incomplete ARP entries;" in v6.33.5 ROS).

Pinging an non-existing IP from LAN ads an ARP entry in the ARP table with 00:00:00:00:00:00 MAC Address which is ok since it's Mikrotick's way of saying there is no such thing as that IP on your LAN (as opposed to CISCO for example). But my problem is, how can I make an ARP entry expire (get cleared) much faster than it is now? Because right now it seems it takes minutes and not seconds to expire (as defined in IP-> Settings-> ARP Timeout). Right now anything from a net sniffer/scanner to a simple host ping can "take over" that IP messing with the DHCP static leases (DCHP clients receive leases that have 00:00:00:... in the ARP table of the router -> trouble).

The only thing that I can think of for such behavior can be the case where someone on the network is arp querying for that specific IP but what? Testing with any unalocated IP and the results are the same and I'm pretty sure there's no internal LAN scan/sniff taking place.

The static ARP entries + ARP mode reply-only is not a solution for me. I want to allow static DCHP leases+dynamic ones + static IP on LAN clients.

Can anybody point me in the right direction? What am I missing here? Why does RouterOS takes so long to clear an non-existing ARP IP-MAC entry and not 30secs as specified in settings?

I'm running 6.39.1 on a hEX Gr3 (the new one). I guess I'm confused as to why you are having issues. My bridges and interfaces are all at the default ARP = enabled setting. I do not use add-ARP in the DHCP server. When I ping an unknown IP address I get a dynamic ARP entry with a null (no value) MAC address like one would expect and the ARP requests are flooded out all bridge ports for that VLAN. Alternatively I get dynamic and complete entries for all devices on my local bridge without issue for at least 2 months now on the latest code available at the time.

I would start by removing the add-ARP feature from your IP DHCP Server for that network.

I already tested "Add ARP entry" in DHCP before posting. As expected, it has no efect whatsoever. There is no connection i can think of between that option and pinging an un-allocated IP on the network because that's the problem here (ARP table "taking over" an IP address with 00:00...MAC address when pinging for example). So what am I missing here?

LE: Right now it takes waaay to long for the ARP table to "release" that entry (timeout after about 5 to 10 minutes) and the ARP Timeout in IP settings is at 30 secs. Anybody?

So nobody? What am i doing wrong? Is this normal behaviour or a bug RouterOS?

For those of you who need it, here's the answer i got from Mikrotik support:

Support: In RouterOS similarly to other Linux systems incomplete ARP entries show up when you can not reach requested IP address from router.
Regarding this incomplete entry two different timeouts should be taken into account.
1) If the entry has not been used and is stale for X seconds (arp-timeout), it should be eligible to be removed;
2) If X seconds has passed and entry is marked as okay to be removed, it will be removed when the garbage collector runs (usually after Y seconds);

This works perfectly and remove incomplete ARP within few seconds. Now the problem is that the neighbor entry will not be deleted if it is being referenced. The main thing that you're going to have problems with is the reference from the ipv4 routing table. There is a lot of complicated garbage collection stuff, but the important thing to note is that the garbage collector for the route cache only expires entries every 5 minutes. In result ARP entry will disappear only after 5 to 10 minutes. If you want to avoid this, then under "IP/Settings" you must disable route-cache and reboot router.

Although router has given out DHCP lease and ARP table in router points to a null MAC, when communication starts, ARP entry is immediately updated with correct MAC and no packets are sent to blackhole - this has been tested.

Question: Are you saying that even if nobody/nothing is referencing that specific IP (X seconds have passed) and the ARP entry is marked for deletion Y seconds HAVE TO PASS untill OS garbage collector actually deletes the entry?

Support: That is right. This avoids consuming extra CPU resources if the table is small to see any real benefit in terms of memory.

Question: ...And if so, and this is caused by route cache, what would be the impact on overall routing performance if i try to disable it?

Support: Disabling routing cache does not allow using Fast Path feature, so it is not recommended. We are not aware that there is any bug related to it.

I think we are seeing a similar problem - but it only just appears to have started.

We have a Cloud Core Router (CCR) set up as a DHCP server.
When we connect a new client, it assigned a dynamic IP address from a small pool - 10.10.20.30 to 10.10.20.50
We then manually go into the DHCP lease table, make the lease static and then assign a specific IP address that is fixed for that client eg: 10.10.20.181
The client router then picks this up as soon as the dynamic lease expires.

We are now running into problems as we start to transfer clients from a wifi network to fibre connections.
We're using HAP AC's so simply insert an SFP module, disconnect the WiFi ethernet cable from Port 1, reconfigure the HAP AC for SFP and then plug in the fibre

In this example we're doing this for the router that was set to 10.10.20.181
The HAP AC now picks up a new IP address from the pool (eg. 10.10.20.39) - work fine, access to internet OK
At the CCR we delete the old static entry (10.10.20.181), make the new (dynamic) IP address static and remap it to 10.10.20.181
When the lease expires the HAP pick up the new IP address and immediately loses connection to the internet

Some detective work...
ARP table in the CCR still has the old MAC address in it (the one for the ethernet port on the HAP), so traffic's getting routed to the wrong MAC address.
Even though there's no traffic on this port it's not being removed from the ARP table - even after several days.
If we manually remove it from the table the new MAC address is correctly stored.

ARP timeout on all interfaces is set to default.

Any ideas? We've followed this process successfully for a number of clients that were swapped to fibre, but in the last couple of weeks have been running into the problem.

Thanks

I'm wondering if this really is related to DHCP. Seems like it is but would be nice to prove it.

Might be worth trying to run a separate DHCP server. I use the ISC DHCP server.

I have no ARP issues but not a big network either.

I cannot reproduce this on my end.

Even if the IP is in incomplete state in the ARP table, the moment the device gets its IP from DHCP the ARP entry is updated and the device gets access to the network immediately.

So, a bit more detective work.

We've set up a brand new router, connected it to the network, assigned a static IP address 10.10.20.244
Entry appears in the ARP table
Disconnect the router.
- First time I tried this the entry disappears.
- Second time entry persisted (complete with MAC address) but after a while the "C" (Complete) flag was cleared and eventually it disappears from the table.

If I then ping 10.10.20.244 the entry reappears in the ARP table but this time with MAC address 00:00:00:00:00:00 flagged as "D" (Dynamic) but not Complete. I believe this is the correct behaviour and the 00:00:00:00:00:00 entry gets cleared up after about 5 minutes by the garbage collector routine.

10.10.20.181 is in the ARP table with the old MAC address, but that router has been disconnected
Double check by pinging 10.10.20.181 from CCR - returns with "timeout" or "host unreachable"
Manually remove the entry from the table
It immediately reappears
Flagged as "D" (Dynamic) but not "C" (Complete)
MAC address is now 00:00:00:00:00:00

So is something pinging 10.10.20.181?
And if so, why is the ARP table hanging onto the old MAC address and flagging it as "C" (Complete) rather than setting it to 00:00:00:00:00:00?
The garbage collector also seems to be a bit slow getting rid of the 00:00:00:00:00:00 entries in the table, although this does not appear to be an issue, as these are replaced by the correct MAC address if a device is then connected and assigned the associated IP address.

CCR is running 6.38.5
Will upgrade to 6.39.2 overnight

I don't know if this has been solved, I had a some what similar problem.

We have a static subnet 64.57.184.0/24 on lan bridge1 of a RB1100AHx2 with 64.57.184.1/24 on the interface of the router meant to be the default gateway for other machines on the same lan.

In the arp table all entries of the subnet were filled with 00:00:00:00:00:00 except for a few IP's that were in use.

I added a new machine and gave it 64.57.184.105 and it would talk fine to every machine on the lan except 184.1

Pings from 184.105 to 184.1 were sent and returned by 184.1, tcpdump showed replies coming into the
eth0 of 184.105, but nothing showed up on the console. Neither 184.1 nor 184.105 could ping the other, although pings were being sent
and received per tcpdump which sees raw data on the eth0 interfaces.

Then I used tcpdump -nn -e icmp to see the mac addresses on the pings, and the destination of the pings was 00:00:00:00:00:00

This was true for pings being sent from the mik at 184.1 to the desktop at 184.105. When the pings to 184.105 were started no arp request
was made for 184.105, the pings were simply sent by 184.1, got to 184.105 and were dropped, probably as the mac was wrong!

I cleared the arp entry for 184.105 on the 184.1 router and everything started working again. I then cleared the whole table, and things seem to be normal.

This is what I see going. Every IP on all public subnets are being arp requested repeatedly and continuously, I can see these on every machine on that same broadcast domain. People say this is because someone or something is scanning the subnet, but this simply is not true as torch, packet sniffer and tcpdump amply testify. As the arp answers do not come back on the unused IP's, the arp table gets 00 for the mac.

Now BEFORE I cleared the table totally, new pings to unused IP's did not generate an arp request, and the ping was sent out anyhow on the interface with mac 00! This is nuts.

After I cleared the table, even with all unused IP's again set to 00 because of the auto arping of the entire subnet, the instant I pinged the 184.105 address an arp request WAS sent out, and if the IP was now used, the 00 in the arp table was immediately replaced by a correct MAC address.

So it would seem that the proper sequence is if an incoming packet to a desired IP has 00 in the arp table, do the arp again, and if it responds put
the correct mac in the table and send the ping. Thus time out is not important, as it really only applies in theory if there are no incoming packets to the IP, whether it was available or not.

Homer

We had similar problem today. RB3011 with 6.39.1.

A Laptop connects to APs, and get 172.16.1.19 or 172.16.1.20 from the DHCP server. The client had arping to 172.16.0.1, but didn't have ping, and the Mikrotik had also arping to the client with no ping (firewalls were down). At that moment I saw that my arping returns different MAC address compared to the ARP records, but it didn't ring no bells to me.

The problem was resolved with giving the laptop static IP address 172.16.1.21.

After few hours I had enough time to go deep into the problem, and I saw, that there were ARP records for those 2 IPs, I searched all L2 Switches and no one had record on any port for those 2 MACs. I tried to ping them, arping them with no result. After that I disabled and enabled the bridge (which eventually cleared the ARP records). Within the LOG I see that those MAC addresses requested DHCP and the server gave them .11 and .18, I know that those mac addresses belong to technicians on site, so they aren't some rouge devices.

This 00:00:00:00:00:00 in the ARP cache is a red herring - in fact, it's a useful indication of something else being wrong - that being that the ARP process is not working for certain hosts sometimes.

I've had arp-related issues from time to time in my career.
There's the classic case where you replace a device, and the new one has the same config but a different MAC address. Some other hosts on the network may take a while to clear their ARP caches before they learn that the MAC for the device's IP address changed. In that interim time period, things are "broken." Let's call this the "arp cache issue"

When having the "arp cache issue" you will have the wrong MAC address in your ARP table. (it will be flagged DC). The fix is to just remove the incorrect entry and let the device send ARP requests again. This will quickly cause the Mikrotik to learn the new MAC address of some IP that changed hosts. (of course, in windows, the equivalent dos command is arp -d x.x.x.x).

When it's not an "arp cache issue", it's almost always a lower-layer issue either at layer2 or layer1. I.e. the Ethernet switches were borking something, or there was a dirty link that wasn't properly transmitting data. Etc.

All of the complaints in this thread don't actually match the "arp cache issue." This is because the incomplete ARP indication (all-zeros MAC address) does not work the same way a completed ARP cache entry does. Basically, the all-zeroes MAC in ARP table is there merely as a convenience for the router administrator. It's there so you can see that the Mikrotik is in fact trying to ARP for some IP, and is not getting any reply. Meanwhile, the Mikrotik is still attempting to ARP for that IP. This indication is useful to you because you now know that the host you can't ping has not successfully replied to an ARP request from the Mikrotik.

I conclusively tested this just now with a Mikrotik and wireshark. I started a ping to an IP address known to not exist on the local LAN. The Mikrotik put the incomplete ARP entry into the arp table (all zeros for the MAC address) and started scrolling "ping timed out" messages. Meanwhile, wireshark was scrolling up ARP request frames from the Mikrotik's MAC address. Thus, the Mikrotik continues to send ARPs for incomplete entries. Only completed entries suppress sending ARP packets.

If you have incomplete arps and you know that the IP address exists on your network, then it's time to dig deeper into your network - are the ARP request broadcasts actually making it all the way to the device-that-won't-arp? Is it sending replies? Are the replies actually reaching the Mikrotik? (If so, then I'd be quite puzzled at this point) Does the host actually think it has the IP address configured on its interface? (i.e. did the DHCP transaction complete and bind itself on both the server and the client?) Does changing the IP address to something else work? etc...

The all-zeroes thing is just there to tell you that there was an ARP sent out for such-and-such an IP address, but no reply was heard.

Our problem was a little different, we had the wrong MAC, not only 00s.
So to my understanding (without actual test with wireshark) - the Client was sending packet to the Gateway, but the Gateway send back answer with the wrong MAC. The proper behavior should be to send back packet with dst mac, the src mac of the request, EXCEPT when there is a static arp entry for this src ip. This will resolve our issue, and also the 00s issue.

I played around a bit and found that if I changed the MAC address of a device already in the ARP cache of the Mikrotik router, this is what happens:

Device and Mikrotik have each other in ARP cache.
Pings just happen - no ARP on the wire.
I change the MAC of the device and ping the Mikrotik (no IP change, just MAC)
Mikrotik replies to the ping using the new src MAC
Mikrotik sends ARP request for the IP
Device replies with its new MAC
MAC address updates in table on the Mikrotik.

The timing of these events could be off due to the fact that I'm running all of this in GNS3, but I don't see how Wireshark would receive these packets out of order.

Interestingly, the Cisco sends gratuitous ARP requests whenever I change the MAC address. (to make sure there's no IP duplication I suppose....)
Just this is enough to trigger the Mikrotik to update its ARP cache to the new MAC. (the Mikrotik doesn't even send its own ARP request out in this case)

Nobody is arguing that this should be the normal behaviour of a router:

- Try to connect with IP of an unexisting device - ARP should be populated with 00s mac
- Connect a device with this IP address - on the first packet the Mikrotik hears a packet with src mac the mac (for this example: 11:11:11:11:11:11) therefore it populates the ARP with the new record.

The problem with other users on this forum is that even when there is a new device with mac 11:11:11:11:11:11 the ARP table record is not updated, and the Mikrotik is sending packets with dst mac 00s. Our problem was almost the same, except that the ARP wasn't populated with 00s, but with real mac address of some device.

The problem should be with the code of Mikrotik which for some reason in not so often occasions doesn't want to rewrite the record with the mac address of the src mac of the packet it receives. This sounds like a bug to me.

I'm not sure how we could reproduce is, if I see it again I'll make more investigations.

Interestingly, the Cisco sends gratuitous ARP requests whenever I change the MAC address. (to make sure there's no IP duplication I suppose....)
Just this is enough to trigger the Mikrotik to update its ARP cache to the new MAC. (the Mikrotik doesn't even send its own ARP request out in this case)

I don't know if Cisco maybe has fixed a longstanding problem in recent versions... but in Cisco the issue always was:
- the default ARP timeout was very very long (I think 6 hours)
- the router would not learn remote MAC addresses from incoming ARP requests, only from replies to its own outgoing requests.

Net result: when you replace a device connected to a network with a Cisco router, and assign the new device (with new MAC) the IP address of the old device, it would just sit there like a dead duck until Cisco liked to timeout their ARP entry and re-issue the ARP request. Which in default config could take hours.

The problem with other users on this forum is that even when there is a new device with mac 11:11:11:11:11:11 the ARP table record is not updated, and the Mikrotik is sending packets with dst mac 00s. Our problem was almost the same, except that the ARP wasn't populated with 00s, but with real mac address of some device.

The problem should be with the code of Mikrotik which for some reason in not so often occasions doesn't want to rewrite the record with the mac address of the src mac of the packet it receives. This sounds like a bug to me.

I'm not sure how we could reproduce is, if I see it again I'll make more investigations.

I can guarantee you that the Mikrotik is not sending frames on the wire with a destination MAC address of all zeroes. The all-zeroes MAC in the ARP table is an indication that the ARP process is failing for one reason or another. It could be something in the Mikrotik, it could be something on the host, it could be something on the network in between the two. But it's not that the Mikrotik thinks that the MAC is all zeroes and using that as the destination layer2 address.

I don't know if Cisco maybe has fixed a longstanding problem in recent versions... but in Cisco the issue always was:
- the default ARP timeout was very very long (I think 6 hours)
- the router would not learn remote MAC addresses from incoming ARP requests, only from replies to its own outgoing requests.

In my experience, the opposite has been true for the most part. I've found Cisco routers tend to be very quick to update their ARP data, despite cache times. It seems to me that the Routers update their ARP cache whenever a new packet arrives from the same IP but with a different MAC address. I've seen them create ARP entries across links that were failing to transmit data in one direction- meaning that it was impossible for a complete ARP request / reply path to complete. I have run into the occasion where clearing ARP entries on a router would fix things, but those tended to be in cases where I had changed the ethernet topology and the switches in the middle hadn't updated their CAM tables with the correct ports.

In my experience, the opposite has been true for the most part. I've found Cisco routers tend to be very quick to update their ARP data, despite cache times. It seems to me that the Routers update their ARP cache whenever a new packet arrives from the same IP but with a different MAC address. I've seen them create ARP entries across links that were failing to transmit data in one direction- meaning that it was impossible for a complete ARP request / reply path to complete. I have run into the occasion where clearing ARP entries on a router would fix things, but those tended to be in cases where I had changed the ethernet topology and the switches in the middle hadn't updated their CAM tables with the correct ports.

Well, not my experience... waiting for Cisco ARP table timeout has been very common for me when we still used Cisco (IOS-based routers).
Probably it is different in switching products. Forwarding tables update much quicker and ARP tables are usually flushed on link down/up.
However, the common case of having some system behind a switch and then connected to the router does not cover this: the router does not see a link down/up when the device is swapped.
I think no router will update the ARP table based solely on IP packets, but it is quite common to create an ARP entry on incoming ARP request. (be prepared for the reply)
However, UPDATING an ARP entry based on incoming ARP request is not always done (certainly not by Cisco IOS in my experience), probably because that would make it even easier to tap traffic on a LAN.

Hey ARP is a bandwidth hog on my partial T1s (E1s for my Europeans)! It also consume CPU cycles! Who ever moves a computer anyways they're huge. Oh wait I flashed back to 1992 there for a minute! That's where the long ARP cache timer came from. It can wreak havoc if you're doing something that doesn't GARP to force upstream devices to update their ARP cache. I commonly hit this when I replace firewalls or routers behind an ISP managed Cisco device.

I may or may not have "reseated" the power plug on an ATT layer 3 switch or two in my day over trying to explain to their techs why I need the ARP cache cleared and then how to do it.

I may or may not have "reseated" the power plug on an ATT layer 3 switch or two in my day over trying to explain to their techs why I need the ARP cache cleared and then how to do it.

I may or may not have spit coffee all over my keyboard while reading this post.