Community discussions

MikroTik App
 
User avatar
smyers119
Member Candidate
Member Candidate
Topic Author
Posts: 232
Joined: Sat Feb 27, 2021 8:16 pm
Location: USA

[ZEROTIER BUG]ARPING for public IP on LAN

Sun Dec 05, 2021 3:16 pm

I happened to be running wireshark and noticed my RB4011 is sending ARP's for public IP's on my LAN. I confirmed I do not have proxy-arp running.
Version:
routerboard: yes
model: RB4011iGS+
serial-number: 
firmware-type: al2
factory-firmware: 6.45.8
current-firmware: 6.47.2
upgrade-firmware: 7.1
ARPS:
arpmikrotik.PNG
UPDATE:
This appears to be zerotier related.
You do not have the required permissions to view the files attached to this post.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3253
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: [ZEROTIER BUG]ARPING for public IP on LAN

Mon Dec 06, 2021 11:19 pm

Might want to also upgrade the firmware. That isn't at 7.1 from your picture.

Not saying related, just noticed...

I did notice seem ZeroTier seems aggressive, but hadn't studied it. So not sure what "normal" would like for it ;).

ARP may one way it figures out it's paths, it's protocol is bit complex. Nothing should respond, right? It is Layer 2 tunnel, so could be seeing if there is a "shortcut" using ARP to find a host locally. But I can see how that might introduce whole lot of trouble...
 
User avatar
smyers119
Member Candidate
Member Candidate
Topic Author
Posts: 232
Joined: Sat Feb 27, 2021 8:16 pm
Location: USA

Re: [ZEROTIER BUG]ARPING for public IP on LAN

Tue Dec 07, 2021 12:03 am

Might want to also upgrade the firmware. That isn't at 7.1 from your picture.

Not saying related, just noticed...

I did notice seem ZeroTier seems aggressive, but hadn't studied it. So not sure what "normal" would like for it ;).

ARP may one way it figures out it's paths, it's protocol is bit complex. Nothing should respond, right? It is Layer 2 tunnel, so could be seeing if there is a "shortcut" using ARP to find a host locally. But I can see how that might introduce whole lot of trouble...
I realize my first post was not clear as I was still discovering the problem. So here is some more information:

The IP's that the router is ARPing for is PLANET and LEAF's from zerotier.

The packet capture is from my LAN, as seen from my home computer. zerotier terminates on the router as a seperate layer 3 interface, so this packet capture would never see zerotier interface traffic.
  • I updated firmware and confirmed the problem still exists.
  • Arp is a layer 2 protocol, and not complex at all. It should not be ARPing for a ip address that is not in the same layer 2 subnet.
  • ARP does not help a router with path resolution. (routing is a layer 3 function)
routerboard: yes
model: RB4011iGS+
serial-number: 
firmware-type: al2
factory-firmware: 6.45.8
current-firmware: 7.1
upgrade-firmware: 7.1
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3253
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: [ZEROTIER BUG]ARPING for public IP on LAN

Tue Dec 07, 2021 2:58 am

Yeah sorry didn't mean to imply necessarily ARP is complicated... Now how ZT uses it and/or integrates into the Mikrotik maybe. ;)

Is this causes M/R/STP to trip or causing some issue, other than ARPs in traces?
Is ZT interface bridged in ANY way on the Mikrotik?
Have any config or diagram?


See an ARP with a different IP isn't necessary "wrong" from L3 POV – multihoming. It's only wrong from a ROS "packet flow"/policy prospective. And, how the ZeroTier package approaches discovery on the ROS is not document by Mikrotik & ZT only has a high-level overview of how it looks to establish it's tunnel paths.

Seems like some bug, but without docs hard to know. Just giving you ideas – I'm not sure how important ZeroTier in their bug list given it's a new V7 feature...
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: [ZEROTIER BUG]ARPING for public IP on LAN

Tue Dec 07, 2021 3:11 am

Hi,

I haven't tested for this with ZeroTier, but I know that outside of ZeroTier this type of issue can happen with regular RouterOS 6.x if you use an interface as the "gateway" instead of an IP in the case where an interface must be used. For instance you could use ether1 as a gateway for a connected route only, otherwise if the route is static or something else, an IP must be specified instead of just "ether1" as ethernet interfaces can have many IPs. If you accidentally add a static route that uses "ether1" as the gateway you can end up with the interface trying to do ARPs for internet addresses, and then the ARP table can fill up.

See this thread: viewtopic.php?t=92767
 
User avatar
smyers119
Member Candidate
Member Candidate
Topic Author
Posts: 232
Joined: Sat Feb 27, 2021 8:16 pm
Location: USA

Re: [ZEROTIER BUG]ARPING for public IP on LAN

Tue Dec 07, 2021 4:01 am

Hi,

I haven't tested for this with ZeroTier, but I know that outside of ZeroTier this type of issue can happen with regular RouterOS 6.x if you use an interface as the "gateway" instead of an IP in the case where an interface must be used. For instance you could use ether1 as a gateway for a connected route only, otherwise if the route is static or something else, an IP must be specified instead of just "ether1" as ethernet interfaces can have many IPs. If you accidentally add a static route that uses "ether1" as the gateway you can end up with the interface trying to do ARPs for internet addresses, and then the ARP table can fill up.

See this thread: viewtopic.php?t=92767
Thanks for the reply.

You are correct when you use a interface route the route becomes active when the interface is up and it considers every destination address as directly connected to that interface, so it will ARP for the destination's filling up your table. But this is not the scenario I am explaining above. The router is sending public IP ARP's out the local interface with a /24 RFC1918 subnet. I have not checked my WAN interface but I guess I should.

Here is the routing table
[[REDACT]@router1] > /ip/route/print brief
Flags: D - DYNAMIC; I, A - ACTIVE; c, o, d, v, y - COPY; H - HW-OFFLOADED
Columns: DST-ADDRESS, GATEWAY, DISTANCE
     DST-ADDRESS      GATEWAY               DISTANCE
D o  0.0.0.0/0        10.172.2.1%zerotier1       110
DAd  0.0.0.0/0        XXX.XXX.228.1                 1    (Public IP)
DAc  10.172.2.0/24    zerotier1                    0
DAo  10.172.10.0/24   10.172.2.1%zerotier1       110
DAc  10.172.255.1/32  loopback0                    0
DAo  10.172.255.2/32  10.172.2.1%zerotier1       110
DAc  69.248.228.0/23  eth1                         0
DIvH 192.168.2.0/24   172.17.2.5                   1
DAc  192.168.2.0/24   bri1                         0
 
User avatar
smyers119
Member Candidate
Member Candidate
Topic Author
Posts: 232
Joined: Sat Feb 27, 2021 8:16 pm
Location: USA

Re: [ZEROTIER BUG]ARPING for public IP on LAN

Tue Dec 07, 2021 4:08 am

My router appears to be acting appropriate on the WAN
[[REDACT]@router1] /tool/sniffer> packet/print detail
 0 time=33.477 num=1 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=FF:FF:FF:FF:FF:FF interface=eth1 protocol=arp size=60 cpu=0

 1 time=33.529 num=2 direction=tx src-mac=C4:AD:34:F8:D6:DF
   dst-mac=00:01:5C:92:AA:46 interface=eth1 protocol=arp size=42 cpu=3

 2 time=33.537 num=3 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=C4:AD:34:F8:D6:DF interface=eth1 protocol=arp size=60 cpu=0

 3 time=46.004 num=4 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=FF:FF:FF:FF:FF:FF interface=eth1 protocol=arp size=60 cpu=0

 4 time=66.935 num=5 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=FF:FF:FF:FF:FF:FF interface=eth1 protocol=arp size=60 cpu=0

 5 time=75.769 num=6 direction=tx src-mac=C4:AD:34:F8:D6:DF
   dst-mac=00:01:5C:92:AA:46 interface=eth1 protocol=arp size=42 cpu=1

 6 time=75.78 num=7 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=C4:AD:34:F8:D6:DF interface=eth1 protocol=arp size=60 cpu=0

 7 time=96.511 num=8 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=FF:FF:FF:FF:FF:FF interface=eth1 protocol=arp size=60 cpu=0

 8 time=118.019 num=9 direction=tx src-mac=C4:AD:34:F8:D6:DF
   dst-mac=00:01:5C:92:AA:46 interface=eth1 protocol=arp size=42 cpu=2

 9 time=118.029 num=10 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=C4:AD:34:F8:D6:DF interface=eth1 protocol=arp size=60 cpu=0

10 time=132.868 num=11 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=FF:FF:FF:FF:FF:FF interface=eth1 protocol=arp size=60 cpu=0

11 time=147.604 num=12 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=FF:FF:FF:FF:FF:FF interface=eth1 protocol=arp size=60 cpu=0

12 time=160.249 num=13 direction=tx src-mac=C4:AD:34:F8:D6:DF
   dst-mac=00:01:5C:92:AA:46 interface=eth1 protocol=arp size=42 cpu=3

13 time=160.256 num=14 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=C4:AD:34:F8:D6:DF interface=eth1 protocol=arp size=60 cpu=0

14 time=184.963 num=15 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=FF:FF:FF:FF:FF:FF interface=eth1 protocol=arp size=60 cpu=0

15 time=202.489 num=16 direction=tx src-mac=C4:AD:34:F8:D6:DF
   dst-mac=00:01:5C:92:AA:46 interface=eth1 protocol=arp size=42 cpu=0

16 time=202.496 num=17 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=C4:AD:34:F8:D6:DF interface=eth1 protocol=arp size=60 cpu=0

17 time=202.699 num=18 direction=rx src-mac=00:01:5C:92:AA:46
   dst-mac=FF:FF:FF:FF:FF:FF interface=eth1 protocol=arp size=60 cpu=0
 
User avatar
smyers119
Member Candidate
Member Candidate
Topic Author
Posts: 232
Joined: Sat Feb 27, 2021 8:16 pm
Location: USA

Re: [ZEROTIER BUG]ARPING for public IP on LAN

Tue Dec 07, 2021 4:43 am


See an ARP with a different IP isn't necessary "wrong" from L3 POV – multihoming. It's only wrong from a ROS "packet flow"/policy prospective. And, how the ZeroTier package approaches discovery on the ROS is not document by Mikrotik & ZT only has a high-level overview of how it looks to establish it's tunnel paths.
ARP only works between devices in the same subnet.

When device A with IP address A needs to send a packet to device B with IP address B, the first thing it does is consult its routing table to determine if IP address B belongs to a subnet it can directly reach through its network interface(s); if it does, then devices A uses ARP to map IP address B to a physical Ethernet address, and then sends an Ethernet frame to that address.

If the two IP Addresses are on different subnets, the device will follow a completely different logic: it will look in its routing table for a route to the destination network, and then it will send its packet to the appropriate router (or to its default gateway if no more specific route is present); in this scenario, ARP will be used to find the hardware address of the router.

So in conclusion if your seeing ARP's for a IP that is not on that subnet then IT IS ALWAYS WRONG.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3253
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: [ZEROTIER BUG]ARPING for public IP on LAN

Tue Dec 07, 2021 5:01 am

You should open a ticket with Mikrotik with the supout.rif, does seem like a bug. Not sure how folks can help if so.

All I was saying is you can have two L3 "subnets" can share a L2 "segment", you'd have ARP looking for two different IP subnets. Not saying it's good, but not entirely surprising. Likely harmless in the short term is what I more trying say.

While you view the WAN as special, ZeroTier may have access to all interfaces outbound, and using them all directly (and perhaps incorrectly).
 
User avatar
smyers119
Member Candidate
Member Candidate
Topic Author
Posts: 232
Joined: Sat Feb 27, 2021 8:16 pm
Location: USA

Re: [ZEROTIER BUG]ARPING for public IP on LAN

Tue Dec 07, 2021 5:09 am

You should open a ticket with Mikrotik with the supout.rif, does seem like a bug. Not sure how folks can help if so.
I am not clear on bug reporting etiquite in this community, but according to this post, The forum is the correct place for beta releases. Is 7.1 still considered beta?

viewtopic.php?t=152006
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3253
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: [ZEROTIER BUG]ARPING for public IP on LAN

Tue Dec 07, 2021 5:41 am

You should open a ticket with Mikrotik with the supout.rif, does seem like a bug. Not sure how folks can help if so.
I am not clear on bug reporting etiquite in this community, but according to this post, The forum is the correct place for beta releases. Is 7.1 still considered beta?

viewtopic.php?t=152006
V7.1 isn't beta is all I know. I believe it's at least release candidate, or released if you go by the changelog page. Hard question.

But yeah support@mikrotik.com has taken bugs for V7 for a while, with supout.rif & details like what you tried to fix it etc. – just the response time may not be what you'd like. Your are kinda trying to prove the forum there is a bug – if you thought it was config related, your specific config help folks double-check that. Just trying to save you time waiting... if bug, only MT can help, or if config issue, your better off trying more things to avoid/minimize the side effects of the non-standard ARP...

Maybe others seen this with ZeroTier, who knows.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3253
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: [ZEROTIER BUG]ARPING for public IP on LAN

Thu Dec 09, 2021 9:49 pm

So I am curious what Mikrotik has to say about your issue...

Since ZeroTier is likely a "routing process" with access "local-in" and "local-out" in the Mikrotik Packet Flow Diagram – it's IP interactions likely follow the Packet Flow Diagram.

But the ARP part is so simple question on how this intersects with Mikrotik/IETF/IEEE understanding's of ARP. Here is what the ZeroTier Protocol Whitepater says about ARP:


Special Handling of IPv4 ARP Broadcasts

IPv4 ARP is built on simple Ethernet broadcast and scales poorly on large or distributed networks. To improve ARP’s scalability ZeroTier generates a unique multicast group for each IPv4 address detected on its system and then transparently intercepts ARP queries and sends them only to the correct group. This converts ARP into effectively a unicast or narrow multicast protocol (like IPv6 NDP) and allows IPv4 ARP to work reliably across wide area networks without excess bandwidth consumption. A similar strategy is implemented under the hood by a number of enterprise switches and WiFi routers designed for deployment on extremely large LANs. This ARP emulation mode is transparent to the OS and application layers, but it does mean that packet sniffers will not see all ARP queries on a virtual network the way they typically can on smaller wired LANs.


This is what I was alluding to on OSI Layer2. While ZeroTier tunnel are at IP (Layer3), the interface is actually Ethernet to the Mikrotik. And as a process, it have low level access to all interfaces – even if Mikrotik enforces each interface being a separate thing at both Layer 2 and Layer 3. So ZeroTier can/may ignore some of Mikrotik's config/logic to implement it's protocol. While I agree your issue looks like a bug, it question of how much freedom does ROS allow ZeroTier to operate.

How these kinda of things are resolved by Mikrotik will be curious to watch, since ZeroTier doesn't so neatly fit into the packet flow diagram.

Who is online

Users browsing this forum: Ahrefs [Bot] and 22 guests