Hi all, Ihave a question about ARP:
---background----
At one of our remote offices we recently elected to have an alarm system installed. We have an RB2011 with dual wan running there. The alarm, which apparently is one of those Christopher Columbus brought with him back in 1493, is supposed to be internet monitored. Because of scheduling issues I couldn't be present for the installation of the system but prepped all the network cabling beforehand so the alarm tech wouldn't have an excuse to lay his greasy paws on my rackmounted equipment.
Long story short, the guy says there's a problem with our networking gear.
The networking cable is fine (triple checked by both of us) but the alarm constantly renegotiates ethernet speed. When forced to 10 mbit half duplex, it maintains a stable link but it won't connect to their server. I told him that this can't be correct (who in the world still produces 10Base-T half duplex gear in 2019!?!?) so he took the equiment back to their shop. They tested it there and contacted the manufacturer and it alledgedly all checked out. As a final test measure he brought back a cheapo "router" with him, plugged the alarm in, and that into our network. Still no connection.
----current status----
Our routerboard gets bombarded with DHCP requests from the alarm (one every 2-3 minutes). This tells me that his "router" isn't routing; merely acting as a switch. I'm also guessing that since the "router" isn't forced to 10 half duplex it's constantly renegotiating, hence the requests. In any case, the alarm is taking the IP and is pingable. Lease time is set to 72 hours.
Now, I'm running the sniffer on the routerboard, streaming to my PC over the VPN and I'm seeing two things. DHCP requests (no surprise there) and ARP requests for the alarm server at 181.XX.XX.XX (I'm guessing the IP is hardcoded into it) but no reply from the router.
Every other piece of hardware on the network (servers, smartphones, laptops, PCs, weather station, ip cameras, radios, swtiches, access points, etc) work perfectly.
----question----
I've always taken ARP for granted so I'm not very knowledgeable on the inner workings of it. My understanding is that either:
A: Host sends out "who has 181.x.x.x? ARP request". If no reply, host sends it to the gateway.
B: Host sends out "who has 181.x.x.x? ARP request", router replies "I know the way, send it to me".
So why is neither of those happening? Could it be that it's taking the IP from DHCP but not the gateway? Anyone ever seen such a thing?
Re: ARP/DHCP issue
You're a bit off on ARP.
On an Ethernet network, every device has a mac address. When packets get sent out over Ethernet, they are actually routed only by their mac address. Not IP address.
Since your PC will connect to remote devices by IP, then it needs to find out who on the network has an IP assigned to it. So it sends out a broadcast packet to all nodes on the Ethernet. Who has IP 192.168.88.100? The node with that IP assigned will respond. The sender now knows it's mac address and will route it via mac-address. A Windows PC will remember that mac/ip binding for about 5 minutes.
Your PC will use it's IP and Subnet to determine if the destination is on the same layer 2 network or not. If the destination IP is outside of the subnet, it'll send the packet to the gateway. Once again, the PC needs to know the mac address of your gateway/router. So yet again, it sends out an ARP request (broadcast), who has <Gateway IP>? Your gateway will respond and the sender sends the packet via mac-address to the gateway.
The packet structure on an Ethernet looks like this: <mac-address><Dest IP><Source IP><Proto>etc.. When your router receives the packet, it'll then route based on IP.
If you're sending between two devices, not only does your PC need to know the mac address of it's destination, each switch (layer 2 device) in between needs to know and will track which port the mac-address is on. When a PC makes the arp request, each node may update it's mac-address table when the arp response passes through the switch. Mac entries in a table typically expire after around 10 minutes in a switch, or when the port is unplugged. In a case where your PC knows the mac address of the destination IP and the Switch does not, then the switch may hold onto the packet while it does it's own ARP lookup.
On an Ethernet network, every device has a mac address. When packets get sent out over Ethernet, they are actually routed only by their mac address. Not IP address.
Since your PC will connect to remote devices by IP, then it needs to find out who on the network has an IP assigned to it. So it sends out a broadcast packet to all nodes on the Ethernet. Who has IP 192.168.88.100? The node with that IP assigned will respond. The sender now knows it's mac address and will route it via mac-address. A Windows PC will remember that mac/ip binding for about 5 minutes.
Your PC will use it's IP and Subnet to determine if the destination is on the same layer 2 network or not. If the destination IP is outside of the subnet, it'll send the packet to the gateway. Once again, the PC needs to know the mac address of your gateway/router. So yet again, it sends out an ARP request (broadcast), who has <Gateway IP>? Your gateway will respond and the sender sends the packet via mac-address to the gateway.
The packet structure on an Ethernet looks like this: <mac-address><Dest IP><Source IP><Proto>etc.. When your router receives the packet, it'll then route based on IP.
If you're sending between two devices, not only does your PC need to know the mac address of it's destination, each switch (layer 2 device) in between needs to know and will track which port the mac-address is on. When a PC makes the arp request, each node may update it's mac-address table when the arp response passes through the switch. Mac entries in a table typically expire after around 10 minutes in a switch, or when the port is unplugged. In a case where your PC knows the mac address of the destination IP and the Switch does not, then the switch may hold onto the packet while it does it's own ARP lookup.
Re: ARP/DHCP issue
If the Alarm system has an IP statically set and it's not on the same subnet as statically set in the alarm system, then the alarm system will do ARP requests for the gateway that's statically set in the Alarm system. Since no device on your network will have that IP, you will only see ARP requests and no responses.
Your alarm system may be detecting brief outages and could explain the frequent DHCP requests and re-nogtiation of link speed. You should also see 4 packets during DHCP. A "discover" packet asking who is a dhcp server, a request for IP (and possibly asking for an IP, usually one it had recently), an offer made by the DHCP server (possibly offering the requested one) and then an acknowledgement by the Alarm system. Does it happen?
Seeing 181.xx.xx.xx arp requests coming from your alarm system while at the same time seeing DHCP requests seems suspect.
Your alarm system may be detecting brief outages and could explain the frequent DHCP requests and re-nogtiation of link speed. You should also see 4 packets during DHCP. A "discover" packet asking who is a dhcp server, a request for IP (and possibly asking for an IP, usually one it had recently), an offer made by the DHCP server (possibly offering the requested one) and then an acknowledgement by the Alarm system. Does it happen?
Seeing 181.xx.xx.xx arp requests coming from your alarm system while at the same time seeing DHCP requests seems suspect.
Re: ARP/DHCP issue
Thanks for replying. I know how layer 2 works for the most part. The specific part I don't know is what happens when the IP is NOT on the network. I.e. there's no MAC address with that IP so no-one is going to reply "that address is mine".You're a bit off on ARP.
On an Ethernet network, every device has a mac address. When packets get sent out over Ethernet, they are actually routed only by their mac address. Not IP address.
Since your PC will connect to remote devices by IP, then it needs to find out who on the network has an IP assigned to it. So it sends out a broadcast packet to all nodes on the Ethernet. Who has IP 192.168.88.100? The node with that IP assigned will respond. The sender now knows it's mac address and will route it via mac-address. A Windows PC will remember that mac/ip binding for about 5 minutes.
Your PC will use it's IP and Subnet to determine if the destination is on the same layer 2 network or not. If the destination IP is outside of the subnet, it'll send the packet to the gateway. Once again, the PC needs to know the mac address of your gateway/router. So yet again, it sends out an ARP request (broadcast), who has <Gateway IP>? Your gateway will respond and the sender sends the packet via mac-address to the gateway.
The packet structure on an Ethernet looks like this: <mac-address><Dest IP><Source IP><Proto>etc.. When your router receives the packet, it'll then route based on IP.
If you're sending between two devices, not only does your PC need to know the mac address of it's destination, each switch (layer 2 device) in between needs to know and will track which port the mac-address is on. When a PC makes the arp request, each node may update it's mac-address table when the arp response passes through the switch. Mac entries in a table typically expire after around 10 minutes in a switch, or when the port is unplugged. In a case where your PC knows the mac address of the destination IP and the Switch does not, then the switch may hold onto the packet while it does it's own ARP lookup.
Or let me rephrase the question. When a host behind a NAT wants to reach an Internet address, how does that work? Does it do an ARP request or does it send the packets straight to the gateway since it already assumes the address not to be on the same L2 network?
Re: ARP/DHCP issue
The only hardcoded IP is the server IP (i.e. the remote Internet address it needs to contact for monitoring). The local IP configuration is set to DHCP. At every DHCP exchange (again, one about every 2-3 minutes) I'm seeing 2 Discover packets from the Alarm, 1 Offer packet from the Router, 2 Request packets from the Alarm and 1 ACK packet from the router. I haven't checked it against other RouterOS DHCP handouts but this looks pretty standard to me?If the Alarm system has an IP statically set and it's not on the same subnet as statically set in the alarm system, then the alarm system will do ARP requests for the gateway that's statically set in the Alarm system. Since no device on your network will have that IP, you will only see ARP requests and no responses.
Your alarm system may be detecting brief outages and could explain the frequent DHCP requests and re-nogtiation of link speed. You should also see 4 packets during DHCP. A "discover" packet asking who is a dhcp server, a request for IP (and possibly asking for an IP, usually one it had recently), an offer made by the DHCP server (possibly offering the requested one) and then an acknowledgement by the Alarm system. Does it happen?
Seeing 181.xx.xx.xx arp requests coming from your alarm system while at the same time seeing DHCP requests seems suspect.
Anyways, what bugs me is that both odd behaviours (ARP requests for an Internet address and DHCP requests every 2 minutes after accepting the offer and replying to ping on the assigned Ip) seem to be related somehow. But I fail to see the connection.
edit: didn't realize both posts came from you Van9018, thanks again for the help
Re: ARP/DHCP issue
- When a host wants to send a packet to an internet address, it will send the packet directly to the gateway. It will NOT do an arp lookup for that internet address.
- You shouldn't see two DHCP discovers and two requests during a DHCP transaction, but not a big deal. Discover, Offer, Request, Ack.
- If DHCP requests are consistent every 2-3 minutes, then it's likely that was an easy and lazy way to design the DHCP client on the system. For example, what happens if the alarm system comes up before the DHCP Server? Or the lease expires? Cable unplugged and plugged back in? Not a big deal, it requests a new IP every 2 minutes and will resolve itself.
Have a spare Mikrotik? Or split off an interface. Set the interface's IP to that of the remote IP. Then the ARP request will resolve, and the alarm system will then send a TCP packet. If that happens, then there is a 2nd IP on the alarm system's LAN interface. Maybe a QA person at the monitoring company used SSH to set a static IP that is on the same L2 network as the monitoring server. They would test a unit before selling it, and maybe he forgot to remove the IP. The user interface would only show the first IP configuration.
And maybe the installers don't know squat. It should be "Plug and Play", they see a green light, so it must be your networking gear. I doubt they contacted the MFG.
- You shouldn't see two DHCP discovers and two requests during a DHCP transaction, but not a big deal. Discover, Offer, Request, Ack.
- If DHCP requests are consistent every 2-3 minutes, then it's likely that was an easy and lazy way to design the DHCP client on the system. For example, what happens if the alarm system comes up before the DHCP Server? Or the lease expires? Cable unplugged and plugged back in? Not a big deal, it requests a new IP every 2 minutes and will resolve itself.
Have a spare Mikrotik? Or split off an interface. Set the interface's IP to that of the remote IP. Then the ARP request will resolve, and the alarm system will then send a TCP packet. If that happens, then there is a 2nd IP on the alarm system's LAN interface. Maybe a QA person at the monitoring company used SSH to set a static IP that is on the same L2 network as the monitoring server. They would test a unit before selling it, and maybe he forgot to remove the IP. The user interface would only show the first IP configuration.
And maybe the installers don't know squat. It should be "Plug and Play", they see a green light, so it must be your networking gear. I doubt they contacted the MFG.
Re: ARP/DHCP issue
There are two kind of routes in the IP protocol:When a host behind a NAT wants to reach an Internet address, how does that work? Does it do an ARP request or does it send the packets straight to the gateway since it already assumes the address not to be on the same L2 network?
- interface routes (or connected routes, that Mikrotik marks with a C when printing)
- gateway routes (that in classic route command are marked with a G)
In gateway routes the traffic is sent to the gateway host, even if it is addressed to other ones. So, if you say
Code: Select all
/ip route add dst-address=192.168.88.0/24 gateway=ether1
to the receiver. If instead you say
Code: Select all
/ip route add distance=1 dst-address=0.0.0.0/0 gateway=192.168.88.1
as 0.0.0.0/0 has the lowest priority, to 192.168.88.1. The machine, then, when trying to send something to,
say, 215.23.1.78, would prepare an IP packet with this destination address, and then look for the ARP
L2 address of 192.168.88.1 and send to it.
Re: ARP/DHCP issue
Awesome. Yeah I've put some wireshark time in yesterday evening and realized that there're no ARP requests for internet traffic on other hosts. Should've done that in the first place lol. But thanks, that answered all my questions.- When a host wants to send a packet to an internet address, it will send the packet directly to the gateway. It will NOT do an arp lookup for that internet address.
- You shouldn't see two DHCP discovers and two requests during a DHCP transaction, but not a big deal. Discover, Offer, Request, Ack.
- If DHCP requests are consistent every 2-3 minutes, then it's likely that was an easy and lazy way to design the DHCP client on the system. For example, what happens if the alarm system comes up before the DHCP Server? Or the lease expires? Cable unplugged and plugged back in? Not a big deal, it requests a new IP every 2 minutes and will resolve itself.
Setting the remote IP and monitor traffic is a great idea! I'll try that and see what happens. They did say there's no network management interface. The device is set up with a "special cable and software". I assume this to mean a serial cable with a propietary plug and some gui software for that serial connection. In any case the tech said there're basically only two options as far as the network interface is concerned: remote server IP and local network configuration with a setting of "DHCP or static".Have a spare Mikrotik? Or split off an interface. Set the interface's IP to that of the remote IP. Then the ARP request will resolve, and the alarm system will then send a TCP packet. If that happens, then there is a 2nd IP on the alarm system's LAN interface. Maybe a QA person at the monitoring company used SSH to set a static IP that is on the same L2 network as the monitoring server. They would test a unit before selling it, and maybe he forgot to remove the IP. The user interface would only show the first IP configuration.
My thoughts exactly... =/ To quote his exact words "there's a light on the side that's red with no connection, amber when there's a connection but the remote server is unreachable and green when it is connected and it should just work".He didn't even know what TCP was when I asked if I was looking for tcp traffic from the box.And maybe the installers don't know squat. It should be "Plug and Play", they see a green light, so it must be your networking gear. I doubt they contacted the MFG.
Anyways, I'm scheduling a trip to the site next week. Probably monday. 200 miles round trip, could be worse lol.
Re: ARP/DHCP issue
Thanks nostromog, crystal clear. Same subnet, ARP. 0.0.0.0/0 traffic, default gateway, no ARP.There are two kind of routes in the IP protocol:
The interface routes are where the ARP mechanism is used, to know which L2 host to address with packets.
- interface routes (or connected routes, that Mikrotik marks with a C when printing)
- gateway routes (that in classic route command are marked with a G)
In gateway routes the traffic is sent to the gateway host, even if it is addressed to other ones. So, if you say
you are telling the machine to use ARP to find out hosts in the given network, and send directly the packetCode: Select all/ip route add dst-address=192.168.88.0/24 gateway=ether1
to the receiver. If instead you say
you are telling the machine to address any packet that it does not know otherwise where to send,Code: Select all/ip route add distance=1 dst-address=0.0.0.0/0 gateway=192.168.88.1
as 0.0.0.0/0 has the lowest priority, to 192.168.88.1. The machine, then, when trying to send something to,
say, 215.23.1.78, would prepare an IP packet with this destination address, and then look for the ARP
L2 address of 192.168.88.1 and send to it.
Re: ARP/DHCP issue [SOLVED]
Well folks, I managed to crack it. For whatever reason the "designer" (if we can call the schmuck who designed this marvellous piece of... erm... tech, that) decided that always doing an ARP request, no matter what the destination IP address is, is a good idea. I'm guessing they did this to surpass any routing issues like cheapo routers not doing their thing for UDP non standard port traffic (because that's what it uses, udp port 8000something). Who knows.
Eventually it occured to me to switch ARP from "enabled" to "proxy-ARP" on the ethernet interface going to the device. Router replied with its own MAC as soon as the request came in and lo and behold, the darn thing sent all its traffic to the router. Connection was up and running in no-time. Saying that I wasn't pissed for not thinking of this from the get-go, would be a major understatement lol.
The DHCP issue was also resolved after the connection to their monitoring server went up. As Van9018 mentioned, this was probably put in as an extra measure of "robustness". I.e. "if it doesn't connect, keep asking dhcp for new leases, perhaps the 999th will work".... *shakehead* *sigh*
Thanks again for all the replies!
Eventually it occured to me to switch ARP from "enabled" to "proxy-ARP" on the ethernet interface going to the device. Router replied with its own MAC as soon as the request came in and lo and behold, the darn thing sent all its traffic to the router. Connection was up and running in no-time. Saying that I wasn't pissed for not thinking of this from the get-go, would be a major understatement lol.
The DHCP issue was also resolved after the connection to their monitoring server went up. As Van9018 mentioned, this was probably put in as an extra measure of "robustness". I.e. "if it doesn't connect, keep asking dhcp for new leases, perhaps the 999th will work".... *shakehead* *sigh*
Thanks again for all the replies!