So… All my staff traffic is on 10 subnet, all guest traffic on 192 subnet. So I am getting this occasional firewall logs for address 192.168.62.185. This is not even in my DHCP pool, ARP, or anywhere that I can find on my network. The machines on the 10 subnet in the photo are in the same office on the vlan30. The natted addresses that it shows they are all going out to are all Microsoft addresses.
So what gives? I can’t figure out what and why I am seeing this? It is several times throughout the day and night. Always same source mac, always same 192.168.62.185 address, always same machines on the 10 subnett. WTF?
Maybe someone from staff has a second fixed IP address set? The source mac is rather strange, as it belongs to ARRIS Group which is a cable modem manufacturer. Maybe they have some auto-aliased internal IP in place.
OK well that makes sense for the source MAC and I should have looked that MAC up to see the manufacturer… But since this firewall rule is on my WAN port the source mac is likely my cable modem. Maybe that will help someone help me figure out why it is doing what it is doing.
Where would I start to look for routing loops? I dont have anything in log files that would indicate routing loop. Is there certain log files I can turn on to show this?
This is caused by a combination of bad ISPs that don’t do BCP38 and bad routers that don’t NAT properly.
An outbound packet from your network goes across the internet to some host behind a poor quality NAT router. The host PC / network responds with an ICMP error (TTL exceeded, port unreachable or similar) but the NAT router fails to translate the returning ICMP packet as it doesn’t consider it related to the inbound connection. It then sends the ICMP message out the WAN interface complete with the original source IP of the host PC, at which point it makes its way across the internet back to your router since the ISP didn’t have any IP spoofing protection in place. Then you see an inbound packet with a private IP source and think WTF!
So from what you are describing this does not seem like any kind of router loop problem as I don’t have excessive LAN traffic, like almost none and no logs that indicate a loop. So the NAT issue you speak of is on the remote (internet) network, not ours?
That’s correct, it’s caused by a non-translated packet exiting from a remote NAT and making it across the internet with an invalid source IP. They’re quite rare, but if you run a busy enough network / website you’ll see quite a lot of them.
Some stats from one of my websites which filter these on INPUT:
pkts bytes target prot opt in out source destination
46 3819 DROP all -- * * 10.0.0.0/8 0.0.0.0/0
0 0 DROP all -- * * 172.16.0.0/12 0.0.0.0/0
190 8704 DROP all -- * * 192.168.0.0/16 0.0.0.0/0
192.168.0.0/16 is the preferred network for many consumer routers which explains why it has the highest count of packets.
Ok makes sense. We have a few hundred host machines on our network and as I look back through the logs I see this same 192.168.62.185 on a few different subnets and the NAT address from the internet varies but when I search these various addresses they all return to Microsoft. The interesting thing is the subnet that has the most log entries, the client machines attached to those log entries, are all using Microsoft Office 365 while everyone else in our organization for the most part uses the locally installed versions of Office. This seems to be too big of a coincidence to me. Thoughts?
I think I was a little too quick with my first assessment. After some more thought I believe this is actually closer to your network. Something in the outbound network path is generating the TTL exceeded messages with the wrong interface / IP address and these are injected back into the internet. You could try a traceroute to see if you can reproduce this, perhaps it only happens on certain routes (eg to Microsoft).
No I can not find this address anywhere on my network, I only use 10 subnet, I do use 192 subnet for guests but this address isnt even in the range of addresses that I use. No sign of this address in my ARP table either.
No not at all, craziest thing! I have seen it happen on four of our machines on our 10 subnet, three of them in the same building on the same VLAN and the other in a different building on a different VLAN. Same 192 address and MAC every time. Always to Microsoft addresses. So weird.
One more thought… We have a static IP from our cable company for internet. Cable modem plugs into router WAN port and is configured for that static address. I went and plugged my laptop directly in to one of the other ports on the cable modem and it handed my laptop a 192.168.0.2 address. I wonder if this might have something to do with it? If cable modem is nating or something and so is my router? Just a thought. Perhaps the 192 address in my logs is from something on the cable company side?
On my Mikrotik, I would think even if there was a device that was outbound with 192.168.62.185.
So… check this out. I think we are getting closer. The first traceroute is from within my network. Isnt that hitting my cable modem? The second traceroute is plugged directly into the cable modem and seems to just time out. Thoughts?
