So.... All my staff traffic is on 10 subnet, all guest traffic on 192 subnet. So I am getting this occasional firewall logs for address 192.168.62.185. This is not even in my DHCP pool, ARP, or anywhere that I can find on my network. The machines on the 10 subnet in the photo are in the same office on the vlan30. The natted addresses that it shows they are all going out to are all Microsoft addresses.

So what gives? I can't figure out what and why I am seeing this? It is several times throughout the day and night. Always same source mac, always same 192.168.62.185 address, always same machines on the 10 subnett. WTF?

Maybe it's just service traffic to Microsoft servers, I dunno.

Maybe someone from staff has a second fixed IP address set? The source mac is rather strange, as it belongs to ARRIS Group which is a cable modem manufacturer. Maybe they have some auto-aliased internal IP in place.

Maybe someone from staff has a second fixed IP address set? The source mac is rather strange, as it belongs to ARRIS Group which is a cable modem manufacturer. Maybe they have some auto-aliased internal IP in place.

OK well that makes sense for the source MAC and I should have looked that MAC up to see the manufacturer..... But since this firewall rule is on my WAN port the source mac is likely my cable modem. Maybe that will help someone help me figure out why it is doing what it is doing.

It means TTL reached 0 during transit, look for routing loops, etc

It means TTL reached 0 during transit, look for routing loops, etc

Where would I start to look for routing loops? I dont have anything in log files that would indicate routing loop. Is there certain log files I can turn on to show this?

This is caused by a combination of bad ISPs that don't do BCP38 and bad routers that don't NAT properly.

An outbound packet from your network goes across the internet to some host behind a poor quality NAT router. The host PC / network responds with an ICMP error (TTL exceeded, port unreachable or similar) but the NAT router fails to translate the returning ICMP packet as it doesn't consider it related to the inbound connection. It then sends the ICMP message out the WAN interface complete with the original source IP of the host PC, at which point it makes its way across the internet back to your router since the ISP didn't have any IP spoofing protection in place. Then you see an inbound packet with a private IP source and think WTF!

This is caused by a combination of bad ISPs that don't do BCP38 and bad routers that don't NAT properly.

An outbound packet from your network goes across the internet to some host behind a poor quality NAT router. The host PC / network responds with an ICMP error (TTL exceeded, port unreachable or similar) but the NAT router fails to translate the returning ICMP packet as it doesn't consider it related to the inbound connection. It then sends the ICMP message out the WAN interface complete with the original source IP of the host PC, at which point it makes its way across the internet back to your router since the ISP didn't have any IP spoofing protection in place. Then you see an inbound packet with a private IP source and think WTF!

So from what you are describing this does not seem like any kind of router loop problem as I don't have excessive LAN traffic, like almost none and no logs that indicate a loop. So the NAT issue you speak of is on the remote (internet) network, not ours?

That's correct, it's caused by a non-translated packet exiting from a remote NAT and making it across the internet with an invalid source IP. They're quite rare, but if you run a busy enough network / website you'll see quite a lot of them.

Some stats from one of my websites which filter these on INPUT:
192.168.0.0/16 is the preferred network for many consumer routers which explains why it has the highest count of packets.

That's correct, it's caused by a non-translated packet exiting from a remote NAT and making it across the internet with an invalid source IP. They're quite rare, but if you run a busy enough network / website you'll see quite a lot of them.

Some stats from one of my websites which filter these on INPUT:

Code: Select all

 pkts bytes target     prot opt in     out     source               destination
   46  3819 DROP       all  --  *      *       10.0.0.0/8           0.0.0.0/0
    0     0 DROP       all  --  *      *       172.16.0.0/12        0.0.0.0/0
  190  8704 DROP       all  --  *      *       192.168.0.0/16       0.0.0.0/0

192.168.0.0/16 is the preferred network for many consumer routers which explains why it has the highest count of packets.

Ok makes sense. We have a few hundred host machines on our network and as I look back through the logs I see this same 192.168.62.185 on a few different subnets and the NAT address from the internet varies but when I search these various addresses they all return to Microsoft. The interesting thing is the subnet that has the most log entries, the client machines attached to those log entries, are all using Microsoft Office 365 while everyone else in our organization for the most part uses the locally installed versions of Office. This seems to be too big of a coincidence to me. Thoughts?

Any thoughts @R1CH?

What is 192.168.62.185? I suspect it is the gateway for the device you posted the logs for?

You can see from the log screenshot posted traffic is coming from a Public IP, but your gateway is reporting this

I think I was a little too quick with my first assessment. After some more thought I believe this is actually closer to your network. Something in the outbound network path is generating the TTL exceeded messages with the wrong interface / IP address and these are injected back into the internet. You could try a traceroute to see if you can reproduce this, perhaps it only happens on certain routes (eg to Microsoft).

What is 192.168.62.185? I suspect it is the gateway for the device you posted the logs for?

You can see from the log screenshot posted traffic is coming from a Public IP, but your gateway is reporting this

No I can not find this address anywhere on my network, I only use 10 subnet, I do use 192 subnet for guests but this address isnt even in the range of addresses that I use. No sign of this address in my ARP table either.

Is it in IP Routes?

Is it in IP Routes?

No not at all, craziest thing! I have seen it happen on four of our machines on our 10 subnet, three of them in the same building on the same VLAN and the other in a different building on a different VLAN. Same 192 address and MAC every time. Always to Microsoft addresses. So weird.

I would then guess the next step is for a diagram of the network and current config of your router

I would then guess the next step is for a diagram of the network and current config of your router

One more thought..... We have a static IP from our cable company for internet. Cable modem plugs into router WAN port and is configured for that static address. I went and plugged my laptop directly in to one of the other ports on the cable modem and it handed my laptop a 192.168.0.2 address. I wonder if this might have something to do with it? If cable modem is nating or something and so is my router? Just a thought. Perhaps the 192 address in my logs is from something on the cable company side?

On my Mikrotik, I would think even if there was a device that was outbound with 192.168.62.185.

So...... check this out. I think we are getting closer. The first traceroute is from within my network. Isnt that hitting my cable modem? The second traceroute is plugged directly into the cable modem and seems to just time out. Thoughts?

Don't give up yet @CZFan and @R1CH

Sorry bud, landed a couple of urgent jobs / projects, will be offline for a bit to focus on that

Do you have VMPlayer, WMWorkstation, VirtualBox etc. installed on any computer in your LAN? These programs create virtual interfaces and assign them "local networks pools" addresses and offer bridging with real interface so you can see packets originating from these virtal interfaces leaking to your LAN.

Do you have VMPlayer, WMWorkstation, VirtualBox etc. installed on any computer in your LAN? These programs create virtual interfaces and assign them "local networks pools" addresses and offer bridging with real interface so you can see packets originating from these virtal interfaces leaking to your LAN.

I thought of that earlier because I do on my particular laptop and disabled those interfaces. I don't believe it exists on the particular LAN segments that this is happening on but will look further into it just in case.

Quick update..... So I ran a traceroute on my Mikrotik to a number of different sites. Take a look..... the offending 192.168.62.185 address is number three on every traceroute. This has to be part of my cable company internet stack. I am just trying to figure out if it is a problem on their end or mine lol.