NAT stops working

mmrcela · August 26, 2011, 9:24am

Hi, guys!

I have a problem regarding NAT. This is the situation: We have a Mikrotik router (450) in datacenter and it’s responsible for connecting few servers (either normal ones, either virtualised) to the Internet. We have also got a range of public ip addresses from our ISP. Some of the servers have assigned public ip addresses, some have private ip addresses which are then translated with NAT. The problem arises with some NAT translations, not with all.

We got from our ISP this range of public addresses: x.x.x.144/28, which gives us the range of available addresses from x.x.x.145 to x.x.x.158. Now, I have assigned x.x.x.147 to one server (NAT involved), x.x.x.149 to another server (no NAT), x.x.x.150 to another server (no NAT), and x.x.x.151 to another server (NAT involved). Recently, I have added another server to our datacenter network, and it got the address x.x.x.152, which was translated with NAT to one private address. Only the needed ports were translated through NAT, and it worked fine for a few days (we tested the server to see if it will give us the desired functionality, and it worked as expected).

After a few days, the server just stopped doing its job. When I inspected the situation, I have found out that NAT rules that were working fine, have just stopped working (nothing would go through them). I tried to use another available public ip address from the subnet, but it still wouldn’t work. It would work, however, when I translated one of the other used public ip addresses towards the server I needed. So, when I would, for example, translate x.x.x.147 to the private ip address of the server that had x.x.x.152, it worked fine, but when I would translate x.x.x.153 (which is not used), then it wouldn’t work. After a lot of work to find out why this has happened, I had no other solution but to conclude that it is some sort of router’s problem. So, I backed up the configuration from the problematic (450) Mikrotik and restored it to another 750 Mikrotik. I connected the 750 just as was 450 connected, and everything started working immediately. Just when I thought that everything is solved, the 750 got the same problem after a few days as the previous, 450 Mikrotik.

Does anybody has any idea why this is happening and how to solve this?

JP_Wireless · August 26, 2011, 10:31am

was there a reboot of the router before the problem start? Is the 450 runing the same version of Router OS with 750? Did you you try restoring the config to the 450 and see if it work again?

mmrcela · August 26, 2011, 11:17am

no, there was no reboot of the router.

the version of 450 is 4.13, and version of 750 is 4.17.

I haven’t removed the configuration from the 450; I have just put that same configuration on 750. since I didn’t remove the configuration from the 450, I don’t see the point of restoring the configuration…

I forgot to mention one more thing. The 750 had problems of its own. For example, sometimes it wouldn’t route between different subnets, although they were all directly connected to the router. For example, if I had addresses 192.168.0.1/24, 192.168.1.1/24 and 192.168.2.1/24 on eth1, and let’s say one of my servers was on 192.168.0.0/24 network, but my computer was on 192.168.1.0/24 network, there were times when I couldn’t contact the server (sometimes I could). The only solution I could find was that I change the address of my computer to the 192.168.2.0/24 network (changing to an address in the same subnet would not work). But next time when my computer was again on the 192.168.1.0/24 network, it could access the server. There was no VLAN-s configured.

This is very strange to me because it seems that router can’t do its basic function - to connect different subnets. Or, to say it better, it can do it sometimes, but sometimes it cannot. How is it possible that router can’t connect two subnets that are directly connected t it?

The only reason why we’ve put this problematic 750 Mikrotik in datacenter was just to test if the configuration of 450 was correct. I am pretty sure that it’s not the configuration problem since NAT worked when we replaced the routers, and since it worked fine in the begining as well. What puzzles me is why is the exactly the same thing happening again, this time with the 750, although 750 never had the NAT problem, but it also had some arbitrary decisions when to do its job?

Currently, I can access the server I want, but only because I have put NAT rule of another server to translate to the ‘problematic’ server as well. This is possible because both servers don’t need the same ports, so I have approximately half NAT rules for one server pointing to one private ip address, and another half pointing to another private ip address…

mmrcela · August 26, 2011, 2:10pm

one correction: the other Mirkotik wasn’t 750, it was also 450…

mmrcela · August 30, 2011, 1:40pm

hmmm… no one seems to be interested…

Well, there was some development. l have added the x.x.x.152 address as an address on the public interface of the router (before, there were only NAT rules). This seems to have solved the problem, but when l disabled that address, it continued to work! Even weirder, it worked for some time like that (a day, or so), and then it stopped working again. After enabling the address again, it started working.

Can someone explain why does the natted address need to be configured on an interface? And if it in fact needs to be configured on an interface, why does it work when it’s not enabled? Also, other public addresses that we have, and which we use with NAT, don’t have their ip addresses configured (only natted), and they work without a problem…

This is very puzzling for me…

fewi · August 30, 2011, 2:12pm

When someone upstream (the next connected router) wants to send a packet to an IP address on a directly connected network it will ARP for the IP address, just like every other node does in basic TCP/IP over Ethernet. If it can’t ARP for the IP address it drops the packet because it doesn’t know where to send it.

If the IP address isn’t implemented on an interface the router doesn’t reply to ARP. If you own the entire subnet you can get around this by turning on proxy ARP on the interface. Other products may hide this by automatically enabling proxy ARP for all IPs used in NAT operations, but this isn’t a problem unique to RouterOS, it’s just how basic TCP/IP works.

If you see this process still working after disabling or removing an IP you’re watching an ARP cache doing its magic. Once a router has resolved an ARP request it will cache the result so it doesn’t have to keep querying. Eventually this cached entry times out.

Other addresses you are implementing might be hardcoded in the ARP cache of the upstream router. Don’t rely on that. Either always implement the IP on an interface (it’s not like there’s penalties for doing so), or turn on proxy ARP (but only if you own the ENTIRE subnet as you’d interfere with others on a shared network).

mmrcela · August 31, 2011, 8:49am

Thanks Fewi! This is really helpful; I guess one learns something new about networks all the time…

The only problem with the explanation is that I already have proxy-arp enabled. It is enabled on bridges (all interfaces are part of bridges), but not on the specific interfaces.

Also, can you please clarify what exactly do you mean by ‘Other addresses you are implementing might be hardcoded in the ARP cache of the upstream router’.

Here is also output of the commands you usually ask. I don’t think there are any other problems, but if you could please take a look, just to be sure everything else is fine…

(for the easier understanding, public addresses have format for example x.x.x.152, and private ones have for example x.x.20.1)

/ip address print detail

0 address=x.x.x.146/28 network=x.x.x.144 broadcast=x.x.x.159
interface=bridge1 actual-interface=bridge1

1 address=x.x.20.1/24 network=x.x.20.0 broadcast=x.x.20.255
interface=bridge2 actual-interface=bridge2

2 address=x.x.250.131/29 network=x.x.250.128 broadcast=x.x.250.135
interface=bridge2 actual-interface=bridge2

3 address=x.x.250.139/29 network=x.x.250.136 broadcast=x.x.250.143
interface=bridge2 actual-interface=bridge2

4 address=x.x.x.152/28 network=x.x.x.144 broadcast=x.x.x.159
interface=bridge1 actual-interface=bridge1

5 D address=x.x.10.1/32 network=x.x.10.2 broadcast=0.0.0.0
interface=datacentar actual-interface=datacentar

/ip route print detail

0 A S dst-address=0.0.0.0/0 gateway=x.x.x.145
gateway-status=x.x.x.145 reachable bridge1 distance=1 scope=30
target-scope=10

1 ADC dst-address=x.x.10.2/32 pref-src=x.x.10.1 gateway=datacentar
gateway-status=datacentar reachable distance=0 scope=10

2 A S dst-address=x.x.4.146/32 gateway=x.x.250.137
gateway-status=x.x.250.137 reachable bridge2 distance=1 scope=30
target-scope=10

3 A S dst-address=x.x.4.147/32 gateway=x.x.250.129
gateway-status=x.x.250.129 reachable bridge2 distance=1 scope=30
target-scope=10

4 ADC dst-address=x.x.250.128/29 pref-src=x.x.250.131 gateway=bridge2
gateway-status=bridge2 reachable distance=0 scope=10

5 A S dst-address=x.x.250.130/32 gateway=bridge2
gateway-status=bridge2 reachable distance=1 scope=30 target-scope=10

6 ADC dst-address=x.x.250.136/29 pref-src=x.x.250.139 gateway=bridge2
gateway-status=bridge2 reachable distance=0 scope=10

7 A S dst-address=x.x.250.138/32 gateway=bridge2
gateway-status=bridge2 reachable distance=1 scope=30 target-scope=10

8 ADC dst-address=x.x.x.144/28 pref-src=x.x.x.146 gateway=bridge1
gateway-status=bridge1 reachable distance=0 scope=10

9 A S dst-address=x.x.x.147/32 gateway=bridge2
gateway-status=bridge2 reachable distance=1 scope=30 target-scope=10

10 A S dst-address=x.x.x.150/32 gateway=bridge2
gateway-status=bridge2 reachable distance=1 scope=30 target-scope=10

11 A S dst-address=x.x.x.151/32 gateway=bridge2
gateway-status=bridge2 reachable distance=1 scope=30 target-scope=10

12 A S dst-address=x.x.209.42/32 gateway=bridge1
gateway-status=bridge1 reachable distance=1 scope=30 target-scope=10

13 A S dst-address=x.x.1.0/24 gateway=datacentar
gateway-status=datacentar reachable distance=1 scope=30
target-scope=10

14 ADC dst-address=x.x.20.0/24 pref-src=x.x.20.1 gateway=bridge2
gateway-status=bridge2 reachable distance=0 scope=10

/interface print detail

0 R ;;;
name=“ether1” type=“ether” mtu=1500 l2mtu=1524

1 R ;;;
name=“ether2” type=“ether” mtu=1500 l2mtu=1524

2 R ;;;
name=“ether3” type=“ether” mtu=1500 l2mtu=1524

3 R ;;;
name=“ether4” type=“ether” mtu=1500 l2mtu=1524

4 R ;;;
name=“ether5” type=“ether” mtu=1500 l2mtu=1524

5 R name=“bridge1” type=“bridge” mtu=1500 l2mtu=1524

6 name=“pptp-in1” type=“pptp-in”

7 name=“GOR-DC” type=“pptp-in”

8 R name=“datacentar” type=“pptp-out” mtu=1460

/ip firewall nat

add action=src-nat chain=srcnat comment=“” disabled=no out-interface=bridge1
src-address=x.x.20.20 to-addresses=x.x.x.152
add action=masquerade chain=srcnat comment=“” disabled=no out-interface=
bridge1 src-address=x.x.20.0/24
add action=masquerade chain=srcnat comment=“” disabled=no out-interface=
bridge1 src-address=x.x.250.128/29
add action=dst-nat chain=dstnat comment=“” disabled=yes dst-address=
x.x.x.147 dst-port=60000 protocol=tcp to-addresses=x.x.250.130
to-ports=22
add action=masquerade chain=srcnat comment=“” disabled=yes out-interface=
bridge1 src-address=x.x.250.136/29
add action=src-nat chain=srcnat comment=“” disabled=no out-interface=bridge1
src-address=x.x.250.130 to-addresses=x.x.x.147
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.250.138 dst-port=5080 in-interface=bridge2 protocol=udp
src-address=x.x.4.146 to-addresses=x.x.250.138 to-ports=5060
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.250.138 dst-port=5060 in-interface=bridge2 protocol=udp
src-address=x.x.4.146 to-addresses=x.x.250.138 to-ports=5060
add action=dst-nat chain=dstnat comment= disabled=no
dst-address=x.x.250.130 dst-port=5060 in-interface=bridge2 protocol=
udp src-address=x.x.4.147 to-addresses=x.x.250.130 to-ports=5080
add action=dst-nat chain=dstnat comment= disabled=no dst-address=
x.x.x.152 dst-port=5062 in-interface=bridge1 protocol=tcp
to-addresses=x.x.20.20 to-ports=5062
add action=dst-nat chain=dstnat comment= disabled=no dst-address=
x.x.x.152 dst-port=5062 in-interface=bridge1 protocol=udp
to-addresses=x.x.20.20 to-ports=5062
add action=dst-nat chain=dstnat comment= disabled=no dst-address=
x.x.x.152 dst-port=448 in-interface=bridge1 protocol=udp
to-addresses=x.x.250.132 to-ports=448
add action=dst-nat chain=dstnat comment= disabled=no dst-address=
x.x.x.152 dst-port=448 in-interface=bridge1 protocol=tcp
to-addresses=x.x.20.20 to-ports=448
add action=dst-nat chain=dstnat comment= disabled=no dst-address=
x.x.x.152 dst-port=443 in-interface=bridge1 protocol=tcp
to-addresses=x.x.20.20 to-ports=443
add action=dst-nat chain=dstnat comment= disabled=no dst-address=
x.x.x.152 in-interface=bridge1 to-addresses=x.x.20.20
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.147 in-interface=bridge1 to-addresses=x.x.250.130
add action=dst-nat chain=dstnat comment= disabled=no dst-address=
x.x.x.151 in-interface=bridge1 to-addresses=x.x.250.138
add action=dst-nat chain=dstnat comment= disabled=no dst-address=
x.x.x.151 dst-port=5000 in-interface=bridge1 protocol=tcp
to-addresses=x.x.250.138 to-ports=5000
add action=src-nat chain=srcnat comment= disabled=no out-interface=bridge1
src-address=x.x.250.138 to-addresses=x.x.x.151
add action=dst-nat chain=dstnat comment= disabled=yes dst-address=
x.x.x.151 dst-port=5090 in-interface=bridge1 protocol=tcp
to-addresses=x.x.250.138 to-ports=5090
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.250.130 dst-port=5060 in-interface=bridge2 protocol=tcp
to-addresses=x.x.250.130 to-ports=5080
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.147 dst-port=80 in-interface=bridge1 protocol=tcp
to-addresses=x.x.250.130 to-ports=80
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.147 dst-port=8443 in-interface=bridge1 protocol=tcp
to-addresses=x.x.x.130 to-ports=8443
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.147 dst-port=5060 in-interface=bridge1 protocol=tcp
to-addresses=x.x.250.130 to-ports=5060
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.147 dst-port=5060 in-interface=bridge1 protocol=udp
to-addresses=x.x.250.130 to-ports=5060
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.147 dst-port=30000-31000 in-interface=bridge1 protocol=udp
to-addresses=x.x.250.130 to-ports=30000-31000
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.147 dst-port=22 in-interface=bridge1 protocol=tcp
to-addresses=x.x.250.130
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.147 in-interface=bridge1 to-addresses=x.x.250.130
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.151 dst-port=3389 protocol=tcp to-addresses=x.x.250.138
to-ports=3389
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.146 dst-port=3389 in-interface=bridge1 protocol=tcp
to-addresses=x.x.20.3 to-ports=3389
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.146 dst-port=33899 in-interface=bridge1 protocol=tcp
to-addresses=x.x.250.138 to-ports=3389
add action=redirect chain=dstnat comment=“” disabled=yes dst-address=
x.x.x.146 dst-port=80 in-interface=bridge1 protocol=tcp to-ports=80
add action=dst-nat chain=dstnat comment=“” disabled=yes dst-address=
x.x.x.146 dst-port=8443 in-interface=bridge1 protocol=tcp
to-addresses=x.x.250.138 to-ports=8443
add action=src-nat chain=srcnat comment=“” disabled=yes out-interface=bridge1
src-address=x.x.250.138 to-addresses=x.x.x.150
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.150 dst-port=80 in-interface=bridge1 protocol=tcp
to-addresses=x.x.250.138 to-ports=80
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.150 dst-port=8443 in-interface=bridge1 protocol=tcp
to-addresses=10.160.250.138 to-ports=8443
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.150 dst-port=443 in-interface=bridge1 protocol=tcp
to-addresses=x.x.250.138 to-ports=443
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.150 in-interface=bridge1 to-addresses=x.x.250.138
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=
x.x.x.150 dst-port=30000-31000 in-interface=bridge1 protocol=udp
to-addresses=x.x.250.138 to-ports=30000-31000

mmrcela · August 31, 2011, 9:02am