Srce-nat 'masquerade' not catching all traffic?

RouterOS General
1

All my client routers (90% SXT, rest SEXTANT or QRT or older boards) are configured as router and connect with the wireless interface to an AP which also assigns an IP to the CPE.

In the CPE we have a srce-nat rule:

 /ip firewall nat
add action=masquerade chain=srcnat comment="Masquerade all traffic leaving wlan1" out-interface=wlan1

wlan interface is dhcp-client:

/ip dhcp-client option
add code=12 name=host value="'R1-027'"
/ip dhcp-client
add default-route-distance=0 dhcp-options=host,clientid disabled=no interface=wlan1

On the ethernet interface we have a dhcp-server assigning addresses to client’s devices:

/ip dhcp-server
add address-pool=DHCP-pool disabled=no interface=ether1-local name=default
/ip dhcp-server network
add address=192.168.50.0/24 dns-server=208.67.222.222,208.67.220.220 gateway=192.168.50.1

What I’d expect and the manual explains, and it is the case in 99% of the cases, it that every package leaving router has been given a src-address from the interface ‘wlan1’, which we also maintain as the ‘client’s IP’.

So, in the AP looking in the registration table we can ‘see’ the CPE and its assigned IP. That makes it easy to copy and paste it to open in winbox in case of need.
We can also see if things are ok.

But what we see occasionally is that the “Last IP” reading at times shows a internal address IP like 192.168.50.xxx
This means it is a package originally coming from the local network but that doesn’t get the ‘public’ IP while leaving the srce-nat chain… How is that possible?

Sometimes when client is browsing we also see other IP’s occasionally. But I’d presume the “Last IP” listing only shows IP’s coming from the client?
Since clients usually don’t run servers there should be no ‘internet initiated’ packages flowing to the client’s CPE? So every package coming from or going to that CPE should have its assigned IP in the header?

Or is there some sort of ‘leak’ that makes some packages go out of the CPE without its public IP but with the original local IP instead?

2

I’ve had the exact same problem and it was caused by “invalid” packets. Since NAT relies on connection tracking, a packet that has an “invalid” connection state can’t be natted and it gets passed through without modification. In my case, I think it’s being caused by ACK and ACK,FIN packets being received after the connection has already expired. Since filter rules get processed before NAT rules, the simple fix is to add a filter DROP rule that matches on connection state: invalid. The more complex fix is to figure out why the packets are occurring.

3

Guessing.. Cpe & client hosts protocols timeouts are probably different, if coontrack kill a connection/stream because timeout and client think it’s still open and speaks ..voilà an invalid packet is born

4

So? The ‘drop invalid connections’ should be the cure?

5

Yep, that’s my guess as well. I just haven’t had time to investigate further.

Yes but you should look for the cause as well. Increasing the connection tracking timeouts might make the invalid packets go away but a side effect might be that connection table entries will hang around a while longer.

6

And what is the problem in that? Eating memory?

7

That and running out of space in the table. I wouldn’t expect either to be an issue but it’s just something to be aware of. In normal Linux, the default nf_conntrack_max is 64k but I don’t see it in RouterOS anywhere.

8

Just FYI… After testing this morning, here are the connection tracking parameters I settled on…

[root@gw1.oaks] > /ip fire conn tracking pr ; /sys resource print 
                   enabled: auto
      tcp-syn-sent-timeout: 1m
  tcp-syn-received-timeout: 30s
   tcp-established-timeout: 1d
      tcp-fin-wait-timeout: 1m
    tcp-close-wait-timeout: 30s
      tcp-last-ack-timeout: 15s
     tcp-time-wait-timeout: 1m
         tcp-close-timeout: 15s
               udp-timeout: 3s
        udp-stream-timeout: 1m
              icmp-timeout: 30s
           generic-timeout: 10m
               max-entries: 88088
             total-entries: 664
                   uptime: 10m28s
                  version: 6.30.4
               build-time: Aug/25/2015 12:59:46
              free-memory: 38.8MiB
             total-memory: 64.0MiB
                      cpu: MIPS 74Kc V4.12
                cpu-count: 1
            cpu-frequency: 600MHz
                 cpu-load: 4%
           free-hdd-space: 98.2MiB
          total-hdd-space: 128.0MiB
  write-sect-since-reboot: 277
         write-sect-total: 175090
               bad-blocks: 0.4%
        architecture-name: mipsbe
               board-name: RB912UAG-2HPnD
                 platform: MikroTik
[root@gw1.oaks] >

This has stopped the invalid packets without too much overhead. Of course, these are tuned for this specific network so don’t just apply them without testing.

9

I would say hypothesis confirmed, well done!

10

Well, all probably very interesting… but what do I test? Where do I look at?

I took same print from one of my CPE’s, just a random one, default setting since I don’t change anything:

  /ip fire conn tracking pr ; /sys resource print 
                   enabled: auto
      tcp-syn-sent-timeout: 5s
  tcp-syn-received-timeout: 5s
   tcp-established-timeout: 1d
      tcp-fin-wait-timeout: 10s
    tcp-close-wait-timeout: 10s
      tcp-last-ack-timeout: 10s
     tcp-time-wait-timeout: 10s
         tcp-close-timeout: 10s
               udp-timeout: 10s
        udp-stream-timeout: 3m
              icmp-timeout: 10s
           generic-timeout: 10m
               max-entries: 88088
             total-entries: 6
                   uptime: 6d22h4m40s
                  version: 6.30.2
               build-time: Jul/22/2015 11:17:58
              free-memory: 44.8MiB
             total-memory: 64.0MiB
                      cpu: MIPS 74Kc V4.12
                cpu-count: 1
            cpu-frequency: 600MHz
                 cpu-load: 2%
           free-hdd-space: 113.1MiB
          total-hdd-space: 128.0MiB
  write-sect-since-reboot: 120
         write-sect-total: 76223
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: SXT Lite5
                 platform: MikroTik

Which parameters should I change? And why?
You guys are probably much better in this than me and I don’t have the time to do long testing because to start with I don’t even know what to test? That the strange IP’s in the “Last IP” registration table goes away?

And this is taken from a CPE, maybe I should change in the AP? Or both?

11

As already suggested by gtj you have only to drop invalid packets in forward chain in cpe, we were only interested in understanding the root cause. I’d leave timeouts as default.

12

Start with 2 rules added to every device you have a NAT rule in…

/ip firewall filter
add action=drop chain=output connection-state=invalid log=yes log-prefix=“out invalid” out-interface=<wan_interface>
add action=drop chain=forward connection-state=invalid log=yes log-prefix=“fwd invalid” out-interface=<wan_interface>

Now watch the counters for those rules and look at the packets in the log. You’ll probably be seeing ACK and ACK,FIN packets being dropped.

The next step I’d take is to increase the tcp timeouts (except tcp-established-timeout) and watch the drop counters, free-memory and total-entries. When the drop counters stop increasing, you’re done. Of course, you have to balance that with the amount of free-memory you have left.

All the parameters are from the standard Linux kernel so there’s plenty of documentation floating around about what they mean and their implications.

13

OK guys, really helpful. Will start playing with your suggestions… thanks…

14

Well guys, I tried your suggestions but things are much worse now. AP ‘Last IP’ listing now shows almost continuous the local IP. Only when I make a winbox session or telnet the CPE’s WAN IP comes in the list…

here’s my code, maybe I’m doing something wrong?

/ip firewall filter
add action=drop chain=output comment="drop invalid outgoing connections and log them" connection-state=invalid log=yes log-prefix="out invalid" out-interface=wlan1
add action=jump chain=forward comment="jump to forward allow chain" jump-target=forward-allow
add action=drop chain=forward-allow comment="drop passing through invalid connections and drop them" connection-state=invalid log=yes log-prefix="fwd invalid" out-interface=wlan1
add chain=forward-allow comment="Accept traffic to LAN network fm \"safe\" address range" out-interface=ether1 src-address-list=safe
add chain=forward-allow comment="allow new connections" connection-state=new
add chain=forward-allow comment="allow established connections" connection-state=established
add chain=forward-allow comment="allow related connections" connection-state=related
add action=drop chain=forward comment="drop everything else" in-interface=ether1
add chain=input comment="accept established connections" connection-state=established
add chain=input comment="accept related connections" connection-state=related
add chain=input comment="allow access to router from known network" src-address-list=safe
add action=drop chain=input comment="drop everything else"

‘safe’ address list address ranges are to allow me access to CPE or behind from everywhere in my network.

15

And oh; the counters as suggested by gtj are not showing anything at all.. they stay “0”.

16

Forget about the last remarks…; I accidentally erased the masq rule in nat.

17

Do the logs show the packets being dropped?

Try moving the drops for invalid packets (both output and forward) after all your accepts.