Connection tracking table not cleared completely after WAN IP address change

RouterOS General
21

We encountered the same issue when using a Starlink dish on the WAN interface.
After receiving a DHCP NAK from the upstream server, the IP address changed, but existing connection tracking entries were not purged, resulting in broken NAT translations and failed outbound connections.
Are there any updates from support?

[admin@_REDACTED] > /ip/firewall/nat/print 
Flags: X - disabled, I - invalid; D - dynamic 
 0    ;;; defconf: masquerade
      chain=srcnat action=masquerade out-interface-list=WAN ipsec-policy=out,none 
[admin@_REDACTED] > /interface/list/member/print 
Columns: LIST, INTERFACE
# LIST  INTERFACE
;;; defconf
0 LAN   bridge   
;;; defconf
1 WAN   ether1 
[admin@_REDACTED] > /log/print
 04-25 04:17:24 dhcp,info dhcp-client on ether1 got IP address YYY.YYY.YYY.YYY
 04-25 04:19:16 dhcp,critical,error dhcp-client on ether1 lost IP address YYY.YYY.YYY.YYY - received NAK from dhcp server ZZZ.ZZZ.ZZ.1
 04-25 04:19:17 dhcp,info dhcp-client on ether1 got IP address ZZZ.ZZZ.ZZ.ZZ
[admin@_REDACTED] > /ip/firewall/connection/print detail 
Flags: E - expected; S - seen-reply; A - assured; C - confirmed; D - dying; F - fasttrack; H - hw-offload; s - srcnat; 
d - dstnat 
 0  SAC      protocol=tcp src-address=10.XXX.X.4:17837 dst-address=10.XXX.X.1:22 reply-src-address=10.XXX.X.1:22 
             reply-dst-address=10.XXX.X.4:17837 tcp-state=established timeout=4m59s orig-packets=581 orig-bytes=43 788 
             orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=479 repl-bytes=112 764 repl-fasttrack-packets=0 
             repl-fasttrack-bytes=0 orig-rate=2.6kbps repl-rate=3.2kbps 

 1    C   s  protocol=icmp src-address=10.XXX.X.4 dst-address=1.0.0.1 reply-src-address=1.0.0.1 
             reply-dst-address=YYY.YYY.YYY.YYY icmp-type=8 icmp-code=0 icmp-id=48984 timeout=9s orig-packets=915 765 
             orig-bytes=26 557 185 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=0 repl-bytes=0 
             repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=464bps repl-rate=0bps 

 2  SAC F s  protocol=udp src-address=10.XXX.X.5:63230 dst-address=AAA.AAA.AAA.AAA:BBBB
             reply-src-address=AAA.AAA.AAA.AAA:BBBB reply-dst-address=ZZZ.ZZZ.ZZ.ZZ:63230 timeout=2m52s orig-packets=38 
             orig-bytes=5 342 orig-fasttrack-packets=7 orig-fasttrack-bytes=3 016 repl-packets=36 repl-bytes=5 019 
             repl-fasttrack-packets=6 repl-fasttrack-bytes=2 632 orig-rate=0bps repl-rate=0bps 

 3  S C F s  protocol=icmp src-address=10.XXX.X.5 dst-address=1.0.0.1 reply-src-address=1.0.0.1 
             reply-dst-address=ZZZ.ZZZ.ZZ.ZZ icmp-type=8 icmp-code=0 icmp-id=48813 timeout=9s orig-packets=2 351 003 
             orig-bytes=68 179 087 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=2 347 992 
             repl-bytes=68 091 768 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=464bps repl-rate=464bps 
[admin@_REDACTED] > /ip/address/print 
Flags: D - DYNAMIC
Columns: ADDRESS, NETWORK, INTERFACE
#   ADDRESS           NETWORK       INTERFACE
;;; defconf
0   10.XXX.X.1/25     10.XXX.X.0    bridge   
1 D ZZZ.ZZZ.ZZ.ZZ/24  ZZZ.ZZZ.ZZ.0  ether1
22

Unfortunately not. I will update this post as soon as I have more information.

23

Yup watching this thread as most expect masquerade to clear connections…otherwise rextended scripts will get extended use LOL.
I would not consider this solved until MT replies with certainty about new behaviour or they forget to do something during programming etc…

24

Yep. But this is the New Linux Way™. The thread where the new behavior was discussed alluded to this use case briefly, with a remark that yeah, this was always intended to be for ppp/pppoe interfaces (which back when this functionality was conceived definitely was the most common access technology where it was anticipated that the address would change.)

Another and I think much better question is whether it could be actually made easy to remove these conntrack entries conntrack-tools style? Because the current way with list building and iteration is really cumbersome, slow and resource intensive. Which is a shame, because the functionality in the kernel that Mikrotik uses. I get that the general filter syntax, etc. would probably have to go, but this is such a common use case.

25

Sorry lurker didnt really understand but you seem to be saying that with the new kernel ( really still an old kernel ) that MT is now using, the unexpected behaviour is normal/expected, much to our shagrin. Furthermore, you are hoping that MT comes up with a built-in easier way to clear the connection tracking table??

26

I think you understood exactly right. The kernel now behaves like this. (If you consider 5.6 old, then please refer to the v6 kernel as “ancient” - just to avoid confusion :slight_smile: ) It now basically expects userspace to initiate/request this deletion to happen. I think this is altogether a good decision. The old way was confusing and often lead to both false negative and false positive events.

Another question is whether this should be incorporated into (maybe it would be more correct to say transferred to?) the dynamic address creation/removal process or into the dhcpcd implementation, so that the user is not burdened with scripting. I think it probably should.

As for the second part. Yes, I still would like some sort of access to “conntrack -D” functionality. :slight_smile:

27

I get the kernel discussion… but Mikrotik does patch a lot of things, so kernel version is not always that telling. The docs and history suggest NAT masquerade should clear conntrack based on IP change… so suggestion DHCP feature for it seems premature (i.e. if DHCP client could do it , so could masquerade).

Has anyone confirm it happens in 7.18.2 and/or 7.19rc1? Some of the posted config refers to 7.18.1.

I’ll test this later tonight, because the docs do say it’s “and/or” on disconnects and IP changes…

Every time when interface disconnects and/or its IP address changes, the router will clear all masqueraded connection tracking entries related to the interface, this way improving system recovery time after public IP change. If srcnat is used instead of masquerade, connection tracking entries remain and connections can simply resume after a link failure.

Most of my use cases, it is the interface (i.e. LTE) that goes down, so connections clear… Now “masquerade clearing” upon a changing IP… I’m sure that worked at some point, but cannot say I’ve checked recently.

28

The point is this: at the time of the change no one volunteered to specify exactly when and exactly which connections should be purged. Any time an IP changes on the interface? If the IP change affects what IP is tied to the masquerade action for the given interface (i.e. if it is SRC_PREF)? What if the masquerade rule itself changes? What do we do when there is a failover without an IP change (e.g. via a routing change)?

The point being: it all depends on why the change is taking place. This the kernel doesn’t know, so any guess that it makes will be flawed in some scenarios. And in came the kernel interface for conntrack-tools, so it was kind of simple to say: let userspace notify the kernel.

This is not a feature request, rather just an adaptation to a new way of doing things.

29

Your asserting the kernel current logic, trumps what the MirkoTik docs do say. Perhaps. Docs could be wrong about “on IP change”, but that be glaring oversight by MikroTik at this point. And it could just be a bug in masquerade (or perhaps some TBD config error elsewhere).

There is now very old presentation from Janis(?) from Mikrotik on V6 that had a lot of good info on masquerade (with presentation subtitle being “My holy war against Masquerade”)
https://mum.mikrotik.com/presentations/EU17/presentation_4058_1490948376.pdf
(starting at page 25)


Any time an IP changes on the interface?

Yes, per docs and historically.


If the IP change affects what IP is tied to the masquerade action for the given interface (i.e. if it is SRC_PREF)?

Yes, on a covered interface (even if covered by list). I’m not sure src-pref is considered, I’d guess it only applies to local packets since src-pref logic has changed from V6 to V7.


What if the masquerade rule itself changes?

Again, docs unclear. I want to say no, but not something I’ve tested but never noticed that. Now if interfaces covered by masquerade NAT changed (i.e. one with connections removed), it should flush.


What do we do when there is a failover without an IP change (e.g. via a routing change)?

Routing happens before NAT, so it’s the interface state that matters, not routing. Now, you can get to bad situation after routing change - in part because that won’t clear connections. Thus why you see scripts to “do more” than masquerade.


The point being: it all depends on why the change is taking place. This the kernel doesn’t know, so any guess that it makes will be flawed in some scenarios. And in came the kernel interface for conntrack-tools, so it was kind of simple to say: let userspace notify the kernel.

Oh, there have always been holes in masquerade scheme. We agree there. Even Mikrotik agrees: “Unfortunately, this can lead to some issues with unstable links when the connection gets routed over different links after the primary link goes down.”.

And while script sometimes may be helpful, for example with netwatch and multiple route tables. It no panacea. More fundamentally with multiwan and especially TCP, there is no free lunch when dealing with WAN failover – it’s never “seamless” (*unless you’re more complex schemes like bonding, BGP, etc. that don’t use NAT conntrack)

(Anyone else chime in if wrong on anything, I need to re-test things since it’s been a while…and things can break/change…)

30

You’re making it awfully hard to disagree with you because you are making valid points :slight_smile:

I’m not trying to make that point. The documentation is very clear, and it is also easily determined that the behavior does not match the description. Obviously that has to be fixed.

In terms of behavior, of course the source code wins, but that’s sort of a tautology.

I’m simply stating that the kernel didn’t just randomly change. I have the suspicion that masquerade as previously known has disappeared and is not coming back. This is not the first firewall action that this has happened to and had either its functionality drastically altered, subsumed by another, etc.

As a user (or downstream) of the kernel, Mikrotik has to find some way of dealing with this. Patching the kernel is of course one, simply adapting to the new way is I think simpler.

I agree with your discussion. The two takeaways for me are:

  • if the dhcp client case was fixed, that would solve the regression for most of the use cases
  • “conntrack -D” is still needed - i am ready to accept that in failover scenarios there will always be disruptions (and there are various ways of dealing with them - none especially nice) but some protocols (notably Wireguard) easily get into a stuck state where they never recover. “Never recover” is not “can lead to unstable links”. But the fact is that for all those atypical failover scenarios, I really would like my conntrack -D.
31

I am curious what you are looking for that is not currently possible with /ip/firewall/connection/remove [find where ] ?

That slide in the MUM deck talking about masquerade vs. src-nat is interesting, because I have used src-nat before in scenarios with multiple (hundreds) PPP interfaces, where if one of the PPP interfaces was torn down, all of the other interfaces would suffer a brief second or two of packet loss & one of the CPU cores would peak out during that time. I can’t remember how this was determined as it was a while ago (definitely ROS 6.x), but eventually it was figured that when an interface was deleted, the entire connection tracking table was being iteratively traversed for related connections, so that they could be removed. This would seem to indicate that it is not just masquerade that does (or, at least, did) this kind of clean-up. We simply stopped running any connection-tracking on our ROS PPP/VPN servers, and for any where we had a need to do so, we offloaded that function to a dedicated box. So it is very bizarre to see an MT staff member claiming that avoiding masquerade on a “PPPoE server” and using src-nat instead is some sort of work-around.

Makes me want to try to lab this up again and see if I was just imagining things, or what.

32

I thought I explained it well with:

Because the current way with list building and iteration is really cumbersome, slow and resource intensive."

Since you suggested a command, just for entertainment:

[admin@xxx1] > /ip/firewall/connection/print count-only where protocol=udp
5895
[admin@xxx1] > /ip/firewall/connection/remove [ find where protocol=udp ]       
no such item (4)
[admin@xxx1] > /ip/firewall/connection/print count-only where protocol=udp
5484

And yes, in fact the deletions are only performed until the first reference is encountered where the connection has timed out while preparing the list of items to delete.

33

Apologies; I missed that.


This is a fair point. Though I don’t necessarily think that they need to throw the actual syntax out. Maybe it would be easier if they did so rather than special-casing it, but if they at least wanted the CLI experience to be consistent, they could decide to not actually treat table entries as traditional ROS list items, and instead just translate the request to do direct table manipulations under-the-hood.

Maybe it would be worth building a statically-linked copy of the ‘conntrack’ userland app, and trying to see if it would work on a jailbroken router… :eyes:

34

Not really necessary. Just look into the standard kernel module xx_conntrack and the usual userland tools (“conntrack-tools”). That should be enough to get you started. There are also some more specialized tools floating around on GitHub if you want extra bells and whistles, but the basics are already there. The ROS v7 control plane uses the same kernel APIs..

35

Though I admit I sound like it sometimes, I’m not all unreasonable. As a first start I would be perfectly happy with a “purge” command that would simply not fail for nonexistent items, but just march on. Better yet would be a “purge” with “find” semantics. Both would eliminate writing unnecessary boilerplate. (I would maybe venture as far as to argue that “remove” itself should not error out on nonexistent connections, because conntrack entries are transient in nature. The only other place I can think of where this can come up would be address-list dynamic entries with timeouts, and maybe this would be the nicer behavior there as well…)

I would be fine with this becasue it’s is not really performance critical. IP addresses change relatively rarely and so if the connections are eventually (whenever) purged, that’s enough for me. Also, purging usually occurs after some configuration change means that these entries will not be recreated (in the same form), so there’s really no rush.

The bigger issue is that the kernel-userland interface supports filtering (and btw deletion) off the shelf (this is where the conntrack syncing daemon and the entire conntrack-tools util set is attached btw.) This should be incorporated (probably opportunistically) into the find/purge process. Which leads me to the thing that bugs me the most: why are addresses and ports concatenated into a string? This obviously hinders any sorting/filtering and any attempt at offloading them to the appropriate place.

This is a classic “show ip route” situation. Manipulating this sort of data generically absolutely has its uses, but there are some common special cases that should not be tossed aside. The only one with regard to the conntrack table that I can think of is purging. For other debug purposes the current way works just fine.

36

That’s what I’m talking about: the userland conntrack tools. Yes, I expect ROS is using the same kernel APIs. In the past, though, I have found statically-linked builds of similar tools to be a requirement when trying to run them on top of ROS’s minimalistic userland, because the shared library binaries that ROS ships with are not necessarily going to be a match for what some rando build of a given tool is expecting to run into. Without having looked into it at all yet, who knows & perhaps the only outside dependency that the ‘conntrack’ utility has is to libc (though since it’s been so long, I can’t remember off-hand if ROS is using glibc or another implementation…I have to believe they’re using one of the more “micro” ones, given the footprint they’re trying to squeeze into).

37

Nathan your hurting my brain, is there any reason to separate connection tracking clearing of change IP and down and change of ISP? and if not, then MT simply needs to ensure the functionality exists that covers both, even if its just a checkbox.

38

Agreed! Overall, how WAN “failover” is handled could be improved more generally.

Personally rather mucking with conntrack-tools/etc… I’d perfer support for adding eBPF code to the existing firewall scheme, so it can be extended however. i.e. Some masquerade2 action could be custom/user-provided eBPF module that be installed like container/package.

39

I’m not arguing about that at all. What I’m doing is what people in the industry call “thinking out-loud about a possible workaround”.


Yes, and until they take two years to implement your checkbox, what are people supposed to do in the meantime?

40

Guys, it finally happened! After all this time I got an E-Mail two days ago, stating that the issue in my ticket was fixed and that I should check out the release notes of coming ROS releases for a corresponding note. Haven’t verified it yet, but I would guess it’s this one for v7.20.7:

firewall - clear relevant masqueraded connection tracking entries on IP address change;
1 Like