Large receive offload breaking TCP on 6to4?

RouterOS General
1

I am running a couple of 6to4 tunnels on a CCR2004-16G-2S+ and since ROS 7.16.2 (and still with 7.18) I seem to have an issue with TCP connections that is making them unusable.

The interesting thing is that it is unobservable from the router itself and we only discovered this by chance from another device.

What happens is that when TCP packets are received on the same connection at any significant enough rate on the tunnel, the payloads are combined (as by large-receive-offload) into a new packet before firewalling/routing. So, we end up with a logical packet whose size is much bigger than any MTU used. After passing through the firewall, it tries to send it through the local interface, and the resulting packet is larger than the MTU. No resegmentation occurs, the packet is dropped and an ICMP packet-too-big message is sent.

In a tcpdump done from the remote side, we can see the packet-too-big message is saying the needed MTU is 1500 (for the internal local network) but the tunnel MTU is 1480 … and the headers of the “too-big” packet included in the ICMP packet is showing a size at least 2x normal payloads, 2700 bytes or more.

If torch or the packet sniffer is activated on the tunnel interface, the problem immediately goes away and is unobservable. The easiest way to see it is to start a download from some remote host and run tcpdump on it. The ICMP packet-too-bigs will be received multiple times per second, and the transfer rate will be terrible. On the machine I test (with 100Mbit connection) I got about a 100kB/s transfer rate, and if I activate the packet sniffer, it increases to 10MB/s

Here are the details:

  • Two 6to4 tunnels to different remote IPs – one directly to a machine I control, another to the HE tunnelbroker – problem happens on both
  • No fasttrack/fastpath used
  • Internal subnets are accessed through VLANs on one of the SFP+ ports
  • No bridge is used for the internal port (but I tried it with a bridge, and there was no difference)
  • Connection tracking is being used
  • No queues active, but using them seems to change nothing
  • External interface doesn’t seem to matter – tried one ISP connection on {Ethernet->VLAN->PPPOE} and another on plain Ethernet and the behaviour is the same on both

Is anyone seeing this behaviour, too? I haven’t found anything searching and it seems that this is enough of a performance killer that others would have noticed. I have thought I have been suffering from poor reputation of HE tunnelbroker addresses for bad IPv6 connectivity for ages and it turns out it is not that…

Is there any setting to turn off the large-receive-offload?

Edit to add : I just noticed that starting torch on any interface at all (including lo) also makes the problem immediately go away – for all tunnels

Thanks

2

Thanks for this dwplx, I responded to you back in my thread. Interesting findings you have for sure.

3

I think I'm hitting the same thing, on an RB4011iGS+ this time. I first noticed it on 7.24.3 and it's still there on 7.26beta3, so it doesn't seem to have been fixed since your post. I've reported it to support as SUP-226686; anyone else still running into it is welcome to refer to that ticket.

What I see is the same as in your first post: the router merges consecutive TCP segments that arrive inside a 6in4 (protocol 41) tunnel, then won't forward the merged packet out a 1500-byte interface. It drops it and sends a "packet too big" back, even though every packet on the wire fit. Downloads lose 6–9% of packets, always in groups of 2–4 segments, and crawl at 1–4 Mbit/s. Unlike in your case, torch and sniffer don't make it go away for me.

It happens both when the tunnel ends on the router (/interface 6to4) and when the router only forwards protocol 41 to a Linux box behind it (dst-nat). In the second case the error quotes protocol-41 packets of 2908 / 4316 bytes, i.e. two or three 1500-byte packets glued together:

198.51.100.10 > 192.0.2.132: ICMP unreachable - need to frag (mtu 1500)
  IP (flags [DF], proto IPv6 (41), length 2908) 192.0.2.132 > 198.51.100.10:
  IP6 2001:db8:a::5.443 > 2001:db8:b::c.34369: Flags [P.], seq ...:..., length 2816

Workaround, for anyone in the forwarding case: clearing DF on the incoming protocol-41 packets makes the router split the merged packets back up properly:

/ip firewall mangle add chain=prerouting in-interface=<wan> protocol=ipv6-encap src-address=<tunnel-server> action=clear-df passthrough=yes place-before=0

With that, my downloads went from ~3 Mbit/s to ~300–560 Mbit/s with zero loss (on 7.26beta3: 57 → 566 Mbit/s for one stream). It doesn't help when the tunnel ends on the router itself, because the inner IPv6 packet has no DF bit to clear, so for now I terminate 6in4 on Linux instead.

A few things I ruled out:

  • Cabling and switches: same result with the WANs plugged straight into the router.
  • The tunnel broker and the ISPs: the same tunnels terminated on a Linux box whose tunnel traffic doesn't pass through the RB4011 run at ~800 Mbit/s with no workaround.
  • My config: CHR (7.15.3, 7.16 and 7.25rc2; x86 and arm64) doesn't reproduce it, even with this router's whole config replayed. CHR never merges forwarded protocol 41 at all.

So for me it looks specific to the RB4011's receive path. Your CCR2004 and the RB4011 in the 6rd thread are both Annapurna Alpine too, so maybe that's the common factor, though I can only show it on mine.

If anyone wants to compare notes, I'm happy to share more captures or the CHR lab scripts I used.

4

Same issue here, also on an RB4011, RouterOS 7.25 stable, with a Hurricane Electric tunnelbroker tunnel via /interface 6to4 terminating on the router. I've reported it to support as SUP-227178

Note: AI was used to generate this writeup as my networking knowledge is not sufficient to fully understand this. This is not how I prefer to communicate so I apologize in advance. However as my issue seems to match this issue I thought it was at least relevent to mention this.

Setup:

  • 6to4: mtu=1280, clamp-tcp-mss=yes, dont-fragment=no
  • IPv4 WAN: VLAN on SFP+, MTU 1500 (DF ping with size 1500 passes)
  • LAN: bridge, MTU 1500

Symptom: IPv6 TCP downloads from LAN clients stall after 50–260 KB, while the same download over IPv4 runs at ~85 MB/s. Every IPv6 bulk download is affected (Hetzner, Akamai, GitHub, Google); in practice, streaming video freezes after a few seconds.

Capture (sniffer on the 6to4 interface plus tcpdump on a LAN client at the same time):

  • MSS 1220 negotiated in both directions.
  • On tunnel rx, segments of the same flow show up coalesced: IPv6 payload length 2448 (32-byte TCP header + 2×1208), and also 3656, 6072, up to 9780 bytes.
  • Many are forwarded fine, but some are dropped: the router sends ICMPv6 Packet Too Big (MTU 1500, source = its own tunnel address) to the remote server, quoting the coalesced packet. 265 PTBs in about 40 seconds; none received from the internet.
  • The server ignores the PTB (its packets are 1280 bytes) and retransmits. The retransmission is coalesced and dropped again, and TCP ends up in RTO backoff. After the sender fell back to 1024-byte segments, two of them were merged into a 2080-byte payload and dropped as well.

Tested without effect:

  • clear-df mangle on incoming protocol 41 (in-interface=WAN, src-address=tunnel server): as antonme said, it doesn't help when the tunnel terminates on the router
  • Sniffer running on the tunnel interface: the problem stays (same as antonme, unlike the original post)
  • IPv6 firewall: no drops for the affected flow (logged), invalid counter 0, raw table empty; no IPv6 FastTrack, IPv6 fast path inactive
  • Bridge use-ip-firewall=no
  • IPv6 mangle change-mss clamp-to-pmtu, and /ipv6 nd mtu=1280

Workaround for now: /ipv6 nd ra-lifetime=none, so clients fall back to IPv4 for internet traffic.