Routing wireguard traffic via separate routing table adds 1.5-3 seconds to TLS handshake

RouterOS General
1

Hello,

I hope you are well.

I encountered a strange performance degradation.

Simplified config (essentially, LAN routed to WAN, however, for specific DHCP pool default route is replaced with wireguard through separate routing table using routing rule)

# * by RouterOS 7.13
# model = CRS309-1G-8S+
/interface bridge add frame-types=admit-only-vlan-tagged name=bridge1 vlan-filtering=yes
/interface ethernet set [ find default-name=sfp-sfpplus1 ] l2mtu=10218
/interface ethernet set [ find default-name=sfp-sfpplus2 ] l2mtu=10218 mtu=10134
/interface wireguard add listen-port=56791 mtu=1420 name=wgCF private-key="*"
/interface vlan add interface=bridge1 mtu=10134 name=lan1 vlan-id=2
/interface vlan add interface=bridge1           name=wan1 vlan-id=3
/interface ethernet switch set 0 l3-hw-offloading=yes qos-hw-offloading=yes
/interface ethernet switch port set sfp-sfpplus1 l3-hw-offloading=no
/ip pool add name=dhcp_pool_lan_devices ranges=192.168.0.0/27
/ip pool add name=dhcp_pool_lan_cf      ranges=192.168.0.32/27
/ip pool add name=dhcp_pool_lan_unknown ranges=192.168.0.96/27
/ip dhcp-server add address-pool=dhcp_pool_lan_unknown authoritative=yes interface=lan1 lease-time=1d name=dhcp_lan
/routing table add fib name=cf
/interface bridge port add bridge=bridge1 comment=WAN1          frame-types=admit-only-untagged-and-priority-tagged interface=sfp-sfpplus1 pvid=3
/interface bridge port add bridge=bridge1 comment=LAN1 edge=yes frame-types=admit-all                               interface=sfp-sfpplus2 pvid=2
/interface ethernet switch l3hw-settings set fasttrack-hw=yes ipv6-hw=yes
/ip settings set allow-fast-path=yes ip-forward=yes
/ipv6 settings set disable-ipv6=no forward=yes
/interface bridge vlan add bridge=bridge1 tagged=bridge1 vlan-ids=2
/interface bridge vlan add bridge=bridge1 tagged=bridge1 vlan-ids=3
/interface wireguard peers add allowed-address=0.0.0.0/0 endpoint-address=* endpoint-port=* interface=wgCF persistent-keepalive=25s public-key="*"
/ip address add address=192.168.0.1/24 interface=lan1 network=192.168.0.0
/ip address add address=172.16.0.2     interface=wgCF network=172.16.0.2
/ip dhcp-client add add-default-route=special-classless default-route-distance=10 interface=wan1 use-peer-dns=no use-peer-ntp=no
/ip dhcp-server lease add address=192.168.0.36 client-id=cf mac-address=* server=dhcp_lan
/ip dhcp-server lease add address=192.168.0.6               mac-address=* server=dhcp_lan
/ip dhcp-server network add address=192.168.0.0/24 comment=lan dns-server=1.1.1.1,1.0.0.1 gateway=192.168.0.1
/ip firewall filter add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
/ip firewall nat add action=masquerade chain=srcnat out-interface=wan1
/ip firewall nat add action=masquerade chain=srcnat out-interface=wgCF
/ip route add dst-address=0.0.0.0/0        gateway=wgCF                     routing-table=cf
/ip route add dst-address=104.16.123.96/32 gateway=wgCF pref-src=172.16.0.2
/routing rule add action=lookup                              min-prefix=0                             table=main
/routing rule add action=lookup-only-in-table interface=lan1              src-address=192.168.0.32/27 table=cf

Note three most relevant parts

/ip dhcp-server lease add address=192.168.0.36 client-id=cf mac-address=* server=dhcp_lan
/ip dhcp-server lease add address=192.168.0.6               mac-address=* server=dhcp_lan

/ip firewall filter add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes

/ip route add dst-address=104.16.123.96/32 gateway=wgCF pref-src=172.16.0.2

Test is being done from macOS device connected to sfp-sfpplus2 using the following commands

curl --resolve www.cloudflare.com:443:104.16.123.96 'https://www.cloudflare.com/cdn-cgi/trace' -v --trace-time
curl --resolve www.cloudflare.com:443:104.16.124.96 'https://www.cloudflare.com/cdn-cgi/trace' -v --trace-time

If the client does not fill client-id, it receives 192.168.0.6 IP. Both curl commands execute instantaneously, and the first result signifies traffic has been through the wireguard interface, the second - not.

If the client sets client-id=cf, it receives 192.168.0.36 IP. The first curl command executes instantaneously, the second curl commands executes with a ~1.5-second delay (see log below) usually during TLS handshake but sometimes before response. Both results signify traffic has been through the wireguard interface.

Deleting the route to 104.16.123.96 adds the delay to execution of the first command. Removing the firewall fasttrack-connection rule removes the delay from both.
Adding either out-interface=!wgCF or src-address=!192.168.0.32/27 to the firewall fasttrack-connection rule does NOT remove the delay.

What could be the problem?

P. S. TLS handshake delay is reproducible with any website that is routed through the separate routing table (e. g. opening a website in browser is delayed, and then images on it are delayed after the website is shown, as new connections with new TLS handshakes need to be established).


14:48:38.984210 * Added www.cloudflare.com:443:104.16.124.96 to DNS cache
14:48:38.984529 * Hostname www.cloudflare.com was found in DNS cache
14:48:38.984641 *   Trying 104.16.124.96:443...
14:48:38.988348 * Connected to www.cloudflare.com (104.16.124.96) port 443
14:48:38.990355 * ALPN: curl offers h2,http/1.1
14:48:38.990609 * TLSv1.3 (OUT), TLS handshake, Client hello (1):
14:48:39.001550 *  CAfile: /opt/local/share/curl/curl-ca-bundle.crt
14:48:39.001577 *  CApath: none
14:48:39.018492 * TLSv1.3 (IN), TLS handshake, Server hello (2):
14:48:39.018921 * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
14:48:39.018962 * TLSv1.3 (IN), TLS handshake, Certificate (11):
14:48:39.022113 * TLSv1.3 (IN), TLS handshake, CERT verify (15):
14:48:39.022339 * TLSv1.3 (IN), TLS handshake, Finished (20):
14:48:39.022407 * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
14:48:39.022464 * TLSv1.3 (OUT), TLS handshake, Finished (20):
14:48:39.022539 * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
14:48:39.022568 * ALPN: server accepted h2
14:48:39.022595 * Server certificate:
14:48:39.022627 *  subject: CN=www.cloudflare.com
14:48:39.022656 *  start date: Apr 25 21:43:30 2024 GMT
14:48:39.022684 *  expire date: Jul 24 21:43:29 2024 GMT
14:48:39.022715 *  subjectAltName: host "www.cloudflare.com" matched cert's "www.cloudflare.com"
14:48:39.022744 *  issuer: C=US; O=Let's Encrypt; CN=E1
14:48:39.022770 *  SSL certificate verify ok.
14:48:39.022800 *   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
14:48:39.022828 *   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
14:48:39.022855 *   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
14:48:39.022957 * using HTTP/2
14:48:39.023015 * [HTTP/2] [1] OPENED stream for https://www.cloudflare.com/cdn-cgi/trace
14:48:39.023042 * [HTTP/2] [1] [:method: GET]
14:48:39.023068 * [HTTP/2] [1] [:scheme: https]
14:48:39.023094 * [HTTP/2] [1] [:authority: www.cloudflare.com]
14:48:39.023119 * [HTTP/2] [1] [:path: /cdn-cgi/trace]
14:48:39.023146 * [HTTP/2] [1] [user-agent: curl/8.6.0]
14:48:39.023172 * [HTTP/2] [1] [accept: */*]
14:48:39.023219 > GET /cdn-cgi/trace HTTP/2
14:48:39.023219 > Host: www.cloudflare.com
14:48:39.023219 > User-Agent: curl/8.6.0
14:48:39.023219 > Accept: */*
14:48:39.023219 > 
14:48:40.292474 * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
14:48:40.292692 * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
14:48:40.292770 * old SSL session ID is stale, removing
14:48:40.295692 < HTTP/2 200 
14:48:40.295761 < date: Mon, 20 May 2024 11:48:39 GMT
14:48:40.295813 < content-type: text/plain
14:48:40.295863 < access-control-allow-origin: *
14:48:40.295912 < server: cloudflare
14:48:40.295961 < cf-ray: 886c1954ece52de7-DME
14:48:40.296017 < x-frame-options: DENY
14:48:40.296066 < x-content-type-options: nosniff
14:48:40.296121 < expires: Thu, 01 Jan 1970 00:00:01 GMT
14:48:40.296171 < cache-control: no-cache
14:48:40.296253 < 
14:48:40.296413 * Connection #0 to host www.cloudflare.com left intact
2

Changing the firewall rule to the following removes the delay, but I’m still not sure if I understand why the delay was there in the first place.

/ip firewall filter add action=jump chain=forward connection-state=established,related jump-target=fwfastt
/ip firewall filter add action=return chain=fwfastt out-interface=wgCF src-address=192.168.0.32/27
/ip firewall filter add action=return chain=fwfastt dst-address=192.168.0.32/27 in-interface=wgCF
/ip firewall filter add action=fasttrack-connection chain=fwfastt hw-offload=yes
3

Wireguard inside wireguard (/ip route add dst-address=[wg2_endpoint_IP] gateway=wgCF /interface wireguard name=wg2 mtu=1340 /interface wireguard peers interface=wg2 endpoint=[wg2_endpoint_IP]) leads to practically indefinite TLS handshake in similar circumstances.

4

The observed behavior is due to the (mis)use of fasttrack. Please read this:

https://help.mikrotik.com/docs/display/ROS/Packet+Flow+in+RouterOS#PacketFlowinRouterOS-FastTrack

Especially these warning sections:


Traffic that belongs to a fast-tracked connection travels in FastPath, which means that it will not be visible by other router L3 facilities (firewall, queues, IPsec, IP accounting, VRF assignment, etc). Fasttrack lookups route before routing marks have been set, so it > works only with the main routing table> .

And one more warning:


FastTrack can process packets > only in the main routing table > so it is the system administrator duty to > not FastTrack connections that are going through non-main routing table > (thus connections that are processed with mangle action=mark-routing rules). Otherwise packets might be > misrouted > though the main routing table.

In your original configuration you had routing rules set up that switch to the non-main routing table “cf” for the source address range 192.168.0.32/27. But you still left this rule in your firewall filter configuration, unrestricted


/ip firewall filter add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes

As a result, of course the packets that were intended to use the cf table will be misrouted by fasttrack and would use the main table instead and would not go through WireGuard. You can see however that the fasttrack rule has the condition connection-state=established,related. Which means it will not be immediately applied to the first packet (sent from your network), that packet will have connection-state=new. Only once the reply-packet is sent back from the other side does the connection get the state of “established” and fasttrack will kick in. That’s also the reason why when you first tried to filter out the fasttrack rule by specifiying src-address=!192.168.0.32/27 it did not work. Because the first packet sent out with src-address=192.168.0.32/27 still have connection-state=new. Only once the return packet arrives, with dst-address=192.168.0.32/27, does connection-state=established,related applied. That’s the reason why it worked when you also specified dst-address=!192.168.0.32/27.

Your newest rules now skip fasttrack for both directions, that’s why it now works.

As for the reason why previously with fasttrack wrongly enabled, the connections still “worked” but with delay, this is due to this quote from the same page above:


to maintain connection tracking entries > some random packets will still be sent to a slow path> . This must be taken into consideration when designing firewalls with enabled “fasttrack”.

While many packets were lost due to being sent through the wrong route, from time to time, some packets will be sent correctly. The observed result would be as if you had a very unreliable connection with a lot of packet lost, where only a small percentage of the packets arrive at the destination. Such connection will result in big observed delays and retries.

5

Thanks, @CGGXANNX. Now it all makes sense.

I remembered that non-main routing tables cannot be FastTracked but implied it would only affect performance (CPU vs HW routing) and not misroute. I.e. that ROS would exclude them from FastTrack automatically. Basically, I missed this quote

so it is the system administrator duty to not FastTrack connections that are going through non-main routing table

Do you have any idea what might be wrong with wireguard inside wireguard?

Essentially, to the simplified config above I add

/interface wireguard add listen-port=56792 mtu=1340 name=wgSF private-key="*"
/ip route add dst-address=[wgSF_ip] gateway=wgCF pref-src=172.16.0.2
/interface wireguard peers add allowed-address=0.0.0.0/0 endpoint-address=[wgSF_ip] endpoint-port=* interface=wgSF persistent-keepalive=25s public-key="*"
/ip address add address=10.14.0.2/16 interface=wgSF network=10.14.0.0
/ip firewall filter add action=return chain=fwfastt out-interface=wgSF src-address=192.168.0.32/27 place-before=2
/ip firewall filter add action=return chain=fwfastt dst-address=192.168.0.32/27 in-interface=wgSF place-before=3
/ip firewall nat add action=masquerade chain=srcnat out-interface=wgSF
/routing table add fib name=sf
/ip route add dst-address=0.0.0.0/0 gateway=wgSF routing-table=sf
/routing rule add action=lookup-only-in-table interface=lan1 src-address=192.168.0.32/27 table=sf place-before=1
/ip route set X dst-address=104.16.123.96/32 gateway=wgSF pref-src=10.14.0.2

where X is the number of route entry for 104.16.123.96/32 that previously went through gateway=wgCF.


From router itself this works fine and signifies that the request went through wgSF which went through wgCF (wgSF is unreachable without wgCF).

:put [ /tool fetch url="https://www.cloudflare.com/cdn-cgi/trace" address=104.16.123.96 mode=https output=user as-value ]

From device with client-id=cf (which now uses routing-table=sf as I placed routing rule before the cf one), curl establishes TCP connection but gets stuck forever on initiating TLS handshake.

I’m wondering if I’m missing more packets that should not go to fasttrack.

6 
:put [ /tool fetch url="https://www.cloudflare.com/cdn-cgi/trace" address=104.16.123.96 mode=https output=user as-value ]

works and signifies traffic went through wgSF.

Both curls get stuck like that

21:52:59.826808 * Added www.cloudflare.com:443:104.16.123.96 to DNS cache
21:52:59.827143 * Hostname www.cloudflare.com was found in DNS cache
21:52:59.827267 *   Trying 104.16.123.96:443...
21:52:59.908752 * Connected to www.cloudflare.com (104.16.123.96) port 443
21:52:59.910547 * ALPN: curl offers h2,http/1.1
21:52:59.910773 * TLSv1.3 (OUT), TLS handshake, Client hello (1):
21:52:59.921722 *  CAfile: /opt/local/share/curl/curl-ca-bundle.crt
21:52:59.921753 *  CApath: none
21:56:15.313131 * Recv failure: Connection reset by peer
21:56:15.313185 * OpenSSL SSL_connect: Connection reset by peer in connection to www.cloudflare.com:443 
21:56:15.313217 * Closing connection
curl: (35) Recv failure: Connection reset by peer

Pinging 104.16.123.96 from client works and from latency I can imply it goes through wgSF.


Entirely disabling fasttrack rule does not change behaviour.


Overall config looks like that

# * by RouterOS 7.13
# model = CRS309-1G-8S+
/interface bridge add frame-types=admit-only-vlan-tagged name=bridge1 vlan-filtering=yes
/interface ethernet set [ find default-name=sfp-sfpplus1 ] l2mtu=10218
/interface ethernet set [ find default-name=sfp-sfpplus2 ] l2mtu=10218 mtu=10134
/interface wireguard add listen-port=56791 mtu=1420 name=wgCF private-key="*"
/interface wireguard add listen-port=56792 mtu=1340 name=wgSF private-key="*"
/interface vlan add interface=bridge1 mtu=10134 name=lan1 vlan-id=2
/interface vlan add interface=bridge1           name=wan1 vlan-id=3
/interface ethernet switch set 0 l3-hw-offloading=yes qos-hw-offloading=yes
/interface ethernet switch port set sfp-sfpplus1 l3-hw-offloading=no
/ip pool add name=dhcp_pool_lan_devices ranges=192.168.0.0/27
/ip pool add name=dhcp_pool_lan_cf      ranges=192.168.0.32/27
/ip pool add name=dhcp_pool_lan_unknown ranges=192.168.0.96/27
/ip dhcp-server add address-pool=dhcp_pool_lan_unknown authoritative=yes interface=lan1 lease-time=1d name=dhcp_lan
/routing table add fib name=cf
/routing table add fib name=sf
/interface bridge port add bridge=bridge1 comment=WAN1          frame-types=admit-only-untagged-and-priority-tagged interface=sfp-sfpplus1 pvid=3
/interface bridge port add bridge=bridge1 comment=LAN1 edge=yes frame-types=admit-all                               interface=sfp-sfpplus2 pvid=2
/interface ethernet switch l3hw-settings set fasttrack-hw=yes ipv6-hw=yes
/ip settings set allow-fast-path=yes ip-forward=yes
/ipv6 settings set disable-ipv6=no forward=yes
/interface bridge vlan add bridge=bridge1 tagged=bridge1 vlan-ids=2
/interface bridge vlan add bridge=bridge1 tagged=bridge1 vlan-ids=3
/interface wireguard peers add allowed-address=0.0.0.0/0 endpoint-address=*         endpoint-port=* interface=wgCF persistent-keepalive=25s public-key="*"
/interface wireguard peers add allowed-address=0.0.0.0/0 endpoint-address=[wgSF_ip] endpoint-port=* interface=wgSF persistent-keepalive=25s public-key="*"
/ip address add address=192.168.0.1/24 interface=lan1 network=192.168.0.0
/ip address add address=172.16.0.2     interface=wgCF network=172.16.0.2
/ip address add address=10.14.0.2/16     interface=wgSF network=10.14.0.0
/ip dhcp-client add add-default-route=special-classless default-route-distance=10 interface=wan1 use-peer-dns=no use-peer-ntp=no
/ip dhcp-server lease add address=192.168.0.36 client-id=cf mac-address=* server=dhcp_lan
/ip dhcp-server lease add address=192.168.0.6               mac-address=* server=dhcp_lan
/ip dhcp-server network add address=192.168.0.0/24 comment=lan dns-server=1.1.1.1,1.0.0.1 gateway=192.168.0.1
/ip firewall filter add action=jump chain=forward connection-state=established,related jump-target=fwfastt
/ip firewall filter add action=return chain=fwfastt out-interface=wgCF src-address=192.168.0.32/27
/ip firewall filter add action=return chain=fwfastt dst-address=192.168.0.32/27 in-interface=wgCF
/ip firewall filter add action=return chain=fwfastt out-interface=wgSF src-address=192.168.0.32/27
/ip firewall filter add action=return chain=fwfastt dst-address=192.168.0.32/27 in-interface=wgSF
/ip firewall filter add action=fasttrack-connection chain=fwfastt hw-offload=yes
/ip firewall nat add action=masquerade chain=srcnat out-interface=wan1
/ip firewall nat add action=masquerade chain=srcnat out-interface=wgCF
/ip firewall nat add action=masquerade chain=srcnat out-interface=wgSF
/ip route add dst-address=0.0.0.0/0        gateway=wgCF                     routing-table=cf
/ip route add dst-address=104.16.123.96/32 gateway=wgSF pref-src=10.14.0.2
/ip route add dst-address=[wgSF_ip]/32     gateway=wgCF pref-src=172.16.0.2
/ip route add dst-address=0.0.0.0/0        gateway=wgSF                     routing-table=sf
/routing rule add action=lookup                              min-prefix=0                             table=main
/routing rule add action=lookup-only-in-table interface=lan1              src-address=192.168.0.32/27 table=sf
/routing rule add action=lookup-only-in-table interface=lan1              src-address=192.168.0.32/27 table=cf
7

It’s most probably a MTU issue because now the MTU inside the sf tunnel is pretty low (1340). If path MTU discovery (PMTUD) doesn’t work correctly, then it will result in problems that you’ve observed. Normal ping works, because the ping packet is not big enough to exceed the MTU value. It also works from the router, because the router knows about the MTU limit of 1340 of the outgoing interface (wgSF) so can correctly fragment the packets when you ran “/tool fetch”. But the devices in the 192.168.0.32/27 range probably have MTU set very large, it looks like you enabled jumbo frames and the MTU on lan1 is 10134. If PMTUD doesn’t work, big packets will not go through. When establishing a TLS connection, the initial packet can be pretty big.

You can try to force reduce the MSS values for TCP connections going through wgSF. See this section in the documentation: https://help.mikrotik.com/docs/display/ROS/Mangle#Mangle-ChangeMSS

In this case we have IPv4, with an MTU of 1340 the MSS should be at most 1300. You can use the rule from the example provided by the page, just need to replace the name of the outgoing interface:


/ip firewall mangle 
add chain=forward out-interface=wgSF protocol=tcp tcp-flags=syn action=change-mss \
    new-mss=1300 tcp-mss=1301-65535

This only apply for TCP and won’t help UDP connections, however. UDP has no concept of MSS. Only working PMTUD will assure that the problem doesn’t affect UDP. If you have drop rules on your firewall, just make sure that ICMP packets are not blocked (remove rules that say “drop ping” or something like that). ICMP is crucial for PMTUD to work properly.

8

@CGGXANNX, thanks for the earlier help.


@CGGXANNX and others,

Could you kindly advise on the follow-up to the initial question, please?

In the initial question, the default interface for the extra routing table was Wireguard. This meant that the default route gateway in the extra routing table was Wireguard device. This allowed for the aforementioned simple firewall exclusion from hardware offloading via in/out-interface=wgCF.

/ip firewall filter add action=jump chain=forward connection-state=established,related jump-target=fwfastt
/ip firewall filter add action=return chain=fwfastt out-interface=wgCF src-address=192.168.0.32/27
/ip firewall filter add action=return chain=fwfastt dst-address=192.168.0.32/27 in-interface=wgCF
/ip firewall filter add action=fasttrack-connection chain=fwfastt hw-offload=yes

Let’s assume that the gateway of the default route in the extra routing table is a direct-connected IP (e.g. 192.168.0.2) now instead. I.e.

/ip route add dst-address=0.0.0.0/0 gateway=192.168.0.2 routing-table=cf

Is there a way to rephrase the firewall hardware offloading exclusion rules for such a case? Essentially, change in/out-interface=wgCF to in/out-address=192.168.0.2 (not to be confused with dst/src-address functionality respectively which is semantically different). But there is no such thing as in/out-address.

I cannot simply change in/out-interface=wgCF to in/out-interface=lan1 as this would also exclude traffic from hardware offloading that would be matched by the first routing rule (i.e. routes with prefix>0 from the main routing table). In the simplified config in the initial post, this is only

#automatically created
#/ip route add dst-address=192.168.0.0/24 gateway=lan1 pref-src=192.168.0.1
/ip route add dst-address=104.16.123.96/32 gateway=wgCF pref-src=172.16.0.2

The first one is performance-sensitive and I don’t want to exclude it from hardware-offloading for 192.168.0.32/27 clients.

Is there a way to rephrase the firewall hardware offloading exclusion rules without first including every route from the main routing table for 192.168.0.32/27? I.e. without doing the following

/ip firewall filter add action=jump chain=forward connection-state=established,related jump-target=fwfastt
/ip firewall filter add action=fasttrack-connection chain=fwfastt src-address=192.168.0.32/27 dst-address=192.168.0.0/24 hw-offload=yes
/ip firewall filter add action=fasttrack-connection chain=fwfastt src-address=192.168.0.0/24 dst-address=192.168.0.32/27 hw-offload=yes
# and so on for every route in the main routing table
/ip firewall filter add action=return chain=fwfastt out-interface=lan1 src-address=192.168.0.32/27
/ip firewall filter add action=return chain=fwfastt dst-address=192.168.0.32/27 in-interface=lan1
/ip firewall filter add action=fasttrack-connection chain=fwfastt hw-offload=yes