Approximately 5s delay in TCP connections when using a static route via an address on bridge
Posted: Mon Dec 28, 2020 8:59 pm
My home router is a hEX RB750Gr3 running RouterOS v6.48. The LAN is 192.168.20.0/24. I'm running WireGuard on a Linux server (eth0: 192.168.20.10 wg0:192.168.21.1) for "roadwarrior" access to my home network as well as linking up with off-site backup hosts. Clients access this Linux server from the internet through a port forward on the RB750Gr3.
If it's helpful, here's a simplified diagram of my network.
The WireGuard network is 192.168.21.0/24. Since I need to access WireGuard hosts from the LAN, I've dispensed with the usual masquerade ifup/ifdown iptables rules on the Linux host and added a static route on the RB750Gr3 for 192.168.21.0/24 via 192.168.20.10. This setup mostly works, but connections take like 5 seconds to start moving data from 192.168.20.0/24 to 192.168.21.0/24 after the initial TCP handshake. This problem does not arise with traffic to and from the internet.
Consider the following example
The WireGuard server is also running an HTTP server for internal use. To take WireGuard out of the picture, we're sending an HTTP GET request to the WireGuard server's wg0 interface (192.168.21.1) rather than something like the offsite backup server (192.168.21.10). We identify the routing on the RB750Gr3 as the culprit by adding a static route on our testing host which bypasses the RB750Gr3. When we do this, the delay goes away.
Here's a screenshot of a representative packet capture. The TCP handshake goes fine, but things go downhill after the testing host sends the HTTP GET.
A lightly redacted copy of my configuration follows:
If it's helpful, here's a simplified diagram of my network.
The WireGuard network is 192.168.21.0/24. Since I need to access WireGuard hosts from the LAN, I've dispensed with the usual masquerade ifup/ifdown iptables rules on the Linux host and added a static route on the RB750Gr3 for 192.168.21.0/24 via 192.168.20.10. This setup mostly works, but connections take like 5 seconds to start moving data from 192.168.20.0/24 to 192.168.21.0/24 after the initial TCP handshake. This problem does not arise with traffic to and from the internet.
Consider the following example
Code: Select all
$ time curl http://192.168.21.1 > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 612 100 612 0 0 91 0 0:00:06 0:00:06 --:--:-- 175
real 0m6.742s
user 0m0.020s
sys 0m0.038s
$ sudo ip route add 192.168.21.0/24 via 192.168.20.10
$ time curl http://192.168.21.1 > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 612 100 612 0 0 199k 0 --:--:-- --:--:-- --:--:-- 298k
real 0m0.046s
user 0m0.022s
sys 0m0.021s
Here's a screenshot of a representative packet capture. The TCP handshake goes fine, but things go downhill after the testing host sends the HTTP GET.
A lightly redacted copy of my configuration follows:
Code: Select all
# dec/28/2020 12:47:30 by RouterOS 6.48
# software id = REDACTED
#
# model = RouterBOARD 750G r3
# serial number = REDACTED
/ip pool
add name=dhcp ranges=192.168.20.10-192.168.20.254
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge name=defconf
/ip address
add address=192.168.20.1/24 comment=defconf interface=ether2 network=192.168.20.0
/ip dhcp-client
add comment=defconf disabled=no interface=ether1
/ip dhcp-server lease
====REDACTED====
/ip dhcp-server network
add address=192.168.20.0/24 comment=defconf gateway=192.168.20.1 netmask=24
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.20.1 comment=defconf name=router.lan
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=dst-nat chain=dstnat comment=wireguard dst-port=51820 protocol=udp to-addresses=192.168.20.10 to-ports=51820
/ip route
add distance=1 dst-address=192.168.21.0/24 gateway=192.168.20.10 pref-src=192.168.20.1
/ip service
====REDACTED====