LAN traffic with local VPN enabled slow to start and going through bridge

RouterOS Beginner Basics
1

Just got the E60iUGS and I’m a bit confused on what’s happening here, presumably some configuration issue on my Windows client but I'm lost and with it happening after the router change this seems like a good place to start.

I have two devices, a client PC and a server, connected through a 2.5Gbit switch that sits behind the router I just set up.
When I have my Proton VPN turned on on my PC (no configuration with it router side), initiating traffic to IPs on the server is very slow and some parts of it I can see as going through the bridge router interface limiting my bandwidth.

The VPN is set to exclude the IPs, and I think excludes the whole private range with my settings anyway, as this has never been a problem before with the setup with the old (1Gbit port) TP-Link router always getting the full 2.5Gbit between the two devices.

When I try iperf I top out at ~600Mbps after connecting for 10 seconds and can see all of the traffic on the bridge, then when I try a samba transfer I get significantly higher speeds but 30Mbps still shows up on the bridge doing something.
When disconnected from the VPN the connections are instant to make; iperf3 sees 2.37 Gbps and samba goes up to about 190-200MB/s from ~150MB/s

A ping gets me <1ms, but connecting to a simple page hosted on the server with curl takes 7s. This happens to all IPs that go through the server's interfaces i.e. all my VMs that are on IPs at 10.1.1.0/24 the router has a route for. Firewall also came to mind but I wouldn't expect the traffic to hit the router in the first place, and I'm not seeing any of the firewall filter rules bytes/packets increasing significantly when running iperf

PS C:\Users\numerlor> ping 192.168.0.21

Pinging 192.168.0.21 with 32 bytes of data:
Reply from 192.168.0.21: bytes=32 time<1ms TTL=64
Reply from 192.168.0.21: bytes=32 time<1ms TTL=64
Reply from 192.168.0.21: bytes=32 time<1ms TTL=64
Reply from 192.168.0.21: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.0.21:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

...

numerlor@DESKTOP-IO3KPUN:/mnt/c/Users/numerlor$ time curl -s -o /dev/null 192.168.0.20

real    0m6.933s
user    0m0.005s
sys     0m0.000s
numerlor@DESKTOP-IO3KPUN:/mnt/c/Users/numerlor$ time curl -s -o /dev/null 192.168.0.2 # vpn still on

real    0m0.038s
user    0m0.005s
sys     0m0.000s
numerlor@DESKTOP-IO3KPUN:/mnt/c/Users/numerlor$ time curl -s -o /dev/null 192.168.0.20 # vpn off

real    0m0.009s
user    0m0.006s
sys     0m0.000s
config 
2025-10-07 01:39:56 by RouterOS 7.20

# software id = 9Q7M-E4GR

# 

# model = E60iUGS

# serial number = HJQ0AY43ZV3

/interface bridge
add admin-mac=04:F4:1C:29:D5:E9 auto-mac=no comment=defconf name=bridge
/interface ethernet
set \[ find default-name=ether1 \] mac-address=A8:42:A1:EF:0F:6C
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/disk settings
set auto-media-interface=bridge auto-media-sharing=yes auto-smb-sharing=yes
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=sfp1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/ip address
add address=192.168.0.1/24 comment=defconf interface=bridge network=192.168.0.0
add address=0/27 interface=ether1 network=0
/ip dhcp-client
add comment=defconf disabled=yes interface=ether1
/ip dhcp-server
add address-pool=default-dhcp disabled=yes interface=bridge name=defconf
/ip dhcp-server network
add address=192.168.0.0/24 comment=defconf dns-server=192.168.0.1 gateway=
192.168.0.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=0
/ip dns static
add address=192.168.0.1 comment=defconf name=router.lan type=A
/ip firewall filter
add action=accept chain=input comment=
"defconf: accept established,related,untracked" connection-state=
established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=
invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=
"defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" 
in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" 
ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" 
ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" 
connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=
"defconf: accept established,related, untracked" connection-state=
established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=
invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" 
connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=
out,none out-interface-list=WAN
/ip route
add comment=VMs disabled=no distance=1 dst-address=10.1.1.0/24 gateway=
192.168.0.21 routing-table=main scope=30 suppress-hw-offload=no 
target-scope=10
add comment=VPN disabled=no dst-address=10.8.0.0/24 gateway=192.186.0.21 
routing-table=main suppress-hw-offload=no
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=
"defconf: accept established,related,untracked" connection-state=
established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=
invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" dst-port=
33434-33534 protocol=udp
add action=accept chain=input comment=
"defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp 
src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 
protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=
ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=
ipsec-esp
add action=accept chain=input comment=
"defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=
"defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=fasttrack-connection chain=forward comment="defconf: fasttrack6" 
connection-state=established,related
add action=accept chain=forward comment=
"defconf: accept established,related,untracked" connection-state=
established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=
invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" 
src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" 
dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" 
hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 
protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=
ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=
ipsec-esp
add action=accept chain=forward comment=
"defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=
"defconf: drop everything else not coming from LAN" in-interface-list=!LAN
/system clock
set time-zone-name=Europe/...
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
server `ip a`

This is a bit of a mess as it's carried over 2 machines, has incus set up, and I stopped with configuring anything on it the moment it worked. But like I said it was fine up until I changed out the router

numerlor@serv:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp104s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 74:56:3c:0f:d6:cc brd ff:ff:ff:ff:ff:ff
    altname enx74563c0fd6cc
    inet 192.168.0.21/24 brd 192.168.0.255 scope global enp104s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::7656:3cff:fe0f:d6cc/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
4: enp104s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 74:56:3c:0f:d6:cd brd ff:ff:ff:ff:ff:ff
    altname enx74563c0fd6cd
    inet 192.168.0.20/24 brd 192.168.0.255 scope global enp104s0f1
       valid_lft forever preferred_lft forever
5: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.8.0.1/24 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::a159:ebfc:f712:292c/64 scope link stable-privacy proto kernel_ll
       valid_lft forever preferred_lft forever
6: incusbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 10:66:6a:56:ec:02 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.1/24 brd 10.1.1.255 scope global incusbr0
       valid_lft forever preferred_lft forever
7: tapea5179ef: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master incusbr0 state UP group default qlen 1000
    link/ether ba:08:33:81:18:22 brd ff:ff:ff:ff:ff:ff
9: vethffdcdff9@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master incusbr0 state UP group default qlen 1000
    link/ether 26:61:36:92:db:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
11: veth84e4d136@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master incusbr0 state UP group default qlen 1000
    link/ether c2:be:bd:d3:5a:ca brd ff:ff:ff:ff:ff:ff link-netnsid 1
13: vethcf55039e@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master incusbr0 state UP group default qlen 1000
    link/ether da:d2:40:ac:87:a9 brd ff:ff:ff:ff:ff:ff link-netnsid 2
14: tapa7fc3e88: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master incusbr0 state UP group default qlen 1000
    link/ether 4e:17:c1:30:d3:df brd ff:ff:ff:ff:ff:ff

iperf

samba transfer

2

Traffic via bridge almost certainly means it's a routed traffic (e.g. to/from internet ... also through your VPN as that's "normal" IP connection for your router).

I somehow doubt that the problem is router's configuration ... but to rule it out, can you try re-doing the tests while VPN is not established? If you get normal results this way, then it's something about configuration on client PC.

3

Hi, yeah it works as expected when the VPN is off, but I'm not sure what could be the cause of this when it's between two local IPs e.g. iperf goes through 192.168.0.103 to 192.168.0.20.
No connections to other local IPs go through the router as pc<->laptop gets full gigabit, and connections to other devices that host management pages are instant while even the simplest page on the server takes seconds to start communicating. It was fully switched before I changed the router even with VPN on as the speeds were 2.5Gbps while the router only had 1Gbit ports

Just for reference the devices I tried speeds with are connected like this, over the same switch going to laptop always works while the server IPs go through router

Drawing1
Drawing11300×800 40.2 KB

4

Thoroughly check setting of (running) VPN. VPNs make it possible to block all local connectivity (or make all traffic pass VPN concentrator for FW inspection ...) and this sometimes (if not most of times) depends on the VPN concentrator settings.

Or it could simply be that VPN concentrator pushes a route towards IP subnet which overlaps your own IP subnet. Or something other.

What I'm trying to write is that running VPN on your PC doesn't alter configuration neither on switches nor on router (and neither on server). And you'd have to verify that traffic between PC and server (when VPN is not active) actually passes router. And if that's so, then something is pushing traffic towards router (router doesn't have proxy-arp enabled and since it has only single interface facing LAN traffic should not pass router if switches do their job properly). PC could send traffic towards server via router, but return traffic would likely go directly.
I'd also check status of the "common switch" (check FDB table if it contains correct entry for server's MAC with correct switch port).

5

Can you run route print on the Windows PC, both when the VPN is not running and when it's running?

You should see routes with some destinations having On-link as gateway, while other routes have an IP address (such as the one of your router) as gateway. From your description, it's look like when the VPN is running, it wrongly installs a route for the exclusion range with the hEX S (2025)'s IP address as gateway instead of On-link. This would explain the behavior:

  • When you run iperf3 as in your screenshot, you are running the test where most of the traffic are being sent from the PC to serv.lan. In the direction from the server to the PC there are only mostly the ACK packets and some iperf3 book-keeping. You are limited by the forwarding performance of the hEX S (2025). 550-600Mbps appears to be its routing limit without fasttrack. And this traffic goes through the router.

    -> Try running the test with the -R parameter to see if it's faster in the reverse direction (server sends).

  • When you download from the SMB share to your PC, your PC are mostly sending ACK packets towards the server, these packets have to be routed through the router (because of the wrong route on the Windows PC), but being only ACKs, they only amount to 30Mbps that need to go through the router. In the other direction, when the server sends the packets with the data, which is the bulk of the traffic, it doesn't have the route problem, and can send the packets on-link, only through the switch and not using the router.

    There is no bottleneck when the server sends packets to your PC.

  • The old router probably didn't have the issue because when it detects that your PC is trying to send packets destined to the same subnet through it, it sends back ICMP redirect packets to your PC to instruct your PC to use the direct route (to use the server's IP address as gateway directly). RouterOS doesn't do that and doesn't send ICMP redirect (that's why the router running RouterOS can be used to filter traffic within the same subnet if it's used as gateway).

  • About the initial delay at the start of the connection, I am not sure about that, but it might have something to do with connection tracking on the router. When the PC sends the SYN packet to the server, while using the wrong route, this packet is sent through the router. The router normally create a connection tracking (conntrack) entry for this TCP connection. The problem is that the SYN,ACK packet that the server sends back never go through the router, but is directly sent to the PC on-link. When the PC send the next packet, this packet might be put on hold because the router is still wating for the SYN,ACK response. The delay might be related to this conntrack setting:

    image
    image504×571 7.77 KB

    After the 5 seconds, the router gives up and forwards the packet that is waiting in the queue. And because of the default loose-tcp-tracking=yes setting, this packet, which is an ACK packet in response to SYN,ACK will switch the conntrack entry's state to established, and further packets are no longer have to wait for tcp-syn-sent-timeout.

    When you immediately make the 2nd curl request to the same server, it reuses the existing TCP connection (probably in TIME_WAIT state or something like that) and doesn't have the initial delay.

    When you send pings, it doesn't have the delay, because it's not TCP there is no need to wait for the duration of tcp-syn-sent-timeout.

In conclusion: You should really check the Windows route table that has been modifed by the VPN client. You might need to manually install On-link routes if the routes are wrong to improve the performance.

1 Like
6

Ah thank you this led me down the right road.

I've had the server IPs excluded from the VPN, which I thought I only added when testing this, but instead I most likely did it when testing DNS a couple days back, and when I disabled the newly added exlusions the previous were still on.

With the local IPs explicitly excluded from the VPN with split tunneling route prints them with

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
     192.168.0.20  255.255.255.255      192.168.0.1     192.168.0.15     40
     192.168.0.21  255.255.255.255      192.168.0.1     192.168.0.15     40

so Windows was directly sending them to the gateway.
The 10.1.1.0/24 addresses are a still though, not sure what I can do about that, they usffer the same problem but that can't be just switched as they go through a static route on the router. I've set up the route manually just now to

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
         10.1.1.0    255.255.255.0      192.168.0.1     192.168.0.15     21
         10.2.0.2  255.255.255.255         On-link          10.2.0.2    256
===========================================================================
7

You can add static routes on the Windows device itself. Adding a persistent route for destination 10.1.1.0/24 gateway 192.168.0.21 on Windows should make it bypass the hEX S (2025).

Something like:

route -p add 10.1.1.0 MASK 255.255.255.0 192.168.0.21
8

hmm nevermind, I thought it'd be an issue for my other clients where I can't set up the routes but it seems to only go through the bridge when I use the VPN's routes

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
         10.1.1.5  255.255.255.255      192.168.0.1     192.168.0.15     40
        10.1.1.10  255.255.255.255      192.168.0.1     192.168.0.15     40
        10.1.1.11  255.255.255.255      192.168.0.1     192.168.0.15     40
        10.1.1.15  255.255.255.255      192.168.0.1     192.168.0.15     40
        10.1.1.20  255.255.255.255      192.168.0.1     192.168.0.15     40
       10.1.1.115  255.255.255.255      192.168.0.1     192.168.0.15     40
         10.2.0.2  255.255.255.255         On-link          10.2.0.2    256
===========================================================================
Persistent Routes:
  None

while the single route I had posted above works fine at full 2.5Gbit. So all's good

9

OK, I was too quick to dismiss this as it may have been Windows doing some funky caching. The windows desktop PC is completely messed up now where I can't understand what it's doing as it behaves differently compared to everything else on the network even what I maually delete the routes.

Main problem - other clients like my TV where I can't modify anything regarding routing get the connection delay problem when going to the 10.1.1.0/24 range of IPs.
The 10.1.1.0/24 is set up as a static route on the router to 192.186.0.21.

@CGGXANNX Changing the Syn Sent Timeout "fixes" it, could it be because the server is ultimately responding out of the 192.186.0.21 interface so the traffic is just switched and bypasses the router on the return path like what you said for the original issue? If that is the case what would be the correct fix - disabling connection tracking on the 10.1.1.0/24 range?

First is from VM, second is from the VM host

root@jellyfin:~# ip route
default via 10.1.1.1 dev enp5s0 proto dhcp src 10.1.1.15 metric 100
10.1.1.0/24 dev enp5s0 proto kernel scope link src 10.1.1.15 metric 100
10.1.1.1 dev enp5s0 proto dhcp scope link src 10.1.1.15 metric 100
root@jellyfin:~#
exit
numerlor@serv:~$ ip route
default via 192.168.0.1 dev enp104s0f0 onlink
10.1.1.0/24 dev incusbr0 proto kernel scope link src 10.1.1.1
10.8.0.0/24 dev tun0 proto kernel scope link src 10.8.0.1
192.168.0.0/24 dev enp104s0f0 proto kernel scope link src 192.168.0.21
192.168.0.0/24 dev enp104s0f1 proto kernel scope link src 192.168.0.20 linkdown
numerlor@serv:~$
10

The server always returns the packet "on-link", because the destinations of the response packets are all in the 192.168.0.0/24 range, which for it don't need the router to be involved.

But the client devices when sending packets towards 10.1.1.0/24 all send through the MikroTik hEX S (2025) router, unless they have a route telling them to use 192.168.0.21 as gateway for those range. For Windows I've added the command in the above post:

The delay is caused by the router waiting for the SYN,ACK response to the initial SYN packet, before forwarding more packets of the connection (in the direction client -> server, the router doesn't see any of the packets in the server -> client direction at all). Because the router never sees those SYN,ACK response, it has to wait for the timeout.

But I don't think it's wise to reduce that delay. What you can do for the devices that don't allow manual route adjustments, like your TV, is to use DHCP to announce the extra routes using DHCP Option 121. Here is an example on MikroTik documentation:

But it's probably better to use a tool like this one to generate the option string for you:

Configuring Classless Static Route Option · Medo's Home Page

image
image1120×679 39.5 KB

For example, to generate the route list for your case, as seen in the screenshot, you need to create this Option 121 entry:

/ip dhcp-server option
add code=121 name=classless-static-route-option value=0x00C0A80001180A0101C0A80015

This option can then be set on the individual leases, or add to the IP -> DHCP Server -> Networks entry corresponding to the subnet. Once the clients renew their addresses (or if forced by you), if they support DHCP Option 121, then they will also have the destination=10.1.1.0/24 gateway=192.168.0.21 route installed.