SMB over Wireguard very slow

dru · August 11, 2024, 11:11am

here comes yet another (i suspect) MTU issue.
I've tried to figure it out and finally got confused, so my tests lost a structure a became a random lock-picking.
Please help to put me back on tracks.

i have two sites: D and M with a wireguard tunnel between them. both sites have: hap ac2 with a NAS attached on LAN.
the NAS-NAS speed doesn't exceed 5.5Mbps [SMB transfers]

BTest between MTs over WG:
tcp
D -> M: 40 Mbps
M -> D: 160 Mbps

udp
D -> M: 51 Mbps
M -> D: 260 Mbps

my first success was few weeks ago when I lowered the WG MTU to ~1260 on both sites and disabled the fasttrack on D (there was no fasttrack on M) and i got some decent speeds.
But later without any obvious changes the speed dropped to current 5.5Mbps. Couldn't replicate the success anymore.

now the setup is:
WG MTU: 1350 [both sites]
fasttrack: both disabled
mangle: add action=change-mss chain=forward new-mss=1350 out-interface=wireguardD passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=!0-1350
speed: 5.5 Mbps

Any ideas?
thank you!

Site D:

2024-08-10 18:47:45 by RouterOS 7.15.2

model = RBD52G-5HacD2HnD

/interface bridge
add admin-mac=aa:94:E4 auto-mac=no comment=defconf name=
"local bridge" port-cost-mode=short

/interface wireguard
add listen-port=13231 mtu=1350 name=wireguardD

/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN

/routing bgp template
set default disabled=no output.network=bgp-networks

/interface bridge port
add bridge="local bridge" comment=defconf ingress-filtering=no interface=
ether2 internal-path-cost=10 path-cost=10
add bridge="local bridge" comment=defconf ingress-filtering=no interface=
ether3 internal-path-cost=10 path-cost=10
add bridge="local bridge" comment=defconf ingress-filtering=no interface=
ether4 internal-path-cost=10 path-cost=10
add bridge="local bridge" comment=defconf ingress-filtering=no interface=
ether5 internal-path-cost=10 path-cost=10
add bridge="local bridge" comment=defconf ingress-filtering=no interface=
wlan1-2GHz internal-path-cost=10 path-cost=10
add bridge="local bridge" comment=defconf ingress-filtering=no interface=
wlan2-5GHz internal-path-cost=10 path-cost=10

/ip firewall connection tracking
set udp-timeout=10s

/ip neighbor discovery-settings
set discover-interface-list=LAN

/ip settings
set max-neighbor-entries=8192

/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192

/interface list member
add comment=defconf interface="local bridge" list=LAN
add comment=defconf interface=ether1-WAN list=WAN
add interface=wireguardD list=LAN

/interface wireguard peers
add allowed-address=9.9.9.0/24,192.168.46.0/28,192.168.100.0/24
endpoint-address=Pub_M_IP endpoint-port=13231 interface=wireguardD
name=wireguardM public-key=
"lll"
add allowed-address=192.168.46.3/32 client-address=192.168.46.3/32 interface=
wireguardD name="" public-key=
"llll"
add allowed-address=192.168.46.4/32 client-address=192.168.46.4/32 interface=
wireguardD name="" public-key=
"llll"

/ip address
add address=9.9.0.1/24 comment=defconf interface="local bridge" network=
9.9.0.0
add address=192.168.2.2/24 interface=ether1-WAN network=192.168.2.0
add address=192.168.46.2/28 comment=wireguardD interface=wireguardD
network=192.168.46.0

/ip dhcp-server network
add address=9.9.0.0/24 comment=defconf gateway=9.9.0.1 netmask=24

/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4

/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan

/ip firewall filter
add action=accept chain=input comment=
"defconf: accept established,related,untracked" connection-state=
established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=
invalid
add action=drop chain=forward comment="defconf: drop invalid"
connection-state=invalid
add action=fasttrack-connection chain=forward comment="defconf: fasttrack"
connection-state=established,related disabled=yes hw-offload=yes
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=
"defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input comment="accept Wireguard" dst-port=13231
protocol=udp
add action=accept chain=input comment="Accept WAN from M" in-interface-list=
WAN src-address=Pub_M_IP
add action=accept chain=forward comment="wireguard - internetwork pass"
in-interface=wireguardD
add action=accept chain=forward out-interface=wireguardD
add action=accept chain=input in-interface=wireguardD
add action=drop chain=input comment="drop all coming from WAN"
in-interface-list=WAN
add action=accept chain=forward comment="defconf: accept in ipsec policy"
ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy"
ipsec-policy=out,ipsec
add action=accept chain=forward comment=
"defconf: accept established,related, untracked" connection-state=
established,related,untracked
add action=accept chain=forward in-interface-list=LAN
add action=drop chain=forward comment=
"defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat
connection-state=new in-interface-list=WAN

/ip firewall mangle
add action=change-mss chain=forward new-mss=1350 out-interface=
wireguardD passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=!0-1350

/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade"
ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat disabled=yes src-address=192.168.46.0/28
add action=src-nat chain=srcnat disabled=yes out-interface=wireguardD
to-addresses=192.168.46.2

/ip route
add disabled=no distance=5 dst-address=0.0.0.0/0 gateway=192.168.2.1
routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=9.9.9.0/24 gateway=192.168.46.1
routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=192.168.100.1/32 gateway=192.168.46.1
routing-table=main scope=30 suppress-hw-offload=no target-scope=10

>
\
\
**Site M:**

> ```text
# 2024-08-11 04:09:33 by RouterOS 7.15.2
#
# model = RBD52G-5HacD2HnD

/interface bridge
add arp=proxy-arp name=bridge1_net port-cost-mode=short


/interface wireguard
add listen-port=13231 mtu=1350 name=wireguardM

/interface list
add name=wlanU_list
add name=LAN
add name=WAN



/interface bridge port
add bridge=bridge1_net ingress-filtering=no interface=wlan1-2.4 \
    internal-path-cost=10 path-cost=10
add bridge=bridge1_net ingress-filtering=no interface="wlan2 - 5" \
    internal-path-cost=10 path-cost=10
add bridge=bridge1_net ingress-filtering=no interface=ether3 \
    internal-path-cost=10 path-cost=10
add bridge=bridge1_net ingress-filtering=no interface=ether4 \
    internal-path-cost=10 path-cost=10
add bridge=bridge1_net ingress-filtering=no interface="ether2"
add bridge=bridge1_net ingress-filtering=no interface="wlan U" \
    internal-path-cost=10 path-cost=10
add bridge=bridge1_net ingress-filtering=no interface=wlanU5 \
    internal-path-cost=10 path-cost=10

/ip firewall connection tracking
set udp-timeout=10s

/ip neighbor discovery-settings
set discover-interface-list=!dynamic

/ip settings
set max-neighbor-entries=8192

/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192



/interface wireguard peers
add allowed-address=9.9.0.0/24,192.168.46.0/28 endpoint-address=Pub_D_IP \
    endpoint-port=13231 interface=wireguardM name=wireguardD public-key=\
    "zz="
add allowed-address=192.168.46.6/32 client-address=192.168.46.6/32 interface=\
    wireguardM name=" " public-key=\
    "zz"
add allowed-address=192.168.46.5/32 client-address=192.168.46.5/32 interface=\
    wireguardM name=" " public-key=\
    "zz"

/ip address
add address=9.9.9.1/24 interface=bridge1_net network=9.9.9.0
add address=192.168.46.1/28 comment=wireguardM interface=wireguardM \
    network=192.168.46.0



/ip dhcp-server network
add address=9.9.9.0/24 dns-server=8.8.8.8 gateway=9.9.9.1

/ip dns
set allow-remote-requests=yes servers=1.2.160.5

/ip firewall filter
add action=passthrough chain=forward
add action=fasttrack-connection chain=forward connection-state=\
    established,related disabled=yes hw-offload=yes
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid log-prefix="all drop___"
add action=drop chain=forward comment="Deny access to NAS from wlanU" \
    dst-address=9.9.9.9 in-interface-list=wlanU_list
add action=drop chain=input comment="Deny access to Mirotik from wlanU" \
    in-interface-list=wlanU_list
add action=drop chain=forward comment="Deny access to DE from wlanU" \
    dst-address=9.9.0.0/24 in-interface-list=wlanU_list
add action=accept chain=input comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked connection-type=""
add action=accept chain=forward out-interface=wireguardM
add action=accept chain=forward in-interface=wireguardM
add action=accept chain=input comment="Accept input from Wireguard" \
    in-interface=wireguardM
add action=accept chain=input comment="Allow Wireguard" dst-port=13231 \
    in-interface-list=WAN protocol=udp
add action=accept chain=input comment="accept Input from D" \
    in-interface-list=WAN src-address=Pub_D_IP
add action=drop chain=input comment="drop all from WAN" in-interface-list=WAN

/ip firewall mangle
add action=change-mss chain=forward connection-mark="" log-prefix=MSS_ \
    new-mss=1350 out-interface=wireguardM passthrough=yes protocol=tcp \
    tcp-flags=syn tcp-mss=!0-1350

/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1_WAN

/ip route
add check-gateway=ping disabled=no distance=1 dst-address=9.9.0.0/24 gateway=\
    192.168.46.2 routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10

anav · August 11, 2024, 11:58am

When you decide which router is server for handshake and which is client for handshake, let me know and can then assist.

dru · August 12, 2024, 5:53pm

Thank you.

WG tunnel currently works, but I probably lack the understanding why current config is NOK.

anyways, i think remote site (M) should be client and my local site (D) should be server.
on server: to remove Endpoint IP in corresponding peer
on client: no change. peer endpoint IP points to WG server.

Should server be remote? any best practice about this?
my logic is that i can locally manage the server and client will always try to dial-in from remote site.

S8T8 · August 12, 2024, 6:07pm

Hi, can you do an MTU test using your main WAN?
Your ISP is using PPPoE? IPv6?

dru · August 12, 2024, 6:38pm

@S8T8, both sites public IPs are IPv4. no PPPoE i know of.

M site WAN is FTTH GPON
D site WAN is something over Coaxial, giving hundreds of Mbps. I don’t know more at this moment.
Home Network v2.1-Page-3.jpg

S8T8 · August 12, 2024, 7:05pm

Is your WAN MTU 1500 on both sites?
https://www.speedguide.net/analyzer.php

dru · August 12, 2024, 8:46pm

All interfaces MTU (except WG) in both routers are 1500. Both WGs are now 1350.

« SpeedGuide.net TCP Analyzer Results »
Tested on: 2024.08.12
Client OS/browser: Windows 10 (Chrome 127.0.0.0)

TCP options string: 0204058c0103030801010402
MSS: 1420
MTU: 1460
TCP Window: 131840 (not multiple of MSS)
RWIN Scaling: 8 bits (2^8=256)
Unscaled RWIN : 515
Recommended RWINs: 65320, 130640, 261280, 522560, 1045120
BDP limit (200ms): 527 Mbps (53 Megabytes/s)
BDP limit (500ms): 211 Mbps (21 Megabytes/s)
MTU Discovery: ON
TTL: 111
Timestamps: OFF
SACKs: ON
IP ToS: 00000000 (0)

anav · August 12, 2024, 9:01pm

Local Site D ( Server for handshake )

/interface bridge
add admin-mac=aa:94:E4 auto-mac=no comment=defconf name=
“local bridge” port-cost-mode=short

/interface wireguard
add listen-port=13231 mtu=1350 name=wireguardD

/ip settings
set max-neighbor-entries=8192

/interface list member
add comment=defconf interface=“local bridge” list=LAN
add comment=defconf interface=ether1-WAN list=WAN
add interface=wireguardD list=LAN

/interface wireguard peers
add allowed-address=192.168.46.2/32,9.9.9.0/24 interface=wireguardD
name=wireguardM public-key=“lll”
add allowed-address=192.168.46.3/32 client-address=192.168.46.3/32 interface=
wireguardD name=“admin-laptop-remote” public-key=“llll”
add allowed-address=192.168.46.4/32 client-address=192.168.46.4/32 interface=
wireguardD name=“admin-smartphone-remote” public-key=“llll”

/ip address
add address=9.9.0.1/24 comment=defconf interface=“local bridge” network=
9.9.0.0
add address=192.168.2.2/24 interface=ether1-WAN network=192.168.2.0
add address=192.168.46**.1/24** comment=wireguardD interface=wireguardD
network=192.168.46.0

REMOVE THIS old default setting.
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan

/ip firewall address-list
add address=9.9.0.X list=Authorized comment=“local admin PC”
add address=9.9.0.X list=Authorized comment=“local admin laptop”
add address=9.9.0.X list=Authorized comment=“local admin smartphone/ipad”
add address=192.168.46.3 list=Authorized comment=“remote wireguard admin laptop”
add address=192.168.46.4 list=Authorized comment=“remote wireguard admin smartphone/ipad”
add address=9.9.9.A list=Authorized comment=“admin at remote router site”

/ip firewall filter
add action=accept chain=input connection-state=established,related,untracked
add action=drop chain=input connection-state=invalid
add action=accept chain=input protocol=icmp
add action=accept chain=input dst-address=127.0.0.1
add action=accept chain=input comment=“wireguard handshake”" dst-port=13231 protocol=udp
add action=accept chain=input comment=“admin access” src-address-list=Authorized
add action=accept chain=input comment=“users to services” in-interface-list=LAN dst-port=53 protocol=udp
add action=accept chain=input comment=“users to services” in-interface-list=LAN dst-port=53 protocol=udp
add action=drop chain=input comment=“Drop all else” { put this rule in last }
++++++++++
add action=fasttrack chain=forward connection-state=established,related
add action=accept chain=forward connection-state=established,related,untracked
add action=drop chain=forward connection-state=invalid
add action=accept chain=forward in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward src-address=9.9.0.0/24 out-interface=wireguardD
add action=accept chain=forward dst-address=9.9.0.0/24 in-interface=wireguardD
add action=accept chain=forward comment=“relay peers” in-interface=wireguardD out-interface=wireguardD { allows remote wg users (admin) to reach Router M }
add action=accept chain=forward comment=“port forwarding” connection-nat-state=dstnat
add action=drop chain=forward comment=“Drop all else”

/ip firewall mangle ( MTU rule only required on the client device…)

/ip firewall nat
add action=masquerade chain=srcnat comment=“defconf: masquerade”
ipsec-policy=out,none out-interface-list=WAN

/ip route
add disabled=no distance=5 dst-address=0.0.0.0/0 gateway=192.168.2.1
routing-table=main scope=30 suppress-hw-offload=no target-scope=10 { This route has no basis, there is no such gateway !! }

add disabled=no distance=1 dst-address=9.9.9.0/24 gateway=wireguardD
routing-table=main scope=30 suppress-hw-offload=no target-scope=10

add disabled=no distance=1 dst-address=192.168.100.1/32 gateway=192.168.46.1 \ { This route has no basis, there is no such destination at Router M !! }
routing-table=main scope=30 suppress-hw-offload=no target

anav · August 12, 2024, 9:10pm

Site M ( Remote

/interface wireguard
add listen-port=13231 mtu=1350 name=wireguardM

/interface list
add name=wlanU_list
add name=LAN
add name=WAN

/interface wireguard peers
add allowed-address=9.9.0.0/24,192.168.46.0/24 endpoint-address=Pub_D_IP
endpoint-port=13231 interface=wireguardM name=wireguardD public-key=
/ip address
add address=9.9.9.1/24 interface=bridge1_net network=9.9.9.0
add address=192.168.46**.2/24** comment=wireguardM interface=wireguardM
network=192.168.46.0

/ip firewall filter
Cannot comment because wlanU list is getting in the way of logic.
What does this list consist of. Users on the 9.9.9.0/subnet?
It would appear that you dont want them to be able to access devices in the same subnet???
It would appear that you dont want them to be able to enter the WG tunnel to access the 9.9.0.0 subnet???

/ip firewall mangle
add action=change-mss chain=forward connection-mark=“” log-prefix=MSS_
new-mss=1350 out-interface=wireguardM passthrough=yes protocol=tcp
tcp-flags=syn tcp-mss=!0-1350

/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1_WAN

/ip route
add check-gateway=ping disabled=no distance=1 dst-address=9.9.0.0/24 gateway=
wireguardM routing-table=main scope=30 suppress-hw-offload=no
target-scope=10

bommi · August 12, 2024, 9:14pm

From my experience SMB is somewhat bad when used through an Internet VPN. SMB traffic gets performance issues when the latency is getting higher, what is your latency using this VPN?

dru · August 12, 2024, 10:36pm

@anav
Local Site D ( Server for handshake )

changed WG addresses so server is …46.1, clients are .2 .3 .4 etc
removed static defcon DNS
changed ip route to wireguardD interface instead of IP
removed MSS Change mangle
/ip route

add disabled=no distance=5 dst-address=0.0.0.0/0 gateway=> 192.168.2.1 >
routing-table=main scope=30 suppress-hw-offload=no target-scope=10 { This route has no basis, there is no such gateway !! }

this is site D WAN gateway.

/ip route

add disabled=no distance=1 dst-address=> 192.168.100.1/32 > gateway=192.168.46.1 \ { This route has no basis, there is no such destination at Router M !! }

this is site M WAN gateway, to be reachable from local D site through WG.

Local Site M ( Clientfor handshake )

changed WG client addresses to .2
wlanU - guest wifi in M site. Tried to isolate them from NAS_M, WG and mtik_M. Allow only internet.
ip route to wireguardM instead of IP
tried Clamp MSS as per your reply on other similar topics + fasttrack=on + MTU1420(or 1380)
tried WG MTU=1380, mangle rule to clamp to pmtu and change to mss=1340, with and without fasttrack
tried WG MTU=1500

Still can’t get more than 5.5Mbps on NAS-NAS smb transfer.

@bommi, NAS-NAS ping is ~47ms.

anav · August 12, 2024, 10:48pm

Why?

/ip route
add disabled=no distance=1 dst-address=192.168.100.1/32 gateway=192.168.46.1 \ { This route has no basis, there is no such destination at Router M !! }

this is site M WAN gateway, to be reachable from local D site through WG.

That is INCORRECT, remove.
If you want to reach the M router from the D router, I provided the method and means via Wireguard, and the input chain rule.
The only person requiring access to the M router for configuration purpose are the admins.

Admin devices at Router D
Remote wireguard Admin device
Admin device at Router M
They are all included in the Authorized List.

You cannot exclude people from the same subnet using wifi as you may think wlanU.
What IP addresses to they get, if you can set them statically it may be possible to block them on the access point but not sure.

See if the hapac2 has something like client isolation.
client-isolation (no | yes) Determines whether client devices connecting to this interface are (by default) isolated from others or not.
This policy can be overridden on a per-client basis using access list rules, so a an AP can have a mixture of isolated and non-isolated clients.
Traffic from an isolated client will not be forwarded to other clients and unicast traffic from a non-isolated client will not be forwarded to an isolated one.
Default: no

I think basically in this case, the wifi user cannot see other wifi users on the same subnet, NOR wired users on the same subnet.
Not sure of the affect of this on traffic for wireguard…
Best to just try it and see what happens with a test connection.

anav · August 12, 2024, 10:58pm

FW Rules M router

/ip firewall address-list
add address=9.9.0.X list=Authorized comment=“remote admin PC on RouterD”
add address=9.9.0.X list=Authorized comment=“remote admin laptop on Router D”
add address=9.9.0.X list=Authorized comment=“remote admin smartphone/ipad on RouterD”
add address=192.168.46.3 list=Authorized comment=“remote wireguard admin laptop”
add address=192.168.46.4 list=Authorized comment=“remote wireguard admin smartphone/ipad”
add address=9.9.9.A list=Authorized comment=“local admin device”

/ip firewall filter
add action=accept chain=input connection-state=established,related,untracked
add action=drop chain=input connection-state=invalid
add action=accept chain=input protocol=icmp
add action=accept chain=input dst-address=127.0.0.1
add action=accept chain=input comment=“admin access” src-address-list=Authorized
add action=accept chain=input comment=“users to services” in-interface-list=LAN dst-port=53 protocol=udp
add action=accept chain=input comment=“users to services” in-interface-list=LAN dst-port=53 protocol=udp
add action=drop chain=input comment=“Drop all else” { put this rule in last }
++++++++++
add action=fasttrack chain=forward connection-state=established,related
add action=accept chain=forward connection-state=established,related,untracked
add action=drop chain=forward connection-state=invalid
add action=accept chain=forward in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward src-address=9.9.9.0/24 out-interface=wireguardM
add action=accept chain=forward dst-address=9.9.9.0/24 in-interface=wireguardM
add action=accept chain=forward comment=“port forwarding” connection-nat-state=dstnat
add action=drop chain=forward comment=“Drop all else”

anav · August 12, 2024, 10:59pm

Would be curious to see what iPerf gives you for a speed across wireguard.

S8T8 · August 12, 2024, 11:11pm

anav is very knowledge, but try also:

/interface wireguard set [find] mtu=1400
/ip firewall mangle add action=change-mss chain=forward new-mss=1360 out-interface=WireGuard protocol=tcp tcp-flags=syn tcp-mss=1361-65535

anav · August 13, 2024, 12:19am

Well for starters both sides of the tunnel should have the same MTU setting.
Then at the client peer for handshake side (M router) side try these two options…

add action=change-mss chain=forward comment=“Clamp MSS to PMTU for Outgoing packets” new-mss=clamp-to-pmtu out-interface=wireguardM passthrough=yes protocol=tcp tcp-flags=syn
OR
add action=change-mss chain=forward new-mss=1380 out-interface=wireguardM protocol=tcp tcp-flags=syn tcp-mss=1381-65535

If that doesnt work then we are left with trying to put wireguard inside an L2TP tunnel, where MTU settings can be massaged.

dru · August 13, 2024, 10:26am

Firstly thank you both for your time to check this!

That is INCORRECT, remove.

this route is required for management of ISP router on M site, that is the next WAN hop for M router. I use it to check/forward ports from the M_public_ip (on ISP GPON router) towards M mikrotik.

You cannot exclude people from the same subnet using wifi as you may think wlanU.

sad news . I’ll have to study and rethink the isolation strategy and probably test it first at home. Thanks for the tips here.

Clamp MSS to PMTU or change MSS to a secific value

i tried playing with this mangle rule in different combinations with different MTUs, but it doesnt seem to bring any improvements.

Would be curious to see what iPerf gives you for a speed across wireguard.

i/m now trying to figure out how to install iperf on my old DNS325 NAS. i’ll share news when i manage to do it

WG over L2TP

I had a working L2TP/IPSec before WG, but also with some speed issues. Are there some limitations Cons for WG over L2TP solution?

anav · August 13, 2024, 10:44am

Limitiations on L2TP dont think so…

You are missing the point, one should NOT manage the router from the www side.
The admin wireguards to the M Router and then accesses the config from inside the M router.