4011 ROS 7.20.5 - slow WireGuard speed in download

Hardware
1

Hello guys, I have a weird issue on 7.20.5 and 4011 - new installation new equipment.

I am on a working 300/150 optics line and behind the router clients get that speed.

However when using WireGuard tunnel I get about 20-30 megabit/s to the router and normal 80-100 megabit/s speed from it when copying files (Synology fileserver behind - functions normally on LAN/WiFi).

I am not sure where to look, I have no rate limiting options or similar and given that clients communicate normally behind the router, not sure where the error might be. I even tried different ports. Might be ROS itself?

Here is the config (IPV& - disabled and removed, WG peeers and BTH peers removed) - as you can see, it is as vanilla as it gets:

/interface bridge
add auto-mac=no name=LOCAL
add name=WIRELESS-GUESTS
/interface ethernet
set [ find default-name=ether1 ] name=ether1-WAN
set [ find default-name=ether2 ] name=ether2-SERVER
/interface wifi
set [ find default-name=wifi1 ] channel.band=5ghz-ac .deprioritize-unii-3-4=yes .skip-dfs-channels=all .width=20/40/80mhz configuration.country=Croatia .installation=indoor .mode=ap .ssid=TEVE
datapath.bridge=LOCAL disabled=no name=wifi1-TEVE security.authentication-types=wpa2-psk,wpa3-psk .ft=yes .ft-over-ds=yes
add configuration.mode=ap .ssid=GUESTS datapath.bridge=WIRELESS-GUESTS .client-isolation=yes disabled=no master-interface=wifi1-TEVE name=wifi2-GUESTS
security.authentication-types=wpa3-psk
/interface wireguard
add comment=back-to-home-vpn listen-port=26917 mtu=1420 name=back-to-home-vpn
add listen-port=13231 mtu=1420 name=wireguard1
add listen-port=53123 mtu=1420 name=wireguard2-53123
/interface list
add name=WAN
add name=LAN
/ip pool
add name=LOCAL ranges=10.11.12.30-10.11.12.254
add name=WIRELESS-GUESTS ranges=10.11.13.2-10.11.13.254
/ip dhcp-server
add address-pool=LOCAL interface=LOCAL name=LOCAL
add address-pool=WIRELESS-GUESTS interface=WIRELESS-GUESTS lease-time=8h30m name=WIRELESS-GUESTS
/port
set 0 name=serial0
set 1 name=serial1
/disk settings
set auto-media-interface=LOCAL auto-media-sharing=yes auto-smb-sharing=yes
/interface bridge port
add bridge=LOCAL comment=defconf interface=ether2-SERVER
add bridge=LOCAL comment=defconf interface=ether3
add bridge=LOCAL comment=defconf interface=ether4
add bridge=LOCAL comment=defconf interface=ether5
add bridge=LOCAL comment=defconf interface=ether6
add bridge=LOCAL comment=defconf interface=ether7
add bridge=LOCAL comment=defconf interface=ether8
add bridge=LOCAL comment=defconf interface=ether9
add bridge=LOCAL comment=defconf interface=ether10
add bridge=LOCAL comment=defconf interface=wifi1-TEVE
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ipv6 settings
set disable-ipv6=yes
/interface list member
add interface=LOCAL list=LAN
add interface=ether1-WAN list=WAN
add interface=wireguard1 list=LAN
add interface=wireguard2-53123 list=LAN
/ip address
add address=10.11.12.1/24 interface=LOCAL network=10.11.12.0
add address=10.11.13.1/24 interface=WIRELESS-GUESTS network=10.11.13.0
add address=172.27.11.1/24 interface=wireguard1 network=172.27.11.0
/ip cloud
set back-to-home-vpn=enabled ddns-enabled=yes ddns-update-interval=1m
/ip dhcp-client
add default-route-tables=main interface=ether1-WAN
/ip dhcp-server lease
add address=10.11.12.10 server=LOCAL
add address=10.11.12.33 server=LOCAL
/ip dhcp-server network
add address=10.11.12.0/24 comment=LOCAL dns-server=10.11.12.1 gateway=10.11.12.1 ntp-server=10.11.12.1
add address=10.11.13.0/24 comment=WIRELESS-GUESTS dns-server=10.11.13.1 gateway=10.11.13.1 ntp-server=10.11.13.1
/ip dns
set allow-remote-requests=yes cache-size=20480KiB servers=1.1.1.3,1.0.0.3
/ip dns adlist
add ssl-verify=no url=https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan type=A
add address=10.11.12.10 name=quad.local type=A
add address=10.11.12.10 name=quad.lan type=A
/ip firewall address-list
add address=0.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=172.16.0.0/12 comment=RFC6890 list=not_in_internet
add address=192.168.0.0/16 comment=RFC6890 list=not_in_internet
add address=10.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=169.254.0.0/16 comment=RFC6890 list=not_in_internet
add address=127.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=224.0.0.0/4 comment=Multicast list=not_in_internet
add address=198.18.0.0/15 comment=RFC6890 list=not_in_internet
add address=192.0.0.0/24 comment=RFC6890 list=not_in_internet
add address=192.0.2.0/24 comment=RFC6890 list=not_in_internet
add address=198.51.100.0/24 comment=RFC6890 list=not_in_internet
add address=203.0.113.0/24 comment=RFC6890 list=not_in_internet
add address=100.64.0.0/10 comment=RFC6890 list=not_in_internet
add address=240.0.0.0/4 comment=RFC6890 list=not_in_internet
add address=192.88.99.0/24 comment="6to4 relay Anycast [RFC 3068]" list=not_in_internet
/ip firewall filter
add action=accept chain=input comment="accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment=" drop invalid" connection-state=invalid
add action=accept chain=input comment="accept ICMP" protocol=icmp
add action=accept chain=input comment="Accept VPN in (Wireguard range)" dst-port=13231 log=yes log-prefix="WireGuard FILTER" protocol=udp
add action=accept chain=input comment="Accept VPN in (Wireguard range)" dst-port=53123 log=yes log-prefix="WireGuard FILTER" protocol=udp
add action=accept chain=input comment="Allow TCP DNS input" dst-port=53 in-interface=!ether1-WAN log-prefix="INPUT - ALLOW PORTS IN" protocol=tcp
add action=accept chain=input comment="Allow TCP UDP input" dst-port=53 in-interface=!ether1-WAN log-prefix="INPUT - ALLOW PORTS IN" protocol=udp
add action=accept chain=input comment="Allow UDP NTP from" dst-port=123 in-interface=!ether1-WAN protocol=udp
add action=accept chain=input comment="Allow UDP DHCP" dst-port=67 in-interface=!ether1-WAN protocol=udp
add action=accept chain=input comment="Allow IP to router" in-interface-list=LAN
add action=drop chain=input comment="Drop all not coming from LAN" in-interface=ether1-WAN
add action=drop chain=input comment="Drop everything else" log-prefix="DROP INPUT ALL ELSE"
add action=accept chain=forward comment="accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment=" fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="drop invalid" connection-state=invalid
add action=accept chain=forward comment="Allow ICMP forwarding" protocol=icmp
add action=accept chain=forward comment="Allow internet traffic" out-interface=ether1-WAN
add action=accept chain=forward comment="Allow BTH WireGuard access" in-interface=back-to-home-vpn out-interface-list=LAN
add action=accept chain=forward comment="DISABLE Enable Port Forwarding" connection-nat-state=dstnat
add action=drop chain=forward comment="Drop BOGONS going out" dst-address-list=not_in_internet out-interface=ether1-WAN
add action=drop chain=forward comment="drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface=ether1-WAN
add action=reject chain=forward in-interface=WIRELESS-GUESTS out-interface=LOCAL reject-with=icmp-admin-prohibited
/ip firewall nat
add action=masquerade chain=srcnat comment=masquerade ipsec-policy=out,none out-interface-list=WAN
/ip service
set ftp disabled=yes
set telnet disabled=yes
set www disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ip ssh
set strong-crypto=yes
/ip upnp
set enabled=yes
/ip upnp interfaces
add interface=LOCAL type=internal
add interface=ether1-WAN type=external
add interface=*F type=internal
/system clock
set time-zone-name=Europe/Zagreb
/system identity
set name="TEVE - GLAVNI ROUTER"
/system logging
add disabled=yes topics=wireguard
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp server
set enabled=yes multicast=yes
/system ntp client servers
add ``address=hr.pool.ntp.org
add ``address=europe.pool.ntp.org
/system routerboard settings
set enter-setup-on=delete-key
/tool bandwidth-server
set enabled=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

2

You’re missing the wireguard tcp mss fix.

3

Can you please point me to an appropriate resource? First time that I hear about this.

4

I searched online, you were thinking about this?

3    ;;; Clamp MSS to PMTU for traffic coming from WGchain=forward action=change-mss new-mss=clamp-to-pmtu passthrough=yes tcp-flags=syn protocol=tcp in-interface=wireguard1 log=no log-prefix=""

4    ;;; Clamp MSS to PMTU for traffic coming from WGchain=forward action=change-mss new-mss=clamp-to-pmtu passthrough=yes tcp-flags=syn protocol=tcp out-interface=wireguard1 log=no log-prefix=""

I got about 10% speed increase. Same file, trasfer measured 2:07 to 1:55 min.

Nice optimisation, but but the real problem is not solved.

5

I would like to update this thread and close it up with a solution, it is quite interesting.

As is, WireGuard is was really good to me. It is very simple, reliable and secure. I implemented a load of these, and they all work fine, some in a very large environments (tens of concurrent users - I know this for some is small, but for me is not).

This issue left me dumbfounded, because the WIreGuard performance was very bad and I couldn’t find the reason why. The proposed MSS fix that you can find in the post above did increase throughput for about 10%. Before I did not searched for it or implemented it simply because a 10% up or down is irrelevant at higher speeds.

However, even with the fix, and/or changing MTU nothing changed speedwise. In the end, it turned out telecom optical cable was attenuated. It was probably inappropriately bent at some point. It took them a week to solve this.

The interesting part was:

  1. perception of speed
  2. speedtest cheating

For 1., if a ping is low and speed is about 10-20 megabits, rarely can people notice the drop in speed.

For 2. speedtest results were wrong. It run with full speed, both upload and download, but when doing actual transfers the speed was abysmal. (Mind that 4011 I used is hardware accelerated, and it should allow for wire speeds for most usage cases.) It seems that telecoms quite strongly prioritise speedtest traffic.

So, it is solved. Luckily, it didn’t have to do anything with me. :slight_smile: I must say I am quite happy with that.

6

Very interesting report, I wonder how one should measure speed at the light of your results.

Once upon a time I used to download a large file (let's say a Linux distro .iso), but on fastish connections the bottleneck often becomes the download server side.

7

Agreed. If SpeedTest had shown that my speed is low, I wouldn’t have issues diagnosing the problem. But SpeedTest (and fast.com) always were great.

I was genuinely puzzled, because I had conflicting input data that pointed to a bad configuration or a bug or hardware issue.

This will change the way I test. I spent several days debugging this. I think the best action would be to actually put up a personal iperf server for testing purposes. I have 2Gbit link and a 5009 on the site, so that shouldn’t be an issue.

As for the usual WireShark diagnosis, I do agree it is an excellent way to debug, but it si time consuming and complex. I use that only for nuts tough to crack.

8

Was that intentional ? :rofl:

9

Where we’re going, we don’t need slow puns. :wink:

10

Just to add from my exp, SMB perfomance works the best with MTU set 1350 on wireguard interface.

And another info, for example i have 2 identical HAP AX3 setups at 2 customers, except one runs DHCP from ISP and other PPPOE, and one with PPPOE has about 300mbit less wireguard perfomance due PPPOE it self is more heavy on CPU, and not always multi-threaded if ever im not even sure at this point with mikrotik..

11

TBH WireGuard is so fast that I don’t bother with optimisation much. Wherever I installed it, it always performed close to wire speed, so didn’t have need to optimise.

Do you do MSS fix?

12

Doesnt do much for me, maybe 10%, while lowering it to 1350 can make massive difference regarding SMB.

13

So, the same result as I got - 10%.

I will test your setting, I am curious what would happen.