Community discussions

MikroTik App
 
OlegSotnikov
just joined
Topic Author
Posts: 4
Joined: Sun May 23, 2021 10:57 am

Bandwidth issues with WireGuard and 7.1beta6

Sun May 23, 2021 11:43 am

Hello everyone!
I'm facing an issue with TCP connections bandwidth between the two CHR.
I have 2 CHR with RouterOS 7.1beta6: one in Russia and one in Kazakhstan, both have a gigabit Internet connection.
Both routers connected with a WireGuard VPN connection. To measure speed I use the bandwidth test tool in RouterOS.
If I use the UDP test - the results are pretty awesome: ~900Mbit for receive and send.
But if I use TCP test:
RU->KZ ~900Mbit
RU<-KZ ~30Mbit

I've done a lot of tests, but cannot figure out where is my issue.
ISP is ok, L2 has no problems, without Miktorik CHR I can get the full speed with no issues.
One of the side effects - I can get up to 140Mbits with TCP test if I use 60+ connections.

Could someone please give me any ideas on how to find the source of my problem?
Thank you.
You do not have the required permissions to view the files attached to this post.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19100
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Bandwidth issues with WireGuard and 7.1beta6

Sun May 23, 2021 4:45 pm

HI there,
Can you let me know how to test this properly.
I have the same setup here in Canada but still on beta5, between two BELL fiber connections about 5 miles apart.
I use ookla speed tests from client PC to server ISP connection via browser and am getting consistently around 75Mbps up and 75Mbps down throughput.

Clearly not seeing the typical OOKLA speed tests from our PC out the standard ISP of about 900 up and 900 down which is also the case for the other partys standard speed tests (without wireguard and using own ISP connection).
Here I was thinking 70Mbps up or down was pretty good for a VPN connection??

Thus how for me to test to see if I can replicate your numbers?
 
User avatar
mozerd
Forum Veteran
Forum Veteran
Posts: 872
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Bandwidth issues with WireGuard and 7.1beta6

Sun May 23, 2021 5:04 pm

Key point: Known Limitations
WireGuard is a protocol that, like all protocols, makes necessary trade-offs. This page summarizes known limitations due to these trade-offs.

TCP Mode
WireGuard explicitly does not support tunneling over TCP, due to the classically terrible network performance of tunneling TCP-over-TCP. Rather, transforming WireGuard's UDP packets into TCP is the job of an upper layer of obfuscation (see previous point), and can be accomplished by projects like udptunnel and udp2raw.

https://www.wireguard.com/known-limitations/
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Bandwidth issues with WireGuard and 7.1beta6

Sun May 23, 2021 6:25 pm

Just guessing... what happens if you swap the roles of the routers in the bandwidth test, is it always the server->client direction (or always the client->server one) that is slow, or it is always the KZ->RU one?

The manual says you should not run the bandwidth test on the router whose throughput you are testing, so what does /tool profile say while you're running the TCP bandwidth test?

Better results with more connections suggest some round-trip delay issue, but if it was the case, both directions should be affected, and they are not.
 
OlegSotnikov
just joined
Topic Author
Posts: 4
Joined: Sun May 23, 2021 10:57 am

Re: Bandwidth issues with WireGuard and 7.1beta6

Sun May 23, 2021 10:42 pm

I tried from another router as well. Same problem, same direction issue.
My guess, that if UDP can fully saturate my wan link, TCP should be able to do almost the same.
My WAN link is a 1 Gigabite connection from both sides, certified data centers, ~60ms delay between.
Both CHR has 2 cores, 512Mb RAM, high priority pool on ESXi and seems this is not an issue - at least one direction is working well.
All configurations are pretty default, MTU is the same, no Fasttrack.
There are Tx Drops and Tx Errors on WG interfaces, but Tx for both routers - not Tx and Rx.

Still open question - why only one way is laggy? Some sort of misconfiguration maybe.
P.S. Submitted support ticket.
You do not have the required permissions to view the files attached to this post.
 
OlegSotnikov
just joined
Topic Author
Posts: 4
Joined: Sun May 23, 2021 10:57 am

Re: Bandwidth issues with WireGuard and 7.1beta6

Mon May 24, 2021 11:01 am

I got some updates.
Today in our DC was a massive blackout and our hypervisor was rebooted.
I still have the same problem, but now in opposite direction!
RU->KZ ~6Mbit
RU<-KZ ~860Mbit

Looks like it's a configuration issue?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11433
Joined: Thu Mar 03, 2016 10:23 pm

Re: Bandwidth issues with WireGuard and 7.1beta6

Mon May 24, 2021 12:11 pm

Just a general observation: TCP doesn't work great with large delays out-of-the-box. That's where TCP windows comes into play and with 40ms delay, TCP window size should be larger than around 5 MB to be able to reach 1Gbps throughput. Default window size of 64kB is only enough for around 13Mbps with that kind of RTT. Indeed most TCP stacks can scale window size (until it reaches max window size or until a retransmission occurs when window size shrinks back to some small value, could be that it shrinks to default value), but that only allows higher speeds in medium to long term. Lossy link affects TCP window scaling very much and if you experience any kind of packet loss, you won't be able to reach full wire speed using WG over TCP. Beware that TCP, being essentially bi-directional (at least ACKs travel in the opposite direction), saturating link in any direction means increase in RTT and will thus throttle throughput down (which is intended behaviour) ...

With WG over TCP window size on clients is only of secondary importance, the bottleneck can be window size on routers terminating WG connection. I'm not sure how this can be changed for WG on ROS ...
 
OlegSotnikov
just joined
Topic Author
Posts: 4
Joined: Sun May 23, 2021 10:57 am

Re: Bandwidth issues with WireGuard and 7.1beta6

Mon May 24, 2021 12:46 pm

I use standard ROS Wireguard with UDP, 1400 MTU.
There is a general consideration, that if I test two different L4 protocols UDP and TCP inside of the same tunnel - the results have to be almost the same (except for TCP overhead).
Also, I see a lot of Tx Drops on my WG interface (3 493 517) for the last 24h.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11433
Joined: Thu Mar 03, 2016 10:23 pm

Re: Bandwidth issues with WireGuard and 7.1beta6

Mon May 24, 2021 1:18 pm

The TCP window settings on both client and server still apply aven if WG is run over UDP (which explains why UDP tests can saturate link). Not sure how that's affected by WG properties. But then Tx drops indicate that WG link is not perfect and that will definitely affect performance of TCP connections through WG tunnel.

What does happen if you limit throughput to something little less than link speed on Tx side of testing suite?
 
Cablenut9
Long time Member
Long time Member
Posts: 542
Joined: Fri Jan 08, 2021 5:30 am

Re: Bandwidth issues with WireGuard and 7.1beta6

Mon May 24, 2021 11:00 pm

Try a regular bandwidth test so you can rule out anything that isn't WG.

Who is online

Users browsing this forum: adrianmartin16, GoogleOther [Bot], rogerioqueiroz, svmk, Valerio5000 and 77 guests