Hey everyone,
I’ve got a VPN issue that I can’t figure out. You can skip the background description if you wish.
Background:
- Main office: Mikrotik RB3011, 30/10 mbps (down/up) on VDSL, Win2008r2 filserver
- Remote office: Mikrotik RB2011uias, 10/10 mbps on 3 hop wireless uplink to fiber, Win2016 fileserver
At some point we’ll upgrade past our physical Windows boxes but for the moment we still rely on them with a SMB (robocopied) backup performed from the remote to the main office.
Ever since I set the VPN up between both sites (4-5 years ago) I’ve experienced slow upload speeds from the remote to the main office. At the time I thought it was due to known issues with SMB (version issues between the 2k8 and 2016 or the protocol being too chatty over WAN). I tinkered around a bit but gave up as the time it took to upload the backups was something I could live with. Last week the size of our backups doubled due to new software and it now takes north of 72 hours to upload a full image. We generally do incrementals but every second week we do full sized backups. Transfer speed hangs around 2 to 3 mbps. So I decided to try out FTP and Windows 2016 to Windows 10 but I didn’t see any improvement.
About a year ago we upgraded the remote office’s connection to a symmetric 10/10 mbps and we now have a solid 25 ms round trip time from one office to the other.
Long story short, I found out it’s a VPN issue with single stream TCP connections and has nothing to do with the application running on top of it (SMB, FTP, other).
The VPN itself is L2TP over IPSec with IKE2 and certificates. I chose L2TP because I needed an interface to assign OSPF to and GRE or IPIP needs static IPs which we don’t have. And I chose IPSec because it supports hardware encryption on the 3011 and is generally more secure using certificates rather than PSKs. Authentication is SHA256 and encryption AES256cbc.
Symptoms:
- Single stream TCP transfers from the 3011 to the 2011 speed maxes out at 10 mbps
- Single stream TCP transfers from the 2011 to the 3011 speed crawls along at 2 to 3 mbps
- Multiple stream TCP transfers from the 2011 to the 3011 are able to max out the 10 mbps.
- CPU load at the 2011 is around 30% when encrypting at 3 mbps.
- CPU load at the 3011 is about 2 to 3% obviously as it only has to route a bit of traffic and the encryption being offloaded when pushing 10 mbps.
- CPU load at the 2011 is about 60-70% when decrypting at 10 mbps
- Single TCP stream internet upload maxes out the 10 mbps
- A second office, also with a RB2011, same setup, 4 mbps upload speed also crawls along at ¼ (1 mbps ish) speed. And is also able to max out the 4 mbps when using multiple streams.
Things I’ve tried:
- AES256 CTR, AES128 CTR, AES128 CBC
- GRE, IPIP and PPTP instead of L2TP
- MTU and MRU dialed down to adequate values
- MSS clamp to PMTU rule
The issue:
Single TCP stream through the VPN goes fine one way but crawls along at ¼ speed in the other. Multiple streams are able to max out the connection. It doesn’t seem to be an encryption or hardware problem but I’m at a loss as to what could be causing this. Any pointers greatly appreciated!!