I have a site to site IPsec tunnel that is established and working but the upload is not utilizing anywhere near my upload capabilities at two of the sites. I have three sites in total but Site C is currently down for renovations. Below is the equipment and basic information about each site. I am using SHA256, AES-256 CTR, and DH Group: modp4096 encryptions. I have watched CPU and memory utilization and they are all low but can jump up to 60% when transferring files (as I would expect). I am using Raw prerouting to reduce resource load and have NAT listed as a backup.
Site A:
CRS328-24P-4S+
Download: 25Mbps
Upload: 5Mbps
CPU: 15%
Site B:
CRS328-24P-4S+
Download: 300Mbps
Upload: 15Mbps
CPU: 5%
All sites show the correct speeds when I run an iPerf3 and/or Mikrotik Bandwidth test to all sites. The Upload speed at site B works as expected and uses 11.6Mbps as expected. Both Site A and C are only using 144-700kbps of the 5Mbps. Pings show the expected ms delay as these sites are not in the same area. I am able to access webpages across the sites just fine but when I access cameras or transfer a file using smb it does not go fast. I have compared the configs and have yet to find the issue or make any headway. Any suggestions on where to start looking? I have dug though several forms and reddit posts but they are not the same issue.
Thanks for taking the time to offer advice or help guide me.
Typically Site B to Site A the ping response is between 49-55 ms
Prior to starting SMB transfer from Site B to Site A
64 bytes from 10.1.2.248: icmp_seq=0 ttl=126 time=52.308 ms
64 bytes from 10.1.2.248: icmp_seq=1 ttl=126 time=52.729 ms
64 bytes from 10.1.2.248: icmp_seq=2 ttl=126 time=51.497 ms
64 bytes from 10.1.2.248: icmp_seq=3 ttl=126 time=51.320 ms
64 bytes from 10.1.2.248: icmp_seq=4 ttl=126 time=50.945 ms
64 bytes from 10.1.2.248: icmp_seq=5 ttl=126 time=54.693 ms
64 bytes from 10.1.2.248: icmp_seq=6 ttl=126 time=50.708 ms
After starting SMB transfer from Site B to Site A
64 bytes from 10.1.2.248: icmp_seq=7 ttl=126 time=62.000 ms
64 bytes from 10.1.2.248: icmp_seq=8 ttl=126 time=61.423 ms
64 bytes from 10.1.2.248: icmp_seq=9 ttl=126 time=81.908 ms
64 bytes from 10.1.2.248: icmp_seq=10 ttl=126 time=76.312 ms
64 bytes from 10.1.2.248: icmp_seq=11 ttl=126 time=87.723 ms
64 bytes from 10.1.2.248: icmp_seq=12 ttl=126 time=79.904 ms
64 bytes from 10.1.2.248: icmp_seq=13 ttl=126 time=108.158 ms
64 bytes from 10.1.2.248: icmp_seq=14 ttl=126 time=113.423 ms
64 bytes from 10.1.2.248: icmp_seq=15 ttl=126 time=50.121 ms
64 bytes from 10.1.2.248: icmp_seq=16 ttl=126 time=127.361 ms
64 bytes from 10.1.2.248: icmp_seq=17 ttl=126 time=140.767 ms
64 bytes from 10.1.2.248: icmp_seq=18 ttl=126 time=55.839 ms
64 bytes from 10.1.2.248: icmp_seq=19 ttl=126 time=152.035 ms
64 bytes from 10.1.2.248: icmp_seq=20 ttl=126 time=160.331 ms
64 bytes from 10.1.2.248: icmp_seq=21 ttl=126 time=173.582 ms
64 bytes from 10.1.2.248: icmp_seq=22 ttl=126 time=166.509 ms
.
I’ve seen issues like this when there’s an MTU mismatch. With the IPsec tunnel you have reduced MTU capacity and I’ve observed transfer speeds reduced to less than 1/4 of expected. Youtube in one case just didn’t stream with the mismatch. Try reducing the MTU on a client machine and start testing with MTU of 1350. Here’s a guide - https://support.zen.co.uk/kb/Knowledgebase/Changing-the-MTU-size-in-Windows-Vista-7-or-8
I don’t necessarily want to start messing with client devices as they are used outside of our office. They just start an L2TP/IPsec tunnel back to the office on their device. I currently have access to a server, one VM, and one laptop at each site to troubleshooting this. I have the issue narrowed to either something with the Mikrotik or the ISP Modem. I am attaching screenshots of the interfaces from Winbox incase that helps determine anything. I know the issue is persistent when using SMB to transfer a file from windows to windows and windows to Mac (vise versa). Site B seems to utilize the full upload speeds when going to either Site A or Site C. Site A and Site C going to Site B is when we notice the issue occurring.
Please note the GRE tunnels across the IPsec peer only handle routing for certain IPs. None of the SMB traffic is going across the GRE tunnel, just the bare IPsec Peer.
The screenshots were captured after hours so there isn’t any traffic and this is when I have been doing my testing/troubleshooting.
I have hard set the MTU to 1500 on both sides of the GRE tunnel but that had no effect on the SMB transfers. Only certain IP addresses traverse the GRE tunnel. All other traffic destine for the other site goes across a bare IPsec peer connection setup between the two MikroTik Routers.
