ROS speed degrade on high-latency WAN

RouterOS General
1

Hello,
I have WAN link 200Mbps to remote Windows server with latency 37 ms.

With direct connected PC over WAN iperf show good speed in both directions:

iperf-2.0.14a-Dec14_20-win.exe -c x.x.x.x -w 1024K -e -i 1 -r

Server listening on TCP port 5001 with pid 4544
Read buffer size: 128 KByte (Dist bin width=16.0 KByte)
TCP window size: 1.00 MByte

Client connecting to x.x.x.x, TCP port 5001 with pid 4544 (1 flows)
Write buffer size: 131072 Byte
TCP window size: 1.00 MByte

[ 1] local z.z.z.z port 50023 connected with x.x.x.x port 5001 (MSS=-1) (sock=380) (ct=46.94 ms)
[ ID] Interval Transfer Bandwidth Write/Err
[ 1] 0.00-1.00 sec 15.5 MBytes 130 Mbits/sec 125/0
[ 1] 1.00-2.00 sec 20.0 MBytes 168 Mbits/sec 160/0
[ 1] 2.00-3.00 sec 21.0 MBytes 176 Mbits/sec 168/0
[ 1] 3.00-4.00 sec 20.0 MBytes 168 Mbits/sec 160/0
[ 1] 4.00-5.00 sec 21.0 MBytes 176 Mbits/sec 168/0
[ 1] 5.00-6.00 sec 21.0 MBytes 176 Mbits/sec 168/0
[ 1] 6.00-7.00 sec 20.0 MBytes 168 Mbits/sec 160/0
[ 1] 7.00-8.00 sec 21.0 MBytes 176 Mbits/sec 168/0
[ 1] 8.00-9.00 sec 20.0 MBytes 168 Mbits/sec 160/0
[ 1] 9.00-10.00 sec 21.0 MBytes 176 Mbits/sec 168/0
[ 1] 0.00-10.02 sec 201 MBytes 168 Mbits/sec 1608/0
[ 2] local z.z.z.z port 5001 connected with x.x.x.x port 59917 (MSS=-1) (sock=364)
[ ID] Interval Transfer Bandwidth Reads=Dist
[ *2] 0.00-1.00 sec 11.4 MBytes 95.5 Mbits/sec 177=51:20:14:1:3:11:23:54
[ *2] 1.00-2.00 sec 20.3 MBytes 170 Mbits/sec 319=97:33:26:1:1:26:34:101
[ *2] 2.00-3.00 sec 20.7 MBytes 174 Mbits/sec 308=92:35:14:1:1:14:38:113
[ *2] 3.00-4.00 sec 21.0 MBytes 176 Mbits/sec 318=86:45:17:2:2:18:46:102
[ *2] 4.00-5.00 sec 20.2 MBytes 169 Mbits/sec 316=87:39:28:1:1:28:39:93
[ *2] 5.00-6.00 sec 20.8 MBytes 175 Mbits/sec 327=87:42:29:2:2:30:46:89
[ *2] 6.00-7.00 sec 21.0 MBytes 176 Mbits/sec 334=83:45:37:1:1:37:46:84
[ *2] 7.00-8.00 sec 20.0 MBytes 168 Mbits/sec 315=81:46:26:2:2:26:46:86
[ *2] 8.00-9.00 sec 21.0 MBytes 176 Mbits/sec 328=76:45:35:4:4:36:48:80
[ *2] 9.00-10.00 sec 20.3 MBytes 170 Mbits/sec 305=73:31:37:2:2:36:34:90
[ *2] 0.00-10.03 sec 198 MBytes 165 Mbits/sec 3060=818:381:264:17:19:264:401:896



but when I place my computer behind a CCR1009 (v6.48) firewall (NAT), the speed is much slower and unstable:

iperf-2.0.14a-Dec14_20-win.exe -c x.x.x.x -w 1024K -e -i 1 -t 20 ------------------------------------------------------------ Client connecting to x.x.x.x, TCP port 5001 with pid 17608 (1 flows) Write buffer size: 131072 Byte TCP window size: 1.00 MByte ------------------------------------------------------------ [ 1] local y.y.y.y port 54415 connected with x.x.x.x port 5001 (MSS=-1) (sock=352) (ct=48.54 ms) [ ID] Interval Transfer Bandwidth Write/Err [ 1] 0.00-1.00 sec 7.50 MBytes 62.9 Mbits/sec 61/0 [ 1] 1.00-2.00 sec 7.13 MBytes 59.8 Mbits/sec 57/0 [ 1] 2.00-3.00 sec 7.75 MBytes 65.0 Mbits/sec 62/0 [ 1] 3.00-4.00 sec 8.25 MBytes 69.2 Mbits/sec 66/0 [ 1] 4.00-5.00 sec 8.25 MBytes 69.2 Mbits/sec 66/0 [ 1] 5.00-6.00 sec 8.50 MBytes 71.3 Mbits/sec 68/0 [ 1] 6.00-7.00 sec 8.38 MBytes 70.3 Mbits/sec 67/0 [ 1] 7.00-8.00 sec 8.75 MBytes 73.4 Mbits/sec 70/0 [ 1] 8.00-9.00 sec 8.38 MBytes 70.3 Mbits/sec 67/0 [ 1] 9.00-10.00 sec 8.38 MBytes 70.3 Mbits/sec 67/0 [ 1] 10.00-11.00 sec 9.00 MBytes 75.5 Mbits/sec 72/0 [ 1] 11.00-12.00 sec 9.63 MBytes 80.7 Mbits/sec 77/0 [ 1] 12.00-13.00 sec 10.3 MBytes 86.0 Mbits/sec 82/0 [ 1] 13.00-14.00 sec 10.1 MBytes 84.9 Mbits/sec 81/0 [ 1] 14.00-15.00 sec 8.50 MBytes 71.3 Mbits/sec 68/0 [ 1] 15.00-16.00 sec 6.50 MBytes 54.5 Mbits/sec 52/0 [ 1] 16.00-17.00 sec 7.13 MBytes 59.8 Mbits/sec 57/0 [ 1] 17.00-18.00 sec 7.25 MBytes 60.8 Mbits/sec 58/0 [ 1] 18.00-19.00 sec 7.75 MBytes 65.0 Mbits/sec 62/0 [ 1] 19.00-20.00 sec 7.88 MBytes 66.1 Mbits/sec 63/0 [ 1] 0.00-20.01 sec 166 MBytes 69.4 Mbits/sec 1326/0 >

also i tried copying files between computers:
PC-PC connection show ~21 MB/s copy speed
PC-CCR-PC show only ~7 MB/s and unstable (see screenshot)

same test with low-latency server (2ms) show good speed in both cases.

Why does the CCR1009 speed degrade so much on high-latency WAN?

filecopy_ccr1009.png
filecopy_ccr1009.png769×578 18.1 KB

Decrease in software quality from mikrotik?
MT support refused to fix issues
2

I tested also on RB3011 and it has the same problem:

iperf-2.0.14a-Dec14_20-win.exe -c x.x.x.x -w 1024K -e -i 1 -t 20 ------------------------------------------------------------ Client connecting to x.x.x.x, TCP port 5001 with pid 17972 (1 flows) Write buffer size: 131072 Byte TCP window size: 1.00 MByte ------------------------------------------------------------ [ 1] local y.y.y.y port 57504 connected with x.x.x.x port 5001 (MSS=-1) (sock=352) (ct=37.80 ms) [ ID] Interval Transfer Bandwidth Write/Err [ 1] 0.00-1.00 sec 8.63 MBytes 72.4 Mbits/sec 70/0 [ 1] 1.00-2.00 sec 9.00 MBytes 75.5 Mbits/sec 72/0 [ 1] 2.00-3.00 sec 10.0 MBytes 83.9 Mbits/sec 80/0 [ 1] 3.00-4.00 sec 10.5 MBytes 88.1 Mbits/sec 84/0 [ 1] 4.00-5.00 sec 10.9 MBytes 91.2 Mbits/sec 87/0 [ 1] 5.00-6.00 sec 10.9 MBytes 91.2 Mbits/sec 87/0 [ 1] 6.00-7.00 sec 11.0 MBytes 92.3 Mbits/sec 88/0 [ 1] 7.00-8.00 sec 11.0 MBytes 92.3 Mbits/sec 88/0 [ 1] 8.00-9.00 sec 11.8 MBytes 98.6 Mbits/sec 94/0 [ 1] 9.00-10.00 sec 12.3 MBytes 103 Mbits/sec 98/0 [ 1] 10.00-11.00 sec 12.8 MBytes 107 Mbits/sec 102/0 [ 1] 11.00-12.00 sec 12.5 MBytes 105 Mbits/sec 100/0 [ 1] 12.00-13.00 sec 6.50 MBytes 54.5 Mbits/sec 52/0 [ 1] 13.00-14.00 sec 7.13 MBytes 59.8 Mbits/sec 57/0 [ 1] 14.00-15.00 sec 7.63 MBytes 64.0 Mbits/sec 61/0 [ 1] 15.00-16.00 sec 7.88 MBytes 66.1 Mbits/sec 63/0 [ 1] 16.00-17.00 sec 8.00 MBytes 67.1 Mbits/sec 64/0 [ 1] 17.00-18.00 sec 4.63 MBytes 38.8 Mbits/sec 37/0 [ 1] 18.00-19.00 sec 4.38 MBytes 36.7 Mbits/sec 35/0 [ 1] 19.00-20.00 sec 4.63 MBytes 38.8 Mbits/sec 37/0 [ 1] 0.00-20.01 sec 182 MBytes 76.4 Mbits/sec 1459/0 >

it looks like this is a bug in RouterOS

3

I don’t know - but for what it’s worth, I use the Hex model with IPSec. My internet speed is 100 mbps, latency is 178 ms from Canada to Europe and I’m able to sustain around 99 mbps over the IPSec tunnel for many hours. My test is a single file transfer via FTP.

While you’re maxed out, a brief packet capture may show packet loss. It would also show the TCP window, although traffic going through a router should have nothing to do with the the TCP window. Or it may show packet fragmentation.

4

This topic not about IPsec.
Try without a tunnel.

  1. do not use fasttrack
  2. one side should be Tile (CCR) or ARM (RB3011)
  3. both PC should be Windows
  4. for testing: copy big file via shared folder
5

Hi mikruser

I am also seeing the same speed problem between two computers over the internet (with or without an IPSec tunnel) with 15ms latency, RB3011 on one side and RB4011 on the other, and I am a little confused about your statement on that you don’t have it if you connect your computers directly to the WAN. I wanted to do a speed test with no MikroTik devices in between, so I installed a Ne-One emulator virtual appliance (https://itrinegy.com/ne-one-range) to simulate lower speed and higher latency.
The test is performed between two computers on my LAN without a MikroTik router.

Speed limit 100 Mbit/s, Latency 1-2ms
hrping 172.16.2.10
Source address is 172.16.2.18; using ICMP echo-request, ID=9412
Pinging 172.16.2.10 [172.16.2.10]
with 32 bytes data (60 bytes IP):

From 172.16.2.10: bytes=60 seq=0001 TTL=128 ID=ffda time=1.536ms
From 172.16.2.10: bytes=60 seq=0002 TTL=128 ID=ffdb time=1.898ms
From 172.16.2.10: bytes=60 seq=0003 TTL=128 ID=ffdc time=1.912ms
From 172.16.2.10: bytes=60 seq=0004 TTL=128 ID=ffdd time=1.546ms

iperf3 -c 172.16.2.10 -w 1024K
Connecting to host 172.16.2.10, port 5201
[ 4] local 172.16.2.18 port 60172 connected to 172.16.2.10 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 9.62 MBytes 80.7 Mbits/sec
[ 4] 1.00-2.00 sec 11.2 MBytes 94.3 Mbits/sec
[ 4] 2.00-3.00 sec 11.5 MBytes 96.4 Mbits/sec
[ 4] 3.00-4.00 sec 11.5 MBytes 96.3 Mbits/sec
[ 4] 4.00-5.01 sec 11.5 MBytes 96.2 Mbits/sec
[ 4] 5.01-6.01 sec 11.5 MBytes 96.2 Mbits/sec
[ 4] 6.01-7.00 sec 11.2 MBytes 94.9 Mbits/sec
[ 4] 7.00-8.01 sec 11.5 MBytes 95.9 Mbits/sec
[ 4] 8.01-9.00 sec 11.4 MBytes 96.2 Mbits/sec
[ 4] 9.00-10.00 sec 11.4 MBytes 95.2 Mbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 112 MBytes 94.2 Mbits/sec sender
[ 4] 0.00-10.00 sec 112 MBytes 93.5 Mbits/sec receiver

Speed limit 100 Mbit/s, Latency 16-19ms
hrping 172.16.2.10
Source address is 172.16.2.18; using ICMP echo-request, ID=d82e
Pinging 172.16.2.10 [172.16.2.10]
with 32 bytes data (60 bytes IP):

From 172.16.2.10: bytes=60 seq=0001 TTL=128 ID=ffde time=18.822ms
From 172.16.2.10: bytes=60 seq=0002 TTL=128 ID=ffdf time=16.978ms
From 172.16.2.10: bytes=60 seq=0003 TTL=128 ID=ffe0 time=18.691ms
From 172.16.2.10: bytes=60 seq=0004 TTL=128 ID=ffe1 time=15.858ms

iperf>iperf3 -c 172.16.2.10 -w 1024K
Connecting to host 172.16.2.10, port 5201
[ 4] local 172.16.2.18 port 60193 connected to 172.16.2.10 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 6.25 MBytes 52.2 Mbits/sec
[ 4] 1.00-2.01 sec 5.88 MBytes 49.1 Mbits/sec
[ 4] 2.01-3.01 sec 7.38 MBytes 61.7 Mbits/sec
[ 4] 3.01-4.01 sec 6.62 MBytes 55.4 Mbits/sec
[ 4] 4.01-5.00 sec 4.50 MBytes 38.1 Mbits/sec
[ 4] 5.00-6.01 sec 6.75 MBytes 56.3 Mbits/sec
[ 4] 6.01-7.00 sec 5.25 MBytes 44.4 Mbits/sec
[ 4] 7.00-8.00 sec 6.00 MBytes 50.3 Mbits/sec
[ 4] 8.00-9.01 sec 5.00 MBytes 41.8 Mbits/sec
[ 4] 9.01-10.01 sec 3.75 MBytes 31.4 Mbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 57.4 MBytes 48.1 Mbits/sec sender
[ 4] 0.00-10.01 sec 56.6 MBytes 47.4 Mbits/sec receiver

Speed limit 100 Mbit/s, Latency 32-36ms
hrping 172.16.2.10
Source address is 172.16.2.18; using ICMP echo-request, ID=802c
Pinging 172.16.2.10 [172.16.2.10]
with 32 bytes data (60 bytes IP):

From 172.16.2.10: bytes=60 seq=0001 TTL=128 ID=8b78 time=32.366ms
From 172.16.2.10: bytes=60 seq=0002 TTL=128 ID=8b79 time=33.970ms
From 172.16.2.10: bytes=60 seq=0003 TTL=128 ID=8b7a time=31.601ms
From 172.16.2.10: bytes=60 seq=0004 TTL=128 ID=8b7b time=35.499ms

iperf3 -c 172.16.2.10 -w 1024K
Connecting to host 172.16.2.10, port 5201
[ 4] local 172.16.2.18 port 60236 connected to 172.16.2.10 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 3.12 MBytes 25.9 Mbits/sec
[ 4] 1.01-2.00 sec 3.12 MBytes 26.6 Mbits/sec
[ 4] 2.00-3.00 sec 3.00 MBytes 25.1 Mbits/sec
[ 4] 3.00-4.00 sec 3.75 MBytes 31.5 Mbits/sec
[ 4] 4.00-5.01 sec 4.25 MBytes 35.4 Mbits/sec
[ 4] 5.01-6.01 sec 4.75 MBytes 39.8 Mbits/sec
[ 4] 6.01-7.00 sec 5.25 MBytes 44.5 Mbits/sec
[ 4] 7.00-8.01 sec 4.75 MBytes 39.7 Mbits/sec
[ 4] 8.01-9.01 sec 5.12 MBytes 42.9 Mbits/sec
[ 4] 9.01-10.01 sec 4.00 MBytes 33.4 Mbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 41.1 MBytes 34.5 Mbits/sec sender
[ 4] 0.00-10.01 sec 40.4 MBytes 33.8 Mbits/sec receiver


The same thing happens when I copy a file over a shared folder. The higher the latency, the slower the speed.
If you try to run iperf with more parallel streams, you can get faster speed even with high latency.

Speed limit 100 Mbit/s, Latency 16-19ms
hrping 172.16.2.10
Source address is 172.16.2.18; using ICMP echo-request, ID=4c3b
Pinging 172.16.2.10 [172.16.2.10]
with 32 bytes data (60 bytes IP):

From 172.16.2.10: bytes=60 seq=0001 TTL=128 ID=c596 time=15.845ms
From 172.16.2.10: bytes=60 seq=0002 TTL=128 ID=c597 time=17.334ms
From 172.16.2.10: bytes=60 seq=0003 TTL=128 ID=c598 time=16.590ms
From 172.16.2.10: bytes=60 seq=0004 TTL=128 ID=c599 time=17.196ms

One single stream
iperf3 -c 172.16.2.10 -w 1024K -t 3
Connecting to host 172.16.2.10, port 5201
[ 4] local 172.16.2.18 port 60306 connected to 172.16.2.10 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 5.50 MBytes 46.1 Mbits/sec
[ 4] 1.00-2.01 sec 4.62 MBytes 38.7 Mbits/sec
[ 4] 2.01-3.01 sec 3.75 MBytes 31.4 Mbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-3.01 sec 13.9 MBytes 38.7 Mbits/sec sender
[ 4] 0.00-3.01 sec 13.0 MBytes 36.3 Mbits/sec receiver

Four parallel streams
iperf3 -c 172.16.2.10 -w 1024K -t 3 -P 4
Connecting to host 172.16.2.10, port 5201
[ 4] local 172.16.2.18 port 60312 connected to 172.16.2.10 port 5201
[ 6] local 172.16.2.18 port 60313 connected to 172.16.2.10 port 5201
[ 8] local 172.16.2.18 port 60314 connected to 172.16.2.10 port 5201
[ 10] local 172.16.2.18 port 60315 connected to 172.16.2.10 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 5.88 MBytes 49.1 Mbits/sec
[ 6] 0.00-1.00 sec 2.62 MBytes 21.9 Mbits/sec
[ 8] 0.00-1.00 sec 1.75 MBytes 14.6 Mbits/sec
[ 10] 0.00-1.00 sec 2.62 MBytes 21.9 Mbits/sec
[SUM] 0.00-1.00 sec 12.9 MBytes 108 Mbits/sec

[ 4] 1.00-2.00 sec 2.88 MBytes 24.2 Mbits/sec
[ 6] 1.00-2.00 sec 3.38 MBytes 28.4 Mbits/sec
[ 8] 1.00-2.00 sec 2.50 MBytes 21.1 Mbits/sec
[ 10] 1.00-2.00 sec 2.75 MBytes 23.2 Mbits/sec
[SUM] 1.00-2.00 sec 11.5 MBytes 96.9 Mbits/sec

[ 4] 2.00-3.01 sec 2.38 MBytes 19.8 Mbits/sec
[ 6] 2.00-3.01 sec 2.75 MBytes 22.9 Mbits/sec
[ 8] 2.00-3.01 sec 3.12 MBytes 26.0 Mbits/sec
[ 10] 2.00-3.01 sec 3.25 MBytes 27.0 Mbits/sec
[SUM] 2.00-3.01 sec 11.5 MBytes 95.7 Mbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-3.01 sec 11.1 MBytes 31.0 Mbits/sec sender
[ 4] 0.00-3.01 sec 10.2 MBytes 28.4 Mbits/sec receiver
[ 6] 0.00-3.01 sec 8.75 MBytes 24.4 Mbits/sec sender
[ 6] 0.00-3.01 sec 7.92 MBytes 22.1 Mbits/sec receiver
[ 8] 0.00-3.01 sec 7.38 MBytes 20.6 Mbits/sec sender
[ 8] 0.00-3.01 sec 6.46 MBytes 18.0 Mbits/sec receiver
[ 10] 0.00-3.01 sec 8.62 MBytes 24.1 Mbits/sec sender
[ 10] 0.00-3.01 sec 7.74 MBytes 21.6 Mbits/sec receiver
[SUM] 0.00-3.01 sec 35.9 MBytes 100 Mbits/sec sender
[SUM] 0.00-3.01 sec 32.3 MBytes 90.0 Mbits/sec receiver


IMHO the speed issue is more related to the way of the communication of iperf or SMB and not to MikroTik devices.