Hi,

I've setup two RouterOS boxes - one in Europe and one in Asia, where the normal Internet latency is about 400 ms between the endpoints.
One of the routers is connected to Internet with PPPoE (MTU/MRU 1460). The other one has a direct ethernet connection to Internet.
Running the Mikrotik Bandwidth tester over a l2tp connection between the routers (ether2-pppoe-l2tp====l2tp-ether2) gives the full expected bandwidth of 10 Mbps (this is the subscribed bandwidth on the PPPoE-connected router), so this works better than expected..
But when I connect a Linux server to ether1 on each router, and copy a file using scp between the servers, the bandwidth is only getting up to 3 Mbps.
I have experimented with different MTU/MRU on both PPoE and L2TP configs, but it doesn't change things dramatically.

Anyone got an idea or explanation? Might it be the large latency (ie the SSH protocol used for the scp command is waiting for regular acks or something..)? Or am I missing something that I need to do on the LAN side of the routers to cater for the lower MxU's on the Internet side..?

Thanks

You may need to increase the TCP window size, google (or search engine of your choice): linux tcp window size

Thanks for your suggestion.
But in the end, there will be lots of servers and users in both ends, so I'm not able to adjust the configs on all those. Any configs must have to be on the Mikrotik routers...?

You might want to test it anyway to see if that is the issue. 400ms is quite a lot of latency. I'm making an assumption that its just getting backed up waiting for acks and the window isn't big enough to handle the delay. Does the latency change a lot (have high jitter)? that can mess up window sizes (or calculations thereof) a bit too. (are all the packets approximately 400ms, or are some 600ms and some 200ms?)

If this is the issue, I don't think there is much, if anything, you can do on the routers to influence client window sizes, but you could use caching proxy servers to try and hide the latency for more popular or common stuff.

Newer clients/operating systems (as higher speeds become common) will hopefully get smarter about picking appropriate window sizes.

I feel a bit on thin ice here, but I did the following in both servers;
echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/rmem_max
echo 256960 > /proc/sys/net/core/wmem_default
echo 256960 > /proc/sys/net/core/wmem_max

(The default was 124928 and max was 131071 before)

No difference though..
A 7 MB file still takes around 23 secs to copy;

services1:~ # scp file01 root@10.1.1.5:/home/hakan
Password:
file01 100% 7101KB 308.7KB/s 00:23


And there is not much jitter;
services1:~ # ping 10.1.1.5
PING 10.1.1.5 (10.1.1.5) 56(84) bytes of data.
64 bytes from 10.1.1.5: icmp_seq=1 ttl=61 time=447 ms
64 bytes from 10.1.1.5: icmp_seq=2 ttl=62 time=450 ms
64 bytes from 10.1.1.5: icmp_seq=3 ttl=62 time=430 ms
64 bytes from 10.1.1.5: icmp_seq=4 ttl=62 time=442 ms
64 bytes from 10.1.1.5: icmp_seq=5 ttl=62 time=444 ms
64 bytes from 10.1.1.5: icmp_seq=6 ttl=62 time=438 ms
64 bytes from 10.1.1.5: icmp_seq=7 ttl=62 time=429 ms
64 bytes from 10.1.1.5: icmp_seq=8 ttl=62 time=430 ms
64 bytes from 10.1.1.5: icmp_seq=9 ttl=62 time=432 ms
64 bytes from 10.1.1.5: icmp_seq=10 ttl=62 time=432 ms
64 bytes from 10.1.1.5: icmp_seq=11 ttl=62 time=444 ms
64 bytes from 10.1.1.5: icmp_seq=12 ttl=62 time=436 ms
64 bytes from 10.1.1.5: icmp_seq=13 ttl=62 time=440 ms
64 bytes from 10.1.1.5: icmp_seq=14 ttl=62 time=444 ms
64 bytes from 10.1.1.5: icmp_seq=15 ttl=62 time=447 ms
64 bytes from 10.1.1.5: icmp_seq=16 ttl=62 time=443 ms
64 bytes from 10.1.1.5: icmp_seq=17 ttl=62 time=447 ms
64 bytes from 10.1.1.5: icmp_seq=18 ttl=62 time=452 ms
64 bytes from 10.1.1.5: icmp_seq=19 ttl=62 time=445 ms

I've noticed this with certain flavors of scp lately, even on wired networks. One scp copy seems to only take some bandwidth, but if you start up additional threads then you can max it out. I am almost wondering if its a flavor of linux (debian?) that does this with scp, not the network itself.

I suppose next would be to try ftp or some other file transfer method, and see what speeds it gets.