Slow FTP upload speed via GRE Tunnel

Hello, I have a problem for which I have not been able to find a solution anywhere on the Internet. The problem is that when I upload some data to my main server via FTP, the upload speed is very bad, even though the internet speed on my computer and on the server is over 500Mbps. I rented a dedicated server from a hosting provider and a block of IP addresses that I tunneled to my mikrotik router via a GRE tunnel.

This is how I did it:

/interface gre
add local-address=109.x.x.x name=gre-2 remote-address=217.x.x.x

/ip address
add address=11.1.1.2/24 interface=gre-2 network=11.1.1.0

/ip firewall address-list
add address=10.11.0.3 list=tunnel-51.x.x.x

/ip firewall filter
add action=accept chain=input in-interface=gre-2
add action=accept chain=forward in-interface=gre-2

/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=tunnel-51.x.x.x \
    passthrough=no src-address-list=tunnel-51.x.x.x

/ip firewall nat
add action=masquerade chain=srcnat comment="Gre tunnel #2" out-interface=gre-2
add action=dst-nat chain=dstnat comment=51.x.x.x dst-address=11.1.1.2 \
    to-addresses=10.11.0.3

/ip route
add distance=1 gateway=11.1.1.1 routing-mark=tunnel-51.x.x.x

I tried to change MTU, MMS but without any success. I also tried using IPIP tunnel with the same configuration but the same problem happens. The speed of the leased dedicated server is 300+ Mbps.

When I run the speedtest myself on my main server that uses the internet from the dedicated ookla speedtest, the speedtest reports a normal speed for both download and upload (300+down, 250+up). On the dedicated server, I use the ubuntu 20.04 system with iptables rules so that everything is redirected properly (if you also need those commands, I can publish them).

When I try to upload a file through our public IP address that the provider gave us, the speed goes perfectly, 500-600mbps, as it should be, but when it goes through the tunnel, it shows 30-40mbps for a moment and immediately drops to 4-5mbps, which is not okay at all.

For mikrotik router I use hEX S with routerOS version v6.49.7

do you have fast-track enabled?

Hello, I don’t have fast-track enabled. I tried enabling it using these commands:

/ip firewall filter add chain=forward action=fasttrack-connection connection-state=established,related
/ip firewall filter add chain=forward action=accept connection-state=established,related

But when I enabled it the upload speed became even worse, from 520KB/s to 40KB/s. So I disabled it again.

I was wondering why with fast-track active ipsec becomes almost unusable. then disable it and restart the routerboard.

Could be wrong, but I don’t think you can use fast-track with IPSec-enabled GRE tunnel.

Also, are you setting MTU lower on the GRE interface, or is MTU 1500? If it’s 1500, you can use ping and don’t fragment to determine the MTU (search here/google for using ping to calculate mtu size).

Relatedly, make sure icmp/ping is not blocked over the GRE link (or anywhere on the path).

gre by default has 1476 MTU. you may need to scale if you have vlans in the transports connecting you to the internet and also if you have pppoe encapsulation. a VLAN involves decreasing MTU by 4bytes and pppoe by 8bytes. So if they deliver you internet with pppoe on VLAN you should set 1488 on the pppoe interface and 1464 on the gre. otherwise you have to test with ping using “do not fragment” . For convenience you can also set a change mss on the forward chain in the mangle.

in any case, even if you have improvements you will never have the result you get by pointing to the public IP, because with gre the data is encrypted

No, you can’t. Default firewall has these two rules

add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec

as first two rules in chain=forward, only 3rd rule is fasttrack rule.

I am aware that I will not be able to have the same results on a gre tunnel as I would have directly on a public ip address. I started a company that deals with hosting and we use GRE tunnel for IP addresses and DDoS protection and our clients have problems with uploading their files to the server, 5-6 Mbps is very slow, especially if it is a website with over 10gb of data…

Did you check the MTU?

If using ipsec, possibly worth looking at the installed SA’s and check it is using hardware encryption.

/ip ipsec installed-sa
print

Flags in the second column, H for HW-AEAD

Also

your dst-nat rule, perhaps restrict it to in-interface=gre-2

The mangle rule, forces all packets from the specified address list to go via the tunnel,
no exception.

You may want to run it via a routing rule, so you can more easily route to local ip addresses.

/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=rule-51.x.x.x \
    passthrough=no src-address-list=tunnel-51.x.x.x


/routing rule
#local known routes just use main table.
add action=lookup disabled=no min-prefix=0 table=main
add action=lookup disabled=no routing-mark=rule-51.x.x.x table=tunnel-51.x.x.x

One other thought. If using ipsec, and there is Nat translation,

I think it is uses transport mode, which will likely only work for
1 client per public ip address for GRE.

wireguard is good

Probably a PMTUD problem. You should not change MTU on the GRE (except to set it lower when the internet MTU is not 1500, e.g. when you use PPPoE).
Instead, make this mange rule to decrease MSS on the TCP connections:

/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes \
    protocol=tcp tcp-flags=syn

I set the MTU for the gre tunnel to 1500 but without any improvement, the default MTU that mikrotik set for me is 1476. I also tried to reduce the mtu to something smaller but without any improvement either. I also tried to use ping to calculate the mtu size, but I didn’t quite understand how because whatever MTU I set it says Packet needs to be fragmented but DF set

C:\Users\Bendza>ping -f -l 1500 51.x.x.232

Pinging 51.x.x.232 with 1500 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

The only thing that interests me is that through the speedtest, the speed is as normal as it should be, but when a file is uploaded via ftp, the speed is significantly lower.

[root@cp1 ~]# speedtest-cli
Retrieving speedtest.net configuration...
Testing from OVH SAS (51.77.82.232)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Deutsche Telekom (Berlin) [300.79 km]: 42.132 ms
Testing download speed................................................................................
Download: 189.25 Mbit/s
Testing upload speed......................................................................................................
Upload: 129.57 Mbit/s

This is a speedtest on the server

Did you already add the mangle rule?

Some thoughts:

It looks like you are not using ipsec, is this correct?

From windows, you need to subtract 28 from your length of ping,
(Windows makes the data in the packet that big, then has 28 bytes of IP header on top).

So for a 1500 byte mtu,
ping -f -l 1472 51.x.x.232

If the MTU the router thinks it has to the other end is incorrect, it can/will cause issues.

I would initially find what the raw outside MTU to the other end of the link is.
Use a PC that doesn’t go via the Gre tunnel, or use the Mikrotik (don’t subtract 28)

(for 1500 from windows)
ping -f -l 1472 217.x.x.x

(for 1492 pppoe from windows)
ping -f -l 1464 217.x.x.x)

If the maximum length of the ping is less, you might need to hunt for it.
eg. Windows: try -f -l 1272 if ok then try -f -l 1372 then…

Often for best efficiency it would be good to set the GRE tunnel MTU to this found
MTU - 24 (the GRE overhead)
eg. If outside MTU=1492, want GRE MTU = 1468
if outside MTU=1500, GRE MTU=1476 (The default)

**Edit: Not this, GRE is layer 3 ** If you have vlan’s running inside your GRE perhaps you want to shrink this MTU by another 4 bytes or more, so the VLan tag information also fits. **

By default the GRE tunnel will do MSS clamping.

Good explanation @rplant. Does seem like MTU issue… i.e. “slow” often a sign of wrong MTU & “slow” also means it’s getting through the firewall so ain’t fw filters… Even if not, it still best to know/confirm your MTU with the ping test.

In similar MTU vain, make sure icmp is allowed by firewall on input to give PMTUD a shot of working. That’s in the default firewall, but if building your own, perhaps ICMP is getting blocked.

Only other thing I can think of is if FTP is not using standard port 21, you’d want to added it to the firewall helper for FTP in /ip/firewall/service-port