Hello every one
I have an RB4011 that I connected to my ESXi server that I have Debian 11 as a VM on, which is hosting my docker/docker containers. everything is okay except that docker containers (not Debian itself) cannot transfer/upload (TX) to the internet but receiving/downloading (RX) is totally fine.
So, for example, I have a container that is a speed test (using speedtest.net) and I start a test it will ping and download but when it comes to uploading it fails.
This only happens when I use my IKEV2 or L2TP VPN for that VM, when using PPTP there is no problem and the speed test doesn't fail.
I test this with another router with the same setup and everything was normal so I think it's a problem with my RB4011 configuration.
The IP address for Debian VM is 192.168.11.69
Any help is highly appreciated
My configurations:

For PPTP there is an address list called "Individual Local VPN 2" and for L2TP we have "Individual Local VPN"

Anyone?

As you say only upload is affected, it sounds like an MTU issue to me. Both ping and download test work fine, which means that routing is OK; ping packets are small in both directions and upload packets are small for download tests, so the MTU issue seems to be related to the VM->internet direction.

As you say the Debian itself works fine, have you tried to upload a file via the tunnel from the Debian, and then the same file from the container, to see the difference under same conditions?

PPTP, L2TP, and IPsec each use a different amount of header bytes, and the PPTP uses the least of all of them. If indeed Debian itself doesn't have any problem while containers do, I'd assume there is an issue with handling of "fragmentation needed" ICMP messages at the Docker. So a simultaneous tcpdump in the container and on the Debian itself should reveal some hint.

As a brute force workaround, a change-mss rule in mangle chain forward might help. I've seen cases where it didn't because the remote device was ignoring the MSS value received, but here, you would indicate a smaller MSS to the container, so it should work.

@sindy
Thanks for the answer man
It's exactly like you said, on the Debian level it's working as well but not on docker and I assume it's because docker have it's own network that all containers are behind it.

I tried to change the mss like you said but when trying to add "Couldn't add New Mangle Rule - tcp mss change works only on tcp syn packets (6)".
P.s. If it's a docker issue should I try to fix it at the docker level?

If it's a docker issue should I try to fix it at the docker level?

This is obviously the preferred approach. The change-mss rule was a suggestion of a workaround that could help until you find how to fix it in docker.

I tried to change the mss like you said but when trying to add "Couldn't add New Mangle Rule - tcp mss change works only on tcp syn packets (6)".

The mangle rule must say
chain=forward protocol=tcp tcp-flags=syn tcp-mss=1381-9999 out-interface=docker-facing-interface action=change-mss new-mss=1380

The value of 1380 may not be sufficiently small - so if it doesn't help, try with 1200 and don't forget to adjust the lower limit of the match range too. Then you can experiment to find the largest value with which it still works.

@sindy
Thank you very much.
I tried the rule that you said and because my server (which has Debian as a VM and docker inside) was connected to ether3 I selected out-interface=ether3 but gave the following error: But after I changed it to bridge, even with 1380 it worked very well.

But after I changed it to bridge, even with 1380 it worked very well.

The IP firewall only recognizes L3 interfaces (with an IP address attached to them). If an interface becomes a member port of a bridge, the bridge becomes the L3 interface (even if the IP configuration is attached to the member port, which is a configuration mistake that breaks some things).

To get the maximum out of the workaround, i.e. to use the available bandwidth most efficiently, find the largest working value of new-mss using binary search - you already know that 1380 works and 1472 doesn't, so try with the value in the middle between the two, i.e. 1426. If it still works, try the middle between 1426 and 1478, otherwise try the middle between 1380 and 1426. Repeat this until the difference between "working" and "not working" is 1 byte.

The IP firewall only recognizes L3 interfaces (with an IP address attached to them). If an interface becomes a member port of a bridge, the bridge becomes the L3 interface (even if the IP configuration is attached to the member port, which is a configuration mistake that breaks some things).

I'm sorry I don't understand, is there something wrong in my setup?

To get the maximum out of the workaround, i.e. to use the available bandwidth most efficiently, find the largest working value of new-mss using binary search - you already know that 1380 works and 1472 doesn't, so try with the value in the middle between the two, i.e. 1426. If it still works, try the middle between 1426 and 1478, otherwise try the middle between 1380 and 1426. Repeat this until the difference between "working" and "not working" is 1 byte.

I did this and apparently 1382 is the highest value that is working right now.

I'm sorry I don't understand, is there something wrong in my setup?

Nothing wrong in your setup, I only tried to explain why "docker facing interface" has to be bridge rather than ether3.

Thanks a lot man for all your time and effort.

For those of you who will came here with the same problem, the solution is to add custom MTU value in /lib/systemd/system/docker.service file. (you can use nano to edit it)
You need to change the line looking like this:
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
to look like this:
ExecStart=/usr/bin/dockerd --mtu 1450 -H fd:// --containerd=/run/containerd/containerd.sock
After that you need to restart Docker and all will be well with your networking:
sudo systemctl daemon-reload
sudo service docker restart

For me 1499 fixed it but for you it might be different so start with 1499 and if it failed decrease it and test again.

Source: https://www.civo.com/learn/fixing-networking-for-docker