EoIP performance

fgarpe · October 31, 2023, 5:43pm

Hi,
I have an EoIP+IPSec tunnel runing between two locations but I have very poor performance in network speed. I include below the tests performed and my configuration

Machine in site A to public iperf server:

iperf3 -c proof.ovh.net -p 5202
Connecting to host proof.ovh.net, port 5202
[  5] local 172.26.34.20 port 38740 connected to 141.95.207.211 port 5202
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   118 MBytes   987 Mbits/sec  7346   32.5 KBytes       
[  5]   1.00-2.00   sec   114 MBytes   958 Mbits/sec  8271   26.9 KBytes       
[  5]   2.00-3.00   sec   113 MBytes   946 Mbits/sec  6975   42.4 KBytes       
[  5]   3.00-4.00   sec   114 MBytes   957 Mbits/sec  7243    238 KBytes       
[  5]   4.00-5.00   sec   113 MBytes   949 Mbits/sec  6529   45.2 KBytes       
[  5]   5.00-6.00   sec   114 MBytes   954 Mbits/sec  7832    110 KBytes       
[  5]   6.00-7.00   sec   114 MBytes   954 Mbits/sec  8056    140 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   944 Mbits/sec  7383   60.8 KBytes       
[  5]   8.00-9.00   sec   114 MBytes   954 Mbits/sec  7980   41.0 KBytes       
[  5]   9.00-10.00  sec   114 MBytes   954 Mbits/sec  8315   83.4 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.11 GBytes   956 Mbits/sec  75930             sender
[  5]   0.00-10.03  sec  1.11 GBytes   950 Mbits/sec                  receiver

Machine in site B to public iperf server:

iperf3 -c proof.ovh.net -p 5202
Connecting to host proof.ovh.net, port 5202
[  5] local 172.26.34.30 port 35042 connected to 141.95.207.211 port 5202
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  20.8 MBytes   175 Mbits/sec    0   1.16 MBytes       
[  5]   1.00-2.00   sec  42.5 MBytes   357 Mbits/sec  1203    955 KBytes       
[  5]   2.00-3.00   sec  31.2 MBytes   262 Mbits/sec    0   1024 KBytes       
[  5]   3.00-4.00   sec  32.5 MBytes   273 Mbits/sec    0   1.05 MBytes       
[  5]   4.00-5.00   sec  33.8 MBytes   283 Mbits/sec    0   1.08 MBytes       
[  5]   5.00-6.00   sec  33.8 MBytes   283 Mbits/sec    0   1.10 MBytes       
[  5]   6.00-7.00   sec  36.2 MBytes   304 Mbits/sec    0   1.11 MBytes       
[  5]   7.00-8.00   sec  35.0 MBytes   294 Mbits/sec    0   1.11 MBytes       
[  5]   8.00-9.00   sec  35.0 MBytes   294 Mbits/sec    0   1.11 MBytes       
[  5]   9.00-10.00  sec  35.0 MBytes   294 Mbits/sec    0   1.11 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   336 MBytes   282 Mbits/sec  1203             sender
[  5]   0.00-10.03  sec   334 MBytes   279 Mbits/sec                  receiver

Machine at site A to machine at site B through the tunnel

iperf3 -c 172.26.34.30 -p4242
Connecting to host 172.26.34.30, port 4242
[  5] local 172.26.34.20 port 52302 connected to 172.26.34.30 port 4242
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   959 KBytes  7.85 Mbits/sec   37   12.7 KBytes       
[  5]   1.00-2.00   sec   573 KBytes  4.69 Mbits/sec    0   33.9 KBytes       
[  5]   2.00-3.00   sec   573 KBytes  4.69 Mbits/sec   21   14.1 KBytes       
[  5]   3.00-4.00   sec   382 KBytes  3.13 Mbits/sec    5   19.8 KBytes       
[  5]   4.00-5.00   sec   764 KBytes  6.26 Mbits/sec    2   21.2 KBytes       
[  5]   5.00-6.00   sec   573 KBytes  4.69 Mbits/sec    2   32.5 KBytes       
[  5]   6.00-7.00   sec   573 KBytes  4.69 Mbits/sec   13   29.7 KBytes       
[  5]   7.00-8.00   sec   764 KBytes  6.26 Mbits/sec   24   18.4 KBytes       
[  5]   8.00-9.00   sec   382 KBytes  3.13 Mbits/sec   13   14.1 KBytes       
[  5]   9.00-10.00  sec   573 KBytes  4.69 Mbits/sec    4   24.0 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.97 MBytes  5.01 Mbits/sec  121             sender
[  5]   0.00-10.04  sec  5.63 MBytes  4.71 Mbits/sec                  receiver

This is my config at site A
IP Addresses:

/ip address print
Flags: X - disabled, I - invalid, D - dynamic 
 #   ADDRESS            NETWORK         INTERFACE                                                                                                                                                                                            
 0   1.2.3.4/24    1.2.3.0  ether1-WAN
 1   172.26.33.253/24  172.26.33.0    bridge-EOIP-A

EoIP Interface:

/interface eoip print
Flags: X - disabled, R - running
0  R name="eoip-A" mtu=1500 actual-mtu=1500 l2mtu=65535 mac-address=02:6B:BD:C9:12:6F arp=enabled arp-timeout=auto loop-protect=default loop-protect-status=off loop-protect-send-interval=5s loop-protect-disable-time=5m local-address=1.2.3.4 remote-address=4.3.2.1 tunnel-id=150 keepalive=10s,10 dscp=inherit clamp-tcp-mss=yes dont-fragment=no ipsec-secret="XXXXXXXXX" allow-fast-path=no

The bridge:

/interface bridge print
Flags: X - disabled, R - running
 0 R name="bridge-EOIP-A" mtu=1500 actual-mtu=1500 l2mtu=65535 arp=enabled arp-timeout=auto mac-address=06:03:AB:DA:3C:3E protocol-mode=rstp fast-forward=yes igmp-snooping=no auto-mac=yes ageing-time=5m priority=0x8000 max-message-age=20s forward-delay=15s transmit-hold-count=6 vlan-filtering=no dhcp-snooping=no

Bridge ports:

/interface bridge port print
 4     ether5-A                                                                              bridge-EOIP-A                                                                      yes    1     0x80         10                 10       none
 5     eoip-A                                                                                 bridge-EOIP-A                                                                             1     0x80         10                 10       none

The config at site B

IP Addresses:

/ip address print
Flags: X - disabled, I - invalid, D - dynamic
0 D 4.3.2.1/32   4.3.2.0   WAN
1   172.26.33.254/24  172.26.33.0   bridge-EOIP-B

EoIP Interface:

/interface eoip print
Flags: X - disabled, R - running 
 0  R name="eoip-B" mtu=1500 actual-mtu=1500 l2mtu=65535 mac-address=FE:DA:E7:5E:75:15 arp=enabled arp-timeout=auto loop-protect=default loop-protect-status=off loop-protect-send-interval=5s loop-protect-disable-time=5m local-address=4.3.2.1 remote-address=1.2.3.4 tunnel-id=150 keepalive=10s,10 dscp=inherit clamp-tcp-mss=yes dont-fragment=no ipsec-secret="XXXXXXXXX" allow-fast-path=no

The bridge:

/interface bridge print
Flags: X - disabled, R - running
0 R name="bridge-EOIP-B" mtu=auto actual-mtu=1500 l2mtu=1598 arp=enabled arp-timeout=auto mac-address=E4:8D:8C:39:82:30 protocol-mode=rstp fast-forward=yes igmp-snooping=no auto-mac=yes ageing-time=5m priority=0x8000 max-message-age=20s forward-delay=15s transmit-hold-count=6 vlan-filtering=no dhcp-snooping=no

Bridge ports:

/interface bridge port print
Flags: X - disabled, I - inactive, D - dynamic, H - hw-offload 
 #     INTERFACE                                                                               BRIDGE                                                                              HW  PVID PRIORITY  PATH-COST INTERNAL-PATH-COST    HORIZON
 0     ether06                                                                            bridge-EOIP-B                                                              no     1     0x80         10                 10       none
 1     eoip-B                                                                         bridge-EOIP-B                                                                     1     0x80         10                 10       none

The two mikrotiks have direct access to internet and have public ip’s, I played with the encryption algorithms and the best result was obtained with SHA-256 + aes-256-cbc. The mikrotik at site A is a VM of CHR with a p10 license activated, the mikrotik at site B is a 1100AHx2.

I see a lot of posts talking about the MTU, but, in my tests, I have not obtained any improvement by modifying the recommended value (1500). I do not know how to correctly calculate the value I should have.

Please, any help is welcome.

Regards

bpwl · October 31, 2023, 9:24pm

I don’t think 1500 as MTU for EoIP is a good idea here.
EoIP is 42 bytes overhead already. http://forum.mikrotik.com/t/eoip-tunnel-not-getting-1500-mtu/111523/1

The IPSEC overhead looks more complex than a simple number of bytes. It seems to be encryption protocol dependent also: https://packetpushers.net/ipsec-bandwidth-overhead-using-aes/

The only way I managed to get 1500byte packets over a VPN tunnel was with BCP (SSTP, PPTP or L2TP) which allows for a multistream connection and allows a larger L2MTU for the VPN connection. The fragmentation is still there, but on another place.

It is quite common to reduce the MTU to 1400 bytes for the data stream everywhere, with VPN involved. Works very well unless the application insists on having unbroken 1500 byte packets.
A bridge will adjust to the smallest MTU of the ports. But if the packet needs fragmentation somewhere allong the path , it is usually bad for the performance.

Amm0 · October 31, 2023, 9:46pm

IPSec gets complex in sizing, why 1400 is common. Cisco has a calculator the IPSec, which varies based on cypher: https://community.cisco.com/legacyfs/online/legacy/4/8/7/27784-IPSec_Calculator_NAT_GRE-Key.htm

I’d also check to make sure icmp is blocked in firewall.

The other thing that’s effecting this is the “clamp-to-mss” check box. Since MTU is wrong, that’s actually adjusting it wrong – so unchecking/=no clamp-to-mss with 1500 MTU might also get you better results. Although I’d still lower the MTU to 1400 (or 1380, if 1400 doesn’t show improvement) and if lower MTU…you do want clamp-to-mss enabled/=yes.

I suspect also you’d see a different result if you use UDP mode in iPerf.

fgarpe · November 1, 2023, 12:26am

Thank you very much for the answers and information provided. I tried your suggestions changing the MTU to a lower value. I have used these tools to try to adjust the MTU value better:
https://baturin.org/tools/encapcalc/
https://ipsec-overhead-calculator.netsec.us/
Now, I’m going to share the results:

With 1380 MTU:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   818 KBytes  6.70 Mbits/sec   26   31.1 KBytes       
[  5]   1.00-2.00   sec   686 KBytes  5.62 Mbits/sec    7   28.5 KBytes       
[  5]   2.00-3.00   sec   475 KBytes  3.89 Mbits/sec   17   20.8 KBytes       
[  5]   3.00-4.00   sec   747 KBytes  6.12 Mbits/sec    6   19.5 KBytes       
[  5]   4.00-5.00   sec   849 KBytes  6.96 Mbits/sec    0   40.2 KBytes       
[  5]   5.00-6.00   sec  1.03 MBytes  8.64 Mbits/sec   14   29.8 KBytes       
[  5]   6.00-7.00   sec   839 KBytes  6.87 Mbits/sec    6   16.9 KBytes       
[  5]   7.00-8.00   sec   508 KBytes  4.16 Mbits/sec    7   13.0 KBytes       
[  5]   8.00-9.00   sec   445 KBytes  3.64 Mbits/sec    7   16.9 KBytes       
[  5]   9.00-10.00  sec   573 KBytes  4.70 Mbits/sec   14   18.2 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.83 MBytes  5.73 Mbits/sec  104             sender
[  5]   0.00-10.04  sec  6.69 MBytes  5.59 Mbits/sec                  receiver

With 1386 MTU:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.27 MBytes  10.7 Mbits/sec   68   13.0 KBytes       
[  5]   1.00-2.00   sec   375 KBytes  3.07 Mbits/sec    7   16.9 KBytes       
[  5]   2.00-3.00   sec   375 KBytes  3.07 Mbits/sec    6   13.0 KBytes       
[  5]   3.00-4.00   sec   563 KBytes  4.61 Mbits/sec    3   13.0 KBytes       
[  5]   4.00-5.00   sec   375 KBytes  3.07 Mbits/sec    2   24.8 KBytes       
[  5]   5.00-6.00   sec   750 KBytes  6.15 Mbits/sec    3   30.0 KBytes       
[  5]   6.00-7.00   sec   750 KBytes  6.15 Mbits/sec   25   13.0 KBytes       
[  5]   7.00-8.00   sec   375 KBytes  3.07 Mbits/sec    1   24.8 KBytes       
[  5]   8.00-9.00   sec   563 KBytes  4.61 Mbits/sec    9   28.7 KBytes       
[  5]   9.00-10.00  sec   563 KBytes  4.61 Mbits/sec    5   19.5 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.85 MBytes  4.91 Mbits/sec  129             sender
[  5]   0.00-10.04  sec  5.59 MBytes  4.67 Mbits/sec                  receiver

With 1390 MTU:

[  5] local 172.17.150.20 port 45096 connected to 172.17.150.10 port 4242
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  11.3 MBytes  94.9 Mbits/sec   14    520 KBytes       
[  5]   1.00-2.00   sec  11.9 MBytes   100 Mbits/sec    0    580 KBytes       
[  5]   2.00-3.00   sec  11.9 MBytes   100 Mbits/sec    0    623 KBytes       
[  5]   3.00-4.00   sec  12.0 MBytes   101 Mbits/sec   13    457 KBytes       
[  5]   4.00-5.00   sec  9.92 MBytes  83.2 Mbits/sec    6    248 KBytes       
[  5]   5.00-6.00   sec  7.17 MBytes  60.1 Mbits/sec    0    274 KBytes       
[  5]   6.00-7.00   sec  7.96 MBytes  66.8 Mbits/sec   10    204 KBytes       
[  5]   7.00-8.00   sec  5.57 MBytes  46.8 Mbits/sec    4    159 KBytes       
[  5]   8.00-9.00   sec  4.78 MBytes  40.1 Mbits/sec    0    178 KBytes       
[  5]   9.00-10.00  sec  5.57 MBytes  46.8 Mbits/sec    0    197 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  88.2 MBytes  74.0 Mbits/sec   47             sender
[  5]   0.00-10.04  sec  86.3 MBytes  72.1 Mbits/sec                  receiver

Great, seems good!, but after repeat the test 10 seconds after…

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   979 KBytes  8.01 Mbits/sec  114   14.4 KBytes       
[  5]   1.00-2.00   sec   376 KBytes  3.08 Mbits/sec    6   11.8 KBytes       
[  5]   2.00-3.00   sec   564 KBytes  4.63 Mbits/sec    5   17.0 KBytes       
[  5]   3.00-4.00   sec   376 KBytes  3.08 Mbits/sec    4   20.9 KBytes       
[  5]   4.00-5.00   sec   376 KBytes  3.08 Mbits/sec    8   15.7 KBytes       
[  5]   5.00-6.00   sec   564 KBytes  4.63 Mbits/sec    5   19.6 KBytes       
[  5]   6.00-7.00   sec   376 KBytes  3.08 Mbits/sec    8   15.7 KBytes       
[  5]   7.00-8.00   sec   564 KBytes  4.62 Mbits/sec   12   17.0 KBytes       
[  5]   8.00-9.00   sec   376 KBytes  3.08 Mbits/sec    5   14.4 KBytes       
[  5]   9.00-10.00  sec   564 KBytes  4.63 Mbits/sec    8   13.1 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.00 MBytes  4.19 Mbits/sec  175             sender
[  5]   0.00-10.04  sec  4.60 MBytes  3.84 Mbits/sec                  receiver

WTF!!!

With 1392 MTU:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  7.86 MBytes  66.0 Mbits/sec    5    309 KBytes       
[  5]   1.00-2.00   sec  6.87 MBytes  57.6 Mbits/sec    9    243 KBytes       
[  5]   2.00-3.00   sec  5.95 MBytes  49.9 Mbits/sec    8    190 KBytes       
[  5]   3.00-4.00   sec  4.17 MBytes  35.0 Mbits/sec   18   73.3 KBytes       
[  5]   4.00-5.00   sec   942 KBytes  7.72 Mbits/sec   24   11.8 KBytes       
[  5]   5.00-6.00   sec   502 KBytes  4.12 Mbits/sec    3   17.0 KBytes       
[  5]   6.00-7.00   sec   440 KBytes  3.60 Mbits/sec    2   20.9 KBytes       
[  5]   7.00-8.00   sec   502 KBytes  4.12 Mbits/sec   10   15.7 KBytes       
[  5]   8.00-9.00   sec   440 KBytes  3.60 Mbits/sec    5   19.6 KBytes       
[  5]   9.00-10.00  sec   440 KBytes  3.60 Mbits/sec   13   18.3 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  28.0 MBytes  23.5 Mbits/sec   97             sender
[  5]   0.00-10.03  sec  27.0 MBytes  22.5 Mbits/sec                  receiver

The speed drops during the test, when I repeat the test I get the same sad values as in previous tests.

With 1394 MTU:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  6.87 MBytes  57.6 Mbits/sec    0    408 KBytes       
[  5]   1.00-2.00   sec  12.5 MBytes   105 Mbits/sec   22    486 KBytes       
[  5]   2.00-3.00   sec  11.8 MBytes  98.9 Mbits/sec    0    545 KBytes       
[  5]   3.00-4.00   sec  11.4 MBytes  95.9 Mbits/sec    4    415 KBytes       
[  5]   4.00-5.00   sec  9.64 MBytes  80.9 Mbits/sec    6    321 KBytes       
[  5]   5.00-6.00   sec  8.35 MBytes  70.1 Mbits/sec   13    249 KBytes       
[  5]   6.00-7.00   sec  7.37 MBytes  61.8 Mbits/sec    8    193 KBytes       
[  5]   7.00-8.00   sec  5.90 MBytes  49.5 Mbits/sec    0    212 KBytes       
[  5]   8.00-9.00   sec  5.90 MBytes  49.5 Mbits/sec    0    229 KBytes       
[  5]   9.00-10.00  sec  7.37 MBytes  61.8 Mbits/sec    0    250 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  87.1 MBytes  73.1 Mbits/sec   53             sender
[  5]   0.00-10.03  sec  85.3 MBytes  71.3 Mbits/sec                  receiver

Again, after repeat the speed goes down

1396 MTU:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   598 KBytes  4.90 Mbits/sec   22   15.8 KBytes       
[  5]   1.00-2.00   sec   454 KBytes  3.72 Mbits/sec    9   15.8 KBytes       
[  5]   2.00-3.00   sec   530 KBytes  4.34 Mbits/sec    7   18.4 KBytes       
[  5]   3.00-4.00   sec   398 KBytes  3.26 Mbits/sec    6   15.8 KBytes       
[  5]   4.00-5.00   sec   398 KBytes  3.26 Mbits/sec    4   18.4 KBytes       
[  5]   5.00-6.00   sec   470 KBytes  3.85 Mbits/sec    8   13.1 KBytes       
[  5]   6.00-7.00   sec   395 KBytes  3.24 Mbits/sec    3   15.8 KBytes       
[  5]   7.00-8.00   sec   567 KBytes  4.64 Mbits/sec    2   23.6 KBytes       
[  5]   8.00-9.00   sec   635 KBytes  5.20 Mbits/sec    9   21.0 KBytes       
[  5]   9.00-10.00  sec   509 KBytes  4.17 Mbits/sec    3   13.1 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.84 MBytes  4.06 Mbits/sec   73             sender
[  5]   0.00-10.05  sec  4.78 MBytes  3.99 Mbits/sec                  receiver

1398, 1400, 1406, 1408 MTU gets me the same poor values.

1412 MTU:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  7.09 MBytes  59.4 Mbits/sec   11    194 KBytes       
[  5]   1.00-2.00   sec  6.03 MBytes  50.6 Mbits/sec    0    216 KBytes       
[  5]   2.00-3.00   sec  4.94 MBytes  41.4 Mbits/sec    8    169 KBytes       
[  5]   3.00-4.00   sec  4.94 MBytes  41.4 Mbits/sec   14    133 KBytes       
[  5]   4.00-5.00   sec  3.90 MBytes  32.7 Mbits/sec    0    151 KBytes       
[  5]   5.00-6.00   sec  4.88 MBytes  40.9 Mbits/sec    0    173 KBytes       
[  5]   6.00-7.00   sec  4.88 MBytes  40.9 Mbits/sec    0    194 KBytes       
[  5]   7.00-8.00   sec  5.79 MBytes  48.6 Mbits/sec    0    214 KBytes       
[  5]   8.00-9.00   sec  5.49 MBytes  46.0 Mbits/sec    1    178 KBytes       
[  5]   9.00-10.00  sec  5.30 MBytes  44.5 Mbits/sec    0    201 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  53.2 MBytes  44.7 Mbits/sec   34             sender
[  5]   0.00-10.04  sec  52.5 MBytes  43.8 Mbits/sec                  receiver

Again, when repeat test 10 seconds later the speed goes down. I don’t understand nothing…

I also check that icmp is not blocked in firewall

Iperf UDP test:

[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   129 KBytes  1.06 Mbits/sec  97  
[  5]   1.00-2.00   sec   128 KBytes  1.04 Mbits/sec  96  
[  5]   2.00-3.00   sec   129 KBytes  1.06 Mbits/sec  97  
[  5]   3.00-4.00   sec   128 KBytes  1.04 Mbits/sec  96  
[  5]   4.00-5.00   sec   128 KBytes  1.04 Mbits/sec  96  
[  5]   5.00-6.00   sec   129 KBytes  1.06 Mbits/sec  97  
[  5]   6.00-7.00   sec   128 KBytes  1.04 Mbits/sec  96  
[  5]   7.00-8.00   sec   128 KBytes  1.04 Mbits/sec  96  
[  5]   8.00-9.00   sec   129 KBytes  1.06 Mbits/sec  97  
[  5]   9.00-10.00  sec   128 KBytes  1.04 Mbits/sec  96  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.25 MBytes  1.05 Mbits/sec  0.000 ms  0/964 (0%)  sender
[  5]   0.00-10.08  sec  1.25 MBytes  1.04 Mbits/sec  1.321 ms  0/964 (0%)  receiver

1500 MTU with clamp-to-mss unchecked

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   639 KBytes  5.23 Mbits/sec   24   11.3 KBytes       
[  5]   1.00-2.00   sec   382 KBytes  3.13 Mbits/sec    2   18.4 KBytes       
[  5]   2.00-3.00   sec   509 KBytes  4.17 Mbits/sec    4   17.0 KBytes       
[  5]   3.00-4.00   sec   382 KBytes  3.13 Mbits/sec    4   18.4 KBytes       
[  5]   4.00-5.00   sec   382 KBytes  3.13 Mbits/sec    6   19.8 KBytes       
[  5]   5.00-6.00   sec   509 KBytes  4.17 Mbits/sec    7   14.1 KBytes       
[  5]   6.00-7.00   sec   509 KBytes  4.17 Mbits/sec    4   12.7 KBytes       
[  5]   7.00-8.00   sec   382 KBytes  3.13 Mbits/sec    5   17.0 KBytes       
[  5]   8.00-9.00   sec   382 KBytes  3.13 Mbits/sec    5   14.1 KBytes       
[  5]   9.00-10.00  sec   509 KBytes  4.17 Mbits/sec    2   21.2 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.48 MBytes  3.76 Mbits/sec   63             sender
[  5]   0.00-10.04  sec  4.27 MBytes  3.57 Mbits/sec                  receiver

I continue to test and review configurations, any suggestions or ideas are welcome.

fgarpe · November 1, 2023, 10:12pm

Hi!,
I have finally found the error!, although I was changing the MTU on the EoIP interface, for some stupid reason, I had manually set the MTU on the bridge to 1500. Removing this value so that the bridge automatically took the lowest MTU value from the associated interfaces was solved the problem. At the moment the best performance has been obtained with a MTU value of 1412, I will keep testing.

Thank you very much for your help.

Amm0 · November 1, 2023, 10:52pm

Geez, it’s always something. It should take the lowest, but yeah if overridden, likely not.

I’m not sure the exact logic, and likely OS/platform dependent… but client will cache PMTUD results. So you can also change stuff, but clients may still use old MTU calculations. That always a possibility.

I’d also recommend running a TCP test for 30 seconds or more with iPerf to verify things.