Why adding EoIP interface to bridge lowers MTU to 1458, and breaks HTTPS connectivity (timeout errors) for some sites?

RouterOS Beginner Basics
1

On the main router, I have configured EoIP according to the guide in the docs.

However, I also needed to add mtu=1500 in order to make everything on LAN to work correctly. The interfaces status with EoIP having mtu set to 1500:


/interface eoip
add mac-address=FE:XX:XX:XX:XX:XX mtu=1500 name=trunk-client-81 remote-address=10.xx.xx.81 tunnel-id=81

/interface/print
Flags: R - RUNNING; S - SLAVE
Columns: NAME, TYPE, ACTUAL-MTU, L2MTU, MAX-L2MTU, MAC-ADDRESS
 #    NAME                  TYPE    ACTUAL-MTU  L2MTU  MAX-L2MTU  MAC-ADDRESS
 0 R  ether1                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 1 RS ether2                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 2  S ether3                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 3  S ether4                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 4  S ether5                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 5    MYNET.hAP-ac3.1-2G1   wifi                                  2C:XX:XX:XX:XX:XX
 6    MYNET.hAP-ac3.1-2G2   wifi                                  2E:XX:XX:XX:XX:XX
 7    MYNET.hAP-ac3.1-5G1   wifi                                  2C:XX:XX:XX:XX:XX
 8    MYNET.hAP-ac3.1-5G2   wifi                                  2E:XX:XX:XX:XX:XX
 9    MYNET.hAP-ax2.1-2G1   wifi                                  48:XX:XX:XX:XX:XX
10    MYNET.hAP-ax2.1-2G2   wifi                                  4A:XX:XX:XX:XX:XX
11    MYNET.hAP-ax2.1-5G1   wifi                                  48:XX:XX:XX:XX:XX
12    MYNET.hAP-ax2.1-5G2   wifi                                  4A:XX:XX:XX:XX:XX
;;; defconf
13 R  bridge                bridge        1500   1568             78:XX:XX:XX:XX:XX
14 R  guest-vlan-30         vlan          1500   1564             78:XX:XX:XX:XX:XX
15 R  guest-vlan-31         vlan          1500   1564             78:XX:XX:XX:XX:XX
16 RS guest-wifi-2.4G       wifi          1500                    7A:XX:XX:XX:XX:XX
17  S guest-wifi-5G         wifi          1500                    7A:XX:XX:XX:XX:XX
18 R  management-vlan-1010  vlan          1500   1564             78:XX:XX:XX:XX:XX
19 RS trunk-client-81       eoip          1500  65535             FE:XX:XX:XX:XX:XX
20 RS wifi-2.4G             wifi          1500                    78:XX:XX:XX:XX:XX
21 RS wifi-5G               wifi          1500                    78:XX:XX:XX:XX:XX
22 R  wifi1                 wifi          1500                    7A:XX:XX:XX:XX:XX

The wifi1 is dedicated SSID with different PSK to carry trunk (VLANs) to the other building over the air.

Now, when I set mtu=auto, the actual-mtu of many interfaces including the bridge, but not for all interfaces, goes to 1458:


/interface/eoip/set 0 mtu=auto

/interface/print
Flags: R - RUNNING; S - SLAVE
Columns: NAME, TYPE, ACTUAL-MTU, L2MTU, MAX-L2MTU, MAC-ADDRESS
 #    NAME                  TYPE    ACTUAL-MTU  L2MTU  MAX-L2MTU  MAC-ADDRESS
 0 R  ether1                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 1 RS ether2                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 2  S ether3                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 3  S ether4                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 4  S ether5                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 5    MYNET.hAP-ac3.1-2G1   wifi                                  2C:XX:XX:XX:XX:XX
 6    MYNET.hAP-ac3.1-2G2   wifi                                  2E:XX:XX:XX:XX:XX
 7    MYNET.hAP-ac3.1-5G1   wifi                                  2C:XX:XX:XX:XX:XX
 8    MYNET.hAP-ac3.1-5G2   wifi                                  2E:XX:XX:XX:XX:XX
 9    MYNET.hAP-ax2.1-2G1   wifi                                  48:XX:XX:XX:XX:XX
10    MYNET.hAP-ax2.1-2G2   wifi                                  4A:XX:XX:XX:XX:XX
11    MYNET.hAP-ax2.1-5G1   wifi                                  48:XX:XX:XX:XX:XX
12    MYNET.hAP-ax2.1-5G2   wifi                                  4A:XX:XX:XX:XX:XX
;;; defconf
13 R  bridge                bridge        1458   1568             78:XX:XX:XX:XX:XX
14 R  guest-vlan-30         vlan          1458   1564             78:XX:XX:XX:XX:XX
15 R  guest-vlan-31         vlan          1458   1564             78:XX:XX:XX:XX:XX
16 RS guest-wifi-2.4G       wifi          1500                    7A:XX:XX:XX:XX:XX
17  S guest-wifi-5G         wifi          1500                    7A:XX:XX:XX:XX:XX
18 R  management-vlan-1010  vlan          1458   1564             78:XX:XX:XX:XX:XX
19 RS trunk-client-81       eoip          1458  65535             FE:XX:XX:XX:XX:XX
20 RS wifi-2.4G             wifi          1500                    78:XX:XX:XX:XX:XX
21 RS wifi-5G               wifi          1500                    78:XX:XX:XX:XX:XX
22 R  wifi1                 wifi          1500                    7A:XX:XX:XX:XX:XX

Which results in certain sites no longer being loadable with timeout error (ERR_TIMED_OUT in Google Chrome). This happens on locally connected machines via wifiwave2 or ethernet, which is traffic that doesn’t go to the tunnel.

The guide has following note, which speaks about traffic going through the tunnel:


EoIP tunnel adds at least 42 byte overhead (8byte GRE + 14 byte Ethernet + 20 byte IP). MTU should be set to 1500 to eliminate packet fragmentation inside the tunnel (that allows transparent bridging of Ethernet-like networks so that it would be possible to transport full-sized Ethernet frame over the tunnel).

However, this had effect on connectivity outside of tunnel. I guess, the bridge adjusted to the lowest MTU.

But, why slightly lower MTU (1458) broke HTTPS connectivity? And, why only certain sites, and not all of HTTPS traffic?

2

Additionally, there’s a bug (issue) - after setting it back to mtu=1500, the bridge won’t reset the actual-mtu of some ports back to 1500 (still have 1458):


/interface/eoip/set 0 mtu=1500

/interface/print
Flags: R - RUNNING; S - SLAVE
Columns: NAME, TYPE, ACTUAL-MTU, L2MTU, MAX-L2MTU, MAC-ADDRESS
 #    NAME                  TYPE    ACTUAL-MTU  L2MTU  MAX-L2MTU  MAC-ADDRESS
 0 R  ether1                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 1 RS ether2                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 2  S ether3                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 3  S ether4                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 4  S ether5                ether         1500   1568       9214  78:XX:XX:XX:XX:XX
 5    MYNET.hAP-ac3.1-2G1   wifi                                  2C:XX:XX:XX:XX:XX
 6    MYNET.hAP-ac3.1-2G2   wifi                                  2E:XX:XX:XX:XX:XX
 7    MYNET.hAP-ac3.1-5G1   wifi                                  2C:XX:XX:XX:XX:XX
 8    MYNET.hAP-ac3.1-5G2   wifi                                  2E:XX:XX:XX:XX:XX
 9    MYNET.hAP-ax2.1-2G1   wifi                                  48:XX:XX:XX:XX:XX
10    MYNET.hAP-ax2.1-2G2   wifi                                  4A:XX:XX:XX:XX:XX
11    MYNET.hAP-ax2.1-5G1   wifi                                  48:XX:XX:XX:XX:XX
12    MYNET.hAP-ax2.1-5G2   wifi                                  4A:XX:XX:XX:XX:XX
;;; defconf
13 R  bridge                bridge        1500   1568             78:XX:XX:XX:XX:XX
14 R  guest-vlan-30         vlan          1458   1564             78:XX:XX:XX:XX:XX
15 R  guest-vlan-31         vlan          1458   1564             78:XX:XX:XX:XX:XX
16 RS guest-wifi-2.4G       wifi          1500                    7A:XX:XX:XX:XX:XX
17  S guest-wifi-5G         wifi          1500                    7A:XX:XX:XX:XX:XX
18 R  management-vlan-1010  vlan          1458   1564             78:XX:XX:XX:XX:XX
19 RS trunk-client-81       eoip          1500  65535             FE:XX:XX:XX:XX:XX
20 RS wifi-2.4G             wifi          1500                    78:XX:XX:XX:XX:XX
21 RS wifi-5G               wifi          1500                    78:XX:XX:XX:XX:XX
22 R  wifi1                 wifi          1500                    7A:XX:XX:XX:XX:XX

(works fine after reboot)

3

It’s a feature :wink: (or rather flexibility).

The thing is that EOIP has quite a bit of overhead, so its L2MTU would naturally have to be lower than MTU of underlying interface. So the corresponding MTU is set if left to auto. However, EOIP is capable of fragmenting/defragmenting frames, hence you can set L2MTU to almost any (large) value. I guess it’s not done automatically due to additional overhead that fragmentation comes with.

Next: bridge will assume L2MTU equal to lowest L2MTU of member ports (because plain ethernet bridge can not fragment frames) and (L3) MTU will adjust downwards accordingly. I’m not sure how this affects changing MTUs of other bridge ports though. Now, after L2MTU of the offending bridge port is increased, all others (i.e. bridge) have to be increased manually as well, specially L3 MTU (it is quite often that L3 MTU is much lower than L2MTU of underlying interface and that’s perfectly fine).

4

In my case MTU of VLANs got re-adjusted after a reboot.

How does the bridge derive which interface is going to carry EoIP? In principal, underlying interface can be switched on a whim by changes in routing tables.

5

Upgraded home MTIK to faster HW and got problems - many popular sites lagging a lot. This post helped to understand that the reason was MTU=1458 on the main bridge due to disabled but not deleted EoIP to summer house where EoIP MTU was not forced to any particular value, and become 1458 by default. Practical test shows that anything below 1500 is too low for internet to work properly.

I believe by default MTU shall stay 1500 as current logic to decrease it to 1458 creates problems and time waste for the users as it requires investigations why the internet is half functioning - many pages do not load without pressing refresh several times, etc. - the only practical solution is to make EoIP MTU to 1500 to get router back to working state.

In latest RouterOS version bridge MTU went instantly up from 1458 to 1500 after MTU of EoIP was changed to 1500.

MTIK needs to correct the default MTU for EoIP to be 1500 if not entered, NOT 1458. EoIP will need to send 2 frames if larger than 1458 frame enters tunnel and frameSize+EoIP overhead is above physical MTU max. between EoIP endpoints. Until default is not changed, every time after creating EoIP tunnel it is obligatory to correct wrong default and force 1500 for EoIP MTU - otherwise 1458 will be applied to main bridge and that breaks internet stability as too many popular sites refuse to fragment packets with MTU below 1500.

6

In practical terms, the bridge (or VLAN on bridge) has no choice but to adjust MTU to lowest of it peers. e.g. multicast/broadcasts. Basically MTU is negotiated & lowest wins, as an interface…the bridge does this. So that part make sense IMO.

The comment “breaks HTTPS” is just not quite fair: if a TCP stack follows the specs, MTU can be as low as ~576. Now, as observed, some apps/stacks are dumb… so MTU (and resulting TCP MSS) being at 1500 may be needed. A more common way it breaks… is if “ping” (ICMP) is blocked, since that how peers find out the MTU/MSS to use.

In terms of default MTU for EoIP… Setting MTU 1500 means the outer tunnel will fragmentation – that’s not good either. Since you’d have one 1500 bytes, but ~50 byte packet and 2nd packet same overhead of headers being repeated. So if goal was to maximize bandwidth of a large file transfer… you’d want MTU to correct (so 1458 if path has 1500, lower still for if WAN has lower MTU) & a MTU 1500 “waste bandwidth” in duplicated headers for small sized fragments.

It’s settable, so if 1500 work, and fragmentation isn’t a concern (e.g. there isn’t a lot of data), that might be okay/needed… But as to the default… EoIP may have to be adjusted anyway, for a variety of reasons (e.g. MTU might be lower than 1500 before you even add EoIP, like some PPPoE or LTE WANs). But it defaulting to a setting that does NOT cause fragmentation if used over “default” MTU of 1500, make sense to me. Adjust if desired.