I've gone through this several times now, finally getting to the bottom of it this time and I have it reproducible with a very simple config.


>> SIMPLEST SETUP TO REPRODUCE:
1) RB750Gr3 on 6.46.8 reset to default config.

2) Add an eoip interface (settings do not appear to matter at all, originally used valid/working connection to another router):
3) Add that eoip interface to the bridge:

4) Try to load these sites. Browser indicators will spin forever and loading will ultimately fail in FF or Chrome (latest versions):
- https://app.cardconnect-ip.com:8443/ -> Will just display "404 - Nothing Found" when successfully loaded.
- https://icloud.com -> Sometimes slightly/partially loads (but does not finish), so issue is probably connecting to a linked site/CDN/etc.
- https://mail.yahoo.com/ -> Does not load at all.
- https://www.conferencecalling.com/ -> Does not load at all.
-> There are others, but these were the ones I've documented so far.

5) Instantly fixed if eoip bridge port is *disabled* and site is reloaded. If you remove the interface or the bridge port entry, that works too, but this is the minimum change I can find that fixes it.

6) CRITICAL TEST PROCEDURE: If you ever load any site affected by this bug successfully (i.e. without the prerequisites for the "bad" setup) and then try to get back to the broken state again, you must first clear some browser settings (Ctrl-Shift-Del) and reload the site to experience the issue again immediately, so this is required before any possible "fix" to see if that fix works. I've just been clearing everything in the browsers because it's in a VM, but if you want specifics then In Chrome I've narrowed it to clearing only "Cookies and other site data" and in Firefox you must clear "Active Logins" (For time range, in both I'm doing "everything"/"all-time").



>> OTHER TESTING: I have tried the following from the broken state, tested each time using the browser clear procedure (item 6 above).

-> GOOD means the above sites (item 4 above) load, BAD means they spin/don't load.
1) Removed EoIP interface itself (not bridge port entry, which stays there labeled "Unknown") -> GOOD
2) Update to 6.47.7 -> BAD
3) Chrome/Firefox update to latest -> BAD
4) Update to 6.48.beta48 -> BAD



>> OTHER NOTES:
- My CRS125 (mipsbe) does not seem to have this issue as I have the same setup (bridged EoIP tunnels) on a second bridge controlling the last half of the ports, so it may be specific to either the 750Gr3 (mmips) OR to mmips architectures in general.

- Based on this and some subsequent testing, I'm pretty sure the barest prerequisites are just any eoip tunnel added to the bridge on an RB750Gr3 or any MMIPS device and the rest doesn't matter... but hard to test as I don't have a wide variety of devices to test with right now.

Probably a MTU problem. Manually change the bridge MTU to 1500 and see if it works.

Probably a MTU problem. Manually change the bridge MTU to 1500 and see if it works.

I had dismissed that initially, thinking that PMTUD would work around that... but you're right.

That said, I'm still 50/50 on whether or not this should be something people should have to manually change or not. It's not a problem for me, personally, but it it does seem like the bridge's auto-MTU algorithm might need a tweak. I guess I'll have to think on it more and do some research on the pros and cons... I'm sure this debate has happened before.

Because EoIP has overhead, it can't encapsulate a full 1500 byte packet in a 1500 byte packet to go out the normal 1500 byte Ethernet port. Auto default is to limit the packet size to what it can pack in without fragmenting the packet. But EoIP can fragment and reassemble without breaking TCP. By forcing the bridge to 1500 you force EoIP to fragment/reassemble. There are other ways to fix the issue, but I find this the simplest and most compatible, so it's the way I've chosen to do it.

Set Bridge MTU to 1280 and set needed Mangle rule, try if that works

I had dismissed that initially, thinking that PMTUD would work around that... but you're right.

PMTUD works between L3 devices and in case where MTU changes between two interfaces of same device routing traffic (mind that normal router behaviour is to fragment packets if needed and packet is rejected only if its DF flag is set). However, MTU on interfaces connecting to same L2 network should always be the same (on all interfaces connected) because device which gets "strangeled" with too big frame will simply discard such frame without any reaction towards sender.

That being said: if inclusion of EOIP interface reduces MTU of bridge which also spans ethernet interfaces connecting LAN devices, then this is a huge problem not only for traffic passing through said tunnel but also for any traffic between LAN devices and router (which is gatewaying traffic towards other networks).

So yes, IMHO current behaviour of ROS which automatically reduces bridge MTU upon addition of EOIP as bridge port is very dangerous and should be changed. Anybody wanting to streamline tunnel performance (e.g. to avoid fragmentation) can change MTU manually to desired value.

Changing MTU setting of the EoIP tunnel itself to 1500 seems to have the same effect, it is even recommended in the wiki.

Is there any drawback doing so?

I had dismissed that initially, thinking that PMTUD would work around that... but you're right.

PMTUD works between L3 devices and in case where MTU changes between two interfaces of same device routing traffic (mind that normal router behaviour is to fragment packets if needed and packet is rejected only if its DF flag is set). However, MTU on interfaces connecting to same L2 network should always be the same (on all interfaces connected) because device which gets "strangeled" with too big frame will simply discard such frame without any reaction towards sender.

That being said: if inclusion of EOIP interface reduces MTU of bridge which also spans ethernet interfaces connecting LAN devices, then this is a huge problem not only for traffic passing through said tunnel but also for any traffic between LAN devices and router (which is gatewaying traffic towards other networks).

So yes, IMHO current behaviour of ROS which automatically reduces bridge MTU upon addition of EOIP as bridge port is very dangerous and should be changed. Anybody wanting to streamline tunnel performance (e.g. to avoid fragmentation) can change MTU manually to desired value.

Thanks for your comment. I used to be very good about MTU and the like but haven't looked at it (or haven't had to) for so long I think I've got a bit soft in the head on it.

You're of course right about L2 vs L3. My other L2 domains all have matching MTUs as expected normally. I think it goes without saying that the least-common-denominator bridge algorithm is perhaps not the best way to solve the issue, particularly with EoIP being added, as EoIP/GRE are, as far as I remember, more flexible, being designed to fragment as needed internally and invisibly without L3/PMTUD being involved (hence its internal "don't fragment" option), plus it has TCP MSS Clamping enabled by default which helps TCP traffic automatically adjust. I'd almost rather the bridge check the MTUs of added/enabled ports and throw an error to the log, or maybe go to a red, but not disabled, state like other settings that are internally checked before applying, as I can't think of a good reason to have mismatched L2 MTUs.

I suppose having it default to its maximum minus overhead has its advantages, but you'd have to build a specific bridge around that MTU. My main reason for using EoIP is bridging, without setting up a full VPN, for diagnostics and I need it to be as transparent as possible for the diagnostics to be accurate and not affected by the tunnel, so I think I have my answer.

Thanks again, I appreciate your reminder of all I'd forgotten :-).