Hi All

I've run into a very strange situation which has me stumped. In short, I have a Cloud Router with a few bridges (few segregated networks & WAN bridge). So majority of the devices on these segregated networks are directly connected, but there are a few with remote L2TP/IPSec connections that get placed on which ever bridge/network they belong to.

So recently I realized that we were having issues with servers connecting to their Telegram bots for notifications. We usually test comms to Telegram by opening "api.telegram.org/bot " in a browser and it has become a hit and miss thing. Sometimes it will reply and sometimes it will simply time out. After lots of scratching I realized this was only happening to servers on bridges with external connections (L2TP). Tried disconnecting the VPN connections on the bridges we are having issues with and presto, the Telegram connections all stabilized.

Now for the million $ question, why would L2TP/IPSec connections (placed on a bridge and with no additional routing done) affect a server from establishing a simple 443/HTTPS connection to Telegram?

PS. The L2TP connections are from remote sites using LTE.

Thanks,
R

Can you elaborate on the relationship of the L2TP tunnels and the bridges? Are you using BCP (bridge in /ppp profile set to the bridge name)?

Yes, I configure the PPP Profile with the IP pools for local and remote address and then set the bridge I want it to park the connections on.

Sent from my BLA-L09 using Tapatalk

The thing is that the L3 tunnel (IPCP) and the L2 tunnel (BCP) are independent from each other. The local-address and remote-address parameters of the /ppp profile (or /ppp secret) are relevant for the L3 tunnel, whereas the bridge parameter is relevant for the L2 one. So I can imagine that the IP subnets attached to the bridges at the two devices which get interconnected by the BCP tunnel may overlap, and so may the host IP addresses in them, so the ARP responses may come from the "wrong" device - so instead of to the IP address of the local Tik on that bridge which is a default gateway for the servers in that subnet, the packets for outside may be sent to the MAC address of some remote device on the bridge.

What do you mean by the subnets might overlap? All devices, including the remote side, have statically assigned addresses. I also use a different range for the L3 tunnel than what's on the bridge.

Thanks so far

Sent from my BLA-L09 using Tapatalk

"static" and "overlapping" do not exclude each other. Do you have also a Mikrotik (or another device supporting BCP) on the remote end of the tunnel? If so, what is the Mikrotik's IP address assigned to the local bridge and the remote device's IP address assigned to the remote bridge, which are interconnected by the BCP tunnel? If not, I have never tested what happens if one end supports BCP and the other one doesn't. The BCP tunnel should not go up at all but we are looking for a weird behaviour here, right?

Yes, there is a Mikrotik at each end. Here is a quick sketch giving an example of one site. All IP's are statically assigned except for the VPN which gets IP's allocated from a certain pool and which falls within a completely different range.
Just as a side note, I initially thought that the Cloud Router might be "randomly" routing internet traffic out via the VPN to the remote router with LTE outbreak, but when running a packet sniffer on the remote Mikrotik, I never saw any signs of this.

So I have resolved the issue for now although I'm not happy about the method. Someone suggested to me to not rely on ARP discovery but instead use routing. Problem is that would then involve having to use different subnets for each remote site (sounds simple enough but is going to severely complicate my life having to manage not only the main subnets but their alternates as well).

Anyway, so I settled on creating and alternate bridge for every existing bridge with an alternate IP range where all VPN connection get settled. The bridge creates a dynamic route and I just have to create a route on the remote router back to the original bridge/subnet.

For now it works but I'd still like to figure out why these VPN connections would cause trouble with web traffic on a bridge.

@rules, does this still bother you?

Hi sindy

Not at all, I eventually took the time to figure out the OVPN setup and have been using that ever since. At some point I'll just have to make some time and "merge" the bridges again and hopefully it won't have the same effect.

Thanks,
R

I eventually took the time to figure out the OVPN setup and have been using that ever since. At some point I'll just have to make some time and "merge" the bridges again and hopefully it won't have the same effect.

The thing is that I've ran into the same issue in the meantime and realized what the reason was, but I was not able to google up this topic to update it earlier. I could only find it as you've popped up in another topic - I remembered the avatar, not the username.

The actual reason is that the MTU of a bridge interface adjusts to the smallest one of the MTUs of the member ports of the bridge. So once you make an L2 tunnel interface a member port of a bridge, the MTU of that bridge gets reduced, and if ICMP is blocked somewhere on the path from the bridge to the remote server and thus PMTUD cannot work properly, you end up with that kind of issues with TCP servers.

For L2TP with BCP, the remedy is easy, you have to configure the mrru value at both the client (on the /interface l2tp-client row) and at the server (in the /interface l2tp-server server settings) to at least 1504. Doing so will activate use of MLPPP on the L2TP link, which in simple terms activates internal fragmentation of the frames which don't fit, so the MTU of the bridge stays at 1500 once you interconnect the bridges.

Something is telling me that for actual bridging, the mrru has to be set to 1522 in order that vlan-tagged frames carrying 1500-byte IP packets would fit, but that's a step further, which is only interesting if you actually need to link the bridges. Since everything runs fine when you use separate bridges for the client connections, I assume you actually don't need L2-transparency of the tunnels at all as you don't need to extend the bridge in the central office to the clients. If my assumption is correct, just removing the bridge parameter from the /ppp profile rows could do the trick as well.

Adding an OpenVPN tunnel interface in TAP (ethernet) mode to a bridge doesn't reduce the MTU of that bridge, so I assume there is an internal fragmentation mechanism in OpenVPN in TAP mode too. In TUN (IP) mode, the MTU of the tunnel interface is lower than 1500, but that is not surprising.