Hi everyone,
recently I had to apply some change to a network which involves remote locations connected (via LTE routers) to a central router R1. In general the devices behind LTE are connected to central servers at R1 directly via Internet connection.
For management reason all LTE devices connect to R2 using L2TP-IPSec tunnels, since some LTE device is still on v6 and Wireguard is not an option for them.
There is a Wireguard tunnel between R1 and R2, as most of the network is v7 based.
I had to connect a device behind LTE3 directly to R1 without appearing via Internet connection, so that server at R1 would see a private address instead.
I had tried to establish a new L2TP-IPSec connection from LTE3 and R1 and it all worked fine.
The communication from the specific host behind LTE3 and server on R1 (192.168.4.5) went fine as needed. All the other devices behind LTE1, LTE2 were still fine connected to server 192.168.4.2 via public IP connection as usual.
The next day I started to see major disconnections at R2 from LTE1, LTE2 and LTE3 which were loosing their connection to R2.
When I initially established the new L2TP tunnel from LTE3 to R1 I was skeptical about generating a sort of "L3 loop" as R1 and R2 are also connected via Wireguard tunnel, but I decided to try this config anyway.
The funny thing was that whenever each LTE devices was re-establishing L2TP connection to R2 I was seeing as coming from similar public IP, instead of their own.
So then I decided to kill the added L2TP from LTE3 and R1, made all LTE routers to reboot (via watchdog implementation that I have on each LTE) so that they were able to re-establish all L2TP links.
Then to work the specific implementation I routed traffic from LTE3 -> R2 -> R1 -> 192.168.4.5 using the existing L3 tunnel links and everything has worked fine. Even after a day no major disruption has happened so far.
For info the Wireguard tunnel does not allow 172.18.1.0/24 subnet to traverse the tunnel, but allows the other subnets like 192.168.4.0/24 to do so.
I don't know whether other aspects (mostly at the LTE ISP side) might have hit this setup hours later after being implemented which caused such disruption, but I guess I have generated some sort of L3 looping.
Now everything looks stable and running fine, was just curious to share and get some opinion.
Regards