Edit -
After more investigation, I've found that its just the tunnel itself that is much slower than I would expect so the below is just a symptom.
I have 2 hex S routers with an EOIP bridge between them over the internet. Doesn't run on top of a VPN or anything.
Both are on full GB fiber connections and both test at around 1Gbps up/down with a speed test, same ISP and pings between them are 2ms. The bandwidth between the two LANs is close to ~500Mbps/TCP and ~950Mbps UPD which if I understand correctly is about as much as I'm going to get with IPSEC on the tunnel. Which, as an aside, I would love to get rid of, but I can't get the tunnel to work without that enabled.
The problem I'm having occurs when I try to use the foreign gateway for internet access. That is if I use LAN 2 gateway on a LAN1 device the speed tests at about 24Mbps down and 40Mbs up. Same result if I use LAN1 Gateway for a device on LAN2 - which even accounting for the IPSEC overhead is about 1/20th the speed i would expect.
MTUs on the EOIP are set at 1500
I'm not doing anything fancy, but I'm obviously doing something wrong. Any ideas on where to investigate?
I can provide configs and diagrams if needed.
Thanks!