We are working on a project where we have four sites we need to link together, in addition we want all internet and intranet traffic to route from all remote site through the main office internet connection.
We have fiber to the premise at all the remote sites but the service uses PPPoE for login and access from ISP-B, and we have no option for anything else.
The main office has two internet connections, dedicated fibre from a ISP-A (MTU 1500) and a separate connection from ISP-B using the same PPPoE (MTU 1480) connection as the remote sites.
We have tested many different configuration with and without encryption.
GRE only
GRE over IPSEC
L2TP with and and without IPSEC encryption, with and without PPP Multilink
IPIP Tunnel with and without IPSEC encryption
We do not require L2, only L3 connectivity for everything.
The problem we are having is with performance on two fronts, one from enabling encryption, and second from packet fragmentation.
We also have concerns with MTU issues with large UDP packets and fragmentation.
RB4011 units are at the remote location an CCR1036-12G-4S is at the head office.
Raw performance from site to site without tunnel is around 700 Mbps through the PPPoE connections @ 1500 Bytes, and ~940 Mbps @ 1480 Bytes.
Through the tunnels we can reach an average of about 600 Mpbs only for appropriately sized packets to avoid fragmentation, as soon as we put anything larger through the tunnel performance collapses.
Currently we are testing L2TP with PPP Multiple to allow 1500 byes packets to flow without error. There is no perceived fragmentation so everything works, but fragmentation is occurring and we are seeing the performance hit.
Our main culprit appears to be packet fragmentation, and we are looking for methods or recommendations to get the best out of what we have.
If anyone has any suggestions I am option to trying anything. We were not expecting the full 940 Mbps but were hoping for ~400 Mbps @ 1500 Byte packets.
Also if anyone could comment on whether I should even be concerned with the maximum packet size for UDP and as a result toss L2TP in favour of GRE which in our tests was faster.
Thanks