Advice for an encrypted tunnel in the local network

JokeRBY · Thu Jun 09, 2022 2:36 am

Hi all,
I need some help with creating a good performance and compliant with Best Practice configuration for situation described below.

The client has two leased locations in the building, physically separated by a distance of app. 200m. Locations are connected by fiber-optic cable belonging to the owner of the building. And for security reasons this connection must be encrypted.
The network actively uses vlans, same tags may occur on both side.
Intended view of the L1 will look like:

switch A <sfp+> router A <sfp+ with tunnel> router B <sfp+> switch B

Routers in this case: CCR2004-1G-12S+2XS.

The current (test) configuration:

sfp28-1 on both sides used as physical connection between routers
/30 network used on this layer
sfp-sfpplus1 on both sides used as physical connection to switches.
EoIP with IPSec over /30 network is the next layer.
Next, bridge that includes EoIP and sfp-sfpplus1 interfaces. On both sides.
And finally, vlans with bridge as parent interface.

This configuration works correctly, but significantly reduces network performance. Smth about 1600Mbps in BTest. Without IPSec - 5500Mbps. Pure /30 - full 10Gbps.
With that in mind, I have two questions:

How correct is this configuration?
What can be changed in it to improve performance?
What are the optimal MTU values in this configuration and should they be changed at all?
Are there any better options for providing an encrypted tunnel under these conditions?

tdw · Thu Jun 09, 2022 5:19 pm

Why have you added the SFP interfaces to the bridge? The ethernet traffic will be encapsulated by the EoIP interface and routed across the /30 subnet between the SFP interfaces at either end. I'm surprised you do not have a broadcast storm by having both the EoIP and SFP interfaces in the same bridge.

You will have to increase the MTU on the SFP interfaces to accommodate the EoIP overhead (42 bytes) and IPsec encryption (varies, typically 38-73 depending on algorithm selected and padding) without fragmentation to provide better throughput, the IPsec encryption test results for that device suggest you might be able to achieve 2500-3000Mbps at best.

As you have a direct layer 2 connection then MACsec (802.1AE) would be more efficient as it does not require the overhead of IP encapsulation. Mikrotik introduced it in RouterOS 7 under /interface/macsec but there appears to be no documentation and others do not seem to have any luck getting it to work as yet.

JokeRBY · Fri Jun 10, 2022 12:06 am

Why have you added the SFP interfaces to the bridge? The ethernet traffic will be encapsulated by the EoIP interface and routed across the /30 subnet between the SFP interfaces at either end. I'm surprised you do not have a broadcast storm by having both the EoIP and SFP interfaces in the same bridge.

To clarify the situation, sfp28-1 (on which EoIP is running) and sfp-sfpplus1 (direct connection to switch) are not the same interfaces.

Current test configuration looks like:

/ip address add address=172.16.9.1/30 interface=sfp28-1 network=172.16.9.0
/interface ethernet
set [ find default-name=sfp-sfpplus1 ] comment="main network"
set [ find default-name=sfp28-1 ] comment="ptp physical connection"
/interface eoip add allow-fast-path=no comment=tunnel local-address=172.16.9.1 mac-address=FE:8B:86:91:F6:2C name=eoip-tunnel1 remote-address=172.16.9.2 tunnel-id=0
/interface bridge add name=bridge-lan vlan-filtering=yes
/interface bridge port
add bridge=bridge-lan ingress-filtering=no interface=sfp-sfpplus1
add bridge=bridge-lan ingress-filtering=no interface=eoip-tunnel1
/interface vlan
add interface=bridge-lan ...
...
add interface=bridge-lan ...

I'm honestly trying to figure out where and how this configuration can cause a network storm...

About macsec - looks promising, but without proper documentation it's hard to even check this option

tdw · Fri Jun 10, 2022 12:31 am

It was not clear in the original description that they were different bridges, so that is fine.

The EoIP interface should have mtu=1500 per the documentation, a lesser value will break things in odd ways. Then the sfp28-1 interface should have a MTU to accommodate the encapsulation and encryption.

Advice for an encrypted tunnel in the local network

Advice for an encrypted tunnel in the local network

Re: Advice for an encrypted tunnel in the local network

Re: Advice for an encrypted tunnel in the local network

Re: Advice for an encrypted tunnel in the local network

Who is online