two BCP tunnels on a single trunked physical interface - possible?

medix08 · July 5, 2024, 10:44am

Hi,

here’s the situation, as a single pic is worth a thousand words:

I have a backbone network using L2 switches with VLANs and frame tagging. I need to establish an encrypted L2 VPN between sites, a separate tunnel to each of them. There’s a node site as well, equipped with two clients requiring encrypted connection to each of their respective servers.

My question is - is it possible to configure BCP to run two separate instances (bridges) on top of a single physical interface, connected via trunk to the backbone switch?

I did a simpler topology in the past, using BCP when there was only a single client and a single server per each site pair - and all was well until the requirements changed.

sindy · July 5, 2024, 4:23pm

Maybe a picture is worth a 1k words, but quite some words are still missing.

If the VLANs 111 and 112 on the switches interconnected using a FO ring carry the transport traffic of the VPN, there is no reason why each of them could not carry a different BCP tunnel. Even a single VLAN&subnet on a single physical interface can carry many BCP tunnels, that’s not the issue.

If the traffic that lives inside VLANs 111 and 112 should be the payload of the BCP, I don’t get the whole hierarchy, as it kind of requires that VLANs 111 and 112 carry both the payload packets of the BCP and the transport packets carrying those payload ones inside, which makes little sense.

It is indeed possible to use VLAN 111 on the core switch ring and VLAN 111 as a payload of a BCP, but then these two VLAN 111 must be two distinct ones that are not bridged together at any point in the whole network. And then the question is whether the payload of the BCP needs VLAN tagging at all, as you can simply set one tagless BCP tunnel from PC2 to SRV1 (that’s exactly what’s on your diagram, these two share “VLAN 111”) and another BCP tunnel from PC1 to SRV2 (in “VLAN 112”).

I suppose your goal is to migrate from a state where the “payload VLAN 111” and “transport VLAN 111” are one transparent L2 VLAN across your whole topology, so the traffic between PC2 and SRV1 passes through the core switches in plaintext, to a state where the traffic between PC2 and SRV1 passes through the core switches encrypted but other than that almost nothing changes about the setup, is my guess correct?

If it is, a lot depends on the complete topology - I assume you have simplified it to bare bone to make it comprehensible but the “unrelated” parts may decide whether such a migration will be possible or not and how complicated it will be. Are there any other VLANs than 111 and 112 that you are currently sending among the three routers through the core switches, and do you intend to encrypt them too?

medix08 · July 8, 2024, 2:00pm

Hi,

thanks for the reply.

Indeeed, real life setup is a bit more complicated.

https://ibb.co/dGZTVND

This is an industrial setting, and at first, the only thing you had was bare backbone network running on native VLAN1 (just a bunch of switches connected to form a ring). No IPs on switches, just PCs and servers reaching each other based only on their IPs (bad idea, but that’s when I came and messed everything up even worse )

Second iteration was to implement BCP on Mikrotik devices - basically they would act as transparent point-to-point encryption devices that would protect the SRV-CLI conversation without both of them even noticing. BCP was chosen, because each client-server pair worked in separate subnet. This is reflected in the PC3-SRV3 connection on the picture.

Third iteration was to add another layer (beneath) and split each CLI-SRV pair into a separate VLAN. This is where the problem comes in. If I have a location in which I have two clients requiring access to two separate servers I have to either:

Generate two separate VLANs on a single Mikrotik device and trunk them to the backbone switch. This in turn would require Mikrotik to differentiate between two VLAN tags and assign the respective frames to two separate BCP sessions running on it - and the question is - is it at all possible;
Use an extra Mikrotik on a “one SRV/CLI pair = one BCP session = one encryption device on each end” basis and drop the trunk idea altogether - this in turn gobbles up the backbone switches’ physical ports and increases the overall number of devices.

I’ve already tried EoIP, but running an instance of it on hAP Lite eats 99% of CPU - pings have ~1000ms latency and the whole idea simply breaks apart due to hardware requirements.

To make matters worse, there’s some sort of a central location with clients who also need to have server access - when there was no VLAN separation all worked fine, it was only a matter of having one BCP session from local CLI to local SRV and another session from remote (central) CLI to aforementioned local SRV.

sindy · July 8, 2024, 2:18pm

Hm. The EoIP encapsulation and decapsulation process is in fact equally simple like the BCP one so I wonder how it could have such a serious impact on the CPU load as compared to BCP. It rather looks to me like some L2 loop you have created unintentionally.

Anyway, I still don’t get the relationship between the VLANs on the payload side of the BCP tunnels and on their transport side. So let’s let the configurations speak rather than drawings and stories - please post the export of the configurations of the two Mikrotiks that serve VLAN 113; I assume no public IPs are involved so for anonymisation, obfuscate just the usernames, all sorts of passwords/secrets, and maybe the serial numbers before posting.

Use [code] and [/code] tags to surround the exported config, so the result will be similar to

# jul/08/2024 13:14:42 by RouterOS 6.49.13
# software id = <removed>
#
# model = RouterBOARD D52G-5HacD2HnD-TC
# serial number = <removed>
/caps-man datapath
add name=test
add local-forwarding=yes name=bridge-test vlan-id=444 vlan-mode=use-tag

medix08 · July 9, 2024, 9:23am

OK, here it goes:

first one is from the “central” location, the one with PC1 and PC2 - topmost location on the diagram". It works fine, but without any VLANs configured.

# software id = 
#
# model = RB750r2
# serial number = 
/interface bridge
add comment=WAN name=bridge1 protocol-mode=none
add comment="PC1 - SRV1" \
    name=bridge_bcp1 protocol-mode=none
add comment="SRV2 - PC2" \
    name=bridge_bcp2 protocol-mode=none
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec profile
set [ find default=yes ] dh-group=modp1024 enc-algorithm=aes-256 \
    hash-algorithm=sha256
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc
/ip pool
add name=vpnpool ranges=172.22.22.2-172.22.22.50
/ppp profile
add name=vpn remote-address=vpnpool use-compression=yes use-encryption=\
    required use-mpls=no
add bridge=bridge_bcp1 name=bcp_bridge1 remote-address=vpnpool use-compression=\
    yes use-encryption=required use-mpls=no
add bridge=bridge_bcp2 name=bcp_bridge2 remote-address=vpnpool use-compression=\
    yes use-encryption=required use-mpls=no
/interface bridge port
add bridge=bridge1 interface=ether1
add bridge=bridge_bcp1 interface=ether2
add bridge=bridge_bcp2 interface=ether3
/ip neighbor discovery-settings
set discover-interface-list=none
/interface l2tp-server server
set authentication=mschap2 enabled=yes \
    ipsec-secret=whateva mrru=1600 use-ipsec=required
/ip address
add address=110.2.111.13/24 comment=\
    "PC1 - SRV1" interface=bridge_bcp1 \
    network=110.2.111.0
add address=110.2.112.13/24 comment=\
    "PC2 - SRV2" interface=bridge_bcp2 \
    network=110.2.112.0
add address=172.2.2.1/24 comment="WAN side IP for Mikrotik BCP" interface=\
    bridge1 network=172.2.2.0
#management
add address=192.168.254.200/24 interface=ether5 network=192.168.254.0
#
/ip firewall filter
add action=accept chain=input in-interface=bridge1 port=500,4500,1701 \
    protocol=udp src-address=172.2.2.0/24
add action=accept chain=input in-interface=bridge1 protocol=ipsec-esp \
    src-address=172.2.2.0/24
add action=accept chain=input in-interface=bridge1 protocol=ipsec-ah \
    src-address=172.2.2.0/24
add action=accept chain=output dst-address=172.2.2.0/24 out-interface=\
    bridge1 port=500,4500,1701 protocol=udp
add action=accept chain=output dst-address=172.2.2.0/24 out-interface=\
    bridge1 protocol=ipsec-esp
add action=accept chain=output dst-address=172.2.2.0/24 out-interface=\
    bridge1 protocol=ipsec-ah
add action=accept chain=input connection-state=established,related,untracked
add action=accept chain=output connection-state=established,related,untracked
add action=accept chain=input dst-port=2137 protocol=tcp src-address=\
    192.168.254.100
add action=accept chain=output protocol=tcp src-address=192.168.254.100 \
    src-port=2137
add action=drop chain=input
add action=drop chain=output
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh address=192.168.254.100/32
set api disabled=yes
set winbox address=192.168.254.100/32 port=2137
set api-ssl disabled=yes
/ppp secret
add local-address=172.22.22.1 name=whateva password=whateva! profile=\
    bcp_bridge1 service=l2tp
add local-address=172.22.22.1 name=whateva password=whateva! profile=\
    bcp_bridge2 service=l2tp
/system identity
set name="Central_ServerVPN"
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none

Then, there’s SRV1 location:

# software id =
#
# model = RB750r2
# serial number =
/interface bridge
add comment=WAN name=bridge1 protocol-mode=none
add comment="SRV1-PC1" name=\
    bridge_bcp protocol-mode=none
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec profile
set [ find default=yes ] dh-group=modp1024 enc-algorithm=aes-256 \
    hash-algorithm=sha256
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc
/ppp profile
add name=vpn use-compression=yes use-encryption=required use-mpls=no
add bridge=bridge_bcp name=bridge_bcp use-compression=yes use-encryption=\
    required use-mpls=no
/interface l2tp-client
add allow=mschap2 comment="SRV1-Central_location" connect-to=172.2.1.1 disabled=no \
    ipsec-secret=whateva mrru=1600 name=l2tp-outLCS password=\
    whateva! profile=bridge_bcp src-address=172.2.1.11 use-ipsec=yes user=\
    whateva
add allow=mschap2 comment="SRV1-PC1" connect-to=172.2.2.1 disabled=no \
    ipsec-secret=whateva mrru=1600 name=l2tp-outND password=whateva! \
    profile=bridge_bcp use-ipsec=yes user=whateva
/interface bridge port
add bridge=bridge1 interface=ether1
add bridge=bridge_bcp interface=ether2
add bridge=bridge_bcp interface=ether3
/ip neighbor discovery-settings
set discover-interface-list=none
/ip address
add address=172.2.1.11/24 comment="WAN to central location " interface=bridge1 network=\
    172.2.1.0
add address=110.2.111.14/24 comment="SRV1" \
    interface=bridge_bcp network=110.2.111.0
add address=172.2.2.11/24 comment="WAN side SRV1" interface=bridge1 \
    network=172.2.2.0
# management
add address=192.168.254.200/24 comment=mgmt interface=ether4 network=\
    192.168.254.0
#
/ip firewall filter
add action=accept chain=input comment="IN UDP 500,4500,1701" \
    in-interface=bridge1 port=500,4500,1701 protocol=udp src-address=\
    172.2.0.0/16
add action=accept chain=input comment="IN ipsec 50" in-interface=\
    bridge1 protocol=ipsec-esp src-address=172.2.0.0/16
add action=accept chain=input comment="IN ipsec 51" in-interface=\
    bridge1 protocol=ipsec-ah src-address=172.2.0.0/16
add action=accept chain=output comment="OUT 500,4500,1701" \
    dst-address=172.2.0.0/16 out-interface=bridge1 port=500,4500,1701 \
    protocol=udp
add action=accept chain=output comment="OUT ipsec 50" dst-address=\
    172.2.0.0/16 out-interface=bridge1 protocol=ipsec-esp
add action=accept chain=output comment="OUT ipsec 51" dst-address=\
    172.2.0.0/16 out-interface=bridge1 protocol=ipsec-ah
add action=accept chain=input connection-state=established,related,untracked
add action=accept chain=output connection-state=established,related,untracked
add action=accept chain=input dst-port=2137 protocol=tcp src-address=\
    192.168.254.100
add action=accept chain=output connection-state=established,related,untracked \
    protocol=tcp src-address=192.168.254.100 src-port=2137
add action=drop chain=input
add action=drop chain=output
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh address=192.168.254.100/32
set api disabled=yes
set winbox address=192.168.254.100/32 port=2137
set api-ssl disabled=yes
/system identity
set name=SRV1
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none
/tool mac-server ping
set enabled=no

And finally, SRV2 location, all of these work with the assumption of no VLANs being configured on the Mikrotik devices

# software id = 
#
# model = RB750r2
# serial number = 
/interface bridge
add comment=WAN name=bridge1 protocol-mode=none
add comment="SRV2-PC2" name=\
    bridge_bcp protocol-mode=none
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec profile
set [ find default=yes ] dh-group=modp1024 enc-algorithm=aes-256 \
    hash-algorithm=sha256
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc
/ppp profile
add name=vpn use-compression=yes use-encryption=required use-mpls=no
add bridge=bridge_bcp name=bridge_bcp use-compression=yes use-encryption=\
    required use-mpls=no
/interface l2tp-client
add allow=mschap2 comment="central location PC2" connect-to=172.2.1.1 disabled=no \
    ipsec-secret=whateva mrru=1600 name=l2tp-outLCS password=\
    whateva! profile=bridge_bcp src-address=172.2.1.12 use-ipsec=yes user=\
    whateva
add allow=mschap2 comment="local PC2" connect-to=172.2.2.1 disabled=no \
    ipsec-secret=whateva mrru=1600 name=l2tp-outND password=whateva! \
    profile=bridge_bcp use-ipsec=yes user=whateva
/interface bridge port
add bridge=bridge1 interface=ether1
add bridge=bridge_bcp interface=ether2
add bridge=bridge_bcp interface=ether3
/ip neighbor discovery-settings
set discover-interface-list=none
/ip address
add address=172.2.1.12/24 comment="WAN to central location" interface=bridge1 network=\
    172.2.1.0
add address=110.2.112.14/24 comment="SRV2" \
    interface=bridge_bcp network=110.2.112.0
add address=172.2.2.12/24 comment="WAN side SRV2" interface=bridge1 \
    network=172.2.2.0
#management
add address=192.168.254.200/24 comment=mgmt interface=ether5 network=\
    192.168.254.0
#
/ip firewall filter
add action=accept chain=input comment="IN udp 500,4500,1701" \
    in-interface=bridge1 port=500,4500,1701 protocol=udp src-address=\
    172.2.0.0/16
add action=accept chain=input comment="IN ipsec 50" in-interface=\
    bridge1 protocol=ipsec-esp src-address=172.2.0.0/16
add action=accept chain=input comment="IN ipsec 51" in-interface=\
    bridge1 protocol=ipsec-ah src-address=172.2.0.0/16
add action=accept chain=output comment="OUT 500,4500,1701" \
    dst-address=172.2.0.0/16 out-interface=bridge1 port=500,4500,1701 \
    protocol=udp
add action=accept chain=output comment="OUT ipsec 50" dst-address=\
    172.2.0.0/16 out-interface=bridge1 protocol=ipsec-esp
add action=accept chain=output comment="OUT ipsec 51" dst-address=\
    172.2.0.0/16 out-interface=bridge1 protocol=ipsec-ah
add action=accept chain=input connection-state=established,related,untracked
add action=accept chain=output connection-state=established,related,untracked
add action=accept chain=input dst-port=2137 protocol=tcp src-address=\
    192.168.254.100
add action=accept chain=output connection-state=established,related,untracked \
    protocol=tcp src-address=192.168.254.100 src-port=2137
add action=drop chain=input
add action=drop chain=output
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh address=192.168.254.100/32
set api disabled=yes
set winbox address=192.168.254.100/32 port=2137
set api-ssl disabled=yes
/system identity
set name=SRV2
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none
/tool mac-server ping
set enabled=no

sindy · July 9, 2024, 10:40am

OK, so now the following question is why exactly do you need the VLANs. This way (the port for each PC on the central router has its own bridge tunneled to a matching bridge on the server side), the traffic between each PC and its respective server is logically separated from the traffic between the other PC and its respective server to exactly the same extent as if they were using two distinct VLANs on a single set of interconnected bridges. On the transport side, the proper mapping of the client side bridge to the desired server side bridge is ensured only by L2TP username and password since you let RouterOS create the IPsec configuration dynamically, which means that all clients must use a common pre-shared key; if you wanted more robustness against misconfiguration or even intentional hacking on client side, you’d have to configure the IPsec layer manually at least at the L2TP server side, but that would still only work unless configuration for one BCP client would leak to another one.

Just adding VLANs to the protocol hierarchy at transport side would not help much as you can only configure a single L2TP server that listens on all interfaces, so if you wanted to connect every BCP client to an access port to another VLAN as a way to prevent linking incorrect bridges on client and server side (because the hacker would have to change the PVID on the access port of the “core” switch in order to connect to a “wrong” VLAN, not just the configuration of the client), you would have to use PPPoE for the transport layer, as unlike L2TP, the PPPoE servers can be instantiated and linked to individual interfaces (in your case, VLAN ones). But then you would still have to use L2TP over IPsec to convey encrypted BCP because you cannot encapsulate PPPoE into IPsec, only vice versa.

All in all, it seems to me that you have some idea regarding the VLANs that is so far away from my way of thinking that I cannot grasp it. So if you think it is worth the effort to make me understand it, we can switch to a more interactive way of communication, using the guideline here.

medix08 · July 10, 2024, 1:20pm

Thanks for your help - it seems itt cannot be done, which is paradoxically, good news. Let me say the usual excuse - it wasn’t me (regarding VLANs).

The point is - the backbone network design changed. Previously, there was no VLAN segmentation and solution worked on a simple principle - one client and one server in a physical location, multiple clients in remote central location.

Now, rules have changed and I have two servers in a single location connecting to a single Mikrotik, because that’s the network designer’s vision. This single Mikrotik has to differentiate somehow which packets go to which server, only because the designer chose to use a single cable connection. Since said two devices belong to two separate CLI-SRV pairs and the backbone network designer assumed, that each pair of devices (well, triplet if central location client is included) will have its own VLAN - apparently just to ensure any form of communication between any two groups of devices is blocked at the L2 level. That’s the real background behind the whole story.

sindy · July 10, 2024, 2:38pm

Well… I don’t say it cannot be done, what I say is I cannot see much point in doing it, provided that “it” means using two distinct VLANs to carry the transport traffic of two distinct BCP tunnels across a single physical interface. What cannot be done is to link an L2TP server with a dynamically created IPsec “sleeve” to a particular VLAN. But what you can do is the following:

create vlan interfaces on the routers and attach IP addresses from distinct subnets to them
manually create an IPsec peer at L2TP server side for each VLAN, setting the IP address attached to the VLAN interface as local-address of that peer
set up firewall rules in chain input that only accept incoming packets to each of the local IP addresses attached to VLAN interfaces only via these interfaces

With these measures in place, you can use an individual pre-shared key for the IPsec peer in each VLAN on the server side, so it requires a bit more effort to establish a wrong connection, but even that way, it is enough to mess up the L2TP username and password to connect a client to a wrong BCP. So the question is who is the enemy, a mere stupidity or evil intention?