L2 tunnel, proved by ISP through VLAN

RouterOS General
1

A short introduction

There are two branches, each with its own subnet (192.168.200.0/23 in the main one, 192.168.199.0/24 in the second one). The connection between the offices was organized via a wireguard tunnel, routes were routed through these tunnels and everything worked fine. Since data flows increased, it became necessary to separate the traffic between the branches and the Internet traffic, so a direct L2 tunnel was requested from the provider. The ISP wrapped this in a VLAN (3255 - Internet, 3777 - L2 tunnel). There were no problems with the Internet - everything was set up quickly and everything worked, but there were problems with L2, now I will describe the connection diagram and attach the configurations with a description of the problem.

So, here th scheme of BRANCH 1 and BRANCH 2 with some config information (routes through wireguard, this configuration works well):


BRANCH1 distribution router (RB3011)

/interface bridge
add name=bridge_wan port-cost-mode=short
add name=bridge_l2

/interface ethernet
set [ find default-name=ether1 ] comment="ISP office internet"
set [ find default-name=ether2 ] comment="VM1"
set [ find default-name=ether3 ] disabled=yes
set [ find default-name=ether4 ] disabled=yes
set [ find default-name=ether5 ] disabled=yes
set [ find default-name=ether6 ] comment="L2 to SFP3_CCR2004"
set [ find default-name=ether7 ] disabled=yes
set [ find default-name=ether8 ] disabled=yes
set [ find default-name=ether9 ] disabled=yes
set [ find default-name=ether10 ] comment="MGMT lan"
set [ find default-name=sfp1 ] comment="ISP"

/interface vlan
add comment="isp internet" interface=sfp1 name=vlan_3255 vlan-id=3255
add comment="isp l2" interface=sfp1 name=vlan_3777 vlan-id=3777

/interface bridge filter
add action=drop chain=forward in-interface=ether1 mac-protocol=ip src-address=!*.*.*.234/32
add action=drop chain=forward in-interface=ether2 mac-protocol=ip src-address=!*.*.*.69/32
add action=drop chain=forward in-interface=ether3 mac-protocol=ip src-address=!*.*.*.68/32
add action=drop chain=forward in-interface=ether4 mac-protocol=ip src-address=!*.*.*.67/32
add action=drop chain=forward in-interface=ether5 mac-protocol=ip src-address=!*.*.*.66/32

/interface bridge port
add bridge=bridge_wan ingress-filtering=no interface=ether1 internal-path-cost=10 path-cost=10
add bridge=bridge_wan ingress-filtering=no interface=ether2 internal-path-cost=10 path-cost=10
add bridge=bridge_wan ingress-filtering=no interface=ether3 internal-path-cost=10 path-cost=10
add bridge=bridge_wan ingress-filtering=no interface=ether4 internal-path-cost=10 path-cost=10
add bridge=bridge_wan ingress-filtering=no interface=ether5 internal-path-cost=10 path-cost=10
add bridge=bridge_wan interface=vlan_3255

add bridge=bridge_l2 interface=vlan_3777
add bridge=bridge_l2 interface=ether6

BRANCH1 main router (CCR2004):

/interface bridge
add comment="lan bridge" name=bridge1 port-cost-mode=short priority=0x7000

/interface ethernet
set [ find default-name=ether1 ] disabled=yes
set [ find default-name=sfp-sfpplus1 ] comment="wan"
set [ find default-name=sfp-sfpplus2 ] disabled=yes
set [ find default-name=sfp-sfpplus3 ] comment="L2 to eth6_RB3011"
set [ find default-name=sfp-sfpplus4 ] disabled=yes
.....................................

/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1

/interface wireguard peers
add allowed-address=10.100.100.253/32,192.168.199.0/24 interface=wireguard1 name=peer1 \
    persistent-keepalive=10s public-key="some key" responder=yes
    
/interface bridge port
add bridge=bridge1 ingress-filtering=no interface=sfp-sfpplus2 internal-path-cost=10 path-cost=10 \
    trusted=yes
add bridge=bridge1 interface=sfp-sfpplus4 internal-path-cost=10 path-cost=10
add bridge=bridge1 interface=sfp-sfpplus5 internal-path-cost=10 path-cost=10
add bridge=bridge1 interface=sfp-sfpplus6 internal-path-cost=10 path-cost=10
add bridge=bridge1 interface=sfp-sfpplus7 internal-path-cost=10 path-cost=10
add bridge=bridge1 interface=sfp-sfpplus8 internal-path-cost=10 path-cost=10
add bridge=bridge1 interface=sfp-sfpplus9 internal-path-cost=10 path-cost=10

/ip address
add address=*.*.*.234/30 comment="WAN ISP1" interface=sfp-sfpplus1 network=*.*.*.232
add address=192.168.201.254/23 comment="lan" interface=bridge1 network=192.168.200.0
add address=10.100.100.254/24 interface=wireguard1 network=10.100.100.0
add address=10.100.101.254/24 interface=sfp-sfpplus3 network=10.100.101.0

/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=*.*.*.233
add disabled=no distance=1 dst-address=192.168.199.0/24 gateway=wireguard1 routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10

BRANCH 2 main router (RB3011):

/interface bridge
add comment="lan bridge" name=bridge_lan port-cost-mode=short

/interface ethernet
set [ find default-name=ether1 ] disabled=yes
set [ find default-name=ether2 ] disabled=yes
set [ find default-name=ether3 ] comment="dowstream switch"
.........................
set [ find default-name=ether10 ] comment=isp_wan_plus_l2 poe-out=off
set [ find default-name=sfp1 ] disabled=yes

/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1
/interface wireguard peers
add allowed-address=10.100.100.254/32,192.168.200.0/23 endpoint-address=*.*.*.234 endpoint-port=13231 interface=wireguard1 name=peer1 \
    persistent-keepalive=10s public-key="some key"
    
/interface vlan
add comment="isp internet" interface=ether10 name=vlan_3255 vlan-id=3255
add comment="l2 isp" interface=ether10 name=vlan_3777 vlan-id=3777

/interface bridge port
add bridge=bridge_lan ingress-filtering=no interface=ether3 internal-path-cost=10 path-cost=10 trusted=yes

/ip address
add address=192.168.199.254/24 comment="lan" interface=bridge_lan network=192.168.199.0
add address=*.*.*.58/30 comment="isp wan" interface=vlan_3255 network=*.*.*.56
add address=10.100.101.253/24 interface=vlan_3777 network=10.100.101.0
add address=10.100.100.253/24 interface=wireguard1 network=10.100.100.0

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=*.*.*.57 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=192.168.200.0/23 gateway=wireguard1 routing-table=main scope=30 suppress-hw-offload=no target-scope=1

So, here is the thing. In this configuretion everything works fine and traffic beetwee branches goes through wireguard tunnel (routes via wireguard interfaces), also i can ping L2 tonnel endpoints (10.100.101.254/24 and 10.100.101.253/24 both directions) and run BWtest beetween them, but if i try to switch rotes via L2 interfaces

BRACNH 1 main router

/ip route
..................
add disabled=no distance=1 dst-address=192.168.199.0/24 gateway=sfp-sfpplus3 routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=1

BRANCH 2 main router

/ip route
.................
add disabled=no distance=1 dst-address=192.168.200.0/23 gateway=vlan_3777 routing-table=main scope=30 suppress-hw-offload=no target-scope=10

traffic doesnt pass (but still ping L2 endpoints). All issues with firewall were excluded due whole day testing by disabling all drop rules on both sides. Interesting thing, when i was trying to catch ICPM packets in this case (rotes through L2 endpoints) i made a rule for logging ICMP and found out log messages like this on both sides:

prerouting: in:lo out:(unknown 0), connection-state:related proto ICMP (type 3, code 1), 10.100.101.254->10.100.101.254, len 84

on BRANCH 1 main router when try to ping BRANCH 2 subnet and

prerouting: in:lo out:(unknown 0), connection-state:related proto ICMP (type 3, code 1), 10.100.101.253->10.100.101.253, len 84

when vice versa.

Give me some advice, please, before i will write a message to my ISP’s support (i suppose it’s my misunderstanding of routing or something else).
branch1.png
branch2.png

2

Whereas you can use the wireguard interface as a gateway of a route because a wireguard interface is an L3 one so the wireguard instance receives whatever the router sends through the interface, setting an L2 interface (Ethernet or VLAN) as a gateway of a route works different.

If the router chooses such a route, it sends an ARP request asking for a MAC address of a device that has the destination IP address, and if it receives a response, it sends the packet to that MAC address. But you haven’t bridged the L2 tunnel with the local LAN segment at either site (which by itself is rather a positive thing), so the ARP request cannot reach the actual host on the remote site, and therefore no ARP response can arrive and the IP packet is never sent.

As said I don’t think bridging the LAN segments of the two sites together using the L2 tunnel is a good idea. Instead, I would assign an address from a non-conflicting private subnet to each interface, like 10.10.10.199/28 to vlan_3777 on the branch office router and 10.10.10.200/28 to sfp-sfpplus3 at the main office router, and change the gateway values of the routes from vlan_3777 and sfp-sfpplus3 to 10.10.10.200/28 and 10.10.10.199/28, respectively (so that each gateway was the IP address of the remote device in that subnet).

Loosely related, disabling action=drop rules is always a bad idea. If you suspect the firewall to cause issues, it is always better to make a copy of the action=drop rule you suspect with action=log and some meaningful log-prefix value and place it right before (above) the drop rule you have copied it from. Doing so will tell you whether that action=drop rule indeed drops the traffic that “went missing” or there is another reason.

3

Thanks a lot, it’s worked! I was confused and didn’t think to try to specify the IP of another branch from the L2 tunnel interface on the opposite side as a gateway, but I specified it as in the wireguard option. Now routes looks just like this:

BRANCH1 main router


/ip route
add disabled=no distance=1 dst-address=192.168.199.0/24 gateway=10.100.101.253 routing-table=main scope=30 suppress-hw-offload=no target-scope=10

BRANCH2 router


/ip route
add disabled=no distance=1 dst-address=192.168.200.0/23 gateway=10.100.101.254 routing-table=main scope=30 suppress-hw-offload=no target-scope=10

Thank you very much for other advices. Topic closed.