CRS354-48G-4S+2Q+ VLANs over bonding interface

RouterOS General
1

Hi everyone.

I have a quite simple setup: router CCR1016-12G and a switch. Router is connected to switch via 2xEthernet bonding interface.
Until yesterday I used CRS125-24G-1S as a switch with following configuration

CRS125, configured the “old way”:

/interface ethernet switch trunk
add member-ports=ether12-link1-to-ccr,ether13-link2-to-ccr name=\
    trunk1-link-to-ccr
...
    /interface ethernet switch egress-vlan-tag
add tagged-ports=trunk1-link-to-ccr,ether19,switch1-cpu \
    vlan-id=102
add tagged-ports=trunk1-link-to-ccr,ether19,switch1-cpu \
    vlan-id=101
add tagged-ports=trunk1-link-to-ccr,switch1-cpu vlan-id=100
add tagged-ports=trunk1-link-to-ccr,ether19,switch1-cpu \
    vlan-id=103
...
/interface ethernet switch vlan
add ports="trunk1-link-to-ccr,ether19,switch1-cpu" vlan-id=101
add ports="trunk1-link-to-ccr,ether19,switch1-cpu" vlan-id=102
add ports="trunk1-link-to-ccr,ether24,switch1-cpu" vlan-id=100
add ports="trunk1-link-to-ccr,ether19,ether24,switch1-cpu" vlan-id=103

CCR1016:

/interface bridge
add fast-forward=no name=bridge-trunk vlan-filtering=yes
...
/interface bonding
add mode=balance-xor name=bonding1 slaves=ether1,ether2 transmit-hash-policy=\
    layer-2-and-3
    ...
/interface bridge port
add bridge=bridge-trunk hw=no interface=bonding1
...
/interface vlan

add interface=bridge-trunk name=vlan100 vlan-id=100
add interface=bridge-trunk name=vlan101 vlan-id=101
add interface=bridge-trunk name=vlan102 vlan-id=102
add interface=bridge-trunk name=vlan103 vlan-id=103

Everything worked perfectly.
Yesterday CRS125 died, so I repaced it with CRS354-48G-4S+2Q+.
So, 3xx series switches have to have VLANs configured the “new way”, so I changed my config accordingly:

CRS354:

/interface bridge
add fast-forward=no name=bridge1 vlan-filtering=yes
...
/interface bridge port
add bridge=bridge1 interface=bonding1
add bridge=bridge1 interface=ether1
...
add bridge=bridge1 interface=ether11
add bridge=bridge1 interface=ether14
add bridge=bridge1 interface=ether15
...

/interface bonding
add mode=balance-xor name=bonding1 slaves=ether12,ether13 transmit-hash-policy=\
    layer-2-and-3
...
/interface bridge vlan
add bridge=bridge1 tagged="bonding1,bridge1" untagged=ether6 vlan-ids=10
add bridge=bridge1 tagged="bonding1,ether19,bridge1" vlan-ids=101-103
add bridge=bridge1 tagged="bonding1,bridge1" untagged=ether24 vlan-ids=100

No changes were made to CCR1016 configuration.

So, here is the problem: if I have it set up like this, the switch cannot access internet via router, though default gateway is set and switch receives ARP replies from router. All devices, connected to the switch, have internet access no problem, but! ICMP requests sent from those devices recieve duplicated replies:

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=250 time=18.3 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=250 time=18.4 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=1 ttl=250 time=18.4 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=1 ttl=250 time=18.4 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=2 ttl=250 time=18.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=250 time=18.3 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=2 ttl=250 time=18.3 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=2 ttl=250 time=18.3 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=3 ttl=250 time=18.2 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=250 time=18.3 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=3 ttl=250 time=18.3 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=3 ttl=250 time=18.3 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=4 ttl=250 time=18.3 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=250 time=18.3 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=4 ttl=250 time=18.3 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=4 ttl=250 time=18.3 ms (DUP!)

If I replace bonding with just ethernet interface — everything works perfectly, like it should.

So, my question is — is that a bug in RouterOS or I cannot have VLANs over bonding interfaces by design?

Thanks in advance.

2

Do you use any VLAN as management VLAN ?
Since i do not see the whole config,
What i would do is, have a management VLAN e.g. 99 setup on my router and then on the Switch, i would:
Set an IP Address on the VLAN 99 e.g. 192.168.99.2
Set DNS 192.168.99.1 (Router’s MGMT Vlan)
Add the Bridge as Tagged Member, as well as the Bonding Interface for the VID 99…
etc…

No, bonding works with VLANs just fine, there is an example here as well: https://wiki.mikrotik.com/wiki/Manual:CRS3xx_VLANs_with_Bonds#Bonding

3

Hi Zacharias, thanks for replying.


I do not quite follow your point. The problem is that there is no L3 connectivity between router and switch whatsoever when using VLANs over bonding interface.
I skipped most of the config, just assume that I use VLAN100 for management. My config was like you said: L3 interfaces “vlan100”, “vlan101” etc on both devices with IP addresses, added bridge interface and bonding interface as tagged to bridge VLANs. I got lack of L3-connectivity between devices and duplicated packets, and If I remove bonding and add ether interface to bridge instead — everything works like it should (I have L3-connectivity between devices on every VLAN, no packets are duplicated)


Yes, I was following this very guide.

4

I already have a setup on GNS3 for such a scenario, so i wil give you an example with working and tested code…
I ll give you the basic parts of the configuration…
Main Router R1:
Create Interface VLAN for every VID:

/interface vlan
add interface=bridge1 name=vlan10 vlan-id=10
add interface=bridge1 name=vlan20 vlan-id=20
add interface=bridge1 name=vlan30 vlan-id=30
add interface=bridge1 name=vlan99 vlan-id=99

Set an Adress for every Vlan ID:

/ip address
add address=192.168.99.1/24 interface=vlan99 network=192.168.99.0
add address=192.168.10.1/24 interface=vlan10 network=192.168.10.0
add address=192.168.20.1/24 interface=vlan20 network=192.168.20.0
add address=192.168.30.1/24 interface=vlan30 network=192.168.30.0

Add the Trunk ports to the Bridge:

/interface bridge port
add bridge=bridge1 interface=bonding-eth2-eth3

Switch SW2 (Trunk between R1 and SW2):
Trunk Port between switches…
…

/interface vlan
add interface=bridge1 name=vlan99 vlan-id=99

Set an address for the MGMT VLAN:

/ip address
add address=192.168.99.2/24 interface=vlan99 network=192.168.99.0

Add ports to the Bridge as well as Access ports:

/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged \
    ingress-filtering=yes interface=bonding2-3
add bridge=bridge1 frame-types=admit-only-vlan-tagged \
    ingress-filtering=yes interface=bonding4-5
add bridge=bridge1 frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes \
    interface=ether10 pvid=10
add bridge=bridge1 interface=bonding6-7

Here you can see how i give Layer 3 Access to the Switch from my 1st rule:
The Bridge Port is added as Tagged Member for my MGMT Vlan along with the Trunk Ports…

/interface bridge vlan
add bridge=bridge1 tagged=\
    bonding2-3,bonding4-5,bonding6-7,bridge1 vlan-ids=99
add bridge=bridge1 tagged=bonding2-3,bonding4-5,bonding6-7 \
    vlan-ids=10
add bridge=bridge1 tagged=bonding2-3,bonding4-5,bonding6-7 \
    vlan-ids=20
add bridge=bridge1 tagged=bonding2-3,bonding4-5,bonding6-7 \
    vlan-ids=30

Finally SW1 (Trunk Between SW1 and SW2):

/interface vlan
add interface=bridge1 name=vlan99 vlan-id=99

Same as above:

/ip address
add address=192.168.99.3/24 interface=vlan99 network=192.168.99.0

…

/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=bonding-eth2-eth3
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether8 pvid=10
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether9 pvid=20
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether10 pvid=30

…

/interface bridge vlan
add bridge=bridge1 tagged=bonding-eth2-eth3 untagged=ether8 vlan-ids=10
add bridge=bridge1 tagged=bonding-eth2-eth3 untagged=ether9 vlan-ids=20
add bridge=bridge1 tagged=bonding-eth2-eth3 untagged=ether10 vlan-ids=30
add bridge=bridge1 tagged=bridge1,bonding-eth2-eth3 vlan-ids=99

Do not forget though that the Switch needs a DNS as well and a default Route…

/ip dns
set servers=192.168.99.1

And:

/ip route
add distance=1 gateway=192.168.99.1
5

Hi everyone,

quick update on the issue: it was, in fact, a bug in RouterOS - tagged frames did not reach the CPU of the switch.
This bug was fixed in the latest beta 6.48beta12 (2020-Jul-06 13:33):

*) crs3xx - improved Ethernet port group traffic forwarding for CRS354 devices;

6

Yes you are right…it works now!!