CRS326 VLAN leakage to CPU?

lapsio · Mon Jan 07, 2019 12:25 am

I'm using CRS326 as switch (disabled packet forwarding, only management connected to cpu port) but today one machine connected to it sent broadcast and I got firewall alert:

Jan 6 19:37:43 192.168.10.6 firewall,info crs326: X_X val-net: in:ether12 out:(unknown 0), src-mac 00:90:f5:e5:26:14, proto UDP, 192.168.4.6:1716->255.255.255.255:1716, len 978

The thing is - according to my understanding of /interface bridge mechanics that packet should never even reach CPU at all. Here's config export of /interface section:

# jan/06/2019 23:24:43 by RouterOS 6.43.2
# software id = TY91-A3R5
#
# model = CRS326-24G-2S+
# serial number = 763C08DC0959
/interface bridge
add admin-mac=CC:2D:E0:51:8E:E0 auto-mac=no name=br-hardware protocol-mode=none vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether2 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether3 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether4 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether5 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether6 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether7 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether8 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether9 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether10 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether11 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether12 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether13 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether14 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether15 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether16 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether17 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether18 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether19 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether20 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether21 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether22 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether23 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=ether24 ] l2mtu=9112 mtu=9000 speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] l2mtu=9112 mtu=9000 speed=10Gbps
set [ find default-name=sfp-sfpplus2 ] l2mtu=9112 mac-address=CC:2D:E0:51:8E:F8 mtu=9000 speed=10Gbps
/interface vlan
add interface=br-hardware name=vlan10-ccr vlan-id=1002
/interface bonding
add mode=balance-xor mtu=9000 name=bond-crs slaves=sfp-sfpplus1,sfp-sfpplus2 transmit-hash-policy=layer-3-and-4
/interface list
add exclude=dynamic name=discover
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether17 pvid=10
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether18 pvid=11
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether19 pvid=12
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether20 pvid=13
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether21 pvid=14
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether22 pvid=15
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether23 pvid=16
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether24 pvid=17
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether1 pvid=99
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether2 pvid=99
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether3 pvid=99
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether4 pvid=99
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether5 pvid=301
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether6 pvid=302
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether7 pvid=303
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether8 pvid=304
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether9 pvid=401
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether10 pvid=402
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether11 pvid=403
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether12 pvid=404
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether15 pvid=18
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether16 pvid=19
add bridge=br-hardware interface=bond-crs pvid=2
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether13 pvid=3
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether14 pvid=4
/interface bridge vlan
add bridge=br-hardware tagged=br-hardware,bond-crs vlan-ids=1002
add bridge=br-hardware tagged=ether13 untagged=ether9 vlan-ids=401
add bridge=br-hardware tagged=ether14 untagged=ether10 vlan-ids=402
add bridge=br-hardware tagged=ether13 untagged=ether11 vlan-ids=403
add bridge=br-hardware tagged=ether14 untagged=ether12 vlan-ids=404
add bridge=br-hardware tagged=ether13 untagged=ether5 vlan-ids=301
add bridge=br-hardware tagged=ether14 untagged=ether6 vlan-ids=302
add bridge=br-hardware tagged=ether13 untagged=ether7 vlan-ids=303
add bridge=br-hardware tagged=ether14 untagged=ether8 vlan-ids=304
add bridge=br-hardware untagged=ether1,ether2,ether3,ether4 vlan-ids=99
add bridge=br-hardware tagged=bond-crs untagged=ether15,ether16 vlan-ids=4000,4001,4002,4003,4004,4005,4006,4007,4008,4009
add bridge=br-hardware tagged=bond-crs untagged=ether17,ether18,ether19,ether20,ether21,ether22,ether23,ether24 vlan-ids=4030,4031,4032,4033,4034,4035,4036,4037,4038,4039
/interface list member
add interface=vlan10-ccr list=discover

As you can see all ports have enabled ingress filtering (except uplink) and ether12 has assigned VLAN 404 which is only assigned to ether12 and ether14. Only VLAN 1002 is assigned to br-hardware (aka CPU port). Also VLAN filtering is enabled on bridge. So what's going on? Just to make it clear - this packet did not get forwarded via uplink because CCR1009 used for inter-vlan routing dropped that packet. Also it's entirely different network so it wouldn't get forwarded to management anyways. And CRS says that in-interface is ether12 so there's no way this packet came from uplink on VLAN 1002.

I tried to replicate alert but it doesn't seem to be regular issue. It looks like for some reason just this one particular packet got forwarded to CPU for some reason and triggered alert on input firewall. Is it possible that VLAN filtering actually passes every-nth packet to CPU for some statistics purpose or something like that? Fact that it's not reproducible bothers me even more than if it was regular issue. It makes me feel lack of control and mistrust regarding owned hardware....

I can't tell if it's first occurance of alert ever because we configured firewall and rsyslog in switch just few weeks ago (and switch is operational since around June 2017) so it bothers me even more because I assumed switch was secure since traffic from other VLANs than management can't reach it anyways... Actually the only reason I eventually decided to copy firewall config from routers to switch was to have uniform config for easier synchronization...

sebastia · Mon Jan 07, 2019 1:43 am

Hi

Just wondering if these are correct? And if so what is the effect?
Are multiple subnets mixed on these interfaces?

add bridge=br-hardware tagged=bond-crs untagged=ether15,ether16 vlan-ids=4000,4001,4002,4003,4004,4005,4006,4007,4008,4009
add bridge=br-hardware tagged=bond-crs untagged=ether17,ether18,ether19,ether20,ether21,ether22,ether23,ether24 vlan-ids=4030,4031,4032,4033,4034,4035,4036,4037,4038,4039

lapsio · Mon Jan 07, 2019 8:47 am

Hi

Just wondering if these are correct? And if so what is the effect?
Are multiple subnets mixed on these interfaces?
Code: Select all
add bridge=br-hardware tagged=bond-crs untagged=ether15,ether16 vlan-ids=4000,4001,4002,4003,4004,4005,4006,4007,4008,4009
add bridge=br-hardware tagged=bond-crs untagged=ether17,ether18,ether19,ether20,ether21,ether22,ether23,ether24 vlan-ids=4030,4031,4032,4033,4034,4035,4036,4037,4038,4039

It's for VEPA config on virtualization servers. These are 2 pools of MAC based VLANs using switch rules (each such vlan represents separate subnet but 403x and 400x are two different security zones dmz and lab). Since mikrotik doesn't support VEPA as is, it's dirty hack - if each VM from one subnet is attached to different physical interface then there's no need for actual VEPA support on switch and yet all traffic inside single subnet is switched on hardware switch. On CRS326 it has limited benefits but there's similar config on our CRS317 and here switching in silicon instead of software virtual switch makes huge difference.

If you're interested in this topic I made several VEPA related threads containing my observations and notes regarding lack of VEPA support on MikroTik and possible workarounds.

mkx · Mon Jan 07, 2019 9:07 am

Can you guess, from src IP address, to which VLAN that packed actually belongs? One thing, not really obvious: every ports has implicit setting of PVID=1 unless it's explicitly set to other value. And gets implicitly added to list of untagged ports for given bridge.
If frame-types is not set to admit-only-vlan-tagged on trunk ports, then switch will leak untagged frames here and there.

I can't understand what's the meaning of your settings that @sebastia already asked ... I suspect it's not working as you intended. According to this document those VLANs are actually supposed to be tagged on switch? VEPA further assumes there's some inter-VLAN router to return packets, appropriately tagged (to different VLAN), via same physical port. The only thing that really has to be supported by VEPA switch (and that's not out-of-the-box) is returning packets back via same wire even if the target is in the same VLAN ... usually switches don't do that. IMHO that's laziness by implementers of VEPA on the VM platform ... they should implement VLAN-aware bridge in their network virtualization layer.

lapsio · Mon Jan 07, 2019 11:57 am

Can you guess, from src IP address, to which VLAN that packed actually belongs? One thing, not really obvious: every ports has implicit setting of PVID=1 unless it's explicitly set to other value. And gets implicitly added to list of untagged ports for given bridge.
If frame-types is not set to admit-only-vlan-tagged on trunk ports, then switch will leak untagged frames here and there.

I can't understand what's the meaning of your settings that @sebastia already asked ... I suspect it's not working as you intended. According to this document those VLANs are actually supposed to be tagged on switch? VEPA further assumes there's some inter-VLAN router to return packets, appropriately tagged (to different VLAN), via same physical port. The only thing that really has to be supported by VEPA switch (and that's not out-of-the-box) is returning packets back via same wire even if the target is in the same VLAN ... usually switches don't do that. IMHO that's laziness by implementers of VEPA on the VM platform ... they should implement VLAN-aware bridge in their network virtualization layer.

In my case all ports have set explicit pvid (except trunk because I had some troubles with getting LACP to work and I wasn't sure if it may be related to pvid).

Regarding that VEPA config it's basically MAC based VLANs config described here: https://wiki.mikrotik.com/wiki/Manual:C ... Based_VLAN And you're right I just compared configs and it's wrong on CRS326, frame-types should be set to admit-only-untagged-and-priority-tagged for MAC VLANs to work, just like it's set up on our CRS317 (where it's actively used now). We didn't really assign any VMs to gigabit interfaces yet so I guess it's good thing that it's been noticed now. Probably will save some troubleshooting time in the future. That's also why there's no /interface switch rule entries in this config dump - because there's no VMs yet here so there's no MAC addresses to assign. Anyways...

Yes MikroTik doesn't support VEPA aka doesn't return packets to the same port that's why I'm using multiple ports for VMs and the same port is only assign to multiple VMs if they're in different VLANs. There is CCR1009 for inter-vlan routing connected to trunk. There are 2 purposes of VEPA comparing to software bridge and it's not about VLAN aware bridges because that's already implemented here.

The point of VEPA is to move switching from hypervisor to actual switch. It's done for:

1) uniform ACLs management so that all ACLs no matter if for physical machines or virtual machines are applied on switches (not on hypervisors)
2) performance because for example we use Intel X710-DA4 cards. You'll have really bad time trying to bridge 40G of traffic between VMs without jumbo frames in software bridge on CPU. No matter if you use VMWare or KVM it's just no-go. If you want high performance networking in VMs then hardware passthrough or SR-IOV (technically it's also hardware passthrough, just virtual hardware) are the only options realistically.

I'm not familiar with other implementations of VEPA but in case of Intel network cards that support SR-IOV, they allow for creating virtual sub-interfaces that can be passed through to VMs to achieve bare-metal networking performance. Those virtual sub-interfaces have fixed MAC address that can't be changed by VM. So long story short Intel X710-DA4 requires using MAC based VLANs. If you think about it - it makes sense. It allows VMs to use multiple VLANs on such interface and still be distinguishable from other VMs and manageable on switch level.

mkx · Mon Jan 07, 2019 12:18 pm

Ah, I see now. To me it's kind of counter-intiutive that for ingress it's switch chip that does the tagging while for egress bridge is seemingly doing the untagging (as CRS3xx supports HW-offloading also VLAN ops, in reality it's switch chip doing also untagging ... it's just the place where it's configured makes things messy).

Never the less, can you deduct where do the leaked frames come from (which port and which VLAN if tagged)? That would help to figure out if there's some mis-configuration or is there really some bug lurking around.

lapsio · Mon Jan 07, 2019 1:27 pm

Ah, I see now. To me it's kind of counter-intiutive that for ingress it's switch chip that does the tagging while for egress bridge is seemingly doing the untagging (as CRS3xx supports HW-offloading also VLAN ops, in reality it's switch chip doing also untagging ... it's just the place where it's configured makes things messy).

Never the less, can you deduct where do the leaked frames come from (which port and which VLAN if tagged)? That would help to figure out if there's some mis-configuration or is there really some bug lurking around.

Yeah they didn't implement switch rules into bridge yet that's why MAC vlans are still done via /interface switch rule while regular vlans are already in /interface bridge vlan and port pvid.

Like I said it's not regular leak. I tried replicating exactly the same packets on the same port from the same machine crafted with socat and they didn't trigger alert anymore. I think I'll just wait for this incident to repeat. But I think it'll take a while considering that it took like 2 weeks to trigger even though packet which triggered it was KDEConnect discovery message which is sent every single time linux machine with KDE connects using cable so like... at least several times a day.

sebastia · Mon Jan 07, 2019 8:53 pm

Thx for the info lapsio!

lapsio · Sat Mar 09, 2019 9:25 pm

I noticed similar issues with RB2011 switch. Some packets just for some reason leak to CPU even though I use RB2011 ethernet ports as pure switch (in theory VLANs don't even have access to switch-cpu port). Such incidents are extremely rare but they occur repeatedly. Over past several months around 2-3 packets leaked to CPU in RB2011. And also few more in case of CRSes. Extreme rarity of those incidents and fact that it's always exactly 1 packet, no more, makes me feel really uncomfortably xD

CRS326 VLAN leakage to CPU?

CRS326 VLAN leakage to CPU?

Re: CRS326 VLAN leakage to CPU?

Re: CRS326 VLAN leakage to CPU?

Re: CRS326 VLAN leakage to CPU?

Re: CRS326 VLAN leakage to CPU?

Re: CRS326 VLAN leakage to CPU?

Re: CRS326 VLAN leakage to CPU?

Re: CRS326 VLAN leakage to CPU?

Re: CRS326 VLAN leakage to CPU?

Who is online