RB4011 HWoffload + vlan aware bridge issues

Howard7 · Wed Mar 13, 2024 9:52 pm

For a couple months now I've been running my RB4011 as a managed switch and had no major issues.
However recently I started to try and get the RB4011 to use HW offload, this should be possible since I am only using a vlan aware bridge.

Now it originally had IGMP snooping on and this was turned on at all my other switches downstream, now that I am debugging a issue with subnets in my router I decided to turn IGMP off. This instantly made the ports hw offload, this came at the cost of breaking anything not on vlan 1.

If for whatever reason I turn on a feature which disables hw offload everything works again, I can also turn off hw offload on specific ports to fix the issue on said port. I am simply not experienced enough with RouterOS and networking as a whole to figure out just why hw offload breaks vlans on its own.

To be more specific: when HW offload is on a device on a vlan, in this example I made vlan 50 the native lan on port 7, the device will not see anything. It will only see itself and DUDE scanning. As to why it becomes so isolated I have no clue.

If I cannot fix this issue I'll simply turn off hw offloading but it would be a shame to make the cpu work harder than it has to.
My config is as follows:

/interface bridge
add igmp-snooping=yes name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=ether6 ] auto-negotiation=no
set [ find default-name=ether7 ] auto-negotiation=no
/interface bonding
add comment="proxmox link aggregate eth 2+3" mode=802.3ad name=bonding1 slaves=\
    ether2,ether3 transmit-hash-policy=layer-2-and-3
/interface list
add name=WAN
add name=LAN
/port
set 0 name=serial0
set 1 name=serial1
/dude
set enabled=yes
/interface bridge port
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether1
add bridge=bridge interface=ether4
add bridge=bridge interface=ether5
add bridge=bridge interface=ether8
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether9 pvid=200
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether10 pvid=200
add bridge=bridge interface=sfp-sfpplus1 trusted=yes
add bridge=bridge interface=bonding1
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether6
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether7 pvid=50
/ip firewall connection tracking
set udp-timeout=10s
/interface bridge vlan
add bridge=bridge tagged=ether4,ether5,sfp-sfpplus1 vlan-ids=50
add bridge=bridge tagged=ether5,sfp-sfpplus1 vlan-ids=200
add bridge=bridge tagged=sfp-sfpplus1,bonding1 vlan-ids=99
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=ether8 list=LAN
add interface=ether9 list=LAN
add interface=ether10 list=LAN
add interface=sfp-sfpplus1 list=LAN
/ip dhcp-client
# DHCP client can not run on slave or passthrough interface!
add interface=sfp-sfpplus1
/ip dns
set servers=8.8.8.8
/system clock
set time-zone-name=Europe/Amsterdam
/system note
set show-at-login=no
/system routerboard settings
set enter-setup-on=delete-key

Thu Mar 14, 2024 12:16 am

I'm far from the best to help you with your stated HW offload and VLAN issues, but I have to post anyway about several problems in your configuration, on the basis that despite being unable to help with your immediate concerns, I can't help but wonder if clearing some of them up will incidentally improve matters with the behaviors you consider front-most.

/interface ethernet set [ find default-name=ether6 ] auto-negotiation=no

Disabling auto-negotiation without then telling RouterOS what to negotiate instead isn't helpful. If these ports are simply not used, I'd mark them "disabled", not break their configuration this way.

/interface list add name=WAN
/interface list member add interface=ether1 list=WAN

This goes against your claim that you're using the RB4011 as a managed switch, not a router.

If what you meant to say is that you want to use the LAN-side bridge as a managed switch, fine, say so, but don't tell us it's not a router also. It can be both.

/interface bonding …slaves=ether2,ether3…

Those two ports aren't on the bridge, but there isn't any routing information in your config, explicit or implicit, so how does traffic move between Proxmox and the rest of the network?

I'm taking this as a sign that you over-sanitized your configuration and are now expecting us to tell you why it isn't working despite this missing information, because the alternative explanation is that this aspect doesn't work at all, and I think you'd be reporting that as part of your initial list of problems if that were the case.

/ip firewall connection tracking set udp-timeout=10s

I've got a guide to removing this type of configuration flotsam, if you're interested.

# DHCP client can not run on slave or passthrough interface!

This is a warning from RouterOS that you can't put the dhcp-client on sfp-sfpplus1 while it's also part of the bridge. Either move the client to the bridge, or move it to one of the non-bridged ports (e.g. ether1, the WAN port) depending on what it is intended to accomplish.

I'm guessing the former, since you don't show a static IP address assignment for this switch, which then brings us back to the matter of over-sanitizing it and then expecting us to fill in critical details with guesswork.

We don't need your private LAN IP scheme in particular, but we do need some type of IP scheme to hang our thinking on. If you're worried that revealing RFC1918 addressing details will tell us too much about your local configuration, IANA reserved three /24 blocks for documentation and example purposes in RFC5737.

Howard7 · Thu Mar 14, 2024 1:00 am

I appreciate the feedback. Firstly I did not redact anything from my config, I simply ran a config export with the hide sensitive option on and removed the top with the serial code.

But I do understand the confusion, this "switch" was my router before I replaced it with a opnsense router. Now it is serving as my switch which may be why you find references of it being a router.

/interface list add name=WAN
/interface list member add interface=ether1 list=WAN

This goes against your claim that you're using the RB4011 as a managed switch, not a router.

If what you meant to say is that you want to use the LAN-side bridge as a managed switch, fine, say so, but don't tell us it's not a router also. It can be both.

This was unexpected, I never noticed this and it has not been an issue so far. Which is indeed kind of weird as I would not do this on purpose, this must have been a misclick somewhere and I'll correct it. It is however a wonder how my devices have been able to autonegotiate with a port that apparently has it turned off (perhaps it defaults max speed).

/interface ethernet set [ find default-name=ether6 ] auto-negotiation=no
Disabling auto-negotiation without then telling RouterOS what to negotiate instead isn't helpful. If these ports are simply not used, I'd mark them "disabled", not break their configuration this way.

Good question! I have not sanitised my config at all, what you see is what I have. It does work though, I passthrough a native lan and one vlan over that bond and allocate VM's in proxmox. The real routing gets done in opnsense, it has firewall rules between my vlans to isolate them to a certain degree. Native and all vlans come trough the SFP+ port and from there I spread it around the house and other switches. All I need this switch to do is send out certain vlans to certain ports.

/interface bonding …slaves=ether2,ether3…

Those two ports aren't on the bridge, but there isn't any routing information in your config, explicit or implicit, so how does traffic move between Proxmox and the rest of the network?

I'm taking this as a sign that you over-sanitized your configuration and are now expecting us to tell you why it isn't working despite this missing information, because the alternative explanation is that this aspect doesn't work at all, and I think you'd be reporting that as part of your initial list of problems if that were the case.

Perfect! I'll take a look.

/ip firewall connection tracking set udp-timeout=10s

I've got a guide to removing this type of configuration flotsam, if you're interested.

I'll change that tomorrow, that does explain why it did not get a lease from my router, but I set up a static one and did not look back yet. Now I know the root cause.

# DHCP client can not run on slave or passthrough interface!

This is a warning from RouterOS that you can't put the dhcp-client on sfp-sfpplus1 while it's also part of the bridge. Either move the client to the bridge, or move it to one of the non-bridged ports (e.g. ether1, the WAN port) depending on what it is intended to accomplish.

I'll make these changes after work tomorrow, I'll get back with the results once I have them. Thanks!!

Thu Mar 14, 2024 1:28 am

It's best to post your new configuration whole rather than simply report that it's "fixed" by some standard, so we don't have to mentally integrate my partially-mistargeted advice with your prior configuration. It lets us start from the same basis point again.

In the meantime, I suggest that you study these guides:

MikroTik's documented bridge VLAN filtering info. This should allow you to put your ether2 + ether3 LAG into the bridge again. (This isn't critical, though.)
pcunite's VLAN guide, being far easier to understand than the reference-level info in MikroTik's docs.
My own bridge-per switch chip article, which may be an eye-opener in your particular case; you've got three potentially independent LANs in an RB4011, and you need to configure it with that in mind. Bridging everything together as a single switch is possible, but there are consequences.

Howard7 · Fri Mar 15, 2024 1:46 am

I've processed your suggestions but sadly it has not solved the main issue just yet. It did however clean my config up by a lot and fixed some auxillary issues I had like the dhcp client.

My config is now as follows:

# 2024-03-15 00:39:13 by RouterOS 7.14.1
# software id = XXXXXX
#
# model = RB4011iGS+
# serial number = XXXXXX
/interface bridge
add igmp-snooping=yes name=bridge vlan-filtering=yes
/interface bonding
add comment="proxmox link aggregate eth 2+3" mode=802.3ad name=bonding1 slaves=\
    ether2,ether3 transmit-hash-policy=layer-2-and-3
/interface list
add name=WAN
add name=LAN
/port
set 0 name=serial0
set 1 name=serial1
/dude
set enabled=yes
/interface bridge port
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether1
add bridge=bridge interface=ether4
add bridge=bridge interface=ether5
add bridge=bridge interface=ether8
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether9 pvid=200
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether10 pvid=200
add bridge=bridge interface=sfp-sfpplus1 trusted=yes
add bridge=bridge interface=bonding1
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether6
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether7 pvid=50
/interface bridge vlan
add bridge=bridge tagged=ether4,ether5,sfp-sfpplus1 vlan-ids=50
add bridge=bridge tagged=ether5,sfp-sfpplus1 vlan-ids=200
add bridge=bridge tagged=sfp-sfpplus1,bonding1 vlan-ids=99
/interface list member
add interface=ether1 list=LAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=ether8 list=LAN
add interface=ether9 list=LAN
add interface=ether10 list=LAN
add interface=sfp-sfpplus1 list=LAN
/ip dhcp-client
add interface=bridge
/ip dns
set servers=192.168.1.1
/system clock
set time-zone-name=Europe/Amsterdam
/system note
set show-at-login=no
/system routerboard settings
set enter-setup-on=delete-key

It's best to post your new configuration whole rather than simply report that it's "fixed" by some standard, so we don't have to mentally integrate my partially-mistargeted advice with your prior configuration. It lets us start from the same basis point again.

Fri Mar 15, 2024 6:07 am

Yes, that's much cleaner now. The main thing I'd suggest after this is to get rid of the interface lists, which aren't carrying any weight in your new configuration:

/interface bridge
add igmp-snooping=yes name=bridge vlan-filtering=yes
/interface bonding
add comment="proxmox link aggregate eth 2+3" mode=802.3ad name=bonding1 slaves=ether2,ether3 transmit-hash-policy=layer-2-and-3
/dude set enabled=yes
/interface bridge port
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether1
add bridge=bridge interface=ether4
add bridge=bridge interface=ether5
add bridge=bridge interface=ether8
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether9 pvid=200
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether10 pvid=200
add bridge=bridge interface=sfp-sfpplus1 trusted=yes
add bridge=bridge interface=bonding1
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether6
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether7 pvid=50
/interface bridge vlan
add bridge=bridge tagged=ether4,ether5,sfp-sfpplus1 vlan-ids=50
add bridge=bridge tagged=ether5,sfp-sfpplus1 vlan-ids=200
add bridge=bridge tagged=sfp-sfpplus1,bonding1 vlan-ids=99
/ip dhcp-client add interface=bridge
/ip dns set servers=192.168.1.1
/system clock set time-zone-name=Europe/Amsterdam
/system note set show-at-login=no
/system routerboard settings set enter-setup-on=delete-key

With these changes, can you give a list of commands and their output that shows what is happening? That might draw other, better-equipped minds in to help you with this. I try very hard to avoid VLANs myself.

mkx · Fri Mar 15, 2024 11:43 am

The main problem is, that bridge interface is not member of any of tagged VLANs:

/interface bridge vlan
add bridge=bridge tagged=ether4,ether5,sfp-sfpplus1 vlan-ids=50
add bridge=bridge tagged=ether5,sfp-sfpplus1 vlan-ids=200
add bridge=bridge tagged=sfp-sfpplus1,bonding1 vlan-ids=99

If you want router to interact with any of those VLANs, you have to add bridge port as tagged member. It could be you don't need it (you don't have any of VLAN interfaces created), but there's more to it.

However: the peculiarity of all multi-switch-chip devices (RB4011 is one of them) is that there's (AFAIK still) a bug in how bridge offloads things to hardware ... and if bridge is not made tagged member of certain VLAN, then interconnect between switch chip and CPU is not made member of that VLAN. In your case this means that traffic between both switch chips (chip1 controls ether1-5, chip2 controls ether6-10), which should pass both interconnects, doesn't flow. And sfp-sfpplus1 is affected as well because this port is attached directly to CPU.

The "official" work-around this bug is to include bridge as tagged member od all necessary (cross-switch-chip and SFP+) VLANs.

Howard7 · Fri Mar 15, 2024 4:54 pm

This worked perfectly! Thanks million!!!

The main problem is, that bridge interface is not member of any of tagged VLANs:

However: the peculiarity of all multi-switch-chip devices (RB4011 is one of them) is that there's (AFAIK still) a bug in how bridge offloads things to hardware ... and if bridge is not made tagged member of certain VLAN, then interconnect between switch chip and CPU is not made member of that VLAN. In your case this means that traffic between both switch chips (chip1 controls ether1-5, chip2 controls ether6-10), which should pass both interconnects, doesn't flow. And sfp-sfpplus1 is affected as well because this port is attached directly to CPU.

The "official" work-around this bug is to include bridge as tagged member od all necessary (cross-switch-chip and SFP+) VLANs.

RB4011 HWoffload + vlan aware bridge issues [SOLVED]

RB4011 HWoffload + vlan aware bridge issues

Re: RB4011 HWoffload + vlan aware bridge issues

Re: RB4011 HWoffload + vlan aware bridge issues

Re: RB4011 HWoffload + vlan aware bridge issues

Re: RB4011 HWoffload + vlan aware bridge issues

Re: RB4011 HWoffload + vlan aware bridge issues

Re: RB4011 HWoffload + vlan aware bridge issues [SOLVED]

Re: RB4011 HWoffload + vlan aware bridge issues

Who is online