CRS305 Poor VLAN Performance

I have a CRS305 connected between 4 10Gbps devices, with VLANS on all ports, and the performance I’m getting is only about 800Mbps.

My export looks like:

# oct/12/2022 15:45:30 by RouterOS 6.48.6

/interface bridge
add frame-types=admit-only-vlan-tagged name=External-Bridge protocol-mode=none vlan-filtering=yes

/interface vlan
add interface=External-Bridge name=External_bridge_vlan66 vlan-id=66
add interface=External-Bridge name=External_bridge_vlan67 vlan-id=67
add interface=External-Bridge name=External_bridge_vlan68 vlan-id=68
add interface=External-Bridge name=External_bridge_vlan69 vlan-id=69

/interface bridge port
add bridge=External-Bridge interface=sfp-sfpplus1 pvid=666
add bridge=External-Bridge interface=sfp-sfpplus2 pvid=667
add bridge=External-Bridge interface=sfp-sfpplus3 pvid=668
add bridge=External-Bridge interface=sfp-sfpplus4 pvid=669

/interface bridge vlan
add bridge=External-Bridge tagged=External-Bridge,sfp-sfpplus1,sfp-sfpplus3 vlan-ids=66
add bridge=External-Bridge tagged=External-Bridge,sfp-sfpplus1,sfp-sfpplus4 vlan-ids=67
add bridge=External-Bridge tagged=External-Bridge,sfp-sfpplus2,sfp-sfpplus3 vlan-ids=68
add bridge=External-Bridge tagged=External-Bridge,sfp-sfpplus2,sfp-sfpplus4 vlan-ids=69

When I added the interfaces to the bridge I used hw=yes,

add bridge=External-Bridge interface=sfp-sfpplus2 pvid=667 hw=yes

And when I print the port info under bridge I get the correct listing for hw:

#     INTERFACE                    BRIDGE                    HW  PVID 
 1     sfp-sfpplus2                 External-Bridge           yes 667

I’ve turned off rstp. Based on everything I can find it should be doing hardware offloading and so I should be getting better performance.

What have I done wrong?

J

Are you seeing high CPU usage while testing? it could be that the hw offloading is not working properly

Is there a reason you haven’t set frame-types:

frame-types=admit-only-untagged-and-priority-tagged

Next to that, you configure different pvid from vlan-ids.
Perhaps have a look at this help page:
https://help.mikrotik.com/docs/display/ROS/Bridging+and+Switching#BridgingandSwitching-BridgeVLANFiltering

How exactly are you testing? L2 HW offload only works for traffic within same VLAN between different bridge ports. Your config has only a pair of pirts belonging to same VLAN (e.g. sfp-sfpplus 1 and 3 in VID 66 etc.). So make sure tests are done correctly.

For traffic between different VLANs you need router. CRS3xx can offload L3 to HW, in that case it can route wirespeed. The config shown doesn’t show L3 setup, so it’s impossible to say anything about it.

If L3 setup doesn’t exist on this CRS, then the routing between VLANs is done by another device (router) and performance bottleneck is there to be found. And even if L3 setup is present, it doesn’t mean this device will do the routing, it all depends on setup of all involved devices (servers, main router, …)

Yes, I see 100% CPU load when I am pushing 800Mbit between two ports.

I have a device plugged into sfp-sfpplus 1 which is setup to send tagged traffic on VLAN 66 and 67, and another device plugged into sfp-sfpplus 3 which is setup to send tagged traffic on vlan’s 66 and 68. I am using iperf on both of these devices.

For traffic between different VLANs you need router. CRS3xx can offload L3 to HW, in that case it can route wirespeed. The config shown doesn’t show L3 setup, so it’s impossible to say anything about it.

If L3 setup doesn’t exist on this CRS, then the routing between VLANs is done by another device (router) and performance bottleneck is there to be found. And even if L3 setup is present, it doesn’t mean this device will do the routing, it all depends on setup of all involved devices (servers, main router, …)

I am not routing traffic between VLANs, I am using VLAN’s for isolation only. Hence my expectation that this would all be done in the switch chip of the CRS305, and not hit the CPU at all, thus giving me near wire speed performance. This should all be happening at L2 only.

My hope is that I would have only tagged traffic on this switch, there should be no untagged traffic. The PVID is only configured as it seems to be a requirement to do so. Hence why it is different from the VLAN ID.

Thanks

J

In which case you do not need any /interface vlan entries - these provide the link between tagged VLANs in the bridge and services on the Mikrotik itself, via the implicit bridge-to-CPU port. These, plus the bridge ports having tagged=External-Bridge,… will direct any unicast to unknown MAC addresses plus broadcast and multicast traffic in these VLANs to the CPU and require processing.


My hope is that I would have only tagged traffic on this switch, there should be no untagged traffic. The PVID is only configured as it seems to be a requirement to do so. Hence why it is different from the VLAN ID.

The pvid= setting always exists and defaults to 1, if you set frame-types=admit-only-vlan-tagged untagged membership is disabled and the PVID setting ignored.

So:

/interface bridge
add name=External-Bridge ingress-filtering=yes protocol-mode=none vlan-filtering=yes

/interface bridge port
add bridge=External-Bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes hw=yes interface=sfp-sfpplus1 
add bridge=External-Bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes hw=yes interface=sfp-sfpplus2 
add bridge=External-Bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes hw=yes interface=sfp-sfpplus3 
add bridge=External-Bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes hw=yes interface=sfp-sfpplus4 

/interface bridge vlan
add bridge=External-Bridge tagged=sfp-sfpplus1,sfp-sfpplus3 vlan-ids=66
add bridge=External-Bridge tagged=sfp-sfpplus1,sfp-sfpplus4 vlan-ids=67
add bridge=External-Bridge tagged=sfp-sfpplus2,sfp-sfpplus3 vlan-ids=68
add bridge=External-Bridge tagged=sfp-sfpplus2,sfp-sfpplus4 vlan-ids=69

Fantastic, I’ll give that a try in this evening’s maintenance window.

Thank you

J

Well I tried the new config. It’s an improvement. I now get between 996Mbps and 1.07Mbps, vs the previous ~840Mbps.

But the device CPU is still maxing out:

NAME                    CPU        USAGE
www                                   0%
ethernet                             28%
console                               0%
ssh                                   0%
networking                           40%
management                            1%
routing                               0%
profiling                             0%
bridging                           25.5%
unclassified                        5.5%
total                               100%

And I’d expect it to be closer to the 10Gbps line speed. Iperf output:

[  3]  0.0-120.0 sec  14.6 GBytes  1.04 Gbits/sec

Any idea what I’m missing?

Thanks

J

What is the output of the commands /interface bridge port print and /interface bridge vlan print detail?

are you still on routeros 6.48.6 ??

if you want L3 offloading you need to go to Routeros 7

(Everyone who keeps mentioning Layer 3 isn’t reading the thread very well.)

Have you checked to see what speeds the ports are negotiating with the hosts? Are both ends (switch and host) reporting 10Gbps?

Here’s what I would do:

  1. Reset it to defaults. If you access it via Winbox, you should also check the box that says no default config so it’s completely blank.
  2. Enter this super-simple switch-only config and run your host-to-host tests. In this setup it will pass all traffic, tagged or not. Add management IP to ether1 if you like (see next config section if needed).
/interface bridge
add name=External-Bridge protocol-mode=none 

/interface bridge port
add bridge=External-Bridge hw=yes interface=sfp-sfpplus1 
add bridge=External-Bridge hw=yes interface=sfp-sfpplus2 
add bridge=External-Bridge hw=yes interface=sfp-sfpplus3 
add bridge=External-Bridge hw=yes interface=sfp-sfpplus4
  1. Reset again and load the VLAN-filtering config earlier in the thread, testing host-to-host speeds again.
/interface bridge
add name=External-Bridge ingress-filtering=yes protocol-mode=none vlan-filtering=yes

/interface bridge port
add bridge=External-Bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes hw=yes interface=sfp-sfpplus1 
add bridge=External-Bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes hw=yes interface=sfp-sfpplus2 
add bridge=External-Bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes hw=yes interface=sfp-sfpplus3 
add bridge=External-Bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes hw=yes interface=sfp-sfpplus4 

/interface bridge vlan
add bridge=External-Bridge tagged=sfp-sfpplus1,sfp-sfpplus3 vlan-ids=66
add bridge=External-Bridge tagged=sfp-sfpplus1,sfp-sfpplus4 vlan-ids=67
add bridge=External-Bridge tagged=sfp-sfpplus2,sfp-sfpplus3 vlan-ids=68
add bridge=External-Bridge tagged=sfp-sfpplus2,sfp-sfpplus4 vlan-ids=69

# Optional DHCP
/ip dhcp-client add interface=ether1

# Or statically assigned IP; I'm using 10.0.0.1 as an example
/ip address add interface=ether1 address=10.0.0.1/24

That’s all that should be required to make this do what you want.

This same set of configs should work on both 6.48 and later as well as 7. I recommend 7.4.1 for the CRS300 switches (as opposed to 7.5 or 7.6) due to personal experience.



[admin@RouterOS] > /interface bridge port print
Flags: X - disabled, I - inactive, D - dynamic, H - hw-offload 
 #     INTERFACE                    BRIDGE                    HW  PVID PRIORITY  PATH-COST INTERNAL-PATH-COST    HORIZON
 0   H ;;; defconf
       ether1                       Mgmt-Bridge               yes    1     0x80         10                 10       none
 1     sfp-sfpplus1                 External-Bridge           yes    1     0x80         10                 10       none
 2     sfp-sfpplus2                 External-Bridge           yes    1     0x80         10                 10       none
 3     sfp-sfpplus3                 External-Bridge           yes    1     0x80         10                 10       none
 4     sfp-sfpplus4                 External-Bridge           yes    1     0x80         10                 10       none



Flags: X - disabled, D - dynamic 
 #   BRIDGE                        VLAN-IDS  CURRENT-TAGGED                       CURRENT-UNTAGGED                      
 0   External-Bridge               66       sfp-sfpplus1                        
                                             sfp-sfpplus3                        
 1   External-Bridge               67       sfp-sfpplus1                        
                                             sfp-sfpplus4                        
 2   External-Bridge               68       sfp-sfpplus2                        
                                             sfp-sfpplus3                        
 3   External-Bridge               69       sfp-sfpplus2                        
                                             sfp-sfpplus4                        
 4 D External-Bridge               1                                              External-Bridge

Thanks

J

CRS somehow selected “Mgmt-Bridge” to offload. Set ether1 (as bridge port) with hw=no to make “Mgmt-Bridge” non-HWoffloaded. Perhaps you’ll have to reboot switch.

After that you should see a ‘H’ in “External-Bridge” instead of current row.

Or better yet: ditch Mgmt-Bridge altogether (move IP config from bridge to stand-alone ether1 interface after dusbanding bridge).

The problem being: ROS can only offload single bridge. If there’s more than one, ROS selects bridge to offload automatically. It might happen that selection is unfortunate - as it is in your case.
And there’s no sense in having Mgmt-Bridge with single bridge port anyway.

It would have been helpful if you had posted the complete configuration with sensitive data redacted at the start.

The issue is you have two bridges and CRS3xx only supports hardware offloading on one bridge. Ideally remove Mgmt-Bridge and configure an IP address directly on ether1 for local management, or set hw=no on all bridge ports which are not members of External-Bridge



[  3] 0.0000-120.0006 sec   131 GBytes  9.37 Gbits/sec

That did the trick! Thank you.

I’d like to ditch the mgmt bridge, but as this machine has no serial console, I don’t want to block access to myself, so it’s there as a reminder to not do that in future.

J

As I wrote: configure mgmt IP on ether1 directly. If you don’t have any restrictions regarding MAC access, then you’ll be able to connect via ether1 … you really don’t need bridge for that.

And I’ll mirror tdw’s sentiments: it would shorten the pain if you’d post full config immediately. Things are always interdependant and for inexperienced it’s impossible to tell what’s important and what not.

Ironically I was trying to avoid flooding you all with too much info, and I wasn’t sure which bits needed redacting.

Is it documented anywhere that only one bridge can be hw offloaded?

Is there a way to add the mgmt ip to ether1 directly without locking myself out as the only way to access it is over the mgmt ip ?

J

Just like adding any other IP address in ROS to any interface.

And Safe Mode will be your friend.

It is: https://help.mikrotik.com/docs/display/ROS/CRS3xx%2C+CRS5xx%2C+CCR2116%2C+CCR2216+switch+chip+features#CRS3xx,CRS5xx,CCR2116,CCR2216switchchipfeatures-Portswitching


Use winbox and click MAC address … this way entire IP stack is bypassed for current connection. You should keep management access while changing IP settings. When you remove ether1 from bridge, you may get disconnected but you should be able to reconnect. You might need to re-run discovery in case MAC address changes (it might stay the same, depends on how exactly the bridge is defined … huh, we still need to guess and assume … see?).
And no, when tearing down mgmt-bridge, safe mode is not your friend, loosing access is expected and safe mode will then revert the changes, effectively preventing you from doing the right thing.

Excellent, I shall give that a go in the new year. It’s the last working day of the year here, and I don’t want to break anything in the last 2 hours before my break!

Thanks

J