The sfpplus14,15,16 are 10G fiber links with Mellanox cards, linked at 10Gbps
I do a iperf3 test between the interfaces in the “openstack-backend” bridge and I got maximum 1Gbps , no matter how many streams.
I disable the “br-hz” bridge and the traffic goes up to 10Gbps
Why a bridge that’s on ports 2 and 3 influence the forwarding rates of a different bridge that’s on ports 14,15,16 ?
You can see that your ports sfp-sfpplus2 and sfp-sfpplus3 have flag “H” (hw-offload) while ports sfp-sfpplus14, sfp-sfpplus15 and sfp-sfpplus16 do not have such flag. That means they are not hw-offloaded and instead of being processed in switch (and forwarded with wire-speed) they have to pass through CPU. I guess the internal bus between switch chip and CPU is limited to 1Gbit
For the first bridge I don’t need speed therefore i’ve disabled the hw-offloading and the ports in the secondary bridge automatically showed the hw-offloading as active.
If there a limitation to have only one bridge with hw-offloading active ?
I think I read somewhere on the forums that there is indeed a limit to 1 HW-forwarding enabled bridge on the CRS317. I’m pretty sure there’s no other way to get the full speed on all your ports other than using a single bridge and using VLANs or port isolation.
This is well known limitation for simple(r) devices (practically any soho models) that their switch chip (usually limited to 5 ports so devices with higher amount of ports need multiple switch chips) can handle only one hw-accelerated bridge at the time.
This is also mentioned for CRS models:
Note: Currently it is possible to create only one bridge with hardware offloading on CRS3xx series devices.
(I didn’t know it before today but I can’t say I didn’t expect it - it was pretty obvious from your code that frist bridge is blocking hw-acceleration on second bridge)
To be honest, I am surprised that in 6.43, Mikrotik went the way of implementing port-isolation via switch menu instead of using such functionality to separate multiple bridges and make all of them hw-accelerated at the same time. I guess there is some good reason which I don’t see yet
If you don’t need the speed for first bridge, good for you. Disabling HW acceleration might be the way, but keep in mind it might backfire on you later because any traffic will pass via CPU, therefore utilizing both CPU and internal bus.
Having full-blown RouterOS on CRS devices is sweet but it obviously cause these situations. It is important to always keep in mind these devices are nothing more than smart switches which got overkill operating system.
This is very disappointing. Paying for a switch that has 10 Gbps that you cannot get wired speed only 1 Bridge?? Typically you can use it with 1 VLAN / Bridge if you want to have full wire speed.
That doesn’t make any sense at all. If the HW has 10 Gbps I should be able to get that wire speed independently of the number of Bridges.
My understanding is that per VLAN you need an associated bridge. You cannot have a single Bridge with multiple VLANs. Based on that makes this device useless for Enterprise type of solutions (or any RB with the same limitation).
I had this exact same problem and made the change last Thursday night to a single bridge with multiple VLANs. The performance was night and day different.
Prior to that change, I was lucky to get 2Gbps between two servers connected to different switches which were connected with 2 bonded 10Gbps interfaces. After the change, I easily got 9.5Gbps and higher between the servers.
I would like to clarify that the hardware offloading limitation for single bridge is made by intention because entire switch-chip usually is designed as a one bridge entity. The single Host table usually seen in network switches could be mentioned as an example for it and implementing it in RouterOS requires to follow certain rules.
Please note that features like Port Isolation and Bridge VLAN Filtering already provide the main functionality which comes from the need of multiple bridges and this should not be mistaken with older software based VLAN configurations which required separate bridge for each VLAN to ensure proper isolation.
