Bought CRS520 and looks like it does not support hw-offloading for multiple bridges, works only on first one.
[admin@100G] /interface/bridge/port> print
Flags: H - HW-OFFLOAD
Columns: INTERFACE, BRIDGE, HW, PVID, PRIORITY, HORIZON
As you can see "H" is active only on ports 1 and 2(which are in bridge1) ports 3 and 4 are in bridge2 where hw-offloading is not active however it is configured. In case if i disable hw-offloading on ports 1&2 it immediately become active on ports 3&4.
I reported a bug in ticket SUP-182395 but Edgars P. told me that this is "expected behaviour".
Nowhere it was documented as such behaviour and main feature of this box was advertised that it has hw-offloading however it is not and it becomes useless device since if you need 100G switching none of CPUs can handle it.
Is there any chance to configure this switch and utilise hw-offloading, maybe someone know?
This is indeed expected behavior. And exactly correct.
These (and all other) switch chips can only offload one bridge. Most manufacturers just “implicitly” put all ports into a bridge and don’t even expose you to this. Mikrotik allows more than one bridge, but - again - only one can be handled in hardware, all others are done purely in software. (If you want to know the details, google “switch tags”, that should get you started.) And your device behaves correctly, in that it offloads the bridge that doesn’t have it disabled.
To get HW offloaded routing, follow the manual. It clearly states that for that to work, all involved ports have to be in the same bridge. (In case you have more than one, only the one that is offloaded to hardware can do hardware routing.) A consequence of this is that all HW routing is between VLANs, so you have to assign different VLANs to your different L3 domains. (The VLANs don’t have to be “seen” from the outside, it’s perfectly fine for them to be access ports, and for VLANs to only exist internally.)
But really, read the docs and follow them step-by-step to setup HW routing.
Thank you for the reply.
Unfortunately I need to return the box to supplier since my set up needs multiple bridges. Indeed one of bridges will handle most of the traffic but adding more will just kill CPU.
If it supports L3 hw-offloading this unit can be perfect BGP router however it has been positioned as switch for some reason.
Do you care to elaborate the need? So far I can only think of a single case when more than one bridge would be needed (when VLANs are in use, two distinct subnets use same VID and switch admin doesn’t have any control over which VIDs are used in either of those networks). So I’d love to hear about second case …
My bad. I completely thought that you wanted routing.
No, indeed it can only offload a single (VLAN-aware bridge.) All other devices will be the same.
I can’t imagine what you would want to do that would necessarily require two bridges. Roughly here are the cases:
You don’t use VLANs on the bridges. Assign a VLAN internally with only access ports. Easy.
You use VLANs on both bridges, but (as mkx said,) not the same ones. Just use a single bridge with VLAN filtering. Easy.
You use some of the same VLANs on the two bridges. Two solutions. 1. use QinQ (non-0x8100 ethertype) 2. use port isolation (forwarding_vector in linux bridge2 naming) Again, easy.
I will assume then “q-tag” means the normal 802.1q (ethertype 0x8100) tags.
I will also assume that each customer has one physical connection to your switch.
And I take it that you want to connect q-tag-10 between Customer-2 and Customer-3. But that you would want not to connect Customer-2’s q-tag-20 to Customer-1 and Customer-2’s q-tag-20, but only connect it with let’s say Customer-4’s qtag-20.
Is this correct?
In this case the normal VLAN, Q-in-Q and port isolation abstractions break down, and you have to use switch rules to individually specify the switched port groups per ingress port and per vlan.
@lurker888 yes this is correct.
I didn’t really understand from your answer if this is still possible and if it is how to configuration should look like, can you please give me example.
@chechito, call it whatever VPLS, VNI etc. it should be just single broadcast domain where isolation of all traffic including BUM happens
@maxxch Yes, it is possible. And I’m happy to give you an example.
But before I do, could you please carefully reread what I have written, and confirm that I have correctly understood what you want to do. (I am not doubting that you somehow need this, but - at least from what I’ve seen - it’s quite unusual, in an IX even more so.)
@chechito When you have strange requirements, such as decide who to bridge traffic with on a per-vlan basis, or having to translate vlan ids while transiting the switch, what else can you do than parameterize the switch chip manually…
Made another drawing where should be easier to understand what i need.
CE1,3 are connected with tags(0x8100) to respectful bridge domains, so CE1 can talk to CE3 through Bridge3(orange)
CE1,2,3 can talk to each other in Green broadcast domain. and so on.
Also one specific case where we have CE4 and CE5 where different VLAN tags and they need to be connected to Orange(both)
All vlan tags should be stripped of once “injected” into broadcast domains and of course should be put back one leaving the switch.
Pls let me know now things are clear.
On the fourth port you prescribe that a broadcast packet that comes in with tag 500 should be reflected to the same port with tag 501. To further complicate this, a broadcast packet arriving on another port of your bridge4 would have to be sent out twice (multiplied) both with tag 500 and 501. Switch ASICs simply don’t do that.
The green, blue and red can be actually single bridge. The orange is a complication as @lurker wrote. To me it’s moot as to why it’s necessary to use two different VLANs from “the lower SwitchY” to CRS if they are eventually merged into same broadcast domain.
About Orange it is service called re-seller model where lower switchY is not under my administration and therefore to give access to certain peering LAN we might need to use such setup.
Blue also cannot merge with any other, here I’m showing small setup with just 4 ports, in our setup this Bridge1-4 is huge domain with terrabits of traffic and MPLS/VPLS.(and also not 4 bridges but hundreds of them)
It is really sad that this ASIC cannot do it, all BCM can but they are more expensive of course. This switch is perfect for routing(if of course there is no limitation similar to this one.
Can you please show how can I configure at least first three CEs? I’m curious how does it work.
What you actually want is overlay networking of some type. There are many possible standards and realizations to choose from, but currently VXLAN seems to be the weapon of choice across vendors. The ASICs that support this type of operation can sometimes (usually?) be configured to do what you want locally, but the feature set (the silicon that has to be present) is the same, so you will find it in the same category of devices.
As the reference to VXLAN shows, the devices that do this are not really switches, but have extensive L3 capabilities, and then have the necessary L3-to-L2 support. And then - you know, why not - usually support dynamic routing as well, in order to extend these capabilities through DC backbones.
MikroTik is currently starting to do VXLAN in hardware (which would actually do what you want, and at the scale you intend to use it at). I haven’t even tried it out yet, so maybe someone can chip in with some actual information, but I have the impression (and what they actually said is) that this feature is very much in preview. As with all things MikroTik, at what pace this will progress at is anyone’s guess. I’m cautiously optimistic.
