CRS326-24S+2Q 200 MBit/s Maxout

RcRaCk2k · August 24, 2020, 5:59pm

Sorry Guys, but is Mikrotik kidding me?

I have a CRS326-24S+2Q and i having trouble to get 250 MBit/s out of that switch.

All ports are HARDWARE-OFFLOADED, but the CPU goes that high, that the traffic caps at 250 MBit/s.
Core fucked up.JPG
Why is this happening?
Yeah i know i should update firmware because currently i am running on BETA.
But this issue should not happening at all.

  MMM      MMM       KKK                          TTTTTTTTTTT      KKK
  MMMM    MMMM       KKK                          TTTTTTTTTTT      KKK
  MMM MMMM MMM  III  KKK  KKK  RRRRRR     OOOOOO      TTT     III  KKK  KKK
  MMM  MM  MMM  III  KKKKK     RRR  RRR  OOO  OOO     TTT     III  KKKKK
  MMM      MMM  III  KKK KKK   RRRRRR    OOO  OOO     TTT     III  KKK KKK
  MMM      MMM  III  KKK  KKK  RRR  RRR   OOOOOO      TTT     III  KKK  KKK

  MikroTik RouterOS 6.47.2 (c) 1999-2020       http://www.mikrotik.com/

[?]             Gives the list of available commands
command [?]     Gives help on the command and list of arguments

[Tab]           Completes the command/word. If the input is ambiguous,
                a second [Tab] gives possible options

/               Move up to base level
..              Move up one level
/command        Use command at the base level
[rack@CoreSwitch-BreslauerStr] > export compact hide-sensitive 
# aug/29/2020 17:38:03 by RouterOS 6.47.2
# software id = U1HX-NX49
#
# model = CRS326-24S+2Q+
# serial number = AEF60AD4FEF3
/interface ethernet
set [ find default-name=qsfpplus1-1 ] disabled=yes speed=25Gbps
set [ find default-name=qsfpplus1-2 ] disabled=yes speed=25Gbps
set [ find default-name=qsfpplus1-3 ] disabled=yes speed=25Gbps
set [ find default-name=qsfpplus1-4 ] disabled=yes speed=25Gbps
set [ find default-name=qsfpplus2-1 ] disabled=yes speed=25Gbps
set [ find default-name=qsfpplus2-2 ] disabled=yes speed=25Gbps
set [ find default-name=qsfpplus2-3 ] disabled=yes speed=25Gbps
set [ find default-name=qsfpplus2-4 ] disabled=yes speed=25Gbps
set [ find default-name=sfp-sfpplus1 ] disabled=yes name=sfp-sfpplus1-west
set [ find default-name=sfp-sfpplus2 ] disabled=yes name=sfp-sfpplus2-west
set [ find default-name=sfp-sfpplus3 ] disabled=yes name=sfp-sfpplus3-west
set [ find default-name=sfp-sfpplus4 ] disabled=yes name=sfp-sfpplus4-west
set [ find default-name=sfp-sfpplus5 ] name=sfp-sfpplus5-ost
set [ find default-name=sfp-sfpplus6 ] disabled=yes name=sfp-sfpplus6-ost
set [ find default-name=sfp-sfpplus7 ] disabled=yes name=sfp-sfpplus7-ost
set [ find default-name=sfp-sfpplus8 ] disabled=yes name=sfp-sfpplus8-ost
set [ find default-name=sfp-sfpplus9 ] name=sfp-sfpplus9-router
set [ find default-name=sfp-sfpplus10 ] disabled=yes name=sfp-sfpplus10-router
set [ find default-name=sfp-sfpplus11 ] disabled=yes name=sfp-sfpplus11-router
set [ find default-name=sfp-sfpplus12 ] disabled=yes name=sfp-sfpplus12-router
set [ find default-name=sfp-sfpplus13 ] name=sfp-sfpplus13-Juniper-xe5-3-0
set [ find default-name=sfp-sfpplus14 ] name=sfp-sfpplus14-Juniper-xe5-3-1
set [ find default-name=sfp-sfpplus15 ] disabled=yes
set [ find default-name=sfp-sfpplus16 ] disabled=yes
set [ find default-name=sfp-sfpplus17 ] advertise=1000M-full auto-negotiation=no
set [ find default-name=sfp-sfpplus18 ] name=sfp-sfpplus18-Landertsham
set [ find default-name=sfp-sfpplus19 ] name=sfp-sfpplus19-WBB
set [ find default-name=sfp-sfpplus20 ] disabled=yes
set [ find default-name=sfp-sfpplus21 ] disabled=yes
set [ find default-name=sfp-sfpplus22 ] disabled=yes
set [ find default-name=sfp-sfpplus23 ] name=sfp-sfpplus23-Robel
set [ find default-name=sfp-sfpplus24 ] disabled=yes
/interface bridge
add admin-mac=74:4D:28:D2:6C:92 auto-mac=no comment=defconf ingress-filtering=yes name=bridge protocol-mode=none vlan-filtering=yes
/interface vlan
add interface=bridge name=bridge.2250-Management vlan-id=2250
/interface bonding
add comment="Uplink Juniper Router ae1 Interface" min-links=1 mode=802.3ad name=bond-Juniper-ae1 slaves=sfp-sfpplus13-Juniper-xe5-3-0,sfp-sfpplus14-Juniper-xe5-3-1 transmit-hash-policy=layer-3-and-4
add comment="Salzburg, A-5020 Salzburg" lacp-rate=1sec mode=802.3ad name=bond1-west slaves=sfp-sfpplus1-west,sfp-sfpplus2-west,sfp-sfpplus3-west,sfp-sfpplus4-west transmit-hash-policy=layer-3-and-4
add comment="Neukling, D-83416 Saaldorf-Surheim" lacp-rate=1sec mode=802.3ad name=bond2-ost slaves=sfp-sfpplus5-ost,sfp-sfpplus6-ost,sfp-sfpplus7-ost,sfp-sfpplus8-ost transmit-hash-policy=layer-3-and-4
/user group
set full policy=local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,password,web,sniff,sensitive,api,romon,dude,tikapp
/interface bridge port
add bridge=bridge comment=defconf interface=qsfpplus1-1
add bridge=bridge comment=defconf interface=qsfpplus1-2
add bridge=bridge comment=defconf interface=qsfpplus1-3
add bridge=bridge comment=defconf interface=qsfpplus1-4
add bridge=bridge comment=defconf interface=qsfpplus2-1
add bridge=bridge comment=defconf interface=qsfpplus2-2
add bridge=bridge comment=defconf interface=qsfpplus2-3
add bridge=bridge comment=defconf interface=qsfpplus2-4
add bridge=bridge comment=defconf interface=sfp-sfpplus9-router
add bridge=bridge comment=defconf interface=sfp-sfpplus10-router
add bridge=bridge comment=defconf interface=sfp-sfpplus11-router
add bridge=bridge comment=defconf interface=sfp-sfpplus12-router
add bridge=bridge comment=defconf interface=sfp-sfpplus15
add bridge=bridge comment=defconf interface=sfp-sfpplus16
add bridge=bridge comment=defconf interface=sfp-sfpplus17
add bridge=bridge comment=defconf interface=sfp-sfpplus18-Landertsham
add bridge=bridge comment=defconf interface=sfp-sfpplus19-WBB
add bridge=bridge comment=defconf interface=sfp-sfpplus20
add bridge=bridge comment=defconf interface=sfp-sfpplus21
add bridge=bridge comment=defconf interface=sfp-sfpplus22
add bridge=bridge comment=defconf ingress-filtering=yes interface=sfp-sfpplus23-Robel
add bridge=bridge comment=defconf ingress-filtering=yes interface=sfp-sfpplus24
add bridge=bridge interface=bond1-west
add bridge=bridge interface=bond2-ost
add bridge=bridge interface=bond-Juniper-ae1
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/interface bridge vlan
add bridge=bridge untagged=sfp-sfpplus23-Robel,sfp-sfpplus24,bridge vlan-ids=1
add bridge=bridge comment="VLAN Passtrough" tagged=bond1-west,bond2-ost vlan-ids=2-560,563-575,577-578,580-2219,2222-2224,2226-2249,2251-4094
add bridge=bridge comment="BGP Access-Routers Network" tagged=\
    sfp-sfpplus9-router,sfp-sfpplus10-router,sfp-sfpplus11-router,sfp-sfpplus12-router,bond1-west,bond2-ost,sfp-sfpplus18-Landertsham,sfp-sfpplus23-Robel,bond-Juniper-ae1 vlan-ids=2220-2221
add bridge=bridge comment="Inband Management VLAN" tagged=bond1-west,bond2-ost,bridge,sfp-sfpplus18-Landertsham vlan-ids=2250
add bridge=bridge comment="RuhNET IP-Transit via Neukling" tagged=sfp-sfpplus17,bond2-ost vlan-ids=576
add bridge=bridge comment="RuhNET IP-Transit via Salzburg" tagged=sfp-sfpplus17,bond1-west vlan-ids=579
add bridge=bridge comment="WEST Line of Fiber 2 (green to red)" tagged=\
    bond1-west,bond2-ost,sfp-sfpplus9-router,sfp-sfpplus10-router,sfp-sfpplus11-router,sfp-sfpplus12-router,sfp-sfpplus23-Robel,sfp-sfpplus18-Landertsham vlan-ids=2225
add bridge=bridge comment="Weissblau Breitband IP-Transit via Neukling" tagged=sfp-sfpplus19-WBB,bond2-ost vlan-ids=562
add bridge=bridge comment="Weissblau Breitband IP-Transit via Salzburg" tagged=sfp-sfpplus19-WBB,bond1-west vlan-ids=561
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=192.168.88.0
add address=172.16.10.12/24 interface=bridge.2250-Management network=172.16.10.0
/ip dns
set servers=91.205.12.0
/ip route
add distance=1 gateway=172.16.10.1,172.16.10.2
/system clock
set time-zone-name=Europe/Berlin
/system identity
set name=CoreSwitch-BreslauerStr
/system routerboard settings
set boot-os=router-os
[rack@CoreSwitch-BreslauerStr] >

biomesh · August 24, 2020, 11:47pm

Your config is way off. The config really needs to be mainly done via the bridge.

https://wiki.mikrotik.com/wiki/Manual:CRS3xx_series_switches

Here is the link for vlans with bonds on the crs3xx series.

https://wiki.mikrotik.com/wiki/Manual:CRS3xx_VLANs_with_Bonds

Kindis · August 25, 2020, 2:47pm

So this cannot be the complete config. The picture we see do not correlate to the config you pasted.
So even if ports are hardware offloaded conditions must be met. For instance VLAN in this case is not correct and I think this may break the offloading.
You need to implement VLAN filtering on the Bridge and add the add the management IP on the Bridge.

So I have a few CRS326-24G-2s+RM and here is my VLAN config. I have no issues to push full speed on any port.
My Management VLAN is 3000 which is added as PVID on the Bridge

/interface bridge
add name=B_Switch priority=0x2000 pvid=3000 vlan-filtering=yes
/interface bonding
add mode=802.3ad name=Core-LAN-B-LAN slaves=ether3,ether4
add mode=802.3ad name=Core-LAN-B-NAT slaves=ether1,ether2
add mode=802.3ad name=Core-WAN-B-NAT slaves=ether5,ether6
/interface bridge port
add bridge=B_Switch interface=ether24 pvid=3000
add bridge=B_Switch frame-types=admit-only-vlan-tagged interface=sfp-sfpplus1 \
    path-cost=5 priority=0x50
add bridge=B_Switch frame-types=admit-only-untagged-and-priority-tagged \
    interface=Core-LAN-B-NAT pvid=1111
add bridge=B_Switch frame-types=admit-only-vlan-tagged interface=\
    Core-LAN-B-LAN
add bridge=B_Switch frame-types=admit-only-untagged-and-priority-tagged \
    interface=Core-WAN-B-NAT pvid=1111
add bridge=B_Switch frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether7 pvid=3000
add bridge=B_Switch frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether8 pvid=1000
add bridge=B_Switch frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether9 pvid=1011
add bridge=B_Switch frame-types=admit-only-vlan-tagged interface=ether10
add bridge=B_Switch frame-types=admit-only-vlan-tagged interface=ether11
add bridge=B_Switch frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether12 pvid=1007
add bridge=B_Switch frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether13 pvid=1111
/interface bridge vlan
add bridge=B_Switch tagged=sfp-sfpplus1,Core-LAN-B-LAN,ether10,ether11 \
    untagged=ether24,ether7 vlan-ids=3000
add bridge=B_Switch tagged=Core-LAN-B-LAN,sfp-sfpplus1,ether10,ether11 \
    untagged=ether8 vlan-ids=1000
add bridge=B_Switch tagged=Core-LAN-B-LAN,sfp-sfpplus1,ether10 vlan-ids=1003
add bridge=B_Switch tagged=Core-LAN-B-LAN,sfp-sfpplus1 vlan-ids=1005
add bridge=B_Switch tagged=Core-LAN-B-LAN,sfp-sfpplus1 untagged=ether12 \
    vlan-ids=1007
add bridge=B_Switch tagged=Core-LAN-B-LAN,sfp-sfpplus1,ether10 untagged=\
    ether9 vlan-ids=1011
add bridge=B_Switch tagged=Core-LAN-B-LAN,sfp-sfpplus1,ether11,ether10 \
    vlan-ids=1012
add bridge=B_Switch tagged=Core-LAN-B-LAN,sfp-sfpplus1,ether11,ether10 \
    vlan-ids=2000
add bridge=B_Switch tagged=sfp-sfpplus1 untagged=\
    Core-LAN-B-NAT,Core-WAN-B-NAT,ether13 vlan-ids=1111
add bridge=B_Switch tagged=Core-LAN-B-LAN,sfp-sfpplus1,ether11,ether10 \
    vlan-ids=2004
/ip address
add address=10.2.4.16/24 interface=B_Switch network=10.2.4.0

RcRaCk2k · August 26, 2020, 5:46am

This is exactly how i did the configuration.
In most cases, i unpack the switch, and only changing PVIDs and Bridge-Port Settings for untagged / tagged. After that, enabling VLAN-FILTERING on bridge.
I checked up my configuration and is nothing wrong with it.

My posted configuration is a full export.

What is wrong with my VLAN Settings?

I also did.
Management VLAN is 2250.

/interface vlan
add interface=bridge name=bridge.2250-Management vlan-id=2250

I did a reboot at midnight, all went back to normal without changing the configuration.
That drives me crazy. Since using Mikrotik Switches i feel like a 24/7 active Network-Administrator. I thought that ubiquiti products are driving me nuts, but Mikrotik tops that all.

biomesh · August 26, 2020, 11:02am

Looking at your config you are only switching 2 ports - the qsfp+ ports. Everything else is going through the cpu. The bond interfaces need to be added to the bridge as well. Any bonded slave interface will not be in the bridge but only the bond interface itself. Any other standard interface should be a bridge port as well.

Your export does not show any config of the sort work regards to the bridge.

Following the two links I sent before, I have setup mutiple crs3xx series switches with zero performance issues. You might want to revert to a non beta release of you think it had any bearing.

Kindis · August 26, 2020, 12:55pm

You should not create any VLAN under interface but only handle the VLANS in the Bridge.

Can you do a full export with hide-sensitive

RcRaCk2k · August 29, 2020, 3:40pm

Oh yeha, now i see, the config-export was cut in half. Dont know why.
I have updated the initial post with the correct config-export.

Ah okay now i understand. Instead of adding a VLAN-Interface to the Bridge-Interface, you are setting PVID of the Bridge to the Management-VLAN, so you can add IP-Address directly to the bridge1 interface.

That sounds good.

I will adopt these settings on my switches and will see how that will improve the performance of the switch.