Community discussions

MikroTik App
 
dani2015
just joined
Topic Author
Posts: 11
Joined: Mon May 18, 2015 5:50 pm

CRS326-24S+2Q+ : 100% CPU utilization bridging when a port goes up or down

Sun Feb 19, 2023 5:15 am

Hi.

We have 2 CRS326-24S+2Q+
The configuration works OK. It has 1 brige only and the needed ports added there with the corresponding VLAN rules per port (PVID, tagged/untagged).

But the matter is when a server conected reboots, or if you take down a port in Winbox, the CPU spickes to 100% and the problem acording to "profile tool" is that "bridging" takes all the CPU.
The port role in Bridge (RSTP) takes time to be set (don't know it it could be the cause or an effect of the CPU spike).

We first thought that could be MLAG the origin of the problem. We removed MLAG and the related LACP links. Then removed the LACP that was with the 40G ports to connect both switches.
So now is no bonding at all, just standard ports.

When all is fine:
bridge.png
bridge-vlan.png
bridge-ports.png
profile ok.png
If you take down a port in winbox, cpu spikes for a while (could be 20-30s easily) and the switch freezes basically on this moment
profile nok.png
You do it on one switch, but creates a similar reaction in the other switch connected with some seconds of delay.
So maybe is RSTP related

Doing an "interface bridge export":
/interface bridge
add ingress-filtering=no name=bridge-main vlan-filtering=yes
/interface bridge port
add bridge=bridge-main interface=qsfpplus1-1 pvid=2
add bridge=bridge-main interface=sfp-sfpplus1-ceph-px1 pvid=2
add bridge=bridge-main interface=sfp-sfpplus2-ceph-px2 pvid=2
add bridge=bridge-main interface=sfp-sfpplus3-ceph-px3 pvid=2
add bridge=bridge-main interface=sfp-sfpplus4-ceph-px4 pvid=2
add bridge=bridge-main interface=sfp-sfpplus9-int-px1 pvid=3
add bridge=bridge-main interface=sfp-sfpplus10-int-px2 pvid=3
add bridge=bridge-main interface=sfp-sfpplus11-int-px3 pvid=3
add bridge=bridge-main interface=sfp-sfpplus12-int-px4 pvid=3
add bridge=bridge-main interface=sfp-sfpplus17-pub-px1 pvid=4
add bridge=bridge-main interface=sfp-sfpplus18-pub-px2 pvid=4
add bridge=bridge-main interface=sfp-sfpplus19-pub-px3 pvid=4
add bridge=bridge-main interface=sfp-sfpplus20-pub-px4 pvid=4
add bridge=bridge-main interface=sfp-sfpplus22 pvid=3
add bridge=bridge-main interface=qsfpplus2-1 internal-path-cost=15 path-cost=15 \
    pvid=2
/interface bridge vlan
add bridge=bridge-main tagged=sfp-sfpplus22 untagged="sfp-sfpplus1-ceph-px1,sfp-\
    sfpplus2-ceph-px2,sfp-sfpplus3-ceph-px3,sfp-sfpplus4-ceph-px4,qsfpplus1-1,qs\
    fpplus2-1" vlan-ids=2
add bridge=bridge-main tagged=qsfpplus1-1,qsfpplus1-4,sfp-sfpplus22 untagged="sf\
    p-sfpplus17-pub-px1,sfp-sfpplus18-pub-px2,sfp-sfpplus19-pub-px3,sfp-sfpplus2\
    0-pub-px4" vlan-ids=4
add bridge=bridge-main tagged=qsfpplus1-1,qsfpplus2-1 untagged="sfp-sfpplus9-int\
    -px1,sfp-sfpplus10-int-px2,sfp-sfpplus11-int-px3,sfp-sfpplus12-int-px4,sfp-s\
    fpplus22" vlan-ids=3
add bridge=bridge-main tagged="qsfpplus1-1,qsfpplus2-2,sfp-sfpplus17-pub-px1,sfp\
    -sfpplus18-pub-px2,sfp-sfpplus19-pub-px3,sfp-sfpplus20-pub-px4,sfp-sfpplus22\
    " vlan-ids=5
add bridge=bridge-main tagged="qsfpplus1-1,qsfpplus2-1,sfp-sfpplus1-ceph-px1,sfp\
    -sfpplus2-ceph-px2,sfp-sfpplus3-ceph-px3,sfp-sfpplus4-ceph-px4,sfp-sfpplus17\
    -pub-px1,sfp-sfpplus18-pub-px2,sfp-sfpplus19-pub-px3,sfp-sfpplus20-pub-px4,s\
    fp-sfpplus9-int-px1,sfp-sfpplus10-int-px2,sfp-sfpplus11-int-px3,sfp-sfpplus1\
    2-int-px4,sfp-sfpplus22" vlan-ids=7-4094
But takes a pause of 10s or easily between the 1st block (the initial 2 lines - "/interface bridge") and the rest (until "/interface bridge port" starts)

It has RouterOS 7.7 on both switches.
L3 HW Offlloading isn't active, as no L3 is in use.
Management is connected to the 100Mps Ethernet management port, where DHCP Client is active.
Similar (mirrored) conf is in both switches

Doing an export of the conf is painfully slow and the CPU spikes to 100%. In profile is "management" the one using the CPU basically.
profile export.png
When i say slow is that an export could take 4 min easily.
/interface bridge
add ingress-filtering=no name=bridge-main vlan-filtering=yes
/interface ethernet
set [ find default-name=qsfpplus1-1 ] l2mtu=10218 mtu=9000
set [ find default-name=qsfpplus1-2 ] l2mtu=10218
set [ find default-name=qsfpplus1-3 ] l2mtu=10218
set [ find default-name=qsfpplus1-4 ] l2mtu=10218
set [ find default-name=qsfpplus2-1 ] l2mtu=10218 mtu=9000
set [ find default-name=qsfpplus2-2 ] l2mtu=10218
set [ find default-name=qsfpplus2-3 ] l2mtu=10218
set [ find default-name=qsfpplus2-4 ] l2mtu=10218
set [ find default-name=sfp-sfpplus1 ] l2mtu=10218 mtu=9000 name=sfp-sfpplus1-ceph-px1
set [ find default-name=sfp-sfpplus2 ] l2mtu=10218 mtu=9000 name=sfp-sfpplus2-ceph-px2
set [ find default-name=sfp-sfpplus3 ] l2mtu=10218 mtu=9000 name=sfp-sfpplus3-ceph-px3
set [ find default-name=sfp-sfpplus4 ] l2mtu=10218 mtu=9000 name=sfp-sfpplus4-ceph-px4
set [ find default-name=sfp-sfpplus9 ] name=sfp-sfpplus9-int-px1
set [ find default-name=sfp-sfpplus10 ] name=sfp-sfpplus10-int-px2
set [ find default-name=sfp-sfpplus11 ] name=sfp-sfpplus11-int-px3
set [ find default-name=sfp-sfpplus12 ] name=sfp-sfpplus12-int-px4
set [ find default-name=sfp-sfpplus17 ] name=sfp-sfpplus17-pub-px1
set [ find default-name=sfp-sfpplus18 ] name=sfp-sfpplus18-pub-px2
set [ find default-name=sfp-sfpplus19 ] name=sfp-sfpplus19-pub-px3
set [ find default-name=sfp-sfpplus20 ] name=sfp-sfpplus20-pub-px4
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
/interface bridge port
add bridge=bridge-main interface=qsfpplus1-1 pvid=2
add bridge=bridge-main interface=sfp-sfpplus1-ceph-px1 pvid=2
add bridge=bridge-main interface=sfp-sfpplus2-ceph-px2 pvid=2
add bridge=bridge-main interface=sfp-sfpplus3-ceph-px3 pvid=2
add bridge=bridge-main interface=sfp-sfpplus4-ceph-px4 pvid=2
add bridge=bridge-main interface=sfp-sfpplus9-int-px1 pvid=3
add bridge=bridge-main interface=sfp-sfpplus10-int-px2 pvid=3
add bridge=bridge-main interface=sfp-sfpplus11-int-px3 pvid=3
add bridge=bridge-main interface=sfp-sfpplus12-int-px4 pvid=3
add bridge=bridge-main interface=sfp-sfpplus17-pub-px1 pvid=4
add bridge=bridge-main interface=sfp-sfpplus18-pub-px2 pvid=4
add bridge=bridge-main interface=sfp-sfpplus19-pub-px3 pvid=4
add bridge=bridge-main interface=sfp-sfpplus20-pub-px4 pvid=4
add bridge=bridge-main interface=sfp-sfpplus22 pvid=3
add bridge=bridge-main interface=qsfpplus2-1 internal-path-cost=15 path-cost=15 pvid=2
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface bridge vlan
add bridge=bridge-main tagged=sfp-sfpplus22 untagged="sfp-sfpplus1-ceph-px1,sfp-sfpplus2-ceph-px2,sfp-\
    sfpplus3-ceph-px3,sfp-sfpplus4-ceph-px4,qsfpplus1-1,qsfpplus2-1" vlan-ids=2
add bridge=bridge-main tagged=qsfpplus1-1,qsfpplus1-4,sfp-sfpplus22 untagged=\
    sfp-sfpplus17-pub-px1,sfp-sfpplus18-pub-px2,sfp-sfpplus19-pub-px3,sfp-sfpplus20-pub-px4 vlan-ids=\
    4
add bridge=bridge-main tagged=qsfpplus1-1,qsfpplus2-1 untagged="sfp-sfpplus9-int-px1,sfp-sfpplus10-int\
    -px2,sfp-sfpplus11-int-px3,sfp-sfpplus12-int-px4,sfp-sfpplus22" vlan-ids=3
add bridge=bridge-main tagged="qsfpplus1-1,qsfpplus2-2,sfp-sfpplus17-pub-px1,sfp-sfpplus18-pub-px2,sfp\
    -sfpplus19-pub-px3,sfp-sfpplus20-pub-px4,sfp-sfpplus22" vlan-ids=5
add bridge=bridge-main tagged="qsfpplus1-1,qsfpplus2-1,sfp-sfpplus1-ceph-px1,sfp-sfpplus2-ceph-px2,sfp\
    -sfpplus3-ceph-px3,sfp-sfpplus4-ceph-px4,sfp-sfpplus17-pub-px1,sfp-sfpplus18-pub-px2,sfp-sfpplus19\
    -pub-px3,sfp-sfpplus20-pub-px4,sfp-sfpplus9-int-px1,sfp-sfpplus10-int-px2,sfp-sfpplus11-int-px3,sf\
    p-sfpplus12-int-px4,sfp-sfpplus22" vlan-ids=7-4094
/interface ovpn-server server
set auth=sha1,md5
/ip dhcp-client
add interface=ether1
/ip dns
set servers=9.9.9.9,1.1.1.1
/ip route
add check-gateway=ping disabled=no dst-address=0.0.0.0/0 gateway=XX.XX.XX.XXX routing-table=main \
    suppress-hw-offload=no
/system clock
set time-zone-name=Europe/Berlin
/system identity
set name=RouterOS2
/system ntp client
set enabled=yes
/system ntp client servers
add address=pool.ntp.org
/system routerboard settings
set boot-os=router-os
Any idea of what could be wrong?
You do not have the required permissions to view the files attached to this post.
 
dani2015
just joined
Topic Author
Posts: 11
Joined: Mon May 18, 2015 5:50 pm

Re: CRS326-24S+2Q+ : 100% CPU utilization bridging when a port goes up or down

Sun Feb 19, 2023 3:22 pm

Tried to check if removing the FAST FORWARD on the bridge and adding Ingress filtering could help, but they have the the same issue.

The switch becomes unresponsible with those peacks, not only in the management/Winbox/shell, it start loosing packets or not switching at all during a little while.
 
dani2015
just joined
Topic Author
Posts: 11
Joined: Mon May 18, 2015 5:50 pm

Re: CRS326-24S+2Q+ : 100% CPU utilization bridging when a port goes up or down

Sun Feb 19, 2023 3:52 pm

When the peak happens, i've seen that the bridge can even disappear.
Here a put in shutdown several ports, the peak comes and i see brige leaving the interfaces:
no bridge.png
And when it relaxed, it came back:

no bridge back.png

Checked setting RouterOS 7.6 instead of 7.7 and apparently is the same issue.
You do not have the required permissions to view the files attached to this post.
 
dani2015
just joined
Topic Author
Posts: 11
Joined: Mon May 18, 2015 5:50 pm

Re: CRS326-24S+2Q+ : 100% CPU utilization bridging when a port goes up or down

Mon Feb 20, 2023 2:35 am

After checking i found the trigger that cause the problem, even not a real solution.

As it was happening with RouterOS 7.7 and also downgrading to 7.6 i decided, with 7.6 installed, to do a configuration reset.
Then recreate the configuration. Created an LACP bond to link the switches, MLAG LACP bonds.. All ok.

Upgrade to 7.7 then... Same, all OK.

Then when we add the last rule to allow to use some ports as TRUNK ports, so can use (almost) any VLAN in tagged (802.1q) mode, in that case VLAN 7 up to 4094:
vlan brigde.png
the CPU problem described in the post, with "bridging" using all the CPU comes inplace.

It i remove that VLAN rule (vlan 7-4094 as tagged in some interfaces) from the Bridge, then after it's removed all comes back to normal

If i add the same rule with just a few vlans, with 100-300 for example (200 VLAN), there's no problem.

What could be the problem for that to trigger that big issue?
You do not have the required permissions to view the files attached to this post.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CRS326-24S+2Q+ : 100% CPU utilization bridging when a port goes up or down

Mon Feb 20, 2023 7:46 am

you are adding 4.000 VLAN tagged to an interface?, can you explain that?
 
dani2015
just joined
Topic Author
Posts: 11
Joined: Mon May 18, 2015 5:50 pm

Re: CRS326-24S+2Q+ : 100% CPU utilization bridging when a port goes up or down

Tue Feb 21, 2023 11:30 pm

To be able to use the interface as trunk interface.
It's something common.

Usually you add 1 as native and 2 to 4094 as 802.1q ones for fully trunk ports.

Same config works OK on a CRS317 for example.
 
tangent
Forum Guru
Forum Guru
Posts: 1333
Joined: Thu Jul 01, 2021 3:15 pm
Contact:

Re: CRS326-24S+2Q+ : 100% CPU utilization bridging when a port goes up or down

Wed Feb 22, 2023 1:50 am

This feels like a bug in the layer that translates generic ROS configuration rules into specific switch chip programming commands. You’ve done a great job of diagnosing it as far as you can from the user level. It’s time to report it to MT support.

Best distill the essential info into the actual report. Don’t just point to this thread and its irrelevant speculations, backtracking, etc. Show the straight-line path from problem identification to diagnosis only. Link to the thread last, “for more information.”
 
dani2015
just joined
Topic Author
Posts: 11
Joined: Mon May 18, 2015 5:50 pm

Re: CRS326-24S+2Q+ : 100% CPU utilization bridging when a port goes up or down

Sat Feb 25, 2023 4:18 pm

Already reported.

It sound like if the orders to the switch CPU are sent in a seralized way.
Instead of an order for VLAN 2 to 4094 for example, like 4092 separete orders (just an idea as it works ok while it's stable until a port goes down or you change the config).
 
emunt6
Frequent Visitor
Frequent Visitor
Posts: 87
Joined: Fri Feb 02, 2018 7:00 pm

Re: CRS326-24S+2Q+ : 100% CPU utilization bridging when a port goes up or down

Sat Feb 25, 2023 4:42 pm

Hi!
Try the following:
"vlan-ids=7-4094"
replace
"vlan-ids=4095"

The id 4095 is the "trunk" but i don't think it is needed to configure this way,
There is a better solution would be the following:

/interface 
add bridge=bridge-main vlan-filering=yes 

/interface bridge port
# trunk port example
add bridge=bridge-main interface=qsfpplus1-1 pvid=1 frame-types=admint-only-vlan-tagged ingress-filtering=yes
...
# access port example, vlan = pvid = 3 
add bridge=bridge-main interface=sfp-sfpplus1-ceph-px1 pvid=3 frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes
...

The rest of the config I do not understand, why do you want to join vlans (untagged) on interface? ( vlan pruning / vlan filtering ?)
 
rmac1813
just joined
Posts: 2
Joined: Fri Mar 10, 2023 5:40 am

Re: CRS326-24S+2Q+ : 100% CPU utilization bridging when a port goes up or down

Wed May 31, 2023 5:24 pm

Im having the exact issue on a crs354 (and crs310) - all connectivity goes down + 100% cpu when a server (trunk) port comes online and goes offline.
ROS 7.9
Screenshot 2023-05-31 075927.png
You do not have the required permissions to view the files attached to this post.
Last edited by rmac1813 on Wed May 31, 2023 6:20 pm, edited 3 times in total.
 
AlexanderPronichev
just joined
Posts: 1
Joined: Thu Oct 05, 2023 9:38 pm

Re: CRS326-24S+2Q+ : 100% CPU utilization bridging when a port goes up or down

Tue Oct 10, 2023 9:12 pm

Hello!
I have the same issue.
Is there any updates?

Who is online

Users browsing this forum: GoogleOther [Bot], Kanzler, miks and 90 guests