High CPU Load with broadcast traffic

RouterOS General
1

I have a pair of CRS328 switch linked together with fibre at 10Gbit.

I have a number of vlans and everything is running good except for one thing.
One of the VLANs has 100Mbps in both directions of layer 2 broadcast traffic. It connects two devices that use a lot of bandwidth and are normally directly connected together.
The manufacturer specifies that they can be networked so long as they are on their own VLAN and there is no other traffic on that VLAN.
I’ve got them talking to each other but they are occasionally losing clock sync between them and I’ve traced it down to both CRS328 units running at 100% CPU.

The switches show 20% CPU load on the bridge and 80% load on “networking”

I have turned off the following features.

  • network discovery (so there is no accidental traffic on any interfaces).
  • SFP
  • Auto negotiation (the devices require hard coding to 100Mbps)

I have turned on

  • VLAN filtering
  • ingress filtering (allow untagged & priority)

Hardware offloading is turned on and it shows H on the interfaces.

I’m stumped as to what is causing all the CPU load and how I can offload the layer 2 traffic to the switch chip.

Any ideas or suggestions for tracing this issue.

Port 24 is the port I am running this traffic on in both switchs

Here is the config of switch 1

# feb/11/2019 09:27:56 by RouterOS 6.43.8
# software id = FCPZ-JU9G
#
# model = CRS328-24P-4S+
# serial number = 8223086118B3
/interface bridge
add admin-mac=CC:2D:E0:8E:AC:8D auto-mac=no comment=defconf frame-types=\
    admit-only-vlan-tagged ingress-filtering=yes mtu=1592 name=bridge \
    protocol-mode=none pvid=2 vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] mac-address=CC:2D:E0:8E:AC:8D
set [ find default-name=ether12 ] loop-protect-send-interval=10h5s
set [ find default-name=ether20 ] loop-protect=off
set [ find default-name=ether21 ] arp=disabled loop-protect=off \
    rx-flow-control=on tx-flow-control=on
set [ find default-name=ether22 ] arp=disabled loop-protect=off \
    rx-flow-control=on tx-flow-control=on
set [ find default-name=ether23 ] arp=disabled loop-protect=off \
    rx-flow-control=on tx-flow-control=on
set [ find default-name=ether24 ] advertise=100M-full auto-negotiation=no \
    loop-protect=off loop-protect-disable-time=1s loop-protect-send-interval=\
    4d4h40m39s mtu=1592 poe-out=off speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] loop-protect=off
set [ find default-name=sfp-sfpplus2 ] loop-protect=off
/interface vlan
add interface=bridge name=2 vlan-id=2
add interface=bridge name=10 vlan-id=10
add interface=bridge name=20 vlan-id=20
add interface=bridge loop-protect=off name=21 vlan-id=21
add interface=bridge loop-protect=off name=22 vlan-id=22
add interface=bridge loop-protect=off name=23 vlan-id=23
add interface=bridge loop-protect=off mtu=1588 name=24 vlan-id=24
add interface=bridge name=30 vlan-id=30
add interface=bridge name=40 vlan-id=40
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=switch1
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/ip ipsec proposal
set [ find default=yes ] disabled=yes
/queue interface
set ether24 queue=ethernet-default
/routing bgp instance
set default disabled=yes
/routing ospf area
set [ find default=yes ] disabled=yes
/routing ospf instance
set [ find default=yes ] disabled=yes
/interface bridge port
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether1 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether2 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether3 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether4 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether5 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether6 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether7 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether8 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether9 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether10 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether11 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether12 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether13 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether14 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether15 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether16 pvid=10
add bridge=bridge hw=no ingress-filtering=yes interface=ether17 pvid=2
add bridge=bridge hw=no ingress-filtering=yes interface=ether18 pvid=2
add bridge=bridge hw=no ingress-filtering=yes interface=ether19 pvid=2
add bridge=bridge hw=no ingress-filtering=yes interface=ether20 pvid=2
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged hw=no \
    ingress-filtering=yes interface=ether21 pvid=21
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged hw=no \
    ingress-filtering=yes interface=ether22 pvid=22
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=yes interface=ether23 pvid=23
add bridge=bridge edge=yes-discover frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether24 point-to-point=yes pvid=24
add bridge=bridge interface=sfp-sfpplus1 pvid=2
add bridge=bridge interface=sfp-sfpplus2 pvid=2
add bridge=bridge interface=sfp-sfpplus3 pvid=10
add bridge=bridge interface=sfp-sfpplus4 pvid=10
/ip neighbor discovery-settings
set discover-interface-list=none
/ip settings
set icmp-rate-limit=0 secure-redirects=no send-redirects=no
/interface bridge vlan
add bridge=bridge tagged=\
    bridge,sfp-sfpplus1,sfp-sfpplus2,ether20,ether19,ether18,ether17 \
    untagged="ether1,ether2,ether3,ether4,ether7,ether8,ether9,ether10,ether11\
    ,ether12,ether13,ether14,ether15,ether16" vlan-ids=10
add bridge=bridge disabled=yes tagged=bridge untagged=\
    ether17,ether18,ether19,ether20 vlan-ids=2
add bridge=bridge tagged=\
    bridge,sfp-sfpplus1,sfp-sfpplus2,ether17,ether18,ether19,ether20 \
    vlan-ids=20
add bridge=bridge tagged=\
    bridge,sfp-sfpplus1,sfp-sfpplus2,ether17,ether18,ether19,ether20 \
    vlan-ids=30
add bridge=bridge tagged=\
    bridge,sfp-sfpplus1,sfp-sfpplus2,ether17,ether18,ether19,ether20 \
    vlan-ids=40
add bridge=bridge tagged=bridge,sfp-sfpplus1,sfp-sfpplus2 untagged=ether23 \
    vlan-ids=23
add bridge=bridge tagged=bridge,sfp-sfpplus1,sfp-sfpplus2 untagged=ether21 \
    vlan-ids=21
add bridge=bridge tagged=bridge,sfp-sfpplus1,sfp-sfpplus2 untagged=ether22 \
    vlan-ids=22
add bridge=bridge tagged=bridge,sfp-sfpplus1,sfp-sfpplus2 untagged=ether24 \
    vlan-ids=24
/ip address
add address=192.168.1.5/24 interface=10 network=192.168.1.0
/ip dhcp-client
add dhcp-options=hostname,clientid disabled=no interface=10
/ip dns
set servers=192.168.1.1
/ip ipsec policy
set 0 disabled=yes
/ip route
add distance=1 gateway=192.168.1.1
/routing bfd interface
set [ find default=yes ] disabled=yes
/routing rip interface
add interface=10 send=v1-2
/system clock
set time-zone-name=Pacific/Auckland
/system identity
set name=np-fibre-sw-1
/system routerboard settings
set boot-os=router-os
/tool sniffer
set file-limit=100000KiB filter-interface=ether24



Here is the config of switch 2.

# feb/11/2019 09:24:56 by RouterOS 6.43.8
# software id = DZDL-JR2J
#
# model = CRS328-24P-4S+
# serial number = 822308C328EA
/interface bridge
add admin-mac=CC:2D:E0:8E:B0:E8 auto-mac=no comment=defconf name=bridgeLocal \
    protocol-mode=none vlan-filtering=yes
/interface ethernet
set [ find default-name=ether24 ] auto-negotiation=no loop-protect=off \
    poe-out=off speed=100Mbps
/interface vlan
add interface=bridgeLocal loop-protect=off name=2 vlan-id=2
add interface=bridgeLocal loop-protect=off name=10 vlan-id=10
add interface=bridgeLocal loop-protect=off name=20 vlan-id=20
add interface=bridgeLocal loop-protect=off name=24 vlan-id=24
add interface=bridgeLocal loop-protect=off name=30 vlan-id=30
add interface=bridgeLocal loop-protect=off name=40 vlan-id=40
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=switch2
/interface bridge port
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether1 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether2 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether3 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether4 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether5 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether6 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether7 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether8 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether9 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether10 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether11 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether12 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether13 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether14 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether15 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether16 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether17 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether18 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether19 pvid=10
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether20 pvid=10
add bridge=bridgeLocal comment=defconf interface=ether21
add bridge=bridgeLocal comment=defconf interface=ether22
add bridge=bridgeLocal comment=defconf interface=ether23
add bridge=bridgeLocal comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    ether24 pvid=24
add bridge=bridgeLocal comment=defconf interface=sfp-sfpplus1
add bridge=bridgeLocal comment=defconf interface=sfp-sfpplus2
add bridge=bridgeLocal comment=defconf interface=sfp-sfpplus3
add bridge=bridgeLocal comment=defconf interface=sfp-sfpplus4
/ip neighbor discovery-settings
set discover-interface-list=none
/interface bridge vlan
add bridge=bridgeLocal tagged=bridgeLocal,sfp-sfpplus2,sfp-sfpplus1 untagged="\
    ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,eth\
    er11,ether12,ether13,ether14,ether15,ether16" vlan-ids=10
add bridge=bridgeLocal tagged=bridgeLocal,sfp-sfpplus1,sfp-sfpplus2 untagged=\
    ether24 vlan-ids=24
/ip address
add address=192.168.1.6 interface=10 network=192.168.1.6
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=\
    bridgeLocal
add dhcp-options=hostname,clientid disabled=no interface=10
/system clock
set time-zone-name=Pacific/Auckland
/system routerboard settings
set boot-os=router-os

Here is a screenshot of a packet dump from port 24 and also CPU load below
2019-02-11 09_40_27-Window.png
2019-02-11 09_40_27-Window1.png

2

How is anyone supposed to know unless you provide the configurations of both switches?

3

apologies, config added to original post.

4

First thing: in /interface bridge vlan bridge itself only needs to be member of VLAN group for those vlans where CRS needs to interact (L3). That’s only VLAN ID 10 as that’s the only VLAN interface with IP address.

At the same time: I think that list of untagged ports for vlan-ids=10 whould not be inside double quotes, this way ROS might think it’s single port with completely weird name (you can check that with /interface bridge vlan print … does it show 16 ether ports as untagged? It should show 16 lines not 16 ports, separated with comma.

5

ahh, that makes sense. Thankyou so much for your help. I’m from a ubiquiti background and I’ve bought the microtiks as ubiquiti don’t have anything with 4 SFP+ ports so this is a little new to me.
Trying to get my head around the different language and terminology for the same thing :slight_smile:

I’ve removed the bridge from the tagged vlan ports and the output of the command is

[admin@np-fibre-sw-1] >  /interface bridge vlan print
Flags: X - disabled, D - dynamic 
 #   BRIDGE                                                                   VLAN-IDS  CURRENT-TAGGED                                                                   CURRENT-UNTAGGED                                                                  
 0   bridge                                                                   10        bridge                                                                           ether6                                                                            
                                                                                        sfp-sfpplus2                                                                     ether12                                                                           
                                                                                                                                                                         ether8                                                                            
                                                                                                                                                                         ether16                                                                           
                                                                                                                                                                         ether14                                                                           
                                                                                                                                                                         ether15                                                                           
 1   bridge                                                                   2                                                                                          bridge                                                                            
                                                                                                                                                                         sfp-sfpplus2                                                                      
 2   bridge                                                                   20        sfp-sfpplus2                                                                    
 3   bridge                                                                   30        sfp-sfpplus2                                                                    
 4   bridge                                                                   40        sfp-sfpplus2                                                                    
 5   bridge                                                                   23        sfp-sfpplus2                                                                    
 6   bridge                                                                   21        sfp-sfpplus2                                                                    
 7   bridge                                                                   22        sfp-sfpplus2                                                                    
 8   bridge                                                                   24        sfp-sfpplus2                                                                     ether24

After those changes, still showing 100% CPU.

6

Is HW offload actually active for both ether24 and sfp-sfpplus2? You can check that using command /interface bridge port print … if HW offload is active then port shows a “H” in column right after interface index and before the interface name, such as this:

/interface bridge port print
Flags: X - disabled, I - inactive, D - dynamic, H - hw-offload 
 #     INTERFACE          BRIDGE         HW  PVID PRIORITY  PATH-COST INTERNAL-PATH-COST    HORIZON
 0   H ether1             bridge         yes    1     0x80         10                 10       none
 1 I   wifi               bridge                1     0x80         10                 10       none
 2 I   wifi-guest         bridge                1     0x80         10                 10       none
 3 I H ether2             bridge         yes    1     0x80         10                 10       none
 4 I H ether3             bridge         yes    1     0x80         10                 10       none
 5 I H ether4             bridge         yes    1     0x80         10                 10       none
 6 I H ether5             bridge         yes    1     0x80         10                 10       none

All ether ports have HW offload active while wlan interfaces don’t (they don’t support it so that’s expected).

If HW offload is not active, then this is the problem.

7

Yes, Hardware offload is active on both.
2019-02-12 07_48_15-Window.png
2019-02-12 07_52_18-Window.png
here are the traffic flows if that sets off any alarm bells, but I don’t see anything here that should be overloading the CPU.
2019-02-12 08_14_44-Window.png

8

The last screenshot shows that bridge with its vlan24 interface still receives all of vlan24 traffic. This is weird, since bridge is not member of said vlan it should not receive all that traffic.
Did you perform reboot of CRS after you removed bridge from vlan24 member port list? If not, it might be time to do it to clear some possible remnants … also remove vlan24 interface (attached to bridge), you don’t need it.

Another question: is traffic between ether24 and sfp-sfpplus2 truely symmetric? Counters seem to show that …

9

i’ve removed all the vlan interfaces 21-24 and rebooted.
yes the traffic is symetric, the devices send 32 channels of low latency audio in both directions so they should have identical traffic in both directions.

I had a look at the logs after a reboot and can see this…
It is definitely not a loop but the switch’s are obviously detecting the broadcasts and assuming its something bad or a DOS attack.
2019-02-12 10_40_55-Window.png
still showing 100% cpu


2019-02-12 10_42_47-Window.png
2019-02-12 10_43_47-Window.png