I’m new to MikroTik and more used to cisco, mellanox etc. So I’m fairly certain I’m missing something fundamental here, like fully understanding the concept of bridges. I did look at a guide at youtube when configuring this and since thrown pretty much everything I have at it. But there’s something wrong and something I’m missing.
The setup is:
This CRS326-24S+2Q+ connects with 10GbE to a Cisco nexus switch. This over a transit network, 10.0.254.120/30.
One single VLAN on the Switch with (when fully configured) <5 clients of which 1 is 10GbE.
The issues
First when I had hardware offload on clients bridge ports the performance between 2 test 10GbE clients on vlan132 was limited to sub 40MB/sec (iperf tests) and the CPU usage was at 80%
When I disabled Hardware offload I got 10GbE iperf-performance beetween the clients
When traffic goes over the transit network CPU usage again goes to 80% and performance drops to sub 40MB/sec
You will see remains of a bridge for the transit network in the config, atm not in use.
I am most likely missing something obvious here and I appreciate any pointers in the right direction.
Thanks
# oct/25/2022 06:39:09 by RouterOS 7.5
# software id = 4RZJ-MY29
#
# model = CRS326-24S+2Q+
/interface bridge
add comment="*** TRANSIT BRIDGE ***" disabled=yes fast-forward=no name=\
BR-TRANSIT protocol-mode=none
add comment="*** VLAN132 BRIDGE ***" fast-forward=no name=BR-VLAN132 \
protocol-mode=none
/interface ethernet
set [ find default-name=ether1 ] disabled=yes
set [ find default-name=qsfpplus1-1 ] disabled=yes
set [ find default-name=qsfpplus1-2 ] disabled=yes
set [ find default-name=qsfpplus1-3 ] disabled=yes
set [ find default-name=qsfpplus1-4 ] disabled=yes
set [ find default-name=qsfpplus2-1 ] disabled=yes
set [ find default-name=qsfpplus2-2 ] disabled=yes
set [ find default-name=qsfpplus2-3 ] disabled=yes
set [ find default-name=qsfpplus2-4 ] disabled=yes
set [ find default-name=sfp-sfpplus1 ] disabled=yes
set [ find default-name=sfp-sfpplus2 ] disabled=yes
set [ find default-name=sfp-sfpplus3 ] disabled=yes
set [ find default-name=sfp-sfpplus4 ] disabled=yes
set [ find default-name=sfp-sfpplus5 ] disabled=yes
set [ find default-name=sfp-sfpplus6 ] disabled=yes
set [ find default-name=sfp-sfpplus7 ] disabled=yes
set [ find default-name=sfp-sfpplus8 ] disabled=yes
set [ find default-name=sfp-sfpplus9 ] disabled=yes
set [ find default-name=sfp-sfpplus10 ] disabled=yes
set [ find default-name=sfp-sfpplus11 ] disabled=yes
set [ find default-name=sfp-sfpplus12 ] disabled=yes
set [ find default-name=sfp-sfpplus13 ] disabled=yes
set [ find default-name=sfp-sfpplus14 ] auto-negotiation=no comment=\
"*** client dac test ***" speed=10Gbps
set [ find default-name=sfp-sfpplus15 ] disabled=yes
set [ find default-name=sfp-sfpplus16 ] disabled=yes
set [ find default-name=sfp-sfpplus17 ] disabled=yes
set [ find default-name=sfp-sfpplus18 ] disabled=yes
set [ find default-name=sfp-sfpplus19 ] disabled=yes
set [ find default-name=sfp-sfpplus20 ] disabled=yes
set [ find default-name=sfp-sfpplus21 ] disabled=yes
set [ find default-name=sfp-sfpplus22 ] disabled=yes
set [ find default-name=sfp-sfpplus23 ] disabled=yes
set [ find default-name=sfp-sfpplus24 ] auto-negotiation=no comment=\
"*** IP TRANSIT - 10.0.254.120/30 ***" speed=10Gbps
/interface vlan
add comment="*** VLAN132 ***" interface=BR-VLAN132 name=vlan132 vlan-id=132
/interface ethernet switch
set 1 cpu-flow-control=no
/interface list
add name=WAN
add name=LAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
/routing bgp template
set default disabled=no output.network=bgp-networks
/routing ospf instance
/routing ospf area
add disabled=no instance=default-v2 name=backbone-v2
/interface bridge port
add bridge=BR-VLAN132 comment="*** CLIENT ***" hw=no ingress-filtering=no \
interface=sfp-sfpplus14 pvid=132
add bridge=BR-VLAN132 comment="*** TEMP 10G client ***" disabled=yes \
ingress-filtering=no interface=sfp-sfpplus16 pvid=132
add bridge=BR-VLAN132 disabled=yes ingress-filtering=no interface=\
sfp-sfpplus18 pvid=132
add bridge=BR-TRANSIT comment="*** IP TRANSIT - 10.0.254.120/30 ***" \
disabled=yes ingress-filtering=no interface=sfp-sfpplus24
add bridge=BR-VLAN132 comment="*** BRIDGE PORT PVID VLAN132 ***" \
ingress-filtering=no interface=vlan132 pvid=132
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface bridge vlan
add bridge=BR-VLAN132 comment="*** VLAN132 ***" untagged=\
sfp-sfpplus18,sfp-sfpplus16,sfp-sfpplus14 vlan-ids=132
/interface ovpn-server server
set auth=sha1,md5
/ip address
add address=10.0.254.122/30 interface=sfp-sfpplus24 network=\
10.0.254.120
add address=192.168.1.33/27 comment="*** test test ***" interface=BR-VLAN132 \
network=192.168.1.32
/ip cloud
set update-time=no
/ip dhcp-server network
add address=192.168.1.32/27 gateway=192.168.1.33
/ip dns
set servers=x.x.x.x,x.x.x.x
/ip dns static
add address=159.148.147.204 comment=\
"*** mikrotik upgrade ***" name=upgrade.mikrotik.com
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=10.0.254.121
/ip service
set telnet disabled=yes
set ftp disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ip smb
set allow-guests=no
/ip smb shares
set [ find default=yes ] disabled=yes
/system routerboard settings
set boot-os=router-os
IMHO, one bridge is not the “best” solution. This is - the “only” solution. Only in single bridge mode will hardware offloading start working.
This CPU has poor performance for such tasks, so the entire load must be handled by the switching chip. Plus - it is advisable not to monitor the switch (via Winbox or Dude) itself at the time of the test of the exchange rate between clients.
Hmm. So when looking at using only one bridge I lost ability to ping the other end of the transit net. Couldn’t get it to work even after reverting config. rebooted the unit but still didn’t work. I then disabled ‘l3-hw-offloading’ on switch1 and instantly got ping to work. Enabled it again and it still works… Something feels fishy here?
If you need to transit traffic to another device using this switch. And the rest of the ports of this switch to use to connect devices on the local network - is another option.
/interface bridge add disabled=no name=BR-LAN protocol-mode=none
### Isolate transit ports
/interface ethernet switch port-isolation set qsfpplus1-1 forwarding-override=qsfpplus1-2
/interface ethernet switch port-isolation set qsfpplus1-2 forwarding-override=qsfpplus1-1
### Input from Cisco (trunk port)
/interface bridge port add bridge=BR-LAN hw=yes interface=qsfpplus1-1 pvid=1
### Output to other switch (trunk port)
/interface bridge port add bridge=BR-LAN hw=yes interface=qsfpplus1-2 pvid=1
### Access port
/interface bridge port add bridge=BR-LAN hw=yes interface=xxx
/interface bridge vlan add bridge=BR-LAN tagged=qsfpplus1-1,qsfpplus1-2 vlan-ids=132
/interface bridge set BR-LAN vlan-filtering=yes
Quick follow up q before I look at this. This still doesn’t explain the high CPU usage? What in the current config is so wrong that it causes this? And in what way is it wrong?
is it normal though? to get 30 percent just by opening winbox and monitor the traffic, it seems so bad. but the performance is fine, its just bothering me sometimes to open the winbox just to see the usage is high
Something feels fishy here?
