Hello,
we have a CRS317 connected via SFP+ fiber (Allnet / Ubiquity / Intel (server side)) and different Broadcom NICs. Iperf3 tests shows, that I can’t get over 1Gb and so I asking for help. Maybe there is a configuration issue.
The configuration looks like:
# jan/26/2021 13:08:40 by RouterOS 6.47.4
# software id = M7UZ-9NNP
#
# model = CRS317-1G-16S+
/interface ethernet
set [ find default-name=ether1 ] comment=management
set [ find default-name=sfp-sfpplus1 ] comment=fra-test-san speed=10Gbps
set [ find default-name=sfp-sfpplus2 ] comment=fra-test-pmox-06 speed=10Gbps
set [ find default-name=sfp-sfpplus3 ] comment=fra-test-pmox-07 speed=10Gbps
set [ find default-name=sfp-sfpplus4 ] comment=fra-test-pmox-06 speed=10Gbps
set [ find default-name=sfp-sfpplus5 ] comment=fra-test-pmox-07 speed=10Gbps
set [ find default-name=sfp-sfpplus6 ] comment=frei speed=10Gbps
set [ find default-name=sfp-sfpplus7 ] comment=fra-test-pmox-03
set [ find default-name=sfp-sfpplus8 ] comment=frei
set [ find default-name=sfp-sfpplus9 ] comment=fra-test-pmox-08 speed=10Gbps
set [ find default-name=sfp-sfpplus10 ] comment=frei speed=10Gbps
set [ find default-name=sfp-sfpplus11 ] comment=frei speed=10Gbps
set [ find default-name=sfp-sfpplus12 ] comment=frei speed=10Gbps
set [ find default-name=sfp-sfpplus13 ] comment=frei speed=10Gbps
set [ find default-name=sfp-sfpplus14 ] comment=frei speed=10Gbps
set [ find default-name=sfp-sfpplus15 ] comment="Cross connect"
set [ find default-name=sfp-sfpplus16 ] comment=hp-stack
/interface bridge
add admin-mac=D2:xx:xx:xx:xx:xx auto-mac=no name=core-lan vlan-filtering=yes
add name=management
set full policy=local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,password,web,sniff,sensitive,api,romon,dude,tikapp
/interface bridge msti
add bridge=core-lan identifier=1 priority=0x1000 vlan-mapping=800,900,701,702
/interface bridge port
add bridge=management comment="management interface" interface=ether1
add bridge=core-lan comment=fra-test-san edge=yes interface=sfp-sfpplus1
add bridge=core-lan comment=fra-test-pmox-06 edge=yes interface=sfp-sfpplus2
add bridge=core-lan comment=fra-test-pmox-07 edge=yes interface=sfp-sfpplus3
add bridge=core-lan comment=crossconnect edge=yes interface=sfp-sfpplus15
add bridge=core-lan comment=hp-stack edge=yes interface=sfp-sfpplus16
add bridge=core-lan interface=sfp-sfpplus6
add bridge=core-lan interface=sfp-sfpplus4
add bridge=core-lan interface=sfp-sfpplus5
add bridge=core-lan interface=sfp-sfpplus8
add bridge=core-lan interface=sfp-sfpplus7
add bridge=core-lan comment="DMZ fortigate" interface=sfp-sfpplus14
add bridge=core-lan comment="fortigate LAN" interface=sfp-sfpplus13
add bridge=core-lan comment=fra-test-pmox-08 interface=sfp-sfpplus9
/ip neighbor discovery-settings
set discover-interface-list=none
/interface bridge vlan
add bridge=core-lan comment="internal LAN" tagged=sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,sfp-sfpplus6,sfp-sfpplus16,sfp-sfpplus5,sfp-sfpplus13,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9 vlan-ids=701
add bridge=core-lan comment="DMZ LAN" tagged=sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus16,sfp-sfpplus14,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9 vlan-ids=702
add bridge=core-lan comment="deployment LAN" tagged=sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus16,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9 vlan-ids=800
add bridge=core-lan comment="SAN storage" tagged=sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus15,sfp-sfpplus9 vlan-ids=900
/ip address
add address=10.10.10.4/24 interface=management network=10.10.10.0
/ip dns
set servers=172.25.25.5
/ip route
add distance=1 gateway=10.10.10.10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set api disabled=yes
set winbox disabled=yes
set api-ssl disabled=yes
/system routerboard settings
set boot-os=router-os
We have a copper crossconnect with a S+RJ10 to a second CRS317 (for failover) and a uplink to a HP-Stack (1GB) also with a S+RJ10.
A short test looks like:
Connecting to host 10.2.0.11, port 5201
[ 5] local 10.2.0.18 port 57218 connected to 10.2.0.11 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 182 MBytes 1.53 Gbits/sec 41 312 KBytes
[ 5] 1.00-2.00 sec 181 MBytes 1.52 Gbits/sec 50 336 KBytes
[ 5] 2.00-3.00 sec 180 MBytes 1.51 Gbits/sec 31 252 KBytes
[ 5] 3.00-4.00 sec 181 MBytes 1.52 Gbits/sec 23 250 KBytes
[ 5] 4.00-5.00 sec 155 MBytes 1.30 Gbits/sec 36 341 KBytes
[ 5] 5.00-6.00 sec 100 MBytes 843 Mbits/sec 0 523 KBytes
[ 5] 6.00-7.00 sec 102 MBytes 854 Mbits/sec 4 502 KBytes
[ 5] 7.00-8.00 sec 102 MBytes 853 Mbits/sec 16 458 KBytes
[ 5] 8.00-9.00 sec 99.8 MBytes 837 Mbits/sec 0 602 KBytes
[ 5] 9.00-10.00 sec 99.9 MBytes 838 Mbits/sec 8 559 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.35 GBytes 1.16 Gbits/sec 209 sender
[ 5] 0.00-10.00 sec 1.35 GBytes 1.16 Gbits/sec receiver
iperf Done.
Connecting to host 10.2.0.11, port 5201
[ 5] local 10.2.0.18 port 57226 connected to 10.2.0.11 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 95.0 MBytes 797 Mbits/sec 33 254 KBytes
[ 5] 1.00-2.00 sec 96.9 MBytes 812 Mbits/sec 0 462 KBytes
[ 5] 2.00-3.00 sec 96.6 MBytes 810 Mbits/sec 19 297 KBytes
[ 5] 3.00-4.00 sec 95.5 MBytes 801 Mbits/sec 58 295 KBytes
[ 5] 4.00-5.00 sec 95.6 MBytes 802 Mbits/sec 30 268 KBytes
[ 5] 5.00-6.00 sec 94.2 MBytes 790 Mbits/sec 3 369 KBytes
[ 5] 6.00-7.00 sec 95.9 MBytes 805 Mbits/sec 2 384 KBytes
^[[A[ 5] 7.00-8.00 sec 96.2 MBytes 807 Mbits/sec 46 230 KBytes
^[[A^[[A[ 5] 8.00-9.00 sec 95.3 MBytes 800 Mbits/sec 107 91.2 KBytes
[ 5] 9.00-10.00 sec 95.1 MBytes 798 Mbits/sec 2 277 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 956 MBytes 802 Mbits/sec 300 sender
[ 5] 0.00-10.00 sec 955 MBytes 801 Mbits/sec receiver
iperf Done.
We don’t use MTU 9000 yet, because I had trouble mit Corosync (Proxmox). If I do the iperf, “bridging” and “networking” has the highest CPU usage.
# INTERFACE BRIDGE HW PVID PRIORITY PATH-COST INTERNAL-PATH-COST HORIZON
0 H ;;; management interface
ether1 management yes 1 0x80 10 10 none
1 ;;; fra-test-san
sfp-sfpplus1 core-lan yes 1 0x80 10 10 none
2 ;;; fra-test-pmox-06
sfp-sfpplus2 core-lan yes 1 0x80 10 10 none
3 ;;; fra-test-pmox-07
sfp-sfpplus3 core-lan yes 1 0x80 10 10 none
4 ;;; crossconnect
sfp-sfpplus15 core-lan yes 1 0x80 10 10 none
5 ;;; hp-stack
sfp-sfpplus16 core-lan yes 1 0x80 10 10 none
6 sfp-sfpplus6 core-lan yes 1 0x80 10 10 none
7 sfp-sfpplus4 core-lan yes 1 0x80 10 10 none
8 sfp-sfpplus5 core-lan yes 1 0x80 10 10 none
9 sfp-sfpplus8 core-lan yes 1 0x80 10 10 none
10 sfp-sfpplus7 core-lan yes 1 0x80 10 10 none
11 ;;; DMZ fortigate
sfp-sfpplus14 core-lan yes 1 0x80 10 10 none
12 ;;; fortigate LAN
sfp-sfpplus13 core-lan yes 1 0x80 10 10 none
13 ;;; fra-test-pmox-08
sfp-sfpplus9 core-lan yes 1 0x80 10 10 none
Any suggestions ?
Update
Found the reason: HW offload was enabled for the management bridge, so that it wasn’t possible to enable on core-lan. Disable it on management solves the problem:
Connecting to host 10.2.0.11, port 5201
[ 5] local 10.2.0.18 port 57364 connected to 10.2.0.11 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.10 GBytes 9.48 Gbits/sec 0 1.58 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.47 Gbits/sec 0 1.74 MBytes
[ 5] 2.00-3.00 sec 1.10 GBytes 9.46 Gbits/sec 0 1.74 MBytes
[ 5] 3.00-4.00 sec 1.08 GBytes 9.30 Gbits/sec 0 1.74 MBytes
[ 5] 4.00-5.00 sec 1.10 GBytes 9.47 Gbits/sec 0 1.74 MBytes
[ 5] 5.00-6.00 sec 1.10 GBytes 9.46 Gbits/sec 0 1.74 MBytes
[ 5] 6.00-7.00 sec 1.10 GBytes 9.47 Gbits/sec 0 1.74 MBytes
[ 5] 7.00-8.00 sec 1.10 GBytes 9.46 Gbits/sec 0 1.74 MBytes
[ 5] 8.00-9.00 sec 1.10 GBytes 9.47 Gbits/sec 0 1.74 MBytes
[ 5] 9.00-10.00 sec 1.08 GBytes 9.30 Gbits/sec 0 1.74 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.0 GBytes 9.43 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.0 GBytes 9.43 Gbits/sec receiver
iperf Done.
Solution / reson found here: https://wiki.mikrotik.com/wiki/Manual:Layer2_misconfiguration#Bridges_on_a_single_switch_chip