RouterOS-Bandwidth Test between 2 CRS328

CVXDEV · October 4, 2019, 2:24pm

Hi all,
I have a question related to the bandwidth-test tool within RouterOS.
I am operating in a test-setup 2 CRS328-24P-4S, and they are connected directly via a SFP±Fiber Connection.
When I do the Bandwidth-Test within ROS, I get very disappointing results:

Protocol	 Port	Connection Port	TX/RX (Mbps)
UDP	Send	3	Fiber	4	        530
UDP	receive	3	Fiber	4	        435
UDP	both	3	Fiber	4	       174 / 412
TCP	Send	3	Fiber	4	        220
TCP	receive	3	Fiber	4	        74
TCP	both	3	Fiber	4	        36 / 94

Both CRS are running in RouterOS 6.45.6, as a bridge and no further special settings.
Do you have any idea why I can’t even get close to 10Gb and not even reaching 1Gb?
(Connection was autonegiotiated and shows me a 10Gb link).

Any insight is appreciated - thanks

mjezierski · October 4, 2019, 3:20pm

Are you certain the fiber terminations are clean and free of any defects? Only time I’ve seen bandwidth that poor on even a 1G SFP was dirty fiber.

CZFan · October 4, 2019, 4:28pm

CRS devices are switches by design, and not routers, have low CPU specs.
You should test “through” these devices, and not from one to other
Depending on the configuration, i.e. firewall etc will also impact performance

CVXDEV · October 4, 2019, 4:41pm

Thank you both for your inputs.

Fiber should be fine, since I’ve tested it with various cables and with different modules. Also I tested it on all 4 Ports of the CRS

Ok, I additionally tested through them: Server (10Gb) -->Fiber–> Switch 1 -->Fiber–>Switch2 -->LAN (1Gb) → PC
And even there in some set-ups I get below the 1Gb throughput: around 440 Mbps
Tested with iperf3 (iperf3 -c IP -f -m), also the retries are very high (1666).

From the Switch Configuration, I have the most basic test-set-up, e.g. no firewall etc

CZFan · October 4, 2019, 4:44pm

can you share the full config?

CVXDEV · October 4, 2019, 5:14pm

Switch1 & 2:

# RouterOS 6.45.6
# software id = X2ZN-SQMN
#
# model = CRS328-24P-4S+
# serial number = 8223086D9600
/interface bridge
add admin-mac=CC:2D:E0:8D:B2:10 auto-mac=no comment=defconf name=bridge
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=ether8
add bridge=bridge comment=defconf interface=ether9
add bridge=bridge comment=defconf interface=ether10
add bridge=bridge comment=defconf interface=ether11
add bridge=bridge comment=defconf interface=ether12
add bridge=bridge comment=defconf interface=ether13
add bridge=bridge comment=defconf interface=ether14
add bridge=bridge comment=defconf interface=ether15
add bridge=bridge comment=defconf interface=ether16
add bridge=bridge comment=defconf interface=ether17
add bridge=bridge comment=defconf interface=ether18
add bridge=bridge comment=defconf interface=ether19
add bridge=bridge comment=defconf interface=ether20
add bridge=bridge comment=defconf interface=ether21
add bridge=bridge comment=defconf interface=ether22
add bridge=bridge comment=defconf interface=ether23
add bridge=bridge comment=defconf interface=ether24
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=ether8 list=LAN
add interface=ether9 list=LAN
add interface=ether10 list=LAN
add interface=ether11 list=LAN
add interface=ether12 list=LAN
add interface=ether13 list=LAN
add interface=ether14 list=LAN
add interface=ether15 list=LAN
add interface=ether16 list=LAN
add interface=ether17 list=LAN
add interface=ether18 list=LAN
add interface=ether19 list=LAN
add interface=ether20 list=LAN
add interface=ether21 list=LAN
add interface=ether22 list=LAN
add interface=ether23 list=LAN
add interface=ether24 list=LAN
add interface=sfp-sfpplus1 list=LAN
add interface=sfp-sfpplus2 list=LAN
add interface=sfp-sfpplus3 list=LAN
add interface=sfp-sfpplus4 list=LAN
/ip address
add address=192.168.88.111/24 comment=defconf interface=ether2 network=\
    192.168.88.0
/ip dns
set servers=8.8.8.8
/ip route
add distance=1 gateway=192.168.88.1
/system identity
set name=MikroTik-1
/system routerboard settings
set boot-os=router-os
/system swos
set address-acquisition-mode=static allow-from-ports="p1,p2,p3,p4,p5,p6,p7,p8,\
    p9,p10,p11,p12,p13,p14,p15,p16,p17,p18,p19,p20,p21,p22,p23,p24,p25,p26,p27\
    ,p28" identity=MikroTik-1 static-ip-address=192.168.88.111
---------------------------------------------------------------
Switch 2:

# RouterOS 6.45.6
# software id = 2VZH-3A2K
#
# model = CRS328-24P-4S+
# serial number = 822308C8D805
/interface bridge
add admin-mac=CC:2D:E0:8D:86:A4 auto-mac=no comment=defconf name=bridge
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=ether8
add bridge=bridge comment=defconf interface=ether9
add bridge=bridge comment=defconf interface=ether10
add bridge=bridge comment=defconf interface=ether11
add bridge=bridge comment=defconf interface=ether12
add bridge=bridge comment=defconf interface=ether13
add bridge=bridge comment=defconf interface=ether14
add bridge=bridge comment=defconf interface=ether15
add bridge=bridge comment=defconf interface=ether16
add bridge=bridge comment=defconf interface=ether17
add bridge=bridge comment=defconf interface=ether18
add bridge=bridge comment=defconf interface=ether19
add bridge=bridge comment=defconf interface=ether20
add bridge=bridge comment=defconf interface=ether21
add bridge=bridge comment=defconf interface=ether22
add bridge=bridge comment=defconf interface=ether23
add bridge=bridge comment=defconf interface=ether24
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=ether8 list=LAN
add interface=ether9 list=LAN
add interface=ether10 list=LAN
add interface=ether11 list=LAN
add interface=ether12 list=LAN
add interface=ether13 list=LAN
add interface=ether14 list=LAN
add interface=ether15 list=LAN
add interface=ether16 list=LAN
add interface=ether17 list=LAN
add interface=ether18 list=LAN
add interface=ether19 list=LAN
add interface=ether20 list=LAN
add interface=ether21 list=LAN
add interface=ether22 list=LAN
add interface=ether23 list=LAN
add interface=ether24 list=LAN
add interface=sfp-sfpplus1 list=LAN
add interface=sfp-sfpplus2 list=LAN
add interface=sfp-sfpplus3 list=LAN
add interface=sfp-sfpplus4 list=LAN
/ip address
add address=192.168.88.112/24 comment=defconf interface=ether2 network=\
    192.168.88.0
/system identity
set name=MikroTik-2
/system routerboard settings
set boot-os=router-os
/system swos
set allow-from-ports="p1,p2,p3,p4,p5,p6,p7,p8,p9,p10,p11,p12,p13,p14,p15,p16,p\
    17,p18,p19,p20,p21,p22,p23,p24,p25,p26,p27,p28" identity=MikroTik-2 \
    static-ip-address=192.168.88.112

CZFan · October 4, 2019, 5:55pm

IP addresses of the devices should be on bridge interface and not slave interface ether2.

What does Tools–>Profile show when you do transfers through the devices?

CVXDEV · October 8, 2019, 12:18pm

Sorry for my late reply.
I changed the IPs to bridge interface and if i check the tools–>profile, the CPU seems to be normal when I run iperf3.
CPU goes to 5.5.-6 max on usage: Screenshot:

LINKhttps://www.dropbox.com/s/33wreohc86llrpg/tools-screenshot.jpg?dl=0
Left is Switch 1 and right SWITCH-2 while running iperf3.