CRS2xx high ping

RouterOS General
1

Hello,
I have a network lab with the following schema:
rb4011 ↔ crs226-24g-2s+ ↔ computer1
rb4011 ↔ crs326-24g-2s+ ↔ computer2

I have noticed that there is a noticeable delay (>1ms) in ICMP between the RB4011 and the CRS226, whereas it is low (300ns) between RB4011 and CRS326.

Any hint as to what might be happening?
Additionally, how can I verify, from the device itself, the bandwidth in use by the aggregated ports?

As a backround:
I was given a 10Gb SFP+ cage and module, and wanted to test it in the lab.
I configured trunk/bonding from the RB4011 to the CRS226 to aggregate 2Gb using static LACP, and configured 802.3ad LACP between RB4011 and CRS326.

Adding full config for CRS226:

# mar/02/2022 01:30:46 by RouterOS 7.1.2
# software id = WB68-IFYK
#
# model = CRS226-24G-2S+
/interface bridge
add admin-mac=4C:5E:0C:99:0A:BB auto-mac=no comment=defconf name=bridge protocol-mode=none
/interface ethernet
set [ find default-name=sfpplus2 ] disabled=yes
/interface vlan
add interface=bridge name=mgmt vlan-id=99
/interface ethernet switch
set drop-if-invalid-or-src-port-not-member-of-vlan-on-ports=ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,ether11,ether12,ether13,ether14,ether15,ether16,ether17,ether18,ether19,ether20,ether21,ether22,ether23,ether24,sfp-sfpplus1,sfpplus2
/interface ethernet switch trunk
add member-ports=ether23,ether24 name=trunk23_24
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
/system logging action
set 3 remote=192.168.100.15
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=ether8
add bridge=bridge comment=defconf interface=ether9
add bridge=bridge comment=defconf interface=ether10
add bridge=bridge comment=defconf interface=ether11
add bridge=bridge comment=defconf interface=ether12
add bridge=bridge comment=defconf interface=ether13
add bridge=bridge comment=defconf interface=ether14
add bridge=bridge comment=defconf interface=ether15
add bridge=bridge comment=defconf interface=ether16
add bridge=bridge comment=defconf interface=ether17
add bridge=bridge comment=defconf interface=ether18
add bridge=bridge comment=defconf interface=ether19
add bridge=bridge comment=defconf interface=ether20
add bridge=bridge comment=defconf interface=ether21
add bridge=bridge comment=defconf interface=ether22
add bridge=bridge comment=defconf interface=ether23
add bridge=bridge comment=defconf interface=ether24
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfpplus2
/interface ethernet switch egress-vlan-tag
add tagged-ports=trunk23_24,switch1-cpu,ether19,ether20,ether21,ether22,sfp-sfpplus1 vlan-id=99
add tagged-ports=trunk23_24,ether19,ether20,ether21,sfp-sfpplus1 vlan-id=100
add tagged-ports=trunk23_24,ether19,ether20,ether21,ether22,sfp-sfpplus1 vlan-id=101
add tagged-ports=trunk23_24,ether19,ether20,ether21,ether22,sfp-sfpplus1 vlan-id=102
add tagged-ports=trunk23_24,ether19,ether20,ether21,ether22,sfp-sfpplus1 vlan-id=103
/interface ethernet switch ingress-vlan-translation
add new-customer-vid=100 ports=ether2,ether3,ether4,ether1,ether22
add new-customer-vid=101 ports=ether9,ether10
add new-customer-vid=102 ports=ether11,ether12
add new-customer-vid=103 ports=ether13,ether14
/interface ethernet switch vlan
add ports=trunk23_24,switch1-cpu,ether19,ether20,ether21,ether22,sfp-sfpplus1 vlan-id=99
add ports=trunk23_24,ether1,ether2,ether3,ether4,ether19,ether20,ether21,ether22,sfp-sfpplus1 vlan-id=100
add ports=trunk23_24,ether9,ether10,ether19,ether20,ether21,ether22,sfp-sfpplus1 vlan-id=101
add ports=trunk23_24,ether11,ether12,ether19,ether20,ether21,ether22,sfp-sfpplus1 vlan-id=102
add ports=trunk23_24,ether13,ether14,ether19,ether20,ether21,ether22,sfp-sfpplus1 vlan-id=103
/ip address
add address=192.168.99.3/24 interface=mgmt network=192.168.99.0
/ip dns
set servers=192.168.99.1
/ip route
add gateway=192.168.99.1
/lcd
set enabled=no
/system clock
set time-zone-name=Europe/Madrid
/system identity
set name=switch-attic
/system logging
add action=remote topics=info
add action=remote topics=error
add action=remote topics=warning
add action=remote topics=critical

Relevant config for RB4011 bonding:

# mar/02/2022 01:32:48 by RouterOS 7.1.2
# software id = CRIM-P0J0
#
# model = RB4011iGS+
/interface bonding
add mode=balance-xor name=bond1_2 slaves=ether1,ether2 transmit-hash-policy=layer-2-and-3
add mode=802.3ad name=bond3_4 slaves=ether3,ether4
2

Pinging a switch doesn’t prove anything because that is handled by switch general CPU and CRS226 has a pretty slow CPU (400 MHz MIPSBE). What you should test is ping through switch, i.e. between RB4011 and computer1 … only if that ping gets high (more than 1ms), then you should worry.

On a side note: bonding doesn’t lower RTT nor it increases throughput of a single L4 connection, almost every Tx strategy keeps certain traffic on the same physical link (member of bond). Even on the most “meshy” Tx strategy (L3+L4 hash, not supported by every LACP-capable devices) single TCP and UDP connections use same link for every packet. Your configuration (L2+L3 hash) means that all traffic between same pair of peers (e.g. computer1 + computer2) will always use single link and it will always be the same link (because L2 - MAC address and L3 - IP address don’t change).

3

Yes, it’s why I gave up on LACP for GigE links. Now that we have reasonable cost 2.5, 5, and 10 GigE hardware, using LACP to try and get, for example, 4 gig by burning four ports is kinda silly, even when you can manage to configure it to give you the aggregate bandwidth you want.

It’s just too fiddly to be worth it now.

4

Thank you both for your responses.
I will check connectivity between ends and avoid hitting the CRS226 directly.

Also would look into cat6/cat7 instead of bonding, as the hash policy might not be ideal for such small lab.