Currently I am facing the following problem: We are using two CRS518-16XS-2XQ switches in an MLAG configuration. Both switches are running on RouterOS 7.2.3. Connected to the switches are three servers running Proxmox VE 7.4-3. The servers are bonding two of their SFP28 interfaces each in 802.3ad ("layer2" hashing policy) - i.e. one bond for Migration of VMs, one bond for Ceph and one bond for corosync.
I attached an image of the setup (with one server only).
The MLAG seems to be established successfully but running iperf3 between the servers yields just around ~450 MBit/s:
Code: Select all
root@virt01-clus:~# iperf3 -c 10.20.0.2
Connecting to host 10.20.0.2, port 5201
[ 5] local 10.20.0.1 port 38108 connected to 10.20.0.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 56.6 MBytes 475 Mbits/sec 6 856 KBytes
[ 5] 1.00-2.00 sec 55.6 MBytes 466 Mbits/sec 4 821 KBytes
[ 5] 2.00-3.00 sec 53.7 MBytes 451 Mbits/sec 0 1.07 MBytes
[ 5] 3.00-4.00 sec 55.0 MBytes 461 Mbits/sec 5 918 KBytes
[ 5] 4.00-5.00 sec 55.0 MBytes 461 Mbits/sec 0 1.14 MBytes
[ 5] 5.00-6.00 sec 55.0 MBytes 461 Mbits/sec 4 987 KBytes
[ 5] 6.00-7.00 sec 53.8 MBytes 451 Mbits/sec 0 1.20 MBytes
[ 5] 7.00-8.00 sec 55.0 MBytes 461 Mbits/sec 5 1.04 MBytes
[ 5] 8.00-9.00 sec 55.0 MBytes 461 Mbits/sec 3 970 KBytes
[ 5] 9.00-10.00 sec 55.0 MBytes 461 Mbits/sec 4 856 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 550 MBytes 461 Mbits/sec 31 sender
[ 5] 0.00-10.05 sec 547 MBytes 457 Mbits/sec receiver
The Switch configuration is as follows
Code: Select all
/interface bonding
add mlag-id=10 mode=802.3ad name=ceph-bond-prox1 slaves=sfp28-2
add mlag-id=20 mode=802.3ad name=ceph-bond-prox2 slaves=sfp28-4
add mlag-id=30 mode=802.3ad name=ceph-bond-prox3 slaves=sfp28-6
add mlag-id=40 mode=802.3ad name=coro-bond-prox1 slaves=sfp28-1
add mlag-id=50 mode=802.3ad name=coro-bond-prox2 slaves=sfp28-3
add mlag-id=60 mode=802.3ad name=coro-bond-prox3 slaves=sfp28-5
add min-links=1 mlag-id=70 mode=802.3ad mtu=9000 name=migration-bond-prox1 \
slaves=sfp28-8
add min-links=1 mlag-id=80 mode=802.3ad mtu=9000 name=migration-bond-prox2 \
slaves=sfp28-10
add min-links=1 mlag-id=90 mode=802.3ad mtu=9000 name=migration-bond-prox3 \
slaves=sfp28-12
add mode=802.3ad name=mlag-bond slaves=sfp28-15,sfp28-16
Code: Select all
/interface bridge mlag
set bridge=cluster-bridge peer-port=mlag-bond
Code: Select all
/interface bridge port
add bridge=cluster-bridge interface=mlag-bond pvid=99
add bridge=cluster-bridge interface=ceph-bond-prox1 pvid=3000
add bridge=cluster-bridge interface=ceph-bond-prox2 pvid=3000
add bridge=cluster-bridge interface=ceph-bond-prox3 pvid=3000
add bridge=cluster-bridge interface=coro-bond-prox1 pvid=4000
add bridge=cluster-bridge interface=coro-bond-prox2 pvid=4000
add bridge=cluster-bridge interface=coro-bond-prox3 pvid=4000
add bridge=cluster-bridge interface=migration-bond-prox1 pvid=2000
add bridge=cluster-bridge interface=migration-bond-prox2 pvid=2000
add bridge=cluster-bridge interface=migration-bond-prox3 pvid=2000
I am grateful for any insights or possible ideas from your side.
Best regards,
Alex