Community discussions

MikroTik App
 
almi01
just joined
Topic Author
Posts: 6
Joined: Thu Mar 30, 2023 10:59 am

[SOLVED] Extremely low bandwidth in SFP28 / MLAG setup

Thu Mar 30, 2023 11:29 am

Hello all,

Currently I am facing the following problem: We are using two CRS518-16XS-2XQ switches in an MLAG configuration. Both switches are running on RouterOS 7.2.3. Connected to the switches are three servers running Proxmox VE 7.4-3. The servers are bonding two of their SFP28 interfaces each in 802.3ad ("layer2" hashing policy) - i.e. one bond for Migration of VMs, one bond for Ceph and one bond for corosync.

I attached an image of the setup (with one server only).

The MLAG seems to be established successfully but running iperf3 between the servers yields just around ~450 MBit/s:
root@virt01-clus:~# iperf3 -c 10.20.0.2
Connecting to host 10.20.0.2, port 5201
[  5] local 10.20.0.1 port 38108 connected to 10.20.0.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  56.6 MBytes   475 Mbits/sec    6    856 KBytes       
[  5]   1.00-2.00   sec  55.6 MBytes   466 Mbits/sec    4    821 KBytes       
[  5]   2.00-3.00   sec  53.7 MBytes   451 Mbits/sec    0   1.07 MBytes       
[  5]   3.00-4.00   sec  55.0 MBytes   461 Mbits/sec    5    918 KBytes       
[  5]   4.00-5.00   sec  55.0 MBytes   461 Mbits/sec    0   1.14 MBytes       
[  5]   5.00-6.00   sec  55.0 MBytes   461 Mbits/sec    4    987 KBytes       
[  5]   6.00-7.00   sec  53.8 MBytes   451 Mbits/sec    0   1.20 MBytes       
[  5]   7.00-8.00   sec  55.0 MBytes   461 Mbits/sec    5   1.04 MBytes       
[  5]   8.00-9.00   sec  55.0 MBytes   461 Mbits/sec    3    970 KBytes       
[  5]   9.00-10.00  sec  55.0 MBytes   461 Mbits/sec    4    856 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   550 MBytes   461 Mbits/sec   31             sender
[  5]   0.00-10.05  sec   547 MBytes   457 Mbits/sec                  receiver

The Switch configuration is as follows
/interface bonding
add mlag-id=10 mode=802.3ad name=ceph-bond-prox1 slaves=sfp28-2
add mlag-id=20 mode=802.3ad name=ceph-bond-prox2 slaves=sfp28-4
add mlag-id=30 mode=802.3ad name=ceph-bond-prox3 slaves=sfp28-6
add mlag-id=40 mode=802.3ad name=coro-bond-prox1 slaves=sfp28-1
add mlag-id=50 mode=802.3ad name=coro-bond-prox2 slaves=sfp28-3
add mlag-id=60 mode=802.3ad name=coro-bond-prox3 slaves=sfp28-5
add min-links=1 mlag-id=70 mode=802.3ad mtu=9000 name=migration-bond-prox1 \
    slaves=sfp28-8
add min-links=1 mlag-id=80 mode=802.3ad mtu=9000 name=migration-bond-prox2 \
    slaves=sfp28-10
add min-links=1 mlag-id=90 mode=802.3ad mtu=9000 name=migration-bond-prox3 \
    slaves=sfp28-12
add mode=802.3ad name=mlag-bond slaves=sfp28-15,sfp28-16
MLAG-Config:
/interface bridge mlag
set bridge=cluster-bridge peer-port=mlag-bond
Bridge-Ports:
/interface bridge port
add bridge=cluster-bridge interface=mlag-bond pvid=99
add bridge=cluster-bridge interface=ceph-bond-prox1 pvid=3000
add bridge=cluster-bridge interface=ceph-bond-prox2 pvid=3000
add bridge=cluster-bridge interface=ceph-bond-prox3 pvid=3000
add bridge=cluster-bridge interface=coro-bond-prox1 pvid=4000
add bridge=cluster-bridge interface=coro-bond-prox2 pvid=4000
add bridge=cluster-bridge interface=coro-bond-prox3 pvid=4000
add bridge=cluster-bridge interface=migration-bond-prox1 pvid=2000
add bridge=cluster-bridge interface=migration-bond-prox2 pvid=2000
add bridge=cluster-bridge interface=migration-bond-prox3 pvid=2000
When I directly connect two of the servers I get the full 25G over iperf3. The output of "/log print" also show, that the interfaces come up with the negotiated speed of 25GBit/s.
I am grateful for any insights or possible ideas from your side.

Best regards,

Alex
You do not have the required permissions to view the files attached to this post.
Last edited by almi01 on Wed Apr 12, 2023 3:12 pm, edited 1 time in total.
 
almi01
just joined
Topic Author
Posts: 6
Joined: Thu Mar 30, 2023 10:59 am

Re: Extremely low bandwidth in SFP28 / MLAG setup

Tue Apr 11, 2023 5:14 pm

Update:

If I remove LACP and connect two servers to just one switch (i.e. no MLAG no LACP etc.) I also jsut get ~ 450 - 470 MBit/s. I disabled Autonegotiation and set the ports to 25GBit/s Full-Duplex.

Anyone else facing this problem or found a solution? Or has anyone successfully saturated the full speed on this make of switch? I am at a loss here.
 
almi01
just joined
Topic Author
Posts: 6
Joined: Thu Mar 30, 2023 10:59 am

Re: Extremely low bandwidth in SFP28 / MLAG setup  [SOLVED]

Wed Apr 12, 2023 3:11 pm

Update: I got this solved


When leaving the interfaces in the default "bridge" I get the full bandwidth.

When the interfaces are in a newly created bridge (like in my case) the bandwidth is poor. I compared both configs and couldn't find the difference. Apparently I created a new bridge since in order to form a new LACP Channel you have to remove the interfaces from the bridge first.

I didn't readd the interfaces to the default bridge because I thought It would give me a "cleaner" setup if I had a separate bridge for the MLAG Config only.

So I got this solved, in case anyone is losing their mind over this.
 
User avatar
sirbryan
Member
Member
Posts: 303
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: [SOLVED] Extremely low bandwidth in SFP28 / MLAG setup

Wed Apr 12, 2023 8:14 pm

Only one bridge can belong to the hardware switch. Additional bridges are all routed through the CPU, which explains the slower speeds.

Yes, in this scenario, individual LAG (LACP) ports need to be removed from the bridge, added to the LAG, and the LAG (LACP bond) needs to be added to the bridge. Then, apply VLANs and other filtering as necessary to the individual bridge ports.
 
almi01
just joined
Topic Author
Posts: 6
Joined: Thu Mar 30, 2023 10:59 am

Re: [SOLVED] Extremely low bandwidth in SFP28 / MLAG setup

Thu Apr 13, 2023 9:19 am

@sirbryan - thank you very much for the clarification - I didn't know this, even after using MikroTik products for over 7 years now - shame on me. Never used more than one bridge so far.

Have a nice day and best regards,


Alex

Who is online

Users browsing this forum: Amazon [Bot], benlg, xristostsilis and 82 guests