7.1rc6 + mlag + l3hw periodic loss of connectivity

mfedotov · Thu Nov 11, 2021 2:38 am

I am testing mlag with l3hw offloading. I have CRS309 and CRS305 configured as MLAG with a single peer link. The CRS309 is configured as a router. I have two clients with 2 NICs in each connected to the 802.3ad mlag port channel. Each client is in a different VLAN and configured to use the router as the default gateway. It works, but when testing the bandwidth I can see that traffic stalls periodically for about 6 seconds (see lines from seconds 13 in the output from iperf3). It seems to only happen when the peer link is used to transport the traffic (i.e the transmit hash policy causes the client to send the traffic to CRS305, which is not the router, so it sends the packets to the router over the peer link). If I swap the cables on the client, which causes the traffic to go directly to the router, the issue disappears.
Any ideas of what is going on? I tried to limit the bandwidth of iperf3, found that with 3Gb/s it is quite stable with no traffic loss, but 5Gb/s and higher seems to always cause a traffic stall. The stall seems to always last for about 6 seconds.

root@pve5:~# iperf3 -c 10.255.1.11  -t 40
Connecting to host 10.255.1.11, port 5201
[  5] local 10.255.201.9 port 34596 connected to 10.255.1.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.15 GBytes  9.92 Gbits/sec    0   1.30 MBytes
[  5]   1.00-2.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.44 MBytes
[  5]   2.00-3.00   sec   970 MBytes  8.14 Gbits/sec   16   1.54 MBytes
[  5]   3.00-4.00   sec   615 MBytes  5.16 Gbits/sec  179   1.33 MBytes
[  5]   4.00-5.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.57 MBytes
[  5]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.57 MBytes
[  5]   6.00-7.00   sec   595 MBytes  4.99 Gbits/sec    1   8.74 KBytes
[  5]   7.00-8.00   sec   992 MBytes  8.33 Gbits/sec  179   1.37 MBytes
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.51 MBytes
[  5]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec    0   1.53 MBytes
[  5]  10.00-11.00  sec   396 MBytes  3.32 Gbits/sec  180    245 KBytes
[  5]  11.00-12.00  sec  1.15 GBytes  9.89 Gbits/sec    0   1.43 MBytes
[  5]  12.00-13.00  sec  33.8 MBytes   283 Mbits/sec    2   8.74 KBytes
[  5]  13.00-14.00  sec  0.00 Bytes  0.00 bits/sec    1   8.74 KBytes
[  5]  14.00-15.00  sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes
[  5]  15.00-16.00  sec  0.00 Bytes  0.00 bits/sec    1   8.74 KBytes
[  5]  16.00-17.00  sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes
[  5]  17.00-18.00  sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes
[  5]  18.00-19.00  sec   455 MBytes  3.82 Gbits/sec  167   1.34 MBytes
[  5]  19.00-20.00  sec  1.15 GBytes  9.89 Gbits/sec    0   1.56 MBytes
[  5]  20.00-21.00  sec  1.15 GBytes  9.89 Gbits/sec    0   1.56 MBytes
[  5]  21.00-22.00  sec   500 MBytes  4.19 Gbits/sec    2   8.74 KBytes
[  5]  22.00-23.00  sec  1.07 GBytes  9.19 Gbits/sec  208   1.37 MBytes
[  5]  23.00-24.00  sec  1.15 GBytes  9.89 Gbits/sec    0   1.37 MBytes
[  5]  24.00-25.00  sec   240 MBytes  2.01 Gbits/sec    2   8.74 KBytes
[  5]  25.00-26.00  sec  0.00 Bytes  0.00 bits/sec    1   8.74 KBytes
[  5]  26.00-27.00  sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes
[  5]  27.00-28.00  sec  0.00 Bytes  0.00 bits/sec    1   8.74 KBytes
[  5]  28.00-29.00  sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes
[  5]  29.00-30.00  sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes
[  5]  30.00-31.00  sec   115 MBytes   966 Mbits/sec  160   1.25 MBytes
[  5]  31.00-32.00  sec  1.15 GBytes  9.89 Gbits/sec    0   1.33 MBytes
[  5]  32.00-33.00  sec  1.15 GBytes  9.89 Gbits/sec    0   1.55 MBytes
[  5]  33.00-34.00  sec  1.15 GBytes  9.89 Gbits/sec    0   1.55 MBytes
[  5]  34.00-35.00  sec   422 MBytes  3.54 Gbits/sec  180   1.37 MBytes
[  5]  35.00-36.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.46 MBytes
[  5]  36.00-37.00  sec  1.15 GBytes  9.88 Gbits/sec    0   1.54 MBytes
[  5]  37.00-38.00  sec   879 MBytes  7.37 Gbits/sec    1   8.74 KBytes
[  5]  38.00-39.00  sec   719 MBytes  6.03 Gbits/sec  179   1.26 MBytes
[  5]  39.00-40.00  sec  1.15 GBytes  9.89 Gbits/sec    0   1.33 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-40.00  sec  26.3 GBytes  5.64 Gbits/sec  1460             sender
[  5]   0.00-40.00  sec  26.3 GBytes  5.64 Gbits/sec                  receiver

iperf Done.

Thu Nov 11, 2021 10:49 am

Hey there,

Unfortunately, the compatibility layer between MLAG and L3HW is not implemented yet and postponed after RouterOS 7.1 stable release (due to feature freeze). Until then, it is not recommended to use both features together.

7.1rc6 + mlag + l3hw periodic loss of connectivity

7.1rc6 + mlag + l3hw periodic loss of connectivity

Re: 7.1rc6 + mlag + l3hw periodic loss of connectivity

Who is online