I am currently working on setting up a 2 node-storages network with Mellanox 2x 10Gbit cards, connected through one mikrotik switch CRS326-24S+2Q+RM. I have configured bonding in mode 802.3ad with layer 3/4 hashing on each node storage. When I directly connect two node-storage units, everything works correctly with iperf reaching the theoretical 20 Gbits, without any packet retransmission.

However, when I connect the nodes to my switch, I frequently encounter packet retransmissions with a limited throughput. Sometimes, I reach 20 Gbits without any packet loss, and other times, it cannot reach 20 Gbits; iperf displays a lot of packet loss.The issue is intermittent, occurring about half the time during testing. On the switch, for each port to which a node-storage is connected, a bonding is configured, matching the bonding configuration of the node-storages.

Schema : https://pasteboard.co/uNcXSUU5siM9.png

Bonding config of each node-storage : https://pasteboard.co/vtvpEy0nwDXQ.png

Bonding config of the switch for one node storage connected (ports 5/6 in this config) : https://pasteboard.co/4j1n2wg4Wo9H.png

Iperf3 results when connected to the switch:

20 Gbits without retry => https://pastebin.com/raw/sAXGcWMN

15 Gbits with retry => https://pastebin.com/PSVqqMwS

Does someone has an explication of this behavior ?

Thank you for your help