LACP bonding speed problem with Synology NAS

RouterOS General
1

I am trying to speed-up transmision between VMWare and Synology NAS RS3614XS with no success.

My configuration:

Fujitsu-Siemens server with VMWare 5.1

  • dual Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T NIC
  • virtual distributed switch configured with LACP 802.3ad set for both uplink ports of this dual NIC
  • VDS configured as 10Gb with 9000 Jumbo frames
  • Broadcom is not used for management during test - just for transmision

Virtualized Windows7 Prof in FSC used as transmission source

  • VMXNET3 NIC configured as 10Gb with 9000 Jumbo frames

Production VMWare 5.5 connected to port 6 with second virtualized Win7Prof for testing

  • VMXNET3 NIC configured as 10Gb with 9000 Jumbo frames

RB1100AHx2 v6.18 used as switch for bonding in 802.3ad mode - no firewall rules

  • port 2,3 bonded as LACP to RS3614 - interface LACP_RS3614
  • port 4,5 bonded as LACP to VMWare - interface LACP_172_FSC
  • ports 7,8,9,10 beeing slaves of port 6
  • port 6 - bridged with both bondings
  • port 13 used for management - different IP subnet
  • all ports configured as 9100 MTU jumbo frames with 9200 max frame size

Synology RS3614XS with four NIC

  • two NIC bonded in 802.3ad mode
  • bonded link is configured with 9000 jubo frames
  • tird one used for management in different IP subnet

Transmitting data from Win7 inside FSC to Synology saturates LACP bond to 1Gb and data is transmitted only via one ETH link as expected.
Same results are observed during transmission from production VMWare to Synology.

Problem is observed when I start transmission from both sources. Transmission is splitted to both eth bond ports but total speed does not exceeds 1Gb.

Any ideas what could be the cause of such behaviour ?

I know that the problem could not be related to RB1100AHx2 but I have no spare switch with bonding to exclude MicroTik as cause of problem.
The non changeable bridge’s 1598 MTU also does not seem to be problem as it is used for transmision from production VMWare with 1Gb saturation.

/interface bridge
add l2mtu=1598 name=BR172
/interface ethernet
set [ find default-name=ether2 ] l2mtu=9200 mtu=9100 name=ETH2_LACP_RS3614
set [ find default-name=ether3 ] l2mtu=9200 mtu=9100 name=ETH3_LACP_RS3614
set [ find default-name=ether4 ] l2mtu=9200 mtu=9100 name=ETH4_LACP_FSC
set [ find default-name=ether5 ] l2mtu=9200 mtu=9100 name=ETH5_LACP_FSC
set [ find default-name=ether6 ] name=ETH6_172
set [ find default-name=ether13 ] name=ETH13_192
set [ find default-name=ether7 ] master-port=ETH6_172
set [ find default-name=ether8 ] master-port=ETH6_172
set [ find default-name=ether9 ] master-port=ETH6_172
set [ find default-name=ether10 ] master-port=ETH6_172
/interface bonding
add mode=802.3ad mtu=9100 name=LACP_172_FSC slaves=ETH4_LACP_FSC,ETH5_LACP_FSC transmit-hash-policy=layer-2-and-3
add mode=802.3ad mtu=9100 name=LACP_RS3614 slaves=ETH2_LACP_RS3614,ETH3_LACP_RS3614 transmit-hash-policy=layer-2-and-3
/interface bridge port
add bridge=BR172 interface=LACP_RS3614
add bridge=BR172 interface=ETH6_172
add bridge=BR172 interface=LACP_172_FSC
/ip address
add address=172.16.99.249/24 comment=“default configuration” interface=BR172 network=172.16.99.0
add address=192.168.1.249/24 interface=ETH13_192 network=192.168.1.0

bond6.png

Mikrotik and QNAP bounding 2 NIC's
2

Research done. It is MT bonding implementation problem.

I have changed RB1100AHx2 to D-Link DGS-1210-10 switch and … voilà … everything works as expected…two 1Gb streams of data flowing to Synology instead of one agregated.

I am asking MikroTik Team to enhance bonding.

3

Its not a bonding problem. Its the configuration of the 1100ahx2 that will not allow you more traffic if you bond over ports on a single switch chip.
See this diagram: http://i.mt.lv/routerboard/files/Block-RB1100AHx2.pdf

To solve your issues, bond on ether1+ether6 for example, that will give you full bond speed.

4

The question is if AR8327 could do bonding in hardware to not involve CPU in the process ?

5

Anyone knows if the AR8327 of RB1100AHx2 can do 802.3ad (hardware-based) ?

How would I go about creating bonding with vlan on top of it ?
Just asking because I tried so with RB450G, I have been unsuccessful until now but I hope to be able with the RB1100AHx2. Also asking because I see people mentioning bridge and I don’t see why that would be required.

Thank you!

6

Even if it could, this wouldn’t help unless you’re using it as a simple hardware switch at layer2 only, with the other side being another bonded pair of links - because for layer3 (routing), the traffic must go up to the CPU, which only has 1Gbps lane to the Atheros switch chip.

7

That definitely makes sense. Thank you.

Have you implemented bonding on Mikrotik already ?
Just curious to see how I should proceed to configure VLANs over bonding work (still wondering why bridge has been mentioned above).

8

Never personally implemented it on Mikrotik, nope. However, reading the documentation tells me what to expect (in general, anyway)

I suspect that the bonding interface is going to be a cpu-based interface (i.e. won’t work at wire-speed for layer2 forwarding) - so this means that the bonding interface is probably going to be limited to 1Gbps of throughput unless you use one interface from switch1 and another interface from switch2 (as I believe was mentioned earlier in the thread)

To make the router participate in a VLAN on the bundle, you would then apply vlan interfaces to the bonding interface and put IP addresses on the vlan interfaces.

If you’re trying to bridge these, then you can do it a couple of ways:

(Since these are CPU-based interfaces, this precludes the speed advantages of hardware-based switching, so there’s no harm in using bridge interfaces…)

If all members of the bridge should get all VLANs of the bridge as well, then just put the VLAN interfaces on the bridge interface itself and connect the various physical ethernet ports (or bonding interfaces in your case). Creating vlan interfaces is only necessary if you want the router to participate in the VLAN at the IP layer. Any VLAN tags used by devices attached to the other ends of your physical/bonding interfaces can use any VLAN tag they like.

If you want to do untagged interface → tagged interface, then you need to create a vlan interface on the physical/bonding interface that should use tags, and then create a bridge for each VLAN you want to forward. Connect the untagged interface to the appropriate vlan, and the appropriate VLAN interface. If the Router is to participate in IP on this vlan, then put the IP address on the bridge interface. Note that you can only really do one VLAN on an untagged interface - but that makes sense, right? :slight_smile:

Remember, too, that the bonding interface itself might be limited to a maximum throughput per-stream of the full speed of only one physical interface - e.g. if you’re using 802.3ad (LACP) mode, then all packets from host A to host B will be forwarded over the same physical interface, so there’s no way that A->B can exceed one physical link’s speed.

9

Thank you for your very detailed explanation.

I’m waiting for my RB1100Ahx2 and I’ll play with bonding when I get it :slight_smile:

10

Just to add another footnote to the discussion, when we did throughput testing between the CCR1072 and our ESXi VMWARE hosts, we found the LACP implementation that VM Ware uses does not hash well at all. We had multiple discrete IP Source/Destination TCP flows but they all ended up on a single link.

I’ve also seen the same issue with ESXi LACP connected to Cisco Nexus 7010 switches…the hash algorithms for load balancing on the VM Ware side don’t seem to work well as of ESXi 6.x

There are some notes in the blog post we did on LACP between MikroTIk and VM Ware

http://www.stubarea51.net/2015/10/09/mikrotik-ccr1072-1g-8s-review-part-3-80-gbps-throughput-testing/

11

I had a 3 port LACP between the NAS and a Mikrotik router.
Since some update a while ago, I can’t remember just when, the transfer speed went down.
Just now, I dropped the LACP, and replaced it with a plain 1Gb connection, and all is fine.
Since I don’t have many other connection to the nas, I’m really not in need of a LACP bonding - it was just to see how it works. And it don’t. At least not anymore.