Community discussions

MikroTik App
 
dazzaling69
Member Candidate
Member Candidate
Topic Author
Posts: 104
Joined: Wed Feb 22, 2017 12:01 pm

Link Aggregation Only Speeds up in One Direction

Fri Mar 24, 2023 7:40 pm

I have a bonding interface set up between my Synology NAS (which has 2 NICs) and my Mikrotik RB5009 router, which then connects to my PC via 2.5Gbe. Using a speed test tool on the router I get 2Gbps download to the PC but only 1Gbps upload (confirmed also by monitoring the Interface in Winbox. The router and NAS use LACP to configure the link and both NICs are active in the bonded interface.

Clearly the data can be 2Gbps, but why not symmetrically?

It could be something to do with the speedtest software, NAS, or other. However, could this be something to do with either the router (setup) or a fundamental protocol limit?

This is independent of SMB multichannel, which doesn't appear to work when I activate that on the NAS.
 
wiseroute
Member
Member
Posts: 352
Joined: Sun Feb 05, 2023 11:06 am

Re: Link Aggregation Only Speeds up in One Direction

Fri Mar 24, 2023 8:48 pm

hello.
Using a speed test tool on the router I get 2Gbps download to the PC but only 1Gbps upload (confirmed also by monitoring the Interface in Winbox. The router and NAS use LACP to configure the link and both NICs are active in the bonded interface
ok. what method and which devices did you use in performing the speed test? was it b2b router test? or, between 2 end points?

b2b network device or intermediary test could be considered not the real result as you have to know your 2 end points capabilities ie. write speed etc.

from intermediary devices point of view, as long as the lacp bonding can deliver the packet on those ports correctly, they are done. the rest is up to tuning.

just a thought 💭
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Link Aggregation Only Speeds up in One Direction

Fri Mar 24, 2023 10:06 pm

Link aggregation is not a simple "times N" business. There are many different types (or modes) which behave differently. And many types can be configured differently. And often both link partners (e.g. switch and NAS) can use different types but aggregated link still works.

And most importantly: almost all settings affect only transmission of packets. One link partner can not affect the way the other end uses links for their transmission.

For example: there's "balance-rr" which spreads packets evenly over link members. This includes packets of single connection (e.g. TCP iperf connection) .. with potential to screw connection if packets via one of links get processed faster than other packets and get delivered out-of-order (which is something most TCP stacks really hate). ROS supports it, but its usage is discouraged because it's always bound to CPU (and thus slower than HW-offloaded on most devices). "Normal" linux-based gadgets, which perform any link aggregation in software, don't discourage it's use ...
And then there's 802.3ad (the most standardized one) which has configurable transmit policy. This link aggregation type uses single link for transmitting packets of some connection and thus doesn't mess with in-order-delivery. The method of (quickly) determining which packets might belong to same connections is "transmit hash policy" and packets with same hash will pass same link member. Different Tx policies are configured via property transmit-hash-policy and can spread traffic differently between link members. Setting it to layer-3-and-4 means that connections betwee same pair of IP addresses but using different TCP port numbers (e.g. iperf with multiple parallel streams) have potential to spread over all link members while setting policy to layer-2-and-3 will use same link member for all traffic between same pair of hosts. Devices which can perform 802.3ad link aggregation in hardware may not support tx hash policies which involve higher OSI layers (layer3 or layer4) ... but as already mentioned, both link partners can use different tx hash policies without a problem.
 
dazzaling69
Member Candidate
Member Candidate
Topic Author
Posts: 104
Joined: Wed Feb 22, 2017 12:01 pm

Re: Link Aggregation Only Speeds up in One Direction

Sun Mar 26, 2023 8:38 pm

Thanks for the replies. The Synology NAS offers Balance-TCP (IEEE 802.3ad LACP) as its config mode. I match that on the ROS. I tried the different hash policies and none had any effect.

The two connection methods I am trying to assess speed are different,

#1 Use LibreSpeed Docker on the NAS. I get 2Gb/s down and 1Gb/s up
#2 SMB with Multichannel enabled. Trying various combinations of downloads from same/different files on same/different array. In the case I always get a maximum of 112GB/s - not a bit more.

I don't see any variable to set other transmit hash policy, and neither of the 3 options may a difference.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Link Aggregation Only Speeds up in One Direction

Sun Mar 26, 2023 9:22 pm

What I was trying to explain is that link aggregation only rarely speeds up connection between one pair of peers. It works fine if LA connection is a trunk carrying traffic between many pairs of peers. And your description of how things are set up doesn't give me enough clues to explain why connection is faster in one direction and not in the other.
 
dazzaling69
Member Candidate
Member Candidate
Topic Author
Posts: 104
Joined: Wed Feb 22, 2017 12:01 pm

Re: Link Aggregation Only Speeds up in One Direction

Mon Mar 27, 2023 1:15 am

I don’t know what might be relevant in troubleshooting the speed issues. What information might help?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Link Aggregation Only Speeds up in One Direction

Mon Mar 27, 2023 6:25 pm

Relevant for troubleshooting this problem (which IMO it's very probably not an issue but rather wrong expectation) would be to know exactly which bonding mode and exact properties of it are used on Synology. And which exactly settings are used on Mikrotik.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Link Aggregation Only Speeds up in One Direction

Mon Mar 27, 2023 6:28 pm

A nise summary about what Ling Aggregation 802.3ad is: https://www.ieee802.org/3/hssg/public/a ... 1_0407.pdf
 
wiseroute
Member
Member
Posts: 352
Joined: Sun Feb 05, 2023 11:06 am

Re: Link Aggregation Only Speeds up in One Direction

Mon Mar 27, 2023 7:15 pm

@mkx
which IMO it's very probably not an issue but rather wrong expectation
can't be disagree 👍🏻
 
dazzaling69
Member Candidate
Member Candidate
Topic Author
Posts: 104
Joined: Wed Feb 22, 2017 12:01 pm

Re: Link Aggregation Only Speeds up in One Direction

Mon Mar 27, 2023 8:16 pm

I have looked around at the expectation from LAG in these cases and it isn't clear to me (and I'm not an expert), so you may be right I'm expecting the wrong outcome.

Clearly I can get something like N times the performance in some circumstances - speedtest download is one example - but it doesn't make sense to me why that isn't bidirectional. Perhaps I should expect this asymmetric behaviour, but I don't (yet) see why, and none of the documentation I have read suggests this is an expected or necessary outcome.

My ultimate use case is SMB multichannel which, if I read the documentation on it, should work with LAG to provide greater bandwidth - by using multiple channels. Synology NAS users with a similar setup to me are getting around 236MB/sec, whereas I get 112MB/sec in SMB contiguous transfers. It would seem to be possible.

So I might be expecting the impossible, but the asymmetry and the experience of other users suggests that something might be awry in my setup. The Mikrotik router is only one aspect, but it's the biggest difference compared with others' setups where they get faster throughput using LAG.

The challenge is troubleshooting it...
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Link Aggregation Only Speeds up in One Direction

Mon Mar 27, 2023 8:22 pm

You still didn't show exact configuration of both devices in question. Until you do, we can keep talking about weather ...
 
dazzaling69
Member Candidate
Member Candidate
Topic Author
Posts: 104
Joined: Wed Feb 22, 2017 12:01 pm

Re: Link Aggregation Only Speeds up in One Direction

Wed Mar 29, 2023 11:13 pm

Here is the relevant info, as far as I can find. There are only a few things that can be configured in the Synology NAS. I've attached the MT config file.
Network.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: Link Aggregation Only Speeds up in One Direction

Thu Mar 30, 2023 4:57 am

You still didn't show exact configuration of both devices in question. Until you do, we can keep talking about weather ...

your patience is admirable
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: Link Aggregation Only Speeds up in One Direction

Thu Mar 30, 2023 4:59 am

Here is the relevant info, as far as I can find. There are only a few things that can be configured in the Synology NAS. I've attached the MT config file.
@dazzaling69

you must pay attention at mkx explanations, he has fully explained why you cannot achieve more speed
 
dazzaling69
Member Candidate
Member Candidate
Topic Author
Posts: 104
Joined: Wed Feb 22, 2017 12:01 pm

Re: Link Aggregation Only Speeds up in One Direction

Thu Mar 30, 2023 12:32 pm

mkx was good enough to ask for more information, which I've provided.

I am asking is because others have achieved SMB multichannel transfers that are twice as fast using the exact same NAS, LAG and PC network card as I have. As I understand it, and from Microsoft's documentation on it https://learn.microsoft.com/en-us/azure ... erformance, SMB multichannel should allow LAG-enabled connections to speed up transfers between 2 peers - not just multiple peers/connections - over up to 4 channels.

I can get that not all traffic has an N-times speedup, but I don't get why the different traffic is at different speeds in different directions, or why others achieve better results in similar circumstances. This is complex stuff where all the pieces of the puzzle play a part - the Mikrotik router is the biggest point of difference in these setups.

Interestingly, if I use the get info command from the MS documentation, SMB multichannel is enabled, but the link connection speed is reported as 1Gb/sec and that maxchannels=1. I wonder how it determines that? Perhaps it's the Windows side that's the problem, or perhaps the server is not reporting the config properly, or something else. Troubleshooting help is what I'm after.

PS C:\WINDOWS\system32> Get-SmbMultichannelConnection |fl


ServerName : Randy
Selected : True
Failed : False
FailureCount : 0
ClientInterfaceIndex : 7
ClientRSSCapable : False
ClientRdmaCapable : False
ClientLinkSpeed : 1 Gbps
ClientIpAddress : 10.160.100.87
ServerInterfaceIndex : 6
ServerRSSCapable : False
ServerRdmaCapable : False
ServerLinkSpeed : 1 Gbps
ServerIpAddress : 10.160.100.30
MaxChannels : 1
CurrentChannels : 1
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Link Aggregation Only Speeds up in One Direction

Thu Mar 30, 2023 10:20 pm

Quick googling of what exactly bond type "Balance-TCP" does and how it works did not give me any informative results. So for me it's still a black box. FWIW Synology may have created a Frankenstein supporting IEEE 802.3ad LACP (signaling protocol) and use balance-rr as algorithm to spread traffic between bond links. Or (continuing with speculations) it may be actual 802.3ad with transmit-hash-policy=layer3-and-layer4 ... but having SMB server which selects source TCP ports for multichannel SMB so that resulting hashes will spread traffic over bond links evenly (which is only possible if bond is terminated on device running also SMB server which has information about actual state of bond ... a "simple" switch can't do that).

I can only comment on well known algorithms and Mikrotik does stick to them. Those have limits that I already explained and according to it, it's not possible to consistently double the SMB throughput in direction from Mikrotik towards NAS for single TCP client (e.g. SMB) as long as bond type is not "balance-rr" (which is not recommended to be used due to number of good reasons).

And I already explained: each bond partner distributes Tx traffic according to own rules and algorithms which means that symmetric operation is not a norm (but can happen).
 
dazzaling69
Member Candidate
Member Candidate
Topic Author
Posts: 104
Joined: Wed Feb 22, 2017 12:01 pm

Re: Link Aggregation Only Speeds up in One Direction

Thu Mar 30, 2023 11:25 pm

Thanks for helping me with this complicated topic.

Who is online

Users browsing this forum: adimihaix and 79 guests