Community discussions

MikroTik App
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 416
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

LAG 802.3AD slowness

Sat Apr 20, 2019 10:01 am

So recently my cable modem ( Arris SB8200 ) gained LAG..

I created a bonding interface and set it for 802.3AD with 2 ports.. Im using a CCR-1009..

Everythign works. I see the data spread out across both boths..

BUT... My speeds went way down.. I have gigabit service and I speedtest.net at 950Mbps down.. With both links ( wires ) hooked up I only get 250Mbps... If I unhook one wire I get 950Mbps again.. So with one link I get 950Mbps with both links I get 250...

This seems like a really odd result. A LAG 802-3AD interface on a CCR-1009 can do 950Mpbs to one computer right ? Its not limited in some way like no fast track if I use a bonded interface right ?

I should be able to get 950Mbps on a speed test thru a LAG 802.3AD right ?
Last edited by Xymox on Sat Apr 20, 2019 10:55 pm, edited 1 time in total.
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 416
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: LAG 802.3AD slowness

Sat Apr 20, 2019 11:35 am

Maybe its a hardware limitation ? Maybe I need a CCR 1036 vs the CCR1009 ?
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 416
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: LAG 802.3AD slowness

Sat Apr 20, 2019 9:54 pm

Well I turned off all my firewall rules and the problem persists.

I should be able to get 980Mbps thru a 802.3AD bonded set of 2 x 1Gbps ports on a 1009 or 1036 right ?

Bonding for 802.3AD is done in the switch chip ? Its layer 2 ?
 
User avatar
jprietove
Trainer
Trainer
Posts: 212
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: LAG 802.3AD slowness

Sat Apr 20, 2019 10:12 pm

I'm afraid that bonding is hardware offload only in crs3xx series.
In CCR series bonding is done on software. But I've reached 2gbps with bonding on ccr1009 in a transit router. I mean, different IP addresses for source and destination.
So there should be a problem in your setup.

Enviado desde mi Mi A2 mediante Tapatalk

 
User avatar
Xymox
Member
Member
Topic Author
Posts: 416
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: LAG 802.3AD slowness

Sat Apr 20, 2019 10:24 pm

Hash settings make no difference..

So my config is that im using Ether1 and Ether2, in a bonded 803.2AD group. I have played with all the hashes and for the speedtest.net that I am using for testing ( or iperf public servers ) the layer 3/4 seems to have the best results for hashing. I test out the same speed using any test method, 275-280Mbps and my ether1 and ether2 interfaces show 140/140 Mbps..

This might be a cable modem issue. I just need confirmation I am not doing anything wrong and I should be able to get 980Mbps thru a 2 port 802.3AD group using the above config ?
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 416
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: LAG 802.3AD slowness

Sat Apr 20, 2019 10:28 pm

I'm afraid that bonding is hardware offload only in crs3xx series.
In CCR series bonding is done on software. But I've reached 2gbps with bonding on ccr1009 in a transit router. I mean, different IP addresses for source and destination.
So there should be a problem in your setup.

Enviado desde mi Mi A2 mediante Tapatalk
Thank you.

So I should be able to do 980Mbps with one IP set for a test with the 1009 ? 2Gbps with 2 IP tests ? This would be what i expect.
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 416
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: LAG 802.3AD slowness

Sat Apr 20, 2019 10:45 pm

This is from a Speedtest.net test.. You can see its spreading out the load for download and upload nicely across both ports. But they only reach 150/150.

My CPU only shows a 1-2% increase during these tests.

If I unplug either cable the speed test will reach 980Mbps.
LAG.gif
You do not have the required permissions to view the files attached to this post.
 
User avatar
jprietove
Trainer
Trainer
Posts: 212
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: LAG 802.3AD slowness

Sat Apr 20, 2019 11:28 pm

Try using ether 5 and 6, for example. ether 1 to 4 are connected to switch chip and this chip is connected to cpu by 1gbps line.
This may be a problem in performance.
Check block diagram here: https://i.mt.lv/cdn/rb_files/CCR1009-8G ... 140835.png

Enviado desde mi Mi A2 mediante Tapatalk

 
User avatar
Xymox
Member
Member
Topic Author
Posts: 416
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: LAG 802.3AD slowness

Sat Apr 20, 2019 11:57 pm

OoooOo... Interesting... Thank you... I will go reconfigure and try this..

I think it might also be the modem. They just updated firmware and added LAG in the last 2 weeks. So its a brand new feature for them.
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 416
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: LAG 802.3AD slowness

Sun Apr 21, 2019 12:24 am

THAT WAS IT !!!

Problem solved.. I can now pull 996Mbps from the modem.

THANK YOU... Great call.. Don't use ports 1-4 for 802.3AD bonding..
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 416
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: LAG 802.3AD slowness

Sun Apr 21, 2019 2:50 am

Image
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 416
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: LAG 802.3AD slowness

Mon Apr 22, 2019 1:26 am

So to wrap this up, the only way LAG/LACP 802.3AD works at high speeds speeds is to use ports that do not go thru switch chips to CPU via 1Gb internal link ? Thats correct right ? So Mikrotik might want to make that more clear in the specs ?

So for example..

https://mikrotik.com/product/CCR1009-7G-1C-1SplusPC = up to 7 ports in a LAG group for 7 Gbits ?

Image

Im not sure what speed I could reach with this tho. https://mikrotik.com/product/rb4011igs_rm

Image

But this has no way to do high speed LAG ? Or does it with 2 CPUs ?

Image

And no way this will.. https://mikrotik.com/product/RB2011UiAS-RM

Image

There is also the matter of how fast you can go thru the combined ports. With 10 LAG members thru a CPU can you get 10Gbps ? What pps ? Does this vary with the number of cores ?

SO.... I think Mikrotik should add a line to specs to show what can be done with LAG as its not easy to figure out ahead of time without testing it.
 
User avatar
jprietove
Trainer
Trainer
Posts: 212
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: LAG 802.3AD slowness

Mon Apr 22, 2019 11:28 am

Acording to https://wiki.mikrotik.com/wiki/Manual:I ... Offloading only CRS3xxx series supports bonding with hardware offload. It means that other devices do bonding with CPU.
CCR1009-7G-1C-1S+PC has 9 cores so it can take 1 Gbps from each G port and bond them in total of 7Gbps in SFP+ port
RB4011iGS+RM can bond two ports from ether1..ether5 and another two from ether6..ether10 and bond them in total of 5 Gbps that can be transmitted/received over the SFP+ port.
RB3011UiAS can agreggate up to 2 Gbps but it will depend a lot on CPU's usage and port selection. I wouldn't expect too much. This router is not intended to bonding.
RB2011 can bond 5 Ether ports of 100 Mbit and join the traffic in 1 Gbps port or split 1 Gbps in 5 ports of 100 Mbps.
If you want bonding, as it is a layer 2 work, better choose a good switch that can do it by hardware. CRS3xx works very good. And you can configure it with SwOS or RouterOS.
I don't agree with you in MikroTik should explain what to expect with LAG... but it will be very CPU-dependent.
 
User avatar
sri2007
Member Candidate
Member Candidate
Posts: 205
Joined: Wed May 20, 2015 10:14 pm
Location: Lake Grove, NY

Re: LAG 802.3AD slowness

Tue Apr 23, 2019 12:43 am

Wow, that's interesting, but I've done those bondings using the balanced-xor mode instead of 802.3ad using the first 4 ports of the CCR1009, that config works great it goes up to 3.5Gbps (real traffic) being forwarded between two Mikrotiks (CCR1009 + CRS326) -> that's because the lack of 10gig ports at the ccr, and the excessive amount of ports at the CRS317, my solution is to bond these two mikrotiks and I have three 10gig interfaces with a 4gig bonding in the middle.
 
User avatar
jprietove
Trainer
Trainer
Posts: 212
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: LAG 802.3AD slowness

Tue Apr 23, 2019 1:09 am

It depends on the ccr1009 model. Not all shares the same block diagram

Enviado desde mi Mi A2 mediante Tapatalk

 
User avatar
Xymox
Member
Member
Topic Author
Posts: 416
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: LAG 802.3AD slowness

Tue Apr 23, 2019 9:39 am

Hmm... Yes I guess i agree.. Mikrotik should not work on publishing bonding results as that would get really complex.

The reason im bringing it up is that all new Cable modems for DOCSIS 3.1 are going to come with 2 ports and 802.3ad. Cable companies are going to 2Gbps and to 10Gbps "multigig".. A ton of non technical users are soon to realize normal consumer routers don't have the horsepower or LAG WAN capability. So Mikrotik is sure to see more sales to these power users that do not have the technical skills to figure out which models can do it.

So I was just thinking out loud that it might be good to simplify the ability to choose DOCSIS 3.1 Multigig capable LAG routers.
 
User avatar
sri2007
Member Candidate
Member Candidate
Posts: 205
Joined: Wed May 20, 2015 10:14 pm
Location: Lake Grove, NY

Re: LAG 802.3AD slowness

Tue Apr 23, 2019 7:55 pm

It depends on the ccr1009 model. Not all shares the same block diagram

Enviado desde mi Mi A2 mediante Tapatalk
Got you! thanks!! I've tried that with the CCR1009-7G-1C-1S+ using the 6.42.12 long-term version.
 
Adundee89
just joined
Posts: 5
Joined: Wed Jul 24, 2019 3:50 am

Re: LAG 802.3AD slowness

Wed Nov 20, 2019 10:39 pm

Hello, I hope I'm not drudging up or hijacking an old thread, but it seems as though a few months old isn't too far back. However, I'm curious, how can 2.5Gbps be achieved on a 1Gbps chip? I currently have a 4011 with Ether1 and 2 bonded and am seeing a clear bottleneck. My graphs show a plateau at 1Gbps. This seems like it could be the issue, but not if 2.5 Gbps is possible?


EDIT:

Looking at the specs for the block diagram, the 8367 chip supports RGMII, which is 1 Gbps, whereas the 8366 and 8369 support RSGMII which is 2.5Gbps. Could this be an issue?

Also, IF my problem is in fact the chip can only do 1Gbps, can I lag ports from each chip? For example, can I lag ether1 with ether6? Thank you for your help!
Last edited by Adundee89 on Wed Nov 20, 2019 11:26 pm, edited 1 time in total.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11383
Joined: Thu Mar 03, 2016 10:23 pm

Re: LAG 802.3AD slowness

Wed Nov 20, 2019 11:03 pm

Depending on bond mode it may not be possible for single client connecting to single server to use both physical links. Meaning that many times, single link speed will be maximum achievable speed. Bond will see full use when it's used for connections from different clients to different servers, in that case traffic will spread (almost) evenly.
 
User avatar
xvo
Forum Guru
Forum Guru
Posts: 1237
Joined: Sat Mar 03, 2018 1:12 am
Location: Moscow, Russia

Re: LAG 802.3AD slowness

Wed Nov 20, 2019 11:21 pm

Hello, I hope I'm not drudging up or hijacking an old thread, but it seems as though a few months old isn't too far back. However, I'm curious, how can 2.5Gbps be achieved on a 1Gbps chip? I currently have a 4011 with Ether1 and 2 bonded and am seeing a clear bottleneck. My graphs show a plateau at 1Gbps. This seems like it could be the issue, but not if 2.5 Gbps is possible?
How are you testing it?
 
Adundee89
just joined
Posts: 5
Joined: Wed Jul 24, 2019 3:50 am

Re: LAG 802.3AD slowness

Wed Nov 20, 2019 11:29 pm

Sorry, I didn't realize you guys were going to be so quick! This is a production router that we expected to be capable of near 2Gbps. The router is lagged with a NTNX switch and all the traffic is being then pushed out of the 10G SFP+ port. Traffic is being generated from several access points and other back hauls to more towers...

EDIT: Mode is 802.3ad

Looking at the specs for the block diagram, the 8367 chip supports RGMII, which is 1 Gbps, whereas the 8366 and 8369 support RSGMII which is 2.5Gbps. Could this be an issue?

Also, IF my problem is in fact the chip can only do 1Gbps, can I lag ports from each chip? For example, can I lag ether1 with ether6? Thank you for your help!
 
User avatar
xvo
Forum Guru
Forum Guru
Posts: 1237
Joined: Sat Mar 03, 2018 1:12 am
Location: Moscow, Russia

Re: LAG 802.3AD slowness

Thu Nov 21, 2019 12:29 am

Looking at the specs for the block diagram, the 8367 chip supports RGMII, which is 1 Gbps, whereas the 8366 and 8369 support RSGMII which is 2.5Gbps. Could this be an issue?
Looking at the Realtek website, I see that there are different RTL8367 models, and some of them have 2,5Gbps interface.
So I don't think there is a point not to trust the block diagram.
 
szalkerous
newbie
Posts: 35
Joined: Thu Jan 21, 2016 2:30 am
Location: NH, USA

Re: LAG 802.3AD slowness

Tue Apr 26, 2022 8:40 pm

Just wanted to confirm again:

My SB8200 will do LAG to my CCR1009-8G-1S-1S+ (the older model) only on the last 4 ports. I had the exact same experience trying to use eth 1 + 2 as the OP. Thank fully this thread existed to explain away my disappointment. Oddly I think my eth7 is dead/damaged as it would link up/down over and over again. Swapping to 6 and 8 has no issues.

However there exists no guide from Arris or Comcast for advanced 802.3ad configuration on routers like ours, so I wanted to share how mine is set up. Not sure if best (open to all advice) but this so far is working for me with Comcast Xfinity....

Under the bonding interface you've created in the Interfaces window...

General:
MTU - 1500
Arp - Enabled

Bonding:
Mode - 802.3ad
Link Monitoring - mii
Transmit Hash Policy - Layer 3 and 4 (I chose this based on this thread...not sure if it made a difference)
LACP rate - 30s
MII interval - 100 ms
 
szalkerous
newbie
Posts: 35
Joined: Thu Jan 21, 2016 2:30 am
Location: NH, USA

Re: LAG 802.3AD slowness

Tue Apr 26, 2022 8:44 pm

Acording to https://wiki.mikrotik.com/wiki/Manual:I ... Offloading only CRS3xxx series supports bonding with hardware offload. It means that other devices do bonding with CPU.
If you want bonding, as it is a layer 2 work, better choose a good switch that can do it by hardware. CRS3xx works very good.
Is there any updates on newer CCR router models like the CCR2004-1G-12S+2XS using the ARM-based Annapurna Labs AL32400 (4-core 1700Mhz) that support hardware bonding?

Who is online

Users browsing this forum: Bing [Bot] and 99 guests