Community discussions

MikroTik App
 
tmiklas
just joined
Topic Author
Posts: 9
Joined: Mon Apr 02, 2007 12:06 pm

802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Mon May 03, 2021 12:59 pm

Hi all

I want to share something I finally tracked down to the source (or so I believe) yesterday evening.
Thanks to Kev for bringing this performance issue to my attention. Kev noticed it on his hAP AC3, I noticed/replicated the same on my hAP ac and started digging...

I run hAP ac with 350/35 line from ISP, there is no fancy configs on Mikrotik, really basic NAT setup, ethernets and wifi linked on LAN bridge. I tend to run LTS branch these days.

When I run on gigabit ethernet the speedtest.net flatlines immediately at 350-380 download as expected. Problems begin when I go onto wifi - I noticed it depends on which RouterOS is running on the device. I have tested it multiple times, over and over... taking several days to test each variant in different network conditions (time of day) and different speedtest.net servers (also tested against fast.com which is Netflix AFAIR).

Anyway, here are common things that are fairly constant across all the tests:
- radio interface is set up as AP Bridge, 5GHz-only-AC, channel auto, width 20/40/80MHz XXXX, UK, regulatory domain
- my Mac shows that radio link speed is floating between 1170Mbps and 1300Mbps

Here is where things get strange - depending on version of RouterOS (all below are LTS):
- RouterOS 6.44.6 - wired speed 350+, wireless speed 350+ Mbps
- RouterOS 6.45.9 - wired speed 350+, wireless speed 350+ Mbps
- RouterOS 6.46.8 - wired speed 350+, wireless speed 50-80 bursting to 90-100 Mbps for short durations of time
- RouterOS 6.47.9 - wired speed 350+, wireless speed 35-50 bursting to 70-80 Mbps for short durations of time

I was upgrading and downgrading ROS versions and monitoring what changes in behaviour, without touching config on the device and finally settled on 6.45.9 as the last LTS release working at full speed.

Another observation:
I run DNS cache/resolver/forwarder on Mikrotik, handing off queries to Pi-Hole running on home server, so mikrotik asks pi-hole, all machines ask mikrotik... Since update to 6.46.8 I started getting odd DNS timeouts and my network monitoring scripts light up the dashboard like a Christmas tree several times a day, with ROS 6.46.8 or above it happens every 10-15min, sometimes on and off for hours... but again, since going back to ROS 6.45.9 problem disappeared completely.

Now, I have no clue what may be the reason for this, I hope some really smart people here will be able to work this out. For now I run 6.45.9 LTS and I'm happy with the outcome, but my OCD kicks in and I need to understand what caused it. May it be a bug in ROS, maybe some driver change, maybe something else? Fingers crossed someone will know the answer.

Cheer
Tom
You do not have the required permissions to view the files attached to this post.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 1498
Joined: Mon Apr 08, 2019 1:16 am

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Mon May 03, 2021 1:24 pm

Difficult one. Just some hints

(0. My favorite ROS version is 6.45.6 , yes stable, but really stable)
1. With channel on "auto" and "XXXX" sideband you leave quite some variation in your setup that is not related to the performance of a certain release, but dependent on the channel and sideband selected. Either verify what freq are chosen, or set it fixed to a channel and to a specific sideband like Ceee.
2. You looked at the Mac wireless interface status, there is more information in the MT AP "Registration" table, detailed view or print in CLI. Interface rate is expressed in Mbps, not MCS encoding, but that conversion is easy to do (mcsindex.com). AFAIK hAP ac3 has only a NSS of 2 chains.
3. I get higher speeds between 2 MT devices, and via Btest on client. "Speedtest.net" includes quite some more technology in the test path. What is to blame for the performance hit? Even if it boils down to ROS, then what part of it ?
 
tmiklas
just joined
Topic Author
Posts: 9
Joined: Mon Apr 02, 2007 12:06 pm

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Mon May 03, 2021 1:36 pm

Thanks for the response!

Yes, I am fully aware of that and good comments... but even with sideband fluctuations, how otherwise to explain such massive variance in actual bandwidth?

When radio link shows sync over 1gbps (screenshot taken just now), fully saturated ISP line will be 35% of it... yet with certain versions of ROS I am around 5% struggling to reach 10%, so this is massive impact here :-(

When I have a bit more time I will actually set the sideband to fixed and re-run tests across ROS versions. I doubt this will make any difference to be honest, just looking at relative numbers above.

Tom
You do not have the required permissions to view the files attached to this post.
Tom
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 1498
Joined: Mon Apr 08, 2019 1:16 am

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Mon May 03, 2021 3:27 pm

So it seems not to be the wifi connection. Difficult but very interesting case. Looking forward for more information on your experiments. Maybe look into more information (like CPU load, Profile, Sniffer or Wireshark dump to see the packet timing (what side is delayed?) ....).
 
anuser
Long time Member
Long time Member
Posts: 550
Joined: Sat Nov 29, 2014 7:27 pm

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Tue May 04, 2021 11:42 pm

Can you please post your configuration while running 6.47.9? (/export compact)
Do you see any difference between configuration of 6.45.9 and 6.47.9 at all if you compare full /export?
 
tmiklas
just joined
Topic Author
Posts: 9
Joined: Mon Apr 02, 2007 12:06 pm

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Wed May 05, 2021 10:17 am

Can you please post your configuration while running 6.47.9? (/export compact)
Do you see any difference between configuration of 6.45.9 and 6.47.9 at all if you compare full /export?
I'm sorry, but I really don't have that much time for digging to find root cause - I'd like to, but... no time :-(
The only thing I do when testing is upgrade ROS or downgrade - no other changes at all. I have stable config running for years that I do not touch at all (unless ROS edits them up and down with update/downgrade and doesn't tell me). As such, my post is a statement of a fact - something observed and tested multiple times.

BTW I just found one of my other devices (RB951Ui-2HnD) also being affected in the same way. Device worked for 2 years spotless, at some point I upgraded ROS to 6.47.9 and issues started but I did not connect them to the ROS change, blamed it on radio congestion in the area. Downgraded firmware yesterday - problem disappeared, performance back to one from before upgrade. Again, no config changes, only ROS version downgrade.
Tom
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 1498
Joined: Mon Apr 08, 2019 1:16 am

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Wed May 05, 2021 12:12 pm

I would like to know if this is a general performance problem. I do see issues with 6.47.9 @home (my test bed), but I'm not sure if it is related to MT. The issues are non-specific and different for all devices, (like a relative old laptop with only a 802.11n interface , that sees the wAP ac 1/2 of the scans, and the hAP ac2 1/3 of the times in the scan every 8 seconds in WinFi. The Draytek AP and BBOX from the ISP are seen all the time. So the laptop never picks one of the MT AP's). A pure 802.11ac stick sees them all the time in WinFi. Is it MT "ac only, N/AC, A/N/AC related? Changing that didn't help. Even in my recent Android smartphone "Wifi Analyzer" misses the MT AP's until one is very close to the MT AP, never misses the other AP's. Something with "Wifi Analyzer", something with "WinFi" , or something in the MT AP beacon ?

There is more in the config than the user set configuration. From release to release sometimes "defaults" in the config are changed. So indeed you need "export verbose" to know all parameters, and if they are changed or not. ARM cpu frequency stepping is only since 6.48, so it is not in scope, yet.

EDIT: downgrading to 6.45.6 didn't change anything here. My client devices just prefer other brands of AP's for connections. Or is it because they can estimate their capacity (QBSS field in the beacon, missing in the MT beacon). ?
 
toxicfusion
Member Candidate
Member Candidate
Posts: 213
Joined: Mon Jan 14, 2013 6:02 pm

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Mon Jun 21, 2021 7:33 pm

I would like to know if this is a general performance problem. I do see issues with 6.47.9 @home (my test bed), but I'm not sure if it is related to MT. The issues are non-specific and different for all devices, (like a relative old laptop with only a 802.11n interface , that sees the wAP ac 1/2 of the scans, and the hAP ac2 1/3 of the times in the scan every 8 seconds in WinFi. The Draytek AP and BBOX from the ISP are seen all the time. So the laptop never picks one of the MT AP's). A pure 802.11ac stick sees them all the time in WinFi. Is it MT "ac only, N/AC, A/N/AC related? Changing that didn't help. Even in my recent Android smartphone "Wifi Analyzer" misses the MT AP's until one is very close to the MT AP, never misses the other AP's. Something with "Wifi Analyzer", something with "WinFi" , or something in the MT AP beacon ?

There is more in the config than the user set configuration. From release to release sometimes "defaults" in the config are changed. So indeed you need "export verbose" to know all parameters, and if they are changed or not. ARM cpu frequency stepping is only since 6.48, so it is not in scope, yet.

EDIT: downgrading to 6.45.6 didn't change anything here. My client devices just prefer other brands of AP's for connections. Or is it because they can estimate their capacity (QBSS field in the beacon, missing in the MT beacon). ?
@bpwl - interesting find here.

What about with the latest 6.47.10 release; do you still have same issue of devices not seeing and or not picking the MT wifi?

Perhaps we can send this over to MT regarding MT AP's not having QBSS field in their beacon.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 1498
Joined: Mon Apr 08, 2019 1:16 am

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Mon Jun 21, 2021 11:49 pm

The hAp ac2 is on release 6.47.10 , and there is no change in behavior compared to 6.45.6.. QBSS is not there with MT.

Even if I try hard to create "laboratory conditions", this is not really the case. It is my home installation, with ISP modem, Draytek and hAp Lite in the garage, hAP ac2 in the living room and the wAP ac out in the veranda. hAP ac2 , wAp ac and Draytek have common SSID's set. they use channel 1, 6 and 11 respectively. The ISP modem jumps sometimes to another channel. hAP Lite is on channel 6.

Where I use my clients, the signal strength is almost equal in the living room, except for a weaker wAP ac. Even if the signal of the hAP ac2 is somewhat stronger than the Draytek (signal reading is delayed on Android) ... PC and smartphone connect to the Draytek, not the expected hAP ac2.

Too many parameters and influences to conclude anything. (Except that it is annoying behavior.) e.g. basic rate in ISP modem and Draytek is 1 Mbps, and that cannot be changed (even if it is set to operate in g/n or n-only.). Bringing down the TX power in the Draytek from 80% to 20% convinces the laptop to connect to the hAp ac2, but not the smartphone. (while changing SSID's). So it is just not a good enough experiment ! Better controlled environment and better measuring/analyzing tools are needed.

To be clear: if have no degradation issues with any connection!

I do personally miss that channel utilization information in MT. "What is your expected throughput if you only know the interface rates and CCQ?" One can be expecting way too much, and even select the wrong channel. I know there is "Freq Usage" in MT, but that stops transmissions. The MT should already know the channel utilization from the CCA process, while working.
Klembord-2.jpg
You do not have the required permissions to view the files attached to this post.
 
toxicfusion
Member Candidate
Member Candidate
Posts: 213
Joined: Mon Jan 14, 2013 6:02 pm

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Wed Jun 23, 2021 6:50 pm

@tmiklas

Could you try 6.47.10 Long-term release and see if speeds improved? I noticed an improvement on my devices.

Or even goto stable branch and install 6.48.3? Curious if any differences.

Run speed tests a few times, usually the first test will be lower.
 
toxicfusion
Member Candidate
Member Candidate
Posts: 213
Joined: Mon Jan 14, 2013 6:02 pm

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Wed Jun 23, 2021 6:52 pm

The hAp ac2 is on release 6.47.10 , and there is no change in behavior compared to 6.45.6.. QBSS is not there with MT.

Even if I try hard to create "laboratory conditions", this is not really the case. It is my home installation, with ISP modem, Draytek and hAp Lite in the garage, hAP ac2 in the living room and the wAP ac out in the veranda. hAP ac2 , wAp ac and Draytek have common SSID's set. they use channel 1, 6 and 11 respectively. The ISP modem jumps sometimes to another channel. hAP Lite is on channel 6.

Where I use my clients, the signal strength is almost equal in the living room, except for a weaker wAP ac. Even if the signal of the hAP ac2 is somewhat stronger than the Draytek (signal reading is delayed on Android) ... PC and smartphone connect to the Draytek, not the expected hAP ac2.

Too many parameters and influences to conclude anything. (Except that it is annoying behavior.) e.g. basic rate in ISP modem and Draytek is 1 Mbps, and that cannot be changed (even if it is set to operate in g/n or n-only.). Bringing down the TX power in the Draytek from 80% to 20% convinces the laptop to connect to the hAp ac2, but not the smartphone. (while changing SSID's). So it is just not a good enough experiment ! Better controlled environment and better measuring/analyzing tools are needed.

To be clear: if have no degradation issues with any connection!

I do personally miss that channel utilization information in MT. "What is your expected throughput if you only know the interface rates and CCQ?" One can be expecting way too much, and even select the wrong channel. I know there is "Freq Usage" in MT, but that stops transmissions. The MT should already know the channel utilization from the CCA process, while working.
Klembord-2.jpg
What if you enable WMM on your wireless interfaces. This should be QBSS? Let me know if you already had WMM enabled or not?
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 1498
Joined: Mon Apr 08, 2019 1:16 am

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Wed Jun 23, 2021 8:56 pm

I have WMM enabled. Should check the WMM info in the beacon.... just will do it ...

WMM is clearly in the Routerboard (hAP ac2) beacon.
.
Klembord-3.jpg
.
And I see the normal values for "CW min" etc.
.
Klembord-2.jpg
.
But the QBSS is missing, as can be seen from any other AP, Like the Draytek (also same WMM information here)
.
Klembord-4.jpg
.
Interesting information, for client device, and network operators.
.
You do not have the required permissions to view the files attached to this post.
 
toxicfusion
Member Candidate
Member Candidate
Posts: 213
Joined: Mon Jan 14, 2013 6:02 pm

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Wed Jun 23, 2021 9:53 pm

@bpwl Thanks for the great info, as always!...

Perhaps MikroTik can make a few comments here, or I send in a ticket for their review of this thread....

I have a feeling we will have wait for RouterOS 7 & the new hardware that supports it, along with the new wifi driver implementations. But, maybe MT will give us a present and fix lingering issues in the 6.XX train that is supported on the current wireless hardware [probably only fixes to ARM cpu based]??

Gut feeling routerOS 7 wireless will ONLY support ARM based CPU's, and we will see further improvement in hardware + features... wishful??

I have an upcoming client wireless refresh for their locations. I am to quote MikroTik, but i am unsure. I know MT RouterOS very well and so comfortable. However, I might have to quote out the Aruba InstantON range of access points. 50/50 decision
 
anuser
Long time Member
Long time Member
Posts: 550
Joined: Sat Nov 29, 2014 7:27 pm

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Wed Jun 23, 2021 11:34 pm

@bpwl Thanks for the great info, as always!...
Perhaps MikroTik can make a few comments here, or I send in a ticket for their review of this thread....
I have a feeling we will have wait for RouterOS 7 & the new hardware that supports it, along with the new wifi driver implementations. But, maybe MT will give us a present and fix lingering issues in the 6.XX train that is supported on the current wireless hardware [probably only fixes to ARM cpu based]??
Gut feeling routerOS 7 wireless will ONLY support ARM based CPU's, and we will see further improvement in hardware + features... wishful??
I have an upcoming client wireless refresh for their locations. I am to quote MikroTik, but i am unsure. I know MT RouterOS very well and so comfortable. However, I might have to quote out the Aruba InstantON range of access points. 50/50 decision
wifiwave2 package for RouterOS v7.x consists of binary firmware drivers from chipset vendors which MikroTik started to integrate into RouterOS, that´s all. It has nothing to do with the drivers which are used for RouterOS v6

For which environment do you want to replace your access points? Higher density locations like a school, university, enterprise building, small homes? MikroTik is not meant for the first 3 ones.
Have you already created a requirement list for your new accees points?
e.g.
- IEEE 802.11k
- IEEE 802.11v
- IEEE 802.11r
- IEEE 802.11w
- Airtime Fairness feature support (https://www.tp-link.com/ae/support/faq/2095/)
- Fully compliant with IEEE 802.11ax specification
- DL MU-MIMO
- UL-MU-MIMO
- BSS Coloring
- Target Wake Time
- Something like ZeroWait DFS
- Upload and download OFDMA with up to 37 resource units when 80 MHz channel is used
- Central management software or embedded controller?
- Should the access points forward the traffic to a central controller via tunnel?
- On Premise or cloud management?
- Possibility to deny client-to-client connections
- Powered by 802.3af, 802.3at or passive PoE?
- Which multicast optimizations
- How many clients do the access points support in reality you are currently quoting for?
- Are there any high density benchmarks available for that specific access point model you are looking for?
 
toxicfusion
Member Candidate
Member Candidate
Posts: 213
Joined: Mon Jan 14, 2013 6:02 pm

Re: 802.11ac severe speed degradation with ROS above 6.45.9 (LTS)

Thu Jun 24, 2021 5:29 am

@bpwl Thanks for the great info, as always!...
Perhaps MikroTik can make a few comments here, or I send in a ticket for their review of this thread....
I have a feeling we will have wait for RouterOS 7 & the new hardware that supports it, along with the new wifi driver implementations. But, maybe MT will give us a present and fix lingering issues in the 6.XX train that is supported on the current wireless hardware [probably only fixes to ARM cpu based]??
Gut feeling routerOS 7 wireless will ONLY support ARM based CPU's, and we will see further improvement in hardware + features... wishful??
I have an upcoming client wireless refresh for their locations. I am to quote MikroTik, but i am unsure. I know MT RouterOS very well and so comfortable. However, I might have to quote out the Aruba InstantON range of access points. 50/50 decision
wifiwave2 package for RouterOS v7.x consists of binary firmware drivers from chipset vendors which MikroTik started to integrate into RouterOS, that´s all. It has nothing to do with the drivers which are used for RouterOS v6

For which environment do you want to replace your access points? Higher density locations like a school, university, enterprise building, small homes? MikroTik is not meant for the first 3 ones.
Have you already created a requirement list for your new accees points?
e.g.
- IEEE 802.11k
- IEEE 802.11v
- IEEE 802.11r
- IEEE 802.11w
- Airtime Fairness feature support (https://www.tp-link.com/ae/support/faq/2095/)
- Fully compliant with IEEE 802.11ax specification
- DL MU-MIMO
- UL-MU-MIMO
- BSS Coloring
- Target Wake Time
- Something like ZeroWait DFS
- Upload and download OFDMA with up to 37 resource units when 80 MHz channel is used
- Central management software or embedded controller?
- Should the access points forward the traffic to a central controller via tunnel?
- On Premise or cloud management?
- Possibility to deny client-to-client connections
- Powered by 802.3af, 802.3at or passive PoE?
- Which multicast optimizations
- How many clients do the access points support in reality you are currently quoting for?
- Are there any high density benchmarks available for that specific access point model you are looking for?
Understood about your recommendations. I also realize the wireless driver binaries for RouterOS v7. It should help once MikroTik starts utilizing vendor wireless drivers over their own; or they learn from it.
No, I am not rolling out MikroTik Wireless to enterprise environment; that is not their space. I reserve MikroTik wireless for small business clients [Small restaurants, Pizzaria, doctor offices, etc]

I've also had success with TP-lInk Omada AP's [EAP series] over Ubiquiti -- I do not trust UBT anymore. However, when I have a restaurant client that has both indoor and outdoor wireless requirements; I tend to lean toward MikroTik. CapsMan works great.

Enterprise environments that I've done roll outs was aerohive, as well as Aruba. But SMB space, there are so many options...

Who is online

Users browsing this forum: No registered users and 47 guests