Community discussions

MikroTik App
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

My experience and issues in hi-density networks at school

Mon Jan 30, 2023 6:23 pm

For last 4 months I am battling whit WiFi stability issues in our school. Its around 800 student school + staff, but not all are using WiFi, at least not yet (soon most of classrooms will have dedicated devices for student use). Usually I see around 300-400 WIFI devices connected in total (students + staff). We had Unifi deployment (done very poorly, local "professionals" deployed APs in halls, basically no 5ghz band coverage in classrooms (whit some exceptions) and 2.4ghz band cant hold 70-105 devices per AP (smartphones whit poor signal in noisy env, its ridiculous) if coverage is usable, so we got Mikrotik gear (tnx boys, you know who you are) - cap acs, hap ac2s, wap acs + other cool switching stuff.

Here is our school 2nd floor plan. Tik and Unifi deployment.
Ekrānuzņēmums_20230130_144916.png
Overall situation did improve, AP per classroom is a must have nowadays for multiple reasons (to hold acceptable throughput per station, load balance between APs, have a good/strong (relatively clean) signal on mobile devices, that have weaker antennas etc). But there is issues whit APs stability - 5ghz band issues in particular (to be honest I haven't tested 2.4ghz band much, as it will be quite bad in these type of deployments anyway (but will do later as 5ghz is solved), 2.4ghz is available, but needs tweaking). A lot off issues I seen here in forums I have experienced, also have heard stories from fellow network engineer around the globe (internet is a wonderful thing) about this. It boils down to 3 major issues I would say, all are related in one or another way:

1. WiFi interface hang, giving no throughput or very low throughput. (a few mpbs, ~1-3 usually) It recovers when problematic station leaves AP range. Quite often I see this type of hang. If a device connects whit a bad signal, usually less that -70dbm, it can trigger this (happens even whit >-60dbm, or more rarely). I have deployed 7 testing APs as stations in rooms around school (yes, they work fine whit reduced output,max >5db EIRP, and no, they are not causing this), doing ping most of times, or I do short (>30sec) burst throughput tests (they are actually a great troubleshoot and planning tool, not just be a AP). I monitor pings directly thru Winbox if I do intense, close monitoring, or I have Zabbix instance for long term monitoring. In this video you can see, how one station can hang AP temporarily: https://cloud.harijs.id.lv/index.php/s/3QT62GM7dCpCxbE . I did kick a bunch of stations, but only one was causing this, most likely. As we have quite a lot of APs (now 38 tik and 23 Unifi APs deployed), stations usually roam to another AP. At some cases they do connect back and hag AP again. As I mentioned, there is some throughput (not always), as I can get DHCP, whit struggle mostly, some ARP, multicast, broadcast etc. But for web browsing its useless, even one device (as it cant hold those mbps). Sometimes situation is not that bad, ping in in xxx sec and you can get out a few more mbps, but that's it. Other thing that helped us is ACL rx limits, I have set it to -75dmb. It is higher I would like it to be, but if I set it to -80dbm I see this issue happen more often.

2. WiFi interface freeze, it does not recover - this is the worst one. Throughput is 0, no packets passed, even DHCP fails - complete interface halt. You can see broadcasted SSID from station and try to connect, but connection will fail as you wont be able to get IP from DHCP server, station will pop up in AP registration table however. Besides that, Interface hang can be "seen". If you do a spectrum analysis you can see AP emitting strong "noise":
Spektra ar traucējumiem.png
I don't have proper lab equipment (picture is from SXTsq 5 High Power spectral history, love that feature), so I cant tell if its 802.11 decodable or not. Only thing that helps is interface toggle, it can be on/off in Capsman, or AP restart (also capsman restart will help, but it would be stupid to reboot Capsman server to solve one AP issues). It does not recover and airtime it unusable on that and around channels.


3. Insane amount of "radar detects". No other vendor I have seen goes offline as often as these APs. Our Unifies detects radars (not sure if real ones, as my understand is, its somewhat malformed wifi frame that triggers it, it can happen anywhere from anyone, time to time), but it happens 1-2 times per week. This does not happen only in hi density networks, but also my home. What happen is, APs "detects radar" on every station connection attempt (association, when device tries to connect to WiFi AP). I had Mediatek card in my laptop and it did trigger radar detect almost every single time I opened it (If wifi was disconnected in sleep/hybernate it tries to reestablish connection). I switched to Intel card, its works fine (even 20-230cm from AP), but this is not a fix in general, also not one vendor specific issue. Here is a video of one phone we use in school triggering radar detect on every WiFi association attempt. Link: https://cloud.harijs.id.lv/index.php/s/SmswwYdd87esYZw . In this situation AP (hap ac2) is on my table, so around 1m away from me. Yes, that is close, but you know what helps? Changing country. For testing purposes I set APs to "no country" and this issue, I can't say is solved as radar detect have to do its thing, but is now in line whit Unifi events, happens a bit often, but not far of, can be deployment issue here. This phone, or no other device, triggers false detects on WiFi association or not by being close. One shouldn't do this, as you have to follow regulations and set it correctly, but here you go - whit correct country AP is unstable for one more reason.

I have also attached my capsman and one of classrooms APs config. It is messy, as I have tried so much to at least mitigate this. I understand that I cant solve this, this is no config issue, it looks like driver or hardware. Mikrotik staff can comment on this more. My deployment is not perfect, as we are waiting for new cable runs in summer, now APs are put on desks. This deployment can/have its issues, but it should not cause massive stability breakdown. A performance degradation is what I expected, for multiple reasons. Channels are set manually, I have tried auto mode, all (almost) setting to defaults actually. TX power started from 12-14dbm EIRP, I turned off Unifies at start of deployment. This is the best setup I got in 4 months. Today got quite bad day, multiple temp hangs, one freeze. Btw, reboot from time to time helps, but I cant be sure. As school is very dynamic env every day.

Here is icmp loss from multiple rooms today. Worst case was 4 APs having issues, and I monitor just 7 from 32 classrooms + staff. So number, most likely, was higher.
Ekrānuzņēmums_20230130_181754.png
Now a days network connectivity is a must have, like electricity. Now our country is testing online exams as they might be reality in near future, also a big part of our teachers are preparing lessons online - if internet goes down it is causing major issues. Like most of factories, businesses, WiFi is part of network infrastructure, and it must be reliable. Performance can degrade at most of times and that is tolerable, but stability issues are not. If Mikrotik staff can't replicate this in testing lab, feel free to contact me. Liepāja have quite nice hotels, you can stay for a few days.

These are major issue I face daily, there is a lot more stories to tell and I have heard many more. I hope there will be a fix as, rumors say, we most likely wont see wave2 on 40xx devices, aka soc vendor drivers.

My sup ticket number about this is SUP-98735, filled in november.
You do not have the required permissions to view the files attached to this post.
Last edited by maigonis on Sat Jan 20, 2024 4:35 pm, edited 12 times in total.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: My experience and issues in hi-density networks at school

Mon Jan 30, 2023 10:39 pm

I don't have proper lab equipment (its from SXTsq 5 High Power), so i cant tell if its 802.11 decodable or not.
Use "Snooper" to scan that freq band.
Snooper will give channels, AP and SSID, and client devices, as 3 levels of information.
It will recognise 802.11, nstreme and nv2 protocols (Not other brands specific protocols)

The SXTsq will AMPLIFY the received signal with is large antenna gain
-35dBm is near receiver saturation, what will cause all sorts of non-linear behaviour
EG. a wAP in the receive cone of the SXTsq often generates false radar detects, even if the frequency it uses is in a non-DFS channel. (I had phantom reception with 230MHz bias)
Unifi AP will probably be 3dB stronger than MT (MT is not using the TPC level EIRP , but only the 3 dB lower non-TPC EIRP levels)

Using "no_country_set" is moving to a country with fewer DFS channels, so fewer checks!
Klembord-2.jpg
You do not have the required permissions to view the files attached to this post.
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 12:19 am

I don't have proper lab equipment (its from SXTsq 5 High Power), so i cant tell if its 802.11 decodable or not.
Use "Snooper" to scan that freq band.
Snooper will give channels, AP and SSID, and client devices, as 3 levels of information.
It will recognise 802.11, nstreme and nv2 protocols (Not other brands specific protocols)

The SXTsq will AMPLIFY the received signal with is large antenna gain
-35dBm is near receiver saturation, what will cause all sorts of non-linear behaviour
EG. a wAP in the receive cone of the SXTsq often generates false radar detects, even if the frequency it uses is in a non-DFS channel. (I had phantom reception with 230MHz bias)
Unifi AP will probably be 3dB stronger than MT (MT is not using the TPC level EIRP , but only the 3 dB lower non-TPC EIRP levels)

Using "no_country_set" is moving to a country with fewer DFS channels, so fewer checks!

Klembord-2.jpg
SXTsq 5 High Power is n only, so it will not see ac, but I agree that it can amplify signal as it is hi gain antenna. Its not how hot the signal is, fact that it constantly emits something is more important, as it shows that it is misbehaving. We have UAP-AC-HD Unifi dishes, they are +3dbi, but they are not the issue. When i started tik deployment I turned them off, to eliminate any issues whit multivendor env. They are not the cause of this.

As i mentioned, I do channel manually and TX power is kept in line whit locally allowed, even if i can run them hotter (2.4ghz 30dbm EIRP - ouu yeahhh! I did test this one evening, range was insane).
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 12:41 am

SXTsq 5 High Power is n only, so it will not see ac
Is what one expects to happen. But g/n/ac are compatible protocols.
SXTsq 5 High Power sees hAP ac3 in 802.11ac-only mode !
.

Klembord-2.jpg
You do not have the required permissions to view the files attached to this post.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 1:08 am

I haven't used capsman for some time

but with all the recent changes i bet you will be better with a 6.xx version you validated and tested stable and stay with it as long as your devices supports it, until we see how capsman evolve in 7.x and wave2 era
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 1:14 am

Too much devices, too much @40MHz, two vendors, a true essence of self-interference!
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 11:50 am

@bpwl

I did tried sniffing a while a go and something wasn't right. Might revisit this as seeing ac would be beneficial.

@chechito

v6 feels kind of dead, and this issue is not new afaik. But I might try standalone mode.

@rextended

I started whit 20mhz, Unifies off and TX at 12-14dbm EIRP. We have tick walls, so 5ghz cannot penetrate it much. Now, by running TX at 20dbm EIRP I see as much as 2, max 3 APs at busiest places overlapping, by little. Our usage is quite low at the moment, airtime utilization is, on average, up to 30-40% at most in lessons (whit exceptions), break time can increase usage to 50-60% on busiest APs and its channels. So load itself is not that much of a issue, in general. Client count total and whit what signal quality they connect might be issue here. Station count per AP usually is around 1x, it goes up to 40+ connected devices in some classrooms, as those classroom have ipads there for every student.

A funny story why I enabled Unifies. If Unifi APs are off, then situation gets noticeably worse. That's because, as i sad, if station connects whit "bad signal" AP goes nuts. That means as all student go on break time in halls, as there is no APs, stations try to connect to classroom APs. As result, every break time multiple APs hanged. I have heard this before, in hi dense networks you need to plan network as every station have AP to connect to, and have a great signal (good modulation, not overused channel etc). It means, cover every inch of building where people stand for more than 5 sec.
Last edited by maigonis on Tue Jan 31, 2023 11:56 am, edited 1 time in total.
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 11:54 am

I have setup with 30 caps and tops 300 clients and its pure mikrotik setup also getting same issues as in random radios crashes, need to reboot ap to come to senses, random speed drops..
Mikrotik just simple cant handle such dense setups and mix of clients..
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 12:10 pm

I have setup with 30 caps and tops 300 clients and its pure mikrotik setup also getting same issues as in random radios crashes, need to reboot ap to come to senses, random speed drops..
Mikrotik just simple cant handle such dense setups and mix of clients..
A stupid question, but I have to ask, are you using capsman?
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 12:35 pm

Yea Capsman with local forwarding.
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 12:41 pm

Yea Capsman with local forwarding.
As expected and should be using imo.

Just now got temp complete interface hang in my office. After video i kicked all stations, except testing station, and it recovered. All station signal levels where good as those are mobile ipads in my office. Rx from AP side is/was around -50dbm.

https://cloud.harijs.id.lv/index.php/s/LBTt7sXNNNYG7yo
 
Rox169
Member
Member
Posts: 432
Joined: Sat Sep 04, 2021 1:47 am

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 1:33 pm

It is interesting topic. Im just curious how do they do in Tesla :) I have been told that comapany Tesla is using only Mikrotik :)
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 2:15 pm


I did tried sniffing a while a go and something wasn't right. Might revisit this as seeing ac would be beneficial.
Just to be very clear. 802.11n cannot "wireless sniff" total 802.11ac packets, only the header is compatible, and that's what "Snooper" is analysing. That is also what regulates the air-time between transmitters.

All depends on setup and location and area. I have 30 stand-alone AP's in 3 subnets of the same network, for 400 client devices, plus 20 PtMP connections as interconnects, for 2 years now. No hickups, no reboots, no forced disconnects to clean up. Just a puzzle to be finetuned, including avoiding the other foreign 40 AP's.
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 2:56 pm


I did tried sniffing a while a go and something wasn't right. Might revisit this as seeing ac would be beneficial.
Just to be very clear. 802.11n cannot "wireless sniff" total 802.11ac packets, only the header is compatible, and that's what "Snooper" is analysing. That is also what regulates the air-time between transmitters.

All depends on setup and location and area. I have 30 stand-alone AP's in 3 subnets of the same network, for 400 client devices, plus 20 PtMP connections as interconnects, for 2 years now. No hickups, no reboots, no forced disconnects to clean up. Just a puzzle to be finetuned, including avoiding the other foreign 40 AP's.
"stand-alone" might be keyword here. Tonight might setup some APs as standalones.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 6:51 pm

In the AP config ... Fairly complex queue handling, bridge filter, IP filter addition, packet marking ... I hope the AP processing power is not the bottleneck.

Setting priority does things internally, also for wifi WMM priority, but be aware of the lack of "AMPDU enabled" setting in MT by default for other priorities than priority "0",
No AMPDU aggregation for priorities 1 till 7 , if used, will waste a lot of air-time for everyone in the channel and drastically reduce wifi throughput.
I don't know if all those queue throttlings to the WLAN interfaces could not even break AMSDU aggregation by delaying IP packets, what would be a wifi throughput disaster.

viewtopic.php?t=165698#p912622 , viewtopic.php?t=165698#p912792

Removing 6Mbps basic rate can be good tuning, if needed. Otherwise it causes more disconnects and disables weak client associations.
(One AP/SSID at 6Mbps takes only 0.5% of available air-time in the channel. The old 1Mbps basic rate for 802.11b was 3% air-time, and that was often exhausted then with 30 AP/SSID)
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Tue Jan 31, 2023 10:09 pm

In the AP config ... Fairly complex queue handling, bridge filter, IP filter addition, packet marking ... I hope the AP processing power is not the bottleneck.

Setting priority does things internally, also for wifi WMM priority, but be aware of the lack of "AMPDU enabled" setting in MT by default for other priorities than priority "0",
No AMPDU aggregation for priorities 1 till 7 , if used, will waste a lot of air-time for everyone in the channel and drastically reduce wifi throughput.
I don't know if all those queue throttlings to the WLAN interfaces could not even break AMSDU aggregation by delaying IP packets, what would be a wifi throughput disaster.

viewtopic.php?t=165698#p912622 , viewtopic.php?t=165698#p912792

Removing 6Mbps basic rate can be good tuning, if needed. Otherwise it causes more disconnects and disables weak client associations.
(One AP/SSID at 6Mbps takes only 0.5% of available air-time in the channel. The old 1Mbps basic rate for 802.11b was 3% air-time, and that was often exhausted then with 30 AP/SSID)
When you see config in winbox is not that complex. I do packet marking (have to do in bridge as its L2), after that queues. Its a temp solution as network modernization is split in phases. I monitor networking devices whit Zabbix - resources are plenty, there is no buffer overflows etc.

I did test whit all rates, and combinations, it does not solve issues. Im monitor airtime utilization a bit, as i sad, its not that hi. On average >40%, so we have headroom there.
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Thu Feb 02, 2023 10:53 am

Sorry for double post, but interesting find.

Now Im testing standalone mode on a few APs. Just kicked one station whit rx level -88dbm and it was interesting to see, that its "last activity" time was almost 3min long. Kicked device and AP recovered, so devices whit no activity that long must kicked to at least manage this temp drop.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: My experience and issues in hi-density networks at school

Thu Feb 02, 2023 10:59 am

Try to not uselessly quote all previous block.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: My experience and issues in hi-density networks at school

Thu Feb 02, 2023 11:50 am

No activity for 3 min and the client did not drop the connection ? Reason code 4 (https://www.cisco.com/assets/sol/sb/WAP ... odes2.html)
viewtopic.php?t=159071#p787513

An AP will hold a frame to be transmitted for ever. (It will not drop that frame unless disconnected). Could this hold up a queue ?????

There is a "Frame Lifetime" in advanced tab of wireless interface, to drop old frames. Default is zero, never.
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Thu Feb 02, 2023 12:49 pm

No activity for 3 min and the client did not drop the connection ? Reason code 4 (https://www.cisco.com/assets/sol/sb/WAP ... odes2.html)
viewtopic.php?t=159071#p787513

An AP will hold a frame to be transmitted for ever. (It will not drop that frame unless disconnected). Could this hold up a queue ?????

There is a "Frame Lifetime" in advanced tab of wireless interface, to drop old frames. Default is zero, never.
Exactly that happened. I had the same thought, so I adjusted "frame lifetime" to 20sec on one of monitored APs (that I know often have this issue). Lets see what will happen.
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Fri Feb 03, 2023 9:32 am

It does not help.
Ekrānuzņēmums_20230203_093029.png
EDIT: Wait, look at interfaces. Both devices are on virtual interface. Maybe frame lifetime is not altering those.
You do not have the required permissions to view the files attached to this post.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: My experience and issues in hi-density networks at school

Fri Feb 03, 2023 12:24 pm

What is "helping" here? The lingering packet is dropped after 20 sec. The connection is not dropped by the AP. It's up to the client to drop the connection if there is no activity.
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Fri Feb 03, 2023 12:37 pm

What is "helping" here? The lingering packet is dropped after 20 sec. The connection is not dropped by the AP. It's up to the client to drop the connection if there is no activity.
Something must be done in this case, stations will misbehave.
 
gotsprings
Forum Guru
Forum Guru
Posts: 2087
Joined: Mon May 14, 2012 9:30 pm

Re: My experience and issues in hi-density networks at school

Fri Feb 03, 2023 11:07 pm

"You're right... it is funny when it's happening to someone else!"

That's a line from Big Bang Theory that really sums it up.

I got that email back in 2019...
"We will get back to you when we have a fix."
Still waiting.

But I moved on. Went back to my other wireless vendor.

Clients bitched about the Wireless costing more... but trouble calls stopped dead.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: My experience and issues in hi-density networks at school

Sat Feb 04, 2023 4:18 am

gotsprings is the go to guy for a reality check. If you want wifi that will work today - he normally recommends Cambium I believe maybe the XV2-2 for example.
You really need kit designed for high density.......... and stability under load.
 
gotsprings
Forum Guru
Forum Guru
Posts: 2087
Joined: Mon May 14, 2012 9:30 pm

Re: My experience and issues in hi-density networks at school

Sat Feb 04, 2023 12:56 pm

ANAV,

I don't yet recommend Cambium. I have spent over a year with them shaking out bugs. The latest release doesn't have any of the things I have presented to engineering in the last 13 months, as issues.

I have 3 systems in smaller configs in the wild. Dealing with around 200 clients at most.

My go to for the last 12 years has been Ruckus Wireless. Where I have been able to deploy radios and keep things connected where other manufactures couldn't.

Ruckus introduced a few bugs that really screwed up their gear. The worst causes full on reboot loops. But other ones causes connectivity issues with various 2.4 only client devices. These devices are heavily used my restaurant and bar installs. Fortunately... Going back to an old firmware that lacked any CVEs... Didn't have the issue.

Then the issue became I watched and tested releases for the next 17 months that all had that issue. Despite it being reported again and again by lots of other people. However, as stated, as long as I stayed below that firmware I could keep right on deploying the best wifi around.

I had a Cambium unit that I had been using as a competitive. I was trying to use it to simulate an MDU deployment I had done a couple dozen times with Ruckus. I ran head first into an issue I had not seen before. I could reproduce it and took it to their forum and support. I eventually got the attention of engineering and they called me up and had me demonstrate the problem. I spent a few hours with them tweaking and messing. While we messed with it I found how this problem related to another lesser issue that had been reported.

The changes required to fix the issue were to serious for a RC or bugfix. So I worked with them on it... For months. Because they actually took the time and wanted to fix it.

Once upon a time when Ruckus was private... They would have handled this in a few days. But Cambium being the size they are... Well it took just about a year.

Now I have to scale up so more and make sure nothing breaks.

Unfortunately... My failed caps-man deployment made me far more gun shy. But it also made me better and being able to find and track bugs. Ultimately so I could properly explain and demonstrate them. Which Cambium chooses to take advantage of.

Mikrotik... Has admitted in this forum over the years that there are something and environments where they are absolutely not going to work. The post where they flat out stated that the initial poster has picked the wrong solution was the sort of thing that should be a sticky at the top of the forum. Alas it's buried in all he posts where people are complaining about Mikrotik WiFi.

As I have said for years... If caps-man could control a good radio that could actually keep things connected... It would be my absolute go to favorite.

I have been working with integrators for years who always ask... "Could that control a (fill in the blank)?"

I have to always answer, "Nope."
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Mon Feb 06, 2023 12:58 pm

Another interesting find.
Ekrānuzņēmums_20230206_121505.png
Here you can see statistics from one of stations that hanged AP. Hw frames are transmitted a lot, that count increases rapidly. So that means AP is sending something to that station, constantly.

But as you can see down bellow, there is no activity from station, for a long time.
Ekrānuzņēmums_20230206_121449.png
I also noticed today, that not all stations whit long last activity time hangs AP. it happens at some point.
You do not have the required permissions to view the files attached to this post.
 
User avatar
pcunite
Forum Guru
Forum Guru
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: My experience and issues in hi-density networks at school

Mon Feb 06, 2023 2:38 pm

I don't yet recommend Cambium.
...
MikroTik ... Has admitted in this forum over the years that there are some places and environments where they are absolutely not going to work ... As I have said for years ... If caps-man could control a good radio that could actually keep things connected... It would be my absolute go to favorite.

Well, thanks for staying here in the forums. Hopefully MikroTik will eventually address the issues. They used to maintain a wall between us and them. I think that is slowly changing. Nobody wants to be publicly shamed and embarrassed. I think MikroTik as a culture has had a pride issue. Shyness is pride. But they seem to be overcoming it and are able to laugh at themselves now. But as has been noted many times, they are way behind on the software engineering side of their business. They need to hire out and someone from this side of the pond to broaden their culture perspective.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: My experience and issues in hi-density networks at school

Mon Feb 06, 2023 3:03 pm

Unfortunately a pretty common behavior of some technology-driven companies (or departments) with a widespread "not invented here" syndrome. However as you point out, it seems to be slowly turning in the right direction..
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: My experience and issues in hi-density networks at school

Mon Feb 06, 2023 3:13 pm

Another interesting find.
Indeed interesting. HW Frames are supposed to be the number of frames send/received. Frames the number of succesfull frames.

There is a setting in MT to handle failed HW Frames: HW Retries. Is 7 by default, can be set between 1 and 15.
Failed transmissions in the HW Retries range are not acted upon, except retransmitted.
If the HW Retries count is exceeded then the driver will lower the interface rate, or the number of chains (eg from 2S to 1S), and restart counting.
At the lowest rate the driver will retry during the "Disconnect timeout" every "On fail retry time".

"HW Frames/Frames > 15" is not expected unless all Frames fail. But then I expect the interface rate to be at the minimum supported rate.

What goes wrong?
Modified supported rates or basic rates creating unexpected behaviour? Default on 6 Mbps for lowest rates.
"HW retries" changed to 0 for some reason/bug, meaning unlimited failed retries?
Very long or zero (=infinite?) "Disconnect timeout"? Default = 30 sec. (*)
Very small "On fail retry time" ? Default = 0.1 sec. (*)
Interface rate is still higher than 6 Mbps.???
CCQ has not dropped to zero. ???
Signal strength is only given for the 802.11 speeds (6 till 54 Mbps). HT and VHT MCS are missing.

(*) Allows for 300 retries or [ 300 * ("HW Retries"+1) ] retries ???????
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Mon Feb 06, 2023 5:06 pm

Another interesting find.
Indeed interesting. HW Frames are supposed to be the number of frames send/received. Frames the number of succesfull frames.

There is a setting in MT to handle failed HW Frames: HW Retries. Is 7 by default, can be set between 1 and 15.
Failed transmissions in the HW Retries range are not acted upon, except retransmitted.
If the HW Retries count is exceeded then the driver will lower the interface rate, or the number of chains (eg from 2S to 1S), and restart counting.
At the lowest rate the driver will retry during the "Disconnect timeout" every "On fail retry time".

"HW Frames/Frames > 15" is not expected unless all Frames fail. But then I expect the interface rate to be at the minimum supported rate.

What goes wrong?
Modified supported rates or basic rates creating unexpected behaviour? Default on 6 Mbps for lowest rates.
"HW retries" changed to 0 for some reason/bug, meaning unlimited failed retries?
Very long or zero (=infinite?) "Disconnect timeout"? Default = 30 sec. (*)
Very small "On fail retry time" ? Default = 0.1 sec. (*)
Interface rate is still higher than 6 Mbps.???
CCQ has not dropped to zero. ???
Signal strength is only given for the 802.11 speeds (6 till 54 Mbps). HT and VHT MCS are missing.

(*) Allows for 300 retries or [ 300 * ("HW Retries"+1) ] retries ???????
From wiki: "hw-frames: Number of frames sent over wireless link by the driver.". Those are frames sent by interface itself (driver to be specific) and are not passed to network stack. So issues is, again, somewhere in driver itself. It could be somewhere between line you are saying. As you can see in picture, more that 3GB of traffic is sent to that station on hardware level, only around 150kb from network stack. It would be useful to inspect station that hangs AP and see what traffic is received. This station had bad signal, SNR 25db, rx signal level -80dbm. This is why higher rates and rx limits helps a bit, as this issue happens when signal level and quality is poor. But I have seen -50dbm station taking AP down, so it is not limited to low level signals, but its quality.

That AP runs all default rates in standalone mode, nothing was altered. AP should just kick these stations whit "extensive data loss" , but it does not - it hangs and do some, at this point, unknown stuff to it.

Might play whit HW Retries a bit.
 
UpRunTech
Member Candidate
Member Candidate
Posts: 209
Joined: Fri Jul 27, 2012 12:11 pm

Re: My experience and issues in hi-density networks at school

Tue Feb 07, 2023 10:25 pm

As discussed on Discord I too have similar issues with poor CAPSMAN behaviour on medium sized networks at schools.

It's nothing particularly new, I have had similar issues using the older MIPS dual band wAPACs and cAPACs so I think there is a long standing driver problem here. Why? Who knows. C lets you shoot yourself in the foot in so many ways and I think these bigger CAPSMAN systems really start revealing such issues.

All I can suggest is arranging Mikrotik to log into the affected units at the time and try poke around with their debugging tools. I have tried sending supouts of the state of things before and after CAPSMAN failures and Mikrotik didn't see anything in them to explain what's happening.

All I do know is that resetting the CAPSMAN state, which affects both the radio and the CAPSMAN controller, gets things going again assuming you can still log in with Winbox.

* Log into the client and disable then reenable CAPSMAN on the wireless interface.
* Rebooting the client.
* Disabling then re-enabling CAPSMAN in the controller.

These all essentially reset the CAPSMAN state for a radio.

Maybe there is more than one problem! Along with CAPSMAN device driver issues there might be bridge problems as well.

Once site recently I solved a CAPSMAN connectivity issue (clients kept dropping off or data stopped flowing) by removing the cAPACs off a HP PoE switch and plugging them directly into an RB260GSP switch which in turn plugged directly into the RB3011 CAPSMAN controller. All the problems stopped after that. There were no log messages that could help me work out what was going on.

I don't think it helps CAPSMAN is an obfuscated protocol - if such information was at hand the clever Mikrotik users might be able to see what the problem is.

I really love the CAPSMAN concept, I think it's brilliant but the robustness leaves a lot to be desired. I really wish Mikrotik would take it seriously.
 
gotsprings
Forum Guru
Forum Guru
Posts: 2087
Joined: Mon May 14, 2012 9:30 pm

Re: My experience and issues in hi-density networks at school

Wed Feb 08, 2023 1:25 pm

UpRunTech
Do you set forwarding to local?

I was forced into an install yesterday that was really small

HAP AC2 + 2 cAP AC. RouterOS 7.7

No switch... The caps were connected directly to the hAP AC2 with POE Injectors.

Absolutely instant feed back of the roaming and signal levels of the test phone. A wifi call on 5Ghz didn't miss a beat as the tech walked around.

In the past I would set up netwatches and timers to disable radios in caps-man, to get packets moving again.
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: My experience and issues in hi-density networks at school

Wed Feb 08, 2023 2:24 pm

UpRunTech
Do you set forwarding to local?

I was forced into an install yesterday that was really small

HAP AC2 + 2 cAP AC. RouterOS 7.7

No switch... The caps were connected directly to the hAP AC2 with POE Injectors.

Absolutely instant feed back of the roaming and signal levels of the test phone. A wifi call on 5Ghz didn't miss a beat as the tech walked around.

In the past I would set up netwatches and timers to disable radios in caps-man, to get packets moving again.
I have no issue with roaming, i have school with 30 APS i move around entire place and phone roams between 20+ APs and i run ping to gateway with my samsung phone and not single ping drops.

But still on random days AP kicks all clients simultaneously and they reconnect shortly after or connect to nearby AP if close.. (received disassoc sending station leaving (8)), no way 10+ mix of devices decided just to leave AP for no reason while having full signal and no roaming.
other day speed crawls to unusable until you just reboot AP (Gets very same channel, nothing is actually changed other then reboot)
3d day random classroom has no net, clients are connected and show on registration table in capsman but you cant ping nothing in any direction, reboot AP or even just re-toggle from capsman, all starts working again...
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: My experience and issues in hi-density networks at school

Wed Feb 08, 2023 4:30 pm

MT WIFI Random functionality generator - best of class!! :-(
WAKE THE EFF UP MT and hire bpwl for a 6 mos term, all expenses paid, he can stay an Normunds place to save your cheap asses a few bucks.
Even better the multi-millionaire owners can cough up a few buck to make MT WIFI useful and thus marketable and not some YUGO wifi joke. ;-P
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: My experience and issues in hi-density networks at school

Wed Feb 08, 2023 6:49 pm

(received disassoc sending station leaving (8)), no way 10+ mix of devices decided just to leave AP for no reason while having full signal and no roaming.
Reason code 8 : 8 Disassociated because sending STA is leaving (or has left) BSS https://aboutcher.co.uk/2012/07/linux-w ... son-codes/

The stations (not the AP) decided to leave. Why, it's not said in that reason code "8". Having a perfect wifi signal is just not enough to stay connected.

But root cause candidates are:
- Stations missed multiple beacons from AP (remember beacons are not ACKed and are not retried). Basic rate beacon must be succesfull at once.
- Stations cannot reach their test Internet point. Quite some client devices drop their wifi connection if their specific (=OS dependent) "internet available" test fails.
- Possible reason for test failure is a DHCP lease expired/revoked/Nacked (eg by another DHCP server)

I hope nobody is playing https://en.wikipedia.org/wiki/Wi-Fi_dea ... ion_attack, or some "rogue AP detection" AP is just doing that. ( A Fortinet based wifi network would do that if AP's are identified as "Rogue" )

EDIT: But this disconnecting might be CAPsMAN v2 specific. Just reading interesting old posts in this forum. Like: viewtopic.php?t=91002#p514645
Last edited by bpwl on Thu Feb 09, 2023 1:08 am, edited 1 time in total.
 
gotsprings
Forum Guru
Forum Guru
Posts: 2087
Joined: Mon May 14, 2012 9:30 pm

Re: My experience and issues in hi-density networks at school

Wed Feb 08, 2023 7:20 pm


...clients are connected and show on registration table in capsman but you cant ping nothing in any direction, reboot AP or even just re-toggle from capsman, all starts working again...
Thats what I have seen for years at this point.

Thats when i use Netwatch to disable and enable radios.

I would netwach a printer in the room. When the printer stopped pinging... the remotes and IoT devices would all not be working. That disable enable cured it.

We got sick of it and gave up.
 
anuser
Long time Member
Long time Member
Posts: 601
Joined: Sat Nov 29, 2014 7:27 pm

Re: My experience and issues in hi-density networks at school

Wed Feb 08, 2023 9:21 pm

This is a repost from myself from last year:

"This was the main reason why I moved once from MikroTik to Aruba Network AP-5xx access points for high density and later to Grandstream GWN7660 for office environments, plus VRF lite for clients.

viewtopic.php?t=153735#p760146:
Hello,

the short version: CAPSMAN forwarding is full of bugs. The "fun" begins if you have ~1000 clients on the CCR controller, doing some traffice (500 Mbps/30000 sessions)...
Long version: No time today to tell you all things today.

So, let´s make it short:
- CCR will crash in certain high load situations
- CCR will disconnect all caps if you configure settings which let´s the CAPSMAN reprovision the CAP devices
=> They will disconnect an reconnect

On cAP ac clients disconnect, because 5 GHz interface is unstable. Some say the driver module crashes and reloads
Until:
Version 6.46beta59 has been released.
*) ccr - improved general system stability;
*) wireless - improved IPQ4019, QCA9984, QCA9888 wireless interface stability;
I don´t know whether this will solve your problems. Btw, do you also seee thousands of ~DHCP offer no lease messages in your logs?
=> Imagine you sit in front of one cAP, you can see.
=> Your notebook connects to the cAP but it doesn´t get any IP.
=> You disconnect and reconnect but your notebooks doesn´t get any IP.
=> You look on the DHCP server within the CCR and you can see the timer from the offer goes down from 30 seconds to 0 until it restarts at 30 seconds again. It stays on "offered" but nothing happens
=> Until you hit the provisioning button for the cAP and guess what happens your notebook gets an IP address immediately afterwars
=> Now you have this problem on 300 cAP device running on the CCR/CAPSMAN. Sometime the clients work without any problems for some hours until there is more load on the CCR. So starting at ~10:30 am each day I see "DHCP offer no lease" within the logs. 6.46beta59 doesn´t help. Noone knows why and I did everything this forum asked me to do.
=> The solution for me: Drop CAPSMAN, use single access points with local management and local forwarding. Sad? Of course.

The load was distributed to all cores on my CCR1036 devices with CAPSMAN based forwarding enabled.
Hint: Do not buy MikroTik cAP ac access points for office or campus environments. Those do not support 802.11ac MU-MIMO, no air time fairness, no 802.11k/v/r. Avoid CAPSMAN manager forwarding mode with thousands of clients."

=> I still have some MikroTik cap ac, wap ac left with CAPsMAN controller and local forwarding in places where only few clients will connect concurrently.
=> RouterOS is like a swiss knife, you can do a lot of things with it, even felling a tree, but using a chain saw is the tool you should look for if you want to make it right for reliable wifi

Nowadays there is an official OpenWrt port for MikroTik cAP ac devices (https://openwrt.org/toh/mikrotik/rbcapgi-5acd2nd_cap_ac). In different threads in the past there was once mentioned that single client throughput with openwrt is way lower than with RouterOS. But this isn´t what matters in a class room with 30 to 60+ clients connected on e.g. 1 cAP ac.
I haven´t tested OpenWRT with airtime fairness on CAP ac tested in a room full of students and asked whether the surfing experience was better than with a cAP ac and RouterOS as I´m simply tired of testing, perhaps someone here wants to.

As being already said before: Use 20 MHz channels, set static channel frequencies. lower the power for 2.4GHz so your clients always choose 5 GHz, ...
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Thu Feb 09, 2023 12:10 am

This is a repost from myself from last year:

"This was the main reason why I moved once from MikroTik to Aruba Network AP-5xx access points for high density and later to Grandstream GWN7660 for office environments, plus VRF lite for clients.

viewtopic.php?t=153735#p760146:
Hello,

the short version: CAPSMAN forwarding is full of bugs. The "fun" begins if you have ~1000 clients on the CCR controller, doing some traffice (500 Mbps/30000 sessions)...
Long version: No time today to tell you all things today.

So, let´s make it short:
- CCR will crash in certain high load situations
- CCR will disconnect all caps if you configure settings which let´s the CAPSMAN reprovision the CAP devices
=> They will disconnect an reconnect

On cAP ac clients disconnect, because 5 GHz interface is unstable. Some say the driver module crashes and reloads
Until:

I don´t know whether this will solve your problems. Btw, do you also seee thousands of ~DHCP offer no lease messages in your logs?
=> Imagine you sit in front of one cAP, you can see.
=> Your notebook connects to the cAP but it doesn´t get any IP.
=> You disconnect and reconnect but your notebooks doesn´t get any IP.
=> You look on the DHCP server within the CCR and you can see the timer from the offer goes down from 30 seconds to 0 until it restarts at 30 seconds again. It stays on "offered" but nothing happens
=> Until you hit the provisioning button for the cAP and guess what happens your notebook gets an IP address immediately afterwars
=> Now you have this problem on 300 cAP device running on the CCR/CAPSMAN. Sometime the clients work without any problems for some hours until there is more load on the CCR. So starting at ~10:30 am each day I see "DHCP offer no lease" within the logs. 6.46beta59 doesn´t help. Noone knows why and I did everything this forum asked me to do.
=> The solution for me: Drop CAPSMAN, use single access points with local management and local forwarding. Sad? Of course.

The load was distributed to all cores on my CCR1036 devices with CAPSMAN based forwarding enabled.
Hint: Do not buy MikroTik cAP ac access points for office or campus environments. Those do not support 802.11ac MU-MIMO, no air time fairness, no 802.11k/v/r. Avoid CAPSMAN manager forwarding mode with thousands of clients."

=> I still have some MikroTik cap ac, wap ac left with CAPsMAN controller and local forwarding in places where only few clients will connect concurrently.
=> RouterOS is like a swiss knife, you can do a lot of things with it, even felling a tree, but using a chain saw is the tool you should look for if you want to make it right for reliable wifi

Nowadays there is an official OpenWrt port for MikroTik cAP ac devices (https://openwrt.org/toh/mikrotik/rbcapgi-5acd2nd_cap_ac). In different threads in the past there was once mentioned that single client throughput with openwrt is way lower than with RouterOS. But this isn´t what matters in a class room with 30 to 60+ clients connected on e.g. 1 cAP ac.
I haven´t tested OpenWRT with airtime fairness on CAP ac tested in a room full of students and asked whether the surfing experience was better than with a cAP ac and RouterOS as I´m simply tired of testing, perhaps someone here wants to.

As being already said before: Use 20 MHz channels, set static channel frequencies. lower the power for 2.4GHz so your clients always choose 5 GHz, ...
I tested in my office openwrt a bit, also throughput. On 40mhz channel I got 140mbps down and feature set felt limited (I wonder why?!?).


Mikrotik support have responded to my ticket and now its ongoing process to, hopefully, solve this. This is is purely related how ROS handles wireless, so network stack, capsman or anything else is related. At least at this point off understanding. In short, main issue is stations not sending gracefull disconnect to AP (interface toggle, out of range etc.), so AP still tries to send data to it. tx-queue-drop increases and network will start to have hi ping or becomes unresponsive as buffer fills.

A workaround at this point (I was so close) is setting frame-liftetime=1 and hw-retries to 2 or 3. frame-liftetime=1 will clear buffer from stalled connections and hw-retries will reduce data rate faster, so stations will disconnect faster (this will impact roaming). Also acl rx limit helps, as a lot of low signal level stations causes this.
 
Kaldek
Member Candidate
Member Candidate
Posts: 111
Joined: Sat Jul 11, 2015 2:40 pm

Re: My experience and issues in hi-density networks at school

Thu Feb 09, 2023 12:39 am

I am just some SOHO guy, but when I had customised rates enabled my WiFi performance was terrible. It would randomly drop speeds to 30Kbs on the 5Ghz channel.

Went back to default rates, and the problems are gone.
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Thu Feb 09, 2023 9:37 am

I am just some SOHO guy, but when I had customised rates enabled my WiFi performance was terrible. It would randomly drop speeds to 30Kbs on the 5Ghz channel.

Went back to default rates, and the problems are gone.
Messed whit mcs indexes?
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: My experience and issues in hi-density networks at school

Fri Feb 10, 2023 1:55 pm

Here is more every day school experience:

Reboot AP, now selects chan just fine and there is no problems with any regulatory. mismatch and error is gone.

Also i forgot to mention using any DFS chans is 100% unusable, false radar detects all day, i get 10+ wifi blackouts daily, sometimes APS gets stuck on selecting channel forever, similar to what i wrote above and you can see in pics.
You do not have the required permissions to view the files attached to this post.
 
User avatar
Ca6ko
Member
Member
Posts: 498
Joined: Wed May 04, 2022 10:59 pm
Location: Kharkiv, Ukraine

Re: My experience and issues in hi-density networks at school

Sun Feb 12, 2023 5:56 pm

The frequencies only need to be assigned manually. Automatic selection in Mikrotik is not able to arrange the points by frequencies qualitatively. Auto frequency selection works as if there is only one point, it does not take into account neighboring points at all.
In high density conditions Channel 7 frequency 2442 MHz should not be used in any way. Use non-overlapping 1-6-11 if Standard B support is enabled or 1-5-9-13 when Standard B is completely disabled.
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Wed Mar 01, 2023 10:34 am

For a while I have tested workarounds. At this point best results are whit frame-liftetime=1, hw-retries=2 and in acl define minimums rx signal to -75dbm. I tried to reduce rx signal limit, but when I reduced signal to -80dbm situation noticeably got worse, almost instantly two APs that I monitor got issues in first break.

Bellow you can see downtime's from my monitored APs thru last 7 days. Its not much, that 8h downtime was caused by physical issues, so ignore that. But I haven't tested user experience overall just yet, at this point no one have complained.
Ekrānuzņēmums 2023-03-01 100959.png
You do not have the required permissions to view the files attached to this post.
 
kreload
just joined
Posts: 20
Joined: Tue Sep 15, 2020 10:18 am

Re: My experience and issues in hi-density networks at school

Sat Mar 04, 2023 5:33 pm


I tested in my office openwrt a bit, also throughput. On 40mhz channel I got 140mbps down and feature set felt limited (I wonder why?!?).


Mikrotik support have responded to my ticket and now its ongoing process to, hopefully, solve this. This is is purely related how ROS handles wireless, so network stack, capsman or anything else is related. At least at this point off understanding. In short, main issue is stations not sending gracefull disconnect to AP (interface toggle, out of range etc.), so AP still tries to send data to it. tx-queue-drop increases and network will start to have hi ping or becomes unresponsive as buffer fills.

A workaround at this point (I was so close) is setting frame-liftetime=1 and hw-retries to 2 or 3. frame-liftetime=1 will clear buffer from stalled connections and hw-retries will reduce data rate faster, so stations will disconnect faster (this will impact roaming). Also acl rx limit helps, as a lot of low signal level stations causes this.
Probably you test it in router mode or/and without irqbalance package. As dump ap's, mikrotik with openwrt rocks.
 
kreload
just joined
Posts: 20
Joined: Tue Sep 15, 2020 10:18 am

Re: My experience and issues in hi-density networks at school

Sat Mar 04, 2023 5:39 pm

In different threads in the past there was once mentioned that single client throughput with openwrt is way lower than with RouterOS. But this isn´t what matters in a class room with 30 to 60+ clients connected on e.g. 1 cAP ac.
I haven´t tested OpenWRT with airtime fairness on CAP ac tested in a room full of students and asked whether the surfing experience was better than with a cAP ac and RouterOS as I´m simply tired of testing, perhaps someone here wants to.

As being already said before: Use 20 MHz channels, set static channel frequencies. lower the power for 2.4GHz so your clients always choose 5 GHz, ...
It's not way lower but it's way better. I'm a happy mikrotik customer with openwrt installed on about 20 mikrotik devices at work for more than 6 months. For me the alternative when i had Router OS with Capsman installed was to move to another vendor but i tried OpenWRT. 6 months later, not a single wireless problem. For anything else except wireless i use RouterOS.
 
massinia
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Jun 09, 2022 7:20 pm

Re: My experience and issues in hi-density networks at school

Sat Mar 04, 2023 7:01 pm

At this point best results are whit frame-liftetime=1, hw-retries=2 and in acl define minimums rx signal to -75dbm.
Thanks for all the information, they will be very useful to me shortly :D
 
UpRunTech
Member Candidate
Member Candidate
Posts: 209
Joined: Fri Jul 27, 2012 12:11 pm

Re: My experience and issues in hi-density networks at school

Fri Mar 17, 2023 10:23 pm

Do you set forwarding to local?
Absolutely instant feed back of the roaming and signal levels of the test phone. A wifi call on 5Ghz didn't miss a beat as the tech walked around.
In the past I would set up netwatches and timers to disable radios in caps-man, to get packets moving again.
In the past I have tried everything, local, not local, TCP vs UDP and results were variable. It used to be the whole CAPSMAN system would just stall (not much Wifi traffic) when students moved about campus between classes and then it'd start to behave better after 10mins and some WAP reboots.

The large sites I have set up use an RB4011 with all traffic not local-forwarded. The 4011 is connected to CRS328 switch with an SFP+ DAC. I have put in the 2x retry and 1s lifetime on both big school sites and it's looking good. No lockups reported yet and speeds are what I would expect from a busy 40MHz channel. One complaint was the speeds would drop to 5Mbit or so.

I think CAPSMAN has had a lot of problems which has improved over the years so the issues now might just come down to this lockup bug that the timeout and retry might mask or actually fix. I even had WAPACS overheating (hot to touch) and freezing up until I moved them from 50V to 24V in the CRS328 PoE menu. From what I understand from discussions Discord with the default settings sometimes transmit buffers never flushed properly and clogged up with stale frames that were never going to be acked.

I have also noticed some switches don't behave well with CAPSMAN. One site with older HP switches tripped up CAPSMAN until I moved all the CAPACs to an RG260 with PoE and then plugged that direct into the RB3011. Other places were unstable too until I replaced the existing TPLink switches.

I did ask Mikrotik if they use local-forward in their office to which they said they did but they didn't think that !local-forward would be a problem or too different speedwise until there were CPU bottlenecks in the controller. An RB4011 with an SFP+ doesn't see much CPU use even on the busy days at the sites. I am using !local-forward on these larger sites as I have LAN management on VLAN100 and all forwarded CAPSMAN traffic on VLAN101. This makes it easy in the controller to bind new SSIDs to VLANs with a new bridge rather than have to go through all the switches and apply new VLAN IDs to the relevant ports. You also get the benefit of having all Wifi traffic connected to the wired LAN in one place which makes it easy to use torch or bridge filters. Also CAPSMAN reports TX and RX rates properly on the CAP interfaces when in !LF mode instead of just '0'.

Alas at this point CAPSMAN2 only supports local-forward which is a shame.
 
User avatar
carl0s
Member Candidate
Member Candidate
Posts: 179
Joined: Thu Jun 25, 2009 7:18 pm

Re: My experience and issues in hi-density networks at school

Sun Mar 19, 2023 3:17 am

I would have waited for wifiwave2 kit before putting in the Mikrotik stuff. I have been reluctantly putting in Mikrotik Wi-Fi in coffee shops and some offices over the last few years, but for the bigger job (multiple industrial units / factory / offices over an industrial estate), I have delayed and delayed and delayed... and... put in TP-Link Omada. but I told them it's inexpensive and when my 'preferred supplier' finally brings out their up-to-date stuff, we can switch to them. Having said that, I am now wondering if I need to switch at all.
 
User avatar
DanielJB
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Mon May 27, 2013 3:05 pm

Re: My experience and issues in hi-density networks at school

Mon Mar 27, 2023 9:28 am

For last 4 months I am battling whit WiFi stability issues in our school...cap acs, hap ac2s, wap acs + other cool switching stuff.
Firstly, I appreciate you taking the time to document this, since it's worthwhile getting more visibility on some of these issues. I too have been having a challenging journey with Mikrotik 11ac equipment.

A few interesting points:
1. I suggest running without adaptive-noise-immunity=ap-and-client-mode since this will trigger radio microcontroller firmware paths presumably when certain conditions are detected by the radio phy (spurs/low SNR or similar); my guess is this will adjust some phy parameters like sensitivity or filtering. It seems reasonable to assume some of these mechanisms aren't mature as are seldom used, therefore possibly making the radio 'deaf'; this could be scenario #2 you experience. The strong transmission observed could be many retries as the radio doesn't hear the ack.

2. I believe the non-opensource/proprietary WiFi driver here doesn't have airtime fairness and I find low-throughput clients are easily starved by high-throughput clients. Mikrotik's default radio interface queuing discipline SFQ appears to workaroud the issue (perhaps at the expense of aggregation). I suggest using SFQ on the radio interface and performing QoS on the router.

3. The cAP DSCP to WMM mapping is good, however you need to use "/ip/firewall/mangle/set new-priority=from-dscp-high-3-bits" to get the correct mapping; since packets will use radio queues beyond 0, aggregation may be disabled which can hurt air throughput unless you use ampdu-priorities=0,1,2,3,4,5

4. I do recommend putting the user network onto a VLAN (you'll need "/interface/bridge/settings/set use-ip-firewall-for-vlan=yes") to reduce broadcast traffic with

5. Since local forwarding is used on the cAPs, you might be better off managing the layer 2 broadcast domain via bridge horizon

6. I recommend CAPsMAN security group-key-update=1h and disable-pmkid=yes for optimal compatibility with Apple devices (unless things have changed since RouterOS 7.2)

7. There can be quick signal fluctuations due to occlusion so I recommend allow-signal-out-of-range=45s

8. I recommend not having AP and station mode on the CAPs, since this will have less exposure; instead deploy a powered device you can connect and ping

9. You may want to compare "/caps-man/configuration/set hw-protection-mode=rts-cts" vs none

10. I found '/caps-man/configuration/set multicast-helper=full' caused broadcast and multicast traffic to get blocked on a deaf station, so ensure this is set to disabled.

It would be good to see what issues remain with these adjustments on RouterOS 7.8. Beyond that, capturing a supout and decoding to get the WiFi kernel logs may reveal other useful information including radio queue lengths, packets queued per station and related.

Thanks,
Daniel
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: My experience and issues in hi-density networks at school

Mon Mar 27, 2023 10:41 am

For last 4 months I am battling whit WiFi stability issues in our school...cap acs, hap ac2s, wap acs + other cool switching stuff.
A few interesting points:
1. I suggest running without adaptive-noise-immunity=ap-and-client-mode since this will trigger radio microcontroller firmware paths presumably when certain conditions are detected by the radio phy (spurs/low SNR or similar); my guess is this will adjust some phy parameters like sensitivity or filtering. It seems reasonable to assume some of these mechanisms aren't mature as are seldom used, therefore possibly making the radio 'deaf'; this could be scenario #2 you experience. The strong transmission observed could be many retries as the radio doesn't hear the ack.
There is no options to configure or turn on adaptive-noise-immunity on Capsman and its off by default.

Also to all your other points we tried all that and probably 10x times more, nothing helps.
 
User avatar
carl0s
Member Candidate
Member Candidate
Posts: 179
Joined: Thu Jun 25, 2009 7:18 pm

Re: My experience and issues in hi-density networks at school

Mon Mar 27, 2023 11:37 am

For last 4 months I am battling whit WiFi stability issues in our school...cap acs, hap ac2s, wap acs + other cool switching stuff.
Firstly, I appreciate you taking the time to document this, since it's worthwhile getting more visibility on some of these issues. I too have been having a challenging journey with Mikrotik 11ac equipment.

A few interesting points:
1. I suggest running without adaptive-noise-immunity=ap-and-client-mode since this will trigger radio microcontroller firmware paths presumably when certain conditions are detected by the radio phy (spurs/low SNR or similar); my guess is this will adjust some phy parameters like sensitivity or filtering. It seems reasonable to assume some of these mechanisms aren't mature as are seldom used, therefore possibly making the radio 'deaf'; this could be scenario #2 you experience. The strong transmission observed could be many retries as the radio doesn't hear the ack.

2. I believe the non-opensource/proprietary WiFi driver here doesn't have airtime fairness and I find low-throughput clients are easily starved by high-throughput clients. Mikrotik's default radio interface queuing discipline SFQ appears to workaroud the issue (perhaps at the expense of aggregation). I suggest using SFQ on the radio interface and performing QoS on the router.

3. The cAP DSCP to WMM mapping is good, however you need to use "/ip/firewall/mangle/set new-priority=from-dscp-high-3-bits" to get the correct mapping; since packets will use radio queues beyond 0, aggregation may be disabled which can hurt air throughput unless you use ampdu-priorities=0,1,2,3,4,5

4. I do recommend putting the user network onto a VLAN (you'll need "/interface/bridge/settings/set use-ip-firewall-for-vlan=yes") to reduce broadcast traffic with

5. Since local forwarding is used on the cAPs, you might be better off managing the layer 2 broadcast domain via bridge horizon

6. I recommend CAPsMAN security group-key-update=1h and disable-pmkid=yes for optimal compatibility with Apple devices (unless things have changed since RouterOS 7.2)

7. There can be quick signal fluctuations due to occlusion so I recommend allow-signal-out-of-range=45s

8. I recommend not having AP and station mode on the CAPs, since this will have less exposure; instead deploy a powered device you can connect and ping

9. You may want to compare "/caps-man/configuration/set hw-protection-mode=rts-cts" vs none

10. I found '/caps-man/configuration/set multicast-helper=full' caused broadcast and multicast traffic to get blocked on a deaf station, so ensure this is set to disabled.

It would be good to see what issues remain with these adjustments on RouterOS 7.8. Beyond that, capturing a supout and decoding to get the WiFi kernel logs may reveal other useful information including radio queue lengths, packets queued per station and related.

Thanks,
Daniel
That's a very thorough list, thanks for posting!
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: My experience and issues in hi-density networks at school

Mon Mar 27, 2023 12:47 pm

Very interesting list @DanielJB

Can we do better on aggregation? SFQ parameters or other queue type?
Lack of good aggregation wastes a lot of valuable air-time.
 
ivicask
Member
Member
Posts: 417
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: My experience and issues in hi-density networks at school

Mon Mar 27, 2023 1:00 pm

You cant change wireless interface queue type while its managed by capsman.
You do not have the required permissions to view the files attached to this post.
 
User avatar
DanielJB
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Mon May 27, 2013 3:05 pm

Re: My experience and issues in hi-density networks at school

Tue Mar 28, 2023 7:43 am

A few interesting points:
1. I suggest running without adaptive-noise-immunity=ap-and-client-mode since this will trigger radio microcontroller firmware paths presumably when certain conditions are detected by the radio phy (spurs/low SNR or similar); my guess is this will adjust some phy parameters like sensitivity or filtering. It seems reasonable to assume some of these mechanisms aren't mature as are seldom used, therefore possibly making the radio 'deaf'; this could be scenario #2 you experience. The strong transmission observed could be many retries as the radio doesn't hear the ack.
There is no options to configure or turn on adaptive-noise-immunity on Capsman and its off by default.
CAPsMAN overrides the local cAP interface configuration, therefore on the cAPs, use "/interface/wireless/set adaptive-noise-immunity=none ampdu-priorities=0,1,2,3,4,5" when not connected to CAPsMAN. You may want to also specify aggregation on WMM priorities (which map to radio queues) 6 and 7 also to protect against heavy traffic consuming airtime.
 
User avatar
DanielJB
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Mon May 27, 2013 3:05 pm

Re: My experience and issues in hi-density networks at school

Tue Mar 28, 2023 8:08 am

Very interesting list @DanielJB

Can we do better on aggregation? SFQ parameters or other queue type?
Lack of good aggregation wastes a lot of valuable air-time.
To get more aggregation, one could increase sfq-allot to the aggregation limit, however lack of airtime fairness might be problematic for DHCP at least; remember Mikrotik probably use SFQ in all their validation.

The deficit round-robin scheduling in CAKE may further decrease aggregation but increase fairness; I would suggest cake-flowmode=dual-dsthost, maybe ampdu-priorities=0,1,2,3,4,5,7,8 is prudent to protect valuable airtime in case of errant DSCP marking. It would be good to get some site testing and feedback on this actually.

Dan
 
User avatar
DanielJB
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Mon May 27, 2013 3:05 pm

Re: My experience and issues in hi-density networks at school

Tue Mar 28, 2023 8:11 am

You cant change wireless interface queue type while its managed by capsman.
One the cAP, use (eg) "/queue/type/set wireless-default kind=cake cake-flowmode=dual-dsthost cake-diffserv=diffserv4".

Dan
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: My experience and issues in hi-density networks at school

Tue Mar 28, 2023 8:17 pm

It would be good to get some site testing and feedback on this actually.
Difficult to see the AMPDU aggregation, as I understand it is at the driver level. Even my dedicated wifi-sniffing-dongle did not show me the aggregation (as I don't know where to look for the AMPDU size used.)

Need this type of counters, to make it easy: viewtopic.php?t=165698&hilit=aggregation#p816007
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Fri Mar 31, 2023 3:07 pm

For last 4 months I am battling whit WiFi stability issues in our school...cap acs, hap ac2s, wap acs + other cool switching stuff.
Firstly, I appreciate you taking the time to document this, since it's worthwhile getting more visibility on some of these issues. I too have been having a challenging journey with Mikrotik 11ac equipment.

A few interesting points:
1. I suggest running without adaptive-noise-immunity=ap-and-client-mode since this will trigger radio microcontroller firmware paths presumably when certain conditions are detected by the radio phy (spurs/low SNR or similar); my guess is this will adjust some phy parameters like sensitivity or filtering. It seems reasonable to assume some of these mechanisms aren't mature as are seldom used, therefore possibly making the radio 'deaf'; this could be scenario #2 you experience. The strong transmission observed could be many retries as the radio doesn't hear the ack.

2. I believe the non-opensource/proprietary WiFi driver here doesn't have airtime fairness and I find low-throughput clients are easily starved by high-throughput clients. Mikrotik's default radio interface queuing discipline SFQ appears to workaroud the issue (perhaps at the expense of aggregation). I suggest using SFQ on the radio interface and performing QoS on the router.

3. The cAP DSCP to WMM mapping is good, however you need to use "/ip/firewall/mangle/set new-priority=from-dscp-high-3-bits" to get the correct mapping; since packets will use radio queues beyond 0, aggregation may be disabled which can hurt air throughput unless you use ampdu-priorities=0,1,2,3,4,5

4. I do recommend putting the user network onto a VLAN (you'll need "/interface/bridge/settings/set use-ip-firewall-for-vlan=yes") to reduce broadcast traffic with

5. Since local forwarding is used on the cAPs, you might be better off managing the layer 2 broadcast domain via bridge horizon

6. I recommend CAPsMAN security group-key-update=1h and disable-pmkid=yes for optimal compatibility with Apple devices (unless things have changed since RouterOS 7.2)

7. There can be quick signal fluctuations due to occlusion so I recommend allow-signal-out-of-range=45s

8. I recommend not having AP and station mode on the CAPs, since this will have less exposure; instead deploy a powered device you can connect and ping

9. You may want to compare "/caps-man/configuration/set hw-protection-mode=rts-cts" vs none

10. I found '/caps-man/configuration/set multicast-helper=full' caused broadcast and multicast traffic to get blocked on a deaf station, so ensure this is set to disabled.

It would be good to see what issues remain with these adjustments on RouterOS 7.8. Beyond that, capturing a supout and decoding to get the WiFi kernel logs may reveal other useful information including radio queue lengths, packets queued per station and related.

Thanks,
Daniel
Hi. Thanks for joining discussion, to answer to your points:

1. Support confirmed that adaptive-noise-immunity have no effect on Capsman managed devices, nor 40xx SOCs.

2. Whit AP managed queues, I can manage airtime better (per AP). If I do set Queues on router, they are applied to all traffic that is going out and in from our local network (wireless and wired). But what happens if devices send data to each other locally? Queues does not work.

3. DSCP is in to do list.

4. VLANs will be implemented, broadcast/multicast is now limited on switches, also APs drop all multicast on trunk port, as we have no use for it outside AP range. When torching interface I see broadcast around 30pps at busiest hours, also multicast in total is no more than that, usually quite less.

5. It disables hw offload, and as I mentioned, i doubt broadcast/multicast is issue for us. VLANs will help more in future.

6. For us group-key-update=12h for the best. No issues whit Apple, or other devices afaik. Devices on network that are not used outside premises stay connected and does not disconnect whit "group key timeout". disable-pmkid=yes I will set, disabled to troubleshoot this issue.

7. In env like schools they do happen, but this can be set under Capsman.

8. I have no other device to deploy.

9. I will test this more, but at this point I doubt rts/cts will help us, only will burn airtime.

10. I agree, multicast helper makes situation worse.
 
UpRunTech
Member Candidate
Member Candidate
Posts: 209
Joined: Fri Jul 27, 2012 12:11 pm

Re: My experience and issues in hi-density networks at school

Tue May 23, 2023 11:35 pm

@maigonis did you hear anything from Mikrotik since the cAPAC stopped working properly recently?
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Tue May 23, 2023 11:57 pm

@maigonis did you hear anything from Mikrotik since the cAPAC stopped working properly recently?
No, not yet. My task was make APs to hang, but its not that easy to do. Now I have quite optimized schools network and i cant let loose all configs, as a result schools learning process will be interrupted. I managed to hang once of APs, but no response from support.
 
gotsprings
Forum Guru
Forum Guru
Posts: 2087
Joined: Mon May 14, 2012 9:30 pm

Re: My experience and issues in hi-density networks at school

Wed May 24, 2023 2:05 pm

As I state all the time...
I have an email from Mikrotik support saying they know about the density problem. And will get back to me when they have a fix.

It's dated 06/10/2019
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Wed May 24, 2023 3:58 pm

As I state all the time...
I have an email from Mikrotik support saying they know about the density problem. And will get back to me when they have a fix.

It's dated 06/10/2019
It is not density issue, its stations not sending graceful disconnect. It mostly happens in dense networks, yes, but it affects also home users. Just more rarely.
 
gotsprings
Forum Guru
Forum Guru
Posts: 2087
Joined: Mon May 14, 2012 9:30 pm

Re: My experience and issues in hi-density networks at school

Wed May 24, 2023 4:09 pm

The problem occurs a lot in areas where you have a lot of access points from other networks competing for air time.

So in my residential installs... Mikrotik WIRELESS had to be relegated to customers in the middle of nowhere with low client counts and not needing particularly good speed or connectivity.

That's pretty bad.
 
User avatar
andyhenckel
newbie
Posts: 27
Joined: Sun Apr 08, 2018 12:56 am
Location: Missoula, MT

Re: My experience and issues in hi-density networks at school

Thu May 25, 2023 6:39 pm

I've got an apartment complex we manage. It's got 65 units on 4 floors. We installed Mikrotik AP for each unit, hapAC2. I came up with a channel plan, that put adjacent rooms - Right left top bottom and across the hall on different channels to minimize interference and came up with a scheme to apply to each unit. Then the same pattern is staggered slightly for the room across the hall. And staggered, each floor, so Ideally there are very few if any overlapping channels. Seems to work. Here's the config for each unit.

A, B, C, D were the patterns
Attached are the templates we used for each that seemed to work, and allowed up to 250mbit on the 5 G interface. Had to go into each unit and actually label the apartment number, but that's it.

For what it's worth, we don't have any complaints about the wifi at this building. and I've had the experience downtown where wifi won't work more than 5 ft from and AP due to everyone in the building using the same channels with high powered devices.

So with the first apartments, the one across the hall would be on a different channel, and the ones that are adjacent to each are on two different channels. Then the apartments below would be staggered, and the apartments above on the same staggered pattern.
You do not have the required permissions to view the files attached to this post.
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Thu May 25, 2023 7:21 pm

I've got an apartment complex we manage. It's got 65 units on 4 floors. We installed Mikrotik AP for each unit, hapAC2. I came up with a channel plan, that put adjacent rooms - Right left top bottom and across the hall on different channels to minimize interference and came up with a scheme to apply to each unit. Then the same pattern is staggered slightly for the room across the hall. And staggered, each floor, so Ideally there are very few if any overlapping channels. Seems to work. Here's the config for each unit.

A, B, C, D were the patterns
Attached are the templates we used for each that seemed to work, and allowed up to 250mbit on the 5 G interface. Had to go into each unit and actually label the apartment number, but that's it.

For what it's worth, we don't have any complaints about the wifi at this building. and I've had the experience downtown where wifi won't work more than 5 ft from and AP due to everyone in the building using the same channels with high powered devices.

So with the first apartments, the one across the hall would be on a different channel, and the ones that are adjacent to each are on two different channels. Then the apartments below would be staggered, and the apartments above on the same staggered pattern.
General note - if no one complains that does not mean there is no issues. WiFi quality can be quite bad, but if users requirements are met he does not complain (for example LTE link quality can be really bad, but if you can open Instagram and TikTok you are fine). So monitoring and more detailed testing is recommended.
 
gotsprings
Forum Guru
Forum Guru
Posts: 2087
Joined: Mon May 14, 2012 9:30 pm

Re: My experience and issues in hi-density networks at school

Fri May 26, 2023 12:07 pm

Andyhenckle

Since giving up and moving back to the other vendor...

On our MDU systems there is one SSID per property. And customers are given one password each. That password is tied to their VLAN for their room/unit. Each unit has an on wall WAP with a 4 port switch.

A customer from Building 1 can be clear over at parking lot 1. Yet they might as well be in their unit.
 
seriquiti
just joined
Posts: 19
Joined: Wed May 11, 2022 12:55 pm

Re: My experience and issues in hi-density networks at school

Mon Jun 05, 2023 7:50 am

After a client (School) who had Mikrotik AP's contacted me about intermittent wifi speed I did some investigating and testing.

The largest speed killer I found was the SFQ interface queue set on the wireless interfaces. All it takes is one client with bad signal trying to communicate and then all the other connections' speed is dropped to it's level. Prefered to set it to none.

AMSDU should also bet set much smaller than the default depending on how noisy the environment is.

Further than that is the same optimizations you would do with other brand PA's. Channel width, manual frequencies etc.

Actually with these options set Mikrotik's AP performance isn't bad. These AP's are CAP AC's and considering that they are SU-MIMO the performance is what you would expect.
 
Rox169
Member
Member
Posts: 432
Joined: Sat Sep 04, 2021 1:47 am

Re: My experience and issues in hi-density networks at school

Mon Jun 05, 2023 8:12 am

After a client (School) who had Mikrotik AP's contacted me about intermittent wifi speed I did some investigating and testing.

The largest speed killer I found was the SFQ interface queue set on the wireless interfaces. All it takes is one client with bad signal trying to communicate and then all the other connections' speed is dropped to it's level. Prefered to set it to none.

AMSDU should also bet set much smaller than the default depending on how noisy the environment is.

Further than that is the same optimizations you would do with other brand PA's. Channel width, manual frequencies etc.

Actually with these options set Mikrotik's AP performance isn't bad. These AP's are CAP AC's and considering that they are SU-MIMO the performance is what you would expect.
Hi,
how many AP and client has the school? What is the main router?

Tnx
 
seriquiti
just joined
Posts: 19
Joined: Wed May 11, 2022 12:55 pm

Re: My experience and issues in hi-density networks at school

Tue Jun 06, 2023 6:31 am

65 AP's and router is RB5009.

1400 clients peak.
 
gotsprings
Forum Guru
Forum Guru
Posts: 2087
Joined: Mon May 14, 2012 9:30 pm

Re: My experience and issues in hi-density networks at school

Wed Jun 07, 2023 5:47 am

65 AP's and router is RB5009.

1400 clients peak.
Wow... Hope you have a lot of thick walls between WAPs.

So a WAP carries about 22 clients?

What happens when you gather all the kids in the auditorium or similar?
Do you have more WAPs in those common rooms?
 
seriquiti
just joined
Posts: 19
Joined: Wed May 11, 2022 12:55 pm

Re: My experience and issues in hi-density networks at school

Wed Jun 07, 2023 6:30 am

Concrete / Brick walls. They only care about wifi for learners in classes.
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school

Wed Jun 07, 2023 2:09 pm

Concrete / Brick walls. They only care about wifi for learners in classes.
I recommend deploying APs in whole building, for example in halls. It will eliminate issue when stations connect to AP thru walls. If a station connects in these conditions then this will accelerate "graceful disconnect" issue and drag that APs performance down in general, because of bad modulations and TX retries. Yes, ACL whit RX limit do help, but if station count is large that does this you will still notice it. Airtime in classroom env is precious, so you have to use it wisely.

PS: ~20 device count per AP is quite good number (assuming 5ghz ac band is used). In throughput it means that every station gets around 3.5 to 7 mbps.
 
anuser
Long time Member
Long time Member
Posts: 601
Joined: Sat Nov 29, 2014 7:27 pm

Re: My experience and issues in hi-density networks at school

Sun Dec 17, 2023 9:41 am

What ist the current state of airtime fairness feature in wifiwave2 packages. Is it enabled or disabled for all chipsets (older IPQ40xx and new IPQ60xx)?
 
gotsprings
Forum Guru
Forum Guru
Posts: 2087
Joined: Mon May 14, 2012 9:30 pm

Re: My experience and issues in hi-density networks at school

Sun Dec 17, 2023 3:12 pm

I think they take it out of beta this week.
 
maigonis
Member Candidate
Member Candidate
Topic Author
Posts: 180
Joined: Sat Jul 20, 2019 8:16 pm

Re: My experience and issues in hi-density networks at school  [SOLVED]

Sun Dec 17, 2023 9:29 pm

I don't know about air time fairness, but here is update about this situation after wave2 implementation on ipq40xx devices. Quote from one of my posts under version announcements:
Wave2 stability, performance and feature set is now in line whit hardware capabilities, better late than newer. Since first beta I'm testing (carefully whit understanding what I'm doing and having backup plan) ac wave2 in almost 900 students school, we daily have 300-400 STAs connected. WPA3 can and is causing issues, so I recommend starting upgrade whit "legacy security profile" - WPA2 only, MFP off, to be sure and learn how wave2 works as it brings changes.

To summarize my findings: There is no more "graceful disconnect issue", this was major stability issue, mostly noticeable in larger networks. Performance wise 40mhz channel can push around 140-160mbps under load, tested whit 32 ipads updating at the same time. All STAs that do support 2x2 will connect 2x2, and modulations are holding great (will depend on your env and deployment). Also noticed noticeably lower latency/jitter, network in general is more responsive. Wave2 features (802.11kvrw, beam forming, WPA3 etc) bring even more improvements, mostly noticeable in hi density (again) networks, but also smaller networks will see benefits, especially WPA3. As a feature request I would like to see channel utilization stats in capsman, also be able to collect them whit SNMP, so I can see actual ch usage over time. Also radar detect still needs improvement (Mediatek MT7921au based wifi cards still trigger DFS event almost on every interface toggle, also ax lineup), so for now I cant set Latvia as country, again US works like it should, but incorrect country is causing issues, so not recommended. Rate altering would be great, to cut BPSK and QPSK for example, again, useful in hi dense networks, so I can leave ACL RX limits alone (mostly).

Really happy too see wave2 on ac (arm at least) lineup, it brings a lot of stability, performance and feature improvements. To be clear, till stable is released I don't recommend updating larger networks if you don't monitor them and/or don't have knowledge how to do so, but this brings those device further. They now do work whit the same capsman, so you can mix whit ax lineup, even if you are in transition phase.

Great job boys/girls.
Im marking topic solved, but will try update if something meaningful will come up.

Who is online

Users browsing this forum: 4z1st3n, Amazon [Bot], gigabyte091, Kanzler and 23 guests