Wifiwave2 CAPsMAN - full speed on the LAN but internet downloads stall around 150 Mbit

I've been chasing this one for a while now and I'm out of ideas, so I'm hoping
someone here has run into it before.

The short version: on my wifiwave2 CAPsMAN setup any wireless client only gets
about 150 Mbit down from the internet. But that same client, over the same wifi,
does 320-500 Mbit to a machine on my LAN. And a wired PC on the same switch as
the AP gets 500-850 Mbit to the internet. So the wifi radio itself is clearly
fine, the wired/switch/router path is fine, but wifi + internet together tops out
around 150.

My setup:

  • Controller is a hAP ax S (wifi-mediatek), RouterOS 7.23.2
  • Plus one more hAP ax S and two wAP ax (wifi-qcom), all on 7.23.2
  • The slowness happens on both the mediatek and the qcom APs, so it doesn't look
    chipset specific
  • Routing, DHCP and NAT are all done by a separate box (a FRITZ 7590 with its
    wifi turned off). The MikroTiks are pure bridges, no routing on them at all
  • CAPsMAN is local forwarding (datapath bridge = the bridge). I double checked
    with the interface counters: during a client download the controller's ethernet
    carries basically nothing, so the traffic really does go AP -> switch -> router
    directly, it is not tunneled back to the controller
  • 1 Gbit fibre. Sanitized export is attached

The numbers, all taken on the same client at the same moment:

  • wifi client to a LAN host (iperf3, ~2ms RTT): 320-500 Mbit
  • wifi client to the internet (iperf3 to a public server, and plain curl): ~150,
    bouncing anywhere between 60 and 290
  • wired PC on the same switch to the internet: 500-850
  • wired PC straight into the router: ~810

What I can actually see going wrong: during a slow wifi-to-internet download the
TCP connection keeps hitting retransmit timeouts. nstat shows 7 TCPTimeouts and a
few TCPLostRetransmit in a 12 second transfer, but zero out-of-order packets, and
ssthresh stays high with basically no real retransmits. So it isn't congestion and
it isn't reordering, the flow just stalls. On the LAN transfer I actually see about
2% out-of-order but it recovers instantly (2ms RTT) and runs at full speed. The
stalls only bite once the RTT is internet-sized. Ping during the download shows 0%
loss, latency just climbs a bit (some bufferbloat), and there are no scan or roam
events happening during the transfer.

The reason I'm posting rather than just trying things is that I've already ruled
out most of the usual suspects:

  • Channel width: went from 80 to 160 MHz, PHY rate doubled to 1441 Mbit, internet
    throughput stayed at 150
  • Channel/interference: tried ch36 and a clean DFS ch52, identical result
  • RouterOS version: was on 7.20.6, upgraded to 7.23.2, no change
  • AP CPU: about 38% peak under load (2 cores)
  • Wired path, switch, and the switch-to-router uplink: a wired client behind the
    same switch gets 500-850, so none of that is the limit
  • Router and WAN: wired straight in gets 810
  • CAPsMAN forwarding mode: it's local, and the controller carries 0 bytes of
    client traffic during a download
  • MTU: 1500 end to end, DF pings pass, MSS negotiates 1448, no black hole
  • WMM/DSCP: LAN throughput is ~400 at every DSCP value I tested, so it isn't
    aggregation being disabled per priority
  • RTT on its own: I added 50ms of netem delay on the LAN path and throughput did
    NOT drop, so it's genuinely the stalls and not just RTT sensitivity
  • Client hardware: same behaviour on a Realtek laptop and a Galaxy S24FE
  • Client side TCP tuning: BBR vs cubic made no difference, a 32MB receive window
    made no difference
  • IGMP snooping: off on all bridges
  • Multicast flooding is negligible, no leftover queues, use-ip-firewall is no

For context this was a FRITZ mesh before and it did 500-900 to the internet over
wifi, so the line and the WAN can clearly do it, something about the MikroTik wifi
specifically brought this on.

So my actual question: what in wifiwave2 would make a client's downstream TCP
stall/time out only on the higher-RTT internet flows, while handing out full speed
on the low-RTT LAN flows? Is there a known airtime scheduling, power save or
aggregation behaviour at play here, or some setting I should be looking at?

Queues?

Tried it — wifi interfaces use SFQ (wireless-default), swapped to fq-codel on the client's radio, zero change in WiFi→internet throughput (both ~150 Mbit, vs 500 on WiFi→LAN and 850 wired).

Ether1 involved in the path that is gong slow?

The hap Ax S has been reported as having a lot of issues with ether1, often even in a "routing" setup (that you don't have) using ether2 as WAN came out faster than using ether1.

I thought that all issues with ether1 were solved in recent RoS updates, but maybe this is a peculiar case not yet solved.

I'm seeing the issue on both the hAP's and the wAPs, all are wired to switches that lead back to the router, no traffic from one AP travels over the another.

I'm not seeing the issue when using WiFi to access systems and data on the local LAN, only when traversing to the internet.

for certainty, i tested it with the hAP's. didn't help unfortunately

Hi, having the same exact issue, but no CAPsMAN. 2500mbit network here, so WAN is not limited, and router at the other end is a RB5009. AP is a hax S.

A local iperf (to a machine two hops away) shows the same retransmits you're experiencing.

$ iperf3 -c 192.168.90.2 --bidir
Connecting to host 192.168.90.2, port 5201
[ 5] local 192.168.70.100 port 53670 connected to 192.168.90.2 port 5201
[ 7] local 192.168.70.100 port 53674 connected to 192.168.90.2 port 5201
[ ID][Role] Interval Transfer Bitrate Retr Cwnd
[ 5][TX-C] 0.00-1.00 sec 19.2 MBytes 161 Mbits/sec 83 72.1 KBytes
[ 7][RX-C] 0.00-1.00 sec 23.0 MBytes 193 Mbits/sec
[ 5][TX-C] 1.00-2.00 sec 20.1 MBytes 169 Mbits/sec 14 127 KBytes
[ 7][RX-C] 1.00-2.00 sec 34.1 MBytes 286 Mbits/sec
[ 5][TX-C] 2.00-3.00 sec 18.5 MBytes 155 Mbits/sec 15 116 KBytes
[ 7][RX-C] 2.00-3.00 sec 36.5 MBytes 306 Mbits/sec
[ 5][TX-C] 3.00-4.00 sec 19.4 MBytes 163 Mbits/sec 0 208 KBytes
[ 7][RX-C] 3.00-4.00 sec 39.2 MBytes 329 Mbits/sec
[ 5][TX-C] 4.00-5.00 sec 18.9 MBytes 158 Mbits/sec 7 219 KBytes
[ 7][RX-C] 4.00-5.00 sec 41.4 MBytes 347 Mbits/sec
[ 5][TX-C] 5.00-6.00 sec 23.0 MBytes 193 Mbits/sec 4 216 KBytes
[ 7][RX-C] 5.00-6.00 sec 39.5 MBytes 331 Mbits/sec
[ 5][TX-C] 6.00-7.00 sec 23.4 MBytes 196 Mbits/sec 66 202 KBytes
[ 7][RX-C] 6.00-7.00 sec 40.2 MBytes 338 Mbits/sec
[ 5][TX-C] 7.00-8.00 sec 18.1 MBytes 152 Mbits/sec 10 140 KBytes
[ 7][RX-C] 7.00-8.00 sec 42.1 MBytes 353 Mbits/sec
[ 5][TX-C] 8.00-9.00 sec 20.1 MBytes 169 Mbits/sec 10 156 KBytes
[ 7][RX-C] 8.00-9.00 sec 39.8 MBytes 333 Mbits/sec
[ 5][TX-C] 9.00-10.00 sec 18.4 MBytes 154 Mbits/sec 8 148 KBytes
[ 7][RX-C] 9.00-10.00 sec 34.6 MBytes 290 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval Transfer Bitrate Retr
[ 5][TX-C] 0.00-10.00 sec 199 MBytes 167 Mbits/sec 217 sender
[ 5][TX-C] 0.00-10.00 sec 198 MBytes 166 Mbits/sec receiver
[ 7][RX-C] 0.00-10.00 sec 373 MBytes 313 Mbits/sec 157 sender
[ 7][RX-C] 0.00-10.00 sec 370 MBytes 311 Mbits/sec receiver

But what worries me is the cpu usage during the test: CPU is pegged at basically 100%. Usage drops to 0% when traffic stops flowing.

On the interfaces listing, all traffic is flowing directly from the wan port to the wireless interface, with barely any traffic on the bridge.

I don't know if this is supposed to happen, but it's weird as i'd expect the bridge to carry the sum of the packets flowing through all the ports that are in it.

Thanks — your response is valuable: same stall with no CAPsMAN rules CAPsMAN
out entirely (matches my side: the controller's own radios are configured
locally, not via CAPsMAN provisioning, and they stall identically).

I checked your CPU load on my wAP ax (2 cores @ 800 MHz), sampling
/system/resource/cpu once per second during the transfers:

  • idle: ~3%
  • internet download, stalled at 162-183 Mbit: peak 66% / 57% per core
  • LAN download over the same AP-TX air path at 351 Mbit: peak 58%

So at least on my hardware the stall is not CPU: no core gets anywhere near
100%, and the transfer that runs 2x faster actually uses slightly less CPU.
Your 100% peg may be a real ceiling of the ~480 Mbit aggregate your --bidir
test pushes, but that looks like a separate effect from the low-rate stall —
worth re-testing each direction separately, since --bidir on WiFi makes the
two flows fight for airtime and inflates retransmits on its own.

Two smaller notes:

  • The "barely any traffic on the bridge" observation is expected, not a fault:
    the bridge interface counter only counts traffic to/from the CPU (L3) port;
    L2-forwarded traffic between ports shows up only on the port counters.
  • Your retransmit pattern (cwnd collapsing to ~70-220 KB with bursts of Retr)
    is the same TCP-RTO signature I'm seeing; on my side nstat shows it's
    timeouts, not loss or reordering.

The reason for low TCP throughput are retransmissions (as shown by iperf3). Retransmissions really kill TCP performance. And they are due to dropped packets.

Now what causes retransmissions? There are several possibilities, one is processing stalls (either on AP or router) and other is "lossy" link somewhere (either radio ... but normally WiFi's own RF algorithms should ensure loss-less transport ... which might end up in occasional longer delay ... or wired link, can also be due to queues filling up and in this case properly working flow control should help).

It is hard, but not impossible, to determine actual place where those drops occur. One has to sniff traffic off different ports (wireless on end station, wireless/bridge/ether port on AP, ether port on switch/router, WAN port on router). TCP retransmissions do carry information about which TCP packets have to be retransmitted, so it's fairly easy to check different "sniffs" to see where they stop to be present.

Took me a moment to get this tested and ended up somewhere I didn't expect, so this got long. Short
version: it isn't retransmissions, and I think I've found what it actually is. The TCP ACK
stream going back over the radio is costing me 3.5x on downloads.

First, the retransmission question:

Measured on a wAP ax, client on 5 GHz at -60 dBm, HE MCS 8 2SS 80 MHz. Same HTTP source
every run, one direction only, 20 seconds each.

WiFi, single stream        93 Mbit    TcpRetransSegs = 1  in 245 MB
WiFi, 8 parallel streams  163 Mbit    TcpRetransSegs = 2  in 390 MB
Wired host, 8 streams     906 Mbit

AP side during the transfer: wifi2 tx-drop = 0, tx-error = 0. Client side rx_drop stayed at
its idle baseline. Nothing incrementing a drop counter anywhere.

Then the part that convinced me: an iperf3 run from a wired host to the same wireless client,
over the same AP, without the internet leg, gave 389 Mbit with 531 retransmits over 709 MB.
That run is 2.4x faster than the internet run and has roughly 50 times the retransmission
rate. More retransmits, more throughput. So loss isn't what's holding this back.

I also tried the sniffing approach you suggested, and it failed in a way worth mentioning.
Sniffer on the AP's ether2, headers only, filtered to the one flow, written to the AP's own
flash rather than streamed over TZSP. 250 MB is about 172k segments; the AP captured 30.7k
and the client 20.1k. The wAP ax's 2x800 MHz can't sniff at ~12k pps, and GRO on the client
was coalescing segments before tcpdump saw them. Two partial samples, so comparing sequence
numbers between them proves nothing. If anyone repeats this: turn off GRO/LRO on the client
(ethtool -K gro off lro off) and mirror the AP's switch port rather than sniffing on the
AP itself.

What it actually is:

I wired the laptop to the same switch the AP is on, to get a proper same-switch control. That
test tripped over the answer by accident: curl --interface binds the source address, not
the egress device, so with the wired NIC up at a lower metric the download data still came
back over WiFi while the ACKs went out over the cable. That ran at 530 Mbit instead of 151.

So I did it deliberately. 8 streams, 20 seconds, same URL, same minute:

data path   ACK path   throughput   retrans / timeouts
WiFi        WiFi        151 Mbit      24 / 43
WiFi        wired       539 Mbit       2 /  2
wired       wired       938 Mbit      14 /  6

Per-interface counters for the middle run, which is the whole case:

wlan0:  rx 1286 MB / 891545 pkts    tx 0 MB /      5 pkts
eno1 :  rx    3 MB /   3008 pkts    tx 9 MB / 146817 pkts

Every one of those 1286 MB crossed the radio. The only thing that moved to the wire was about
147k tiny ACKs. That alone is worth 3.5x.

Two more that line up with it:

  • UDP downstream over pure WiFi, no ACK stream at all: 363 Mbit delivered (600 offered, 39%
    loss, so it's over-driven, but the air plainly carries a lot more than 151).
  • TCP downstream to a LAN host over pure WiFi, so ACKs still over the radio but only ~2 ms
    RTT: 432 Mbit.

That last one is the interesting bit. ACKs over WiFi are survivable at LAN RTT and fatal at
internet RTT. Which also explains the thing that had me stuck for weeks, that only the
combination of WiFi and internet is slow while WiFi to LAN and wired to internet are both
fine. It was never the downlink. The AP will happily push 539 Mbit to this client.

My reading is that under half duplex contention the client's ACK stream is getting delayed
and compressed, and TCP absorbs that at 2 ms but not at 15 ms, where it turns into RTO
timeouts (43 of them in the pure WiFi run, 2 in the split run). Some ACK penalty is normal on
WiFi, everyone lives with 10-20%, but 3.5x is not normal. That smells like uplink scheduling
for small frames, UL OFDMA or EDCA or TXOP handling. I can't find anything exposed in
/interface/wifi/configuration to test any of that.

Some earlier evidence, now explained:

The receiver's ss output during a slow transfer showed rcv_rtt at 50-63 ms against a true
path minrtt of 13.5-22 ms, so data arriving in bursts with gaps rather than steadily. I'd
been reading that as a transmit side pacing problem in the AP. It's the downstream symptom of
the ACK stalls, not a separate fault.

Also worth flagging for anyone reading ss output on a download: the cwnd:10 you'll see is the
send side congestion window of a socket that only ever emits ACKs (data_segs_out:1), so it's
meaningless in that direction. I cited it earlier in this thread as evidence and I was wrong.

It also explains why none of the loss oriented fixes did anything. Interface queue type made
no difference at all (SFQ vs fq-codel, 80x deeper buffer with AQM, 116-175 vs 148-174 Mbit).
BBR vs cubic, nothing. 32 MB rwnd, nothing. AMPDU, WMM and DSCP, nothing. There was nothing
to fix at the loss layer because nothing was being lost.

Follow up question:

Is there any way to influence uplink scheduling or EDCA/TXOP parameters for small frames on
wifiwave2, or any counter that would show ACK frames queueing on the client-to-AP side? Happy
to run whatever would help narrow it down.

Well, retransmissions and RTT combined govern the TCP throughput ... and that's via TCP Window Size.

TCP Window Size should be large enough to hold all on-the-fly packets ... and with target rate of 900Mbps that's, say, 115kB with RTT of 1ms and proportionally larger with higher RTTs. Default starting TCP window size is slightly different for different OSes, for linux it seems to be around 16kB and for Windows and MacOS it's 64kB. Too small TCP window size means that sender is waiting for ACKs to come back before transmitting more segments ... and that will limit throughput in case of "long" RTTs.

With successful transport of data OSes start to enlarge TCP window size ... so throughput will increase. However, any requested retransmission ("seemingly" dropped packet ... which can mean either truly dropped packet or out-of-order delivery outside OOO window) will shrink TCP window size (depending on TCP stack behaviour even down to default size) which in turn will drop throughput.
Enlarging TCP window size happens with each successfully ACKed segment ... so with shorter RTT even that happens faster. And in case of a dropped packet (which shrinks TCP window) it tends to recover faster ... where "faster" is measured in seconds, not in segments ... but when measuring throughput it's seconds that matter anyway.

Sometimes enlarging TCP window size (which is entirely up to sending party) seems very slow, it might take many seconds to reach window size needed to reach full speed, even (default) 10 second iperf3 test might be too short ... and if that happens, one has to look at results in the last part of test, not the whole test average.

And yes, when working with internet servers it's usual that RTT is longer than on LAN and if throughput is RTT limited, then lower test results are to be expected.

Thanks, that's a fair thing to check and it's easy to measure, so I went and looked at the
actual window during a slow transfer.

I ran a single stream pure WiFi download for 60 seconds (wired NIC down so nothing leaks onto
the cable), sampled throughput every 5 seconds and read rcv_wnd off the socket while it ran:

t=5..25s    160-198 Mbit    rcv_wnd 3.15 MB
t=30..50s   122-196 Mbit    rcv_wnd 5.28 MB

So the receive window autotunes up to 3.15 then 5.28 MB. Path RTT here is about 14 ms
(minrtt 11.8-13.8). A 5 MB window at 14 ms would support somewhere around 2.8 Gbit if the
window were the thing holding it back. It's 5 to 10 times larger than what 900 Mbit needs, and
the transfer still sits at ~180. The window is wide open and the throughput is still capped.

I'd actually tested this from the other direction earlier too: forcing the receive buffer to
32 MB made no difference. Same result, and now I know why, the window was never the bottleneck.

On the "10 second test might be too short, look at the tail" point, fair, so I watched the
whole 60 s. It plateaus by the 5 second mark and stays flat the rest of the way (196 Mbit at
t=50). It isn't still ramping. (The last two samples read zero only because the 1 GB file
finished around 52 s.)

Three other things that point the same way:

  1. Parallel streams don't help. One stream is ~151, eight streams is ~163. A window or BDP
    limit is per flow, so eight flows each get their own window and the aggregate should climb.
    It basically doesn't move, which says the ceiling is a shared resource (airtime), not a per
    flow window.

  2. Same RTT, wired, is 938 Mbit. The wired control to the same server runs at the same
    internet RTT with the same OS and the same autotuning, and hits 938. So the OS window
    demonstrably does grow enough for 900 Mbit at this RTT. "Internet RTT is longer so expect
    less" can't account for a 6x gap when the RTT is identical and only the medium changed.

  3. The split path test from before. Data over WiFi but ACKs pushed out over the wire went from
    151 to 539 with the same window, same RTT, same endpoints. The only variable was which
    medium carried the ACKs.

On the retransmission shrinking the window part, that's real and I'm not disputing the
mechanism in general. But on my slow runs the retransmit count is basically zero, so cwnd
isn't being cut by loss. What the receiver shows instead is bursty arrival, rcv_rtt runs
50-84 ms against a true minrtt of ~14 ms, so data is coming in clumps with gaps. Combined with
the wired-ACK test that fixes it, my read is that the sender can't clock out data because the
ACKs are arriving late and bunched under airtime contention. That's an airtime scheduling
effect, not a window sizing one, which is also why none of the buffer and window tuning I threw
at it moved the needle.

Happy to run anything else that would help pin it down.

Regarding follow-up question: I don't think there's any special handling of ACK packets in wifi driver. To L2 (which is layer where WiFi drivers work), ACKs seem as very usual (albeit short) ethernet frames. It might be possible to play with queues which would treat short frames preferentially ... but it's this exact behaviour (ACKs getting stuck behind larger packets) which warrants running queues against large buffers (existing in all sorts of equipment) - which is sometimes referred to as "buffer bloat". As always, queues work well on transmitting side (so on laptop itself) and have obscure effect when queues are set on receiving side (and only work when target throughputs are below actually available throughputs).

Can't say anything about your test runs (you didn't post iperf3 output), but user @unfoxo posted his iperf3 output and there are some retransmissions. And that's what triggered my posts about retransmissions and window sizes.

As to your particular test case: you can try to run iperf3 tests in UDP mode ... which will entirely avoid the ACK and RTT issue. In my experience one has to be careful about bandwidth setting ... setting it only slightly above whatever throughput is then achieved shows maximum throughputs. Setting it way above whatever realistic results will drop throughput (possibly due to intermediate devices being very busy dropping excessive packets). And in case of UDP tests one must check the report on receiving side, sending side will have only limited (or next to none) visibility to actual test behaviour.

Also perform tests in both directions, it might reveal that WiFi connection is not as symmetrical as one would want (and/or expect).

Done more tests.

Speed test / Bandwidth test, TCP+UDP between Router <-> AP (cable) = full 940mbit symmetric, except when doing both directions simultaneouslt, in that case the limit is CPU on AP (but BW is still ~800mbit on both directions).

Test between AP and my computer. I used btest-rs on my laptop to run a Mikrotik-compatible server ( https://git.manko.yoga/manawenuz/btest-rs/ ).

Results (as seen from the AP)

TCP Download: 335Mbps local-cpu-load:85%
TCP Upload: 54.2Mbps local-cpu-load:90% remote-cpu-load:6%
UDP Download: 54.9Mbps local-cpu-load:7% remote-cpu-load:6%
UDP Upload: 421Mbps local-cpu-load:97% remote-cpu-load:11%

Fair point on the iperf3 output. So I ran your suggested tests: UDP to take ACKs and RTT out of it, both directions,
watching the receiving side, and calibrating the offered rate rather than just blasting.

All of these are pure WiFi to a wired LAN peer, so ~2 ms RTT, no internet leg. Client
is a wAP ax, 5 GHz, -56 dBm, HE MCS 8-9 2SS 80 MHz. AP-TX means the AP transmitting to the
client (the download direction), AP-RX is the client transmitting.

TCP, 4 streams, 15 s:

AP-RX  wifi client -> wired client   [SUM] 564 Mbit sender / 561 receiver, 1813 retr
AP-TX  wired client  -> wifi client  [SUM] 485 Mbit sender / 478 receiver, 1239 retr

UDP AP-TX (wired client -> wifi client, receiver-side figures), offered rate stepped:

offered 250M   247 Mbit received   1.1% loss
offered 300M   288 Mbit received   3.9% loss
offered 350M   327 Mbit received   6.6% loss
offered 400M   368 Mbit received   7.9% loss

A couple of things fall out of this.

The air is not the limit. Over the exact same WiFi hop, in the same AP-TX direction as the
slow download, UDP with no ACK and no RTT delivers 247 Mbit at ~1% loss and keeps climbing to
368 as I push harder. TCP to the LAN peer does 478. The internet download over that same hop
sits at ~150. So the medium comfortably carries 2.5 to 3 times the internet number the moment
you remove the ACK/RTT dependency. That's using your own method, and it points at the same
place I'd landed: it's not raw air capacity and it's not loss.

There is a real asymmetry, as you suspected, though it's the opposite of dramatic: AP-TX 478
vs AP-RX 561 on TCP. The transmit (download) side is the weaker one, which lines up with the
download being the direction that suffers.

On retransmissions again, worth noting these fast LAN runs carry 1200-1800 retransmits each
and still do 478-561 Mbit, while the slow ~150 Mbit internet download carries essentially none.
So in this setup retransmit count and throughput are if anything inversely related, which is
why I don't think retransmissions are the driver here.

One more observation I can't fully nail down but that's suggestive. UDP in the AP-TX direction
(server flooding the client) reports cleanly every time. UDP in the AP-RX direction (client
flooding the server) reliably stalls at the start for me, the iperf3 control channel never gets
going under the flood. The control channel is small TCP sharing the client's uplink airtime, so
when I saturate that uplink the little control exchange can't get through. That's the same shape
as the core problem: small return-path traffic getting starved when the airtime in that
direction is busy. I'm holding that loosely since iperf3 UDP has its own quirks, but it rhymes
with the ACK-starvation behaviour.

Net of it: UDP confirms the air is fine, the asymmetry is real and on the transmit side, and
the thing that actually collapses throughput is the high-RTT ACK dependency, not loss or
capacity. Still very interested if there's any wifiwave2 knob or counter for uplink/return
scheduling.

Running bandwidth test on MT devices is most often the limiting factor as it consumes lots of CPU power which would otherwise be used (or even required) to process traffic. The only "excuse" for running btest is if one runs a (very beefy) CHR.

Experience is that whenever router/AP CPU load exceeds 50% (give or take, also depends on individual CPU core utilization) its overall performance starts to drop.

It's always recommended to run tests "through" tested devices ... which means on two "normal" computers (where one is actually testing the link in question and one is connected beyond the tested link but with connection which by all means known to admins will not be a bottle neck). In this particular case this means one PC using WiFi connection to AP and another computer connected using wired connection to same AP ... and run iperf3 test between those two computers.

This is the test as seen from the RB5009 router. Path is RB5009 -> Ap AX s -> My laptop

Same exact result.

TCP Download: 541Mbps local-cpu-load:39%
TCP Upload: 36.0Mbps local-cpu-load:33% remote-cpu-load:6%
UDP Download: 73.5Mbps local-cpu-load:19% remote-cpu-load:6%
UDP Upload: 389Mbps local-cpu-load:39% remote-cpu-load:9%

During the test, the hAP cpu usage was still high (although not as high as when the test was running on the AP itself):

Columns: NAME, USAGENAMEUSAGE
networking 4.5%
interface-mgmt 0.2%
ethernet 0.2%
bridging 9.7%
console 0%
routing 0%
ipc 0%
wireless 1%
profiling 0.2%
dns 0%
kernel 10.2%
mediatek_switch 0%
ecnt_eth 5.7%
mac80211 21%
mt76 20.5%
mt76_connac_lib 2.2%
mt7915e 7%
led 0.2%
www 0.2%
total 82.8%

Do I understand you correctly that AP-RX means station is trying to push traffic and AP is receiving it (over wireless)? Which would indicate that it's laptop which is behaving weirdly ...

Another thing to consider (and I don't know exactly to go around): in half-duplex shared medium environment (which is what WiFi is), excessive traffic in one direction can very much stall the other direction ... which makes sending ACKs even harder. I.e. if laptop has its Tx buffers full, it may aggressively use airtime (and I'm not sure if AP has any control over how its own stations use it), hence making others (AP as well) to wait for their chance to transmit. This doesn't happen in full-duplex (dedicated medium) as each side can always transmit regardless the traffic in opposite direction.
Which is why one has to set limits on queues (which are supposed to throttle traffic in Rx direction) slightly lower than expected maximum speed ... to accommodate any overhead and ACKs are kind of overhead with regard to application payload.

Seems that mediatek WiFi driver (mt76*, mt7915e and mac80211 combined) used quite big proportion of CPU ... which is to be expected as all traffic to/from WiFi interface(s) is handled by CPU (on all MT devices regardless the architecture, chipset and driver used). So it is what it is.

You're right and I was sloppy with the wording. There's no ACK-specific handling at L2, an
ACK is just a short frame to the driver. I shouldn't have implied the driver "knows" it's an
ACK. So I went and tested the bufferbloat angle properly, since that's the concrete version of
what I was hand-waving at.

First, where the client actually queues. On this Linux client the WiFi netdev qdisc is
"noqueue" - there is no classic tc queue on it at all. The real queuing sits below, in
mac80211, which already runs per-station fq_codel plus AQL (airtime queue limits, threshold
24000 here). So the modern transmit-side anti-bufferbloat machinery you're describing is
already present and active on the client by default.

Then the decisive test, latency under load. I saturated the download over WiFi and pinged the
AP (L2, over the same wlan0 TX path the ACKs use), idle vs loaded:

client -> AP RTT (min/avg/max ms)
  idle                       1.82 / 3.31 / 10.3
  under download, noqueue    1.35 / 2.82 / 6.57
  under download, cake       1.44 / 2.86 / 4.62

The client TX path shows no latency inflation under load at all - loaded is if anything
tighter than idle. If ACKs were stuck behind larger frames in a bloated client transmit queue,
that ping shares the queue and would balloon. It doesn't. So there is no client-side
bufferbloat to fix here, which is exactly what you'd expect given mac80211's fq_codel+AQL is
already doing its job.

I also tried your explicit suggestion, cake with ack-filter on the client egress, plus dropping
the AQL threshold to 5000. Neither moved throughput beyond the (large) single-stream noise, and
the latency was already flat so there was nothing for them to improve. ack-filter in particular
has nothing to bite on: with noqueue the ACKs don't pile up in a qdisc, so there are no
redundant ACKs sitting in a queue to thin.

Which I think refines the mechanism rather than contradicting your airtime point. The penalty
isn't ACK queue latency (no bloat, as above), it's ACK airtime cost. WiFi is half duplex,
so every TXOP the client takes to send its ACK stream is airtime the AP isn't using to send
downlink data. Move those ACKs onto the wire and all that airtime goes back to the download,
which is the 151 -> 539 Mbit jump I measured earlier. That's a medium-access cost, not a buffer
sitting somewhere, so no queue on either side can really recover it - the lever is how
efficiently the AP grants the client uplink airtime/TXOPs for the return stream, which puts it
back on the AP/driver side.

The one client-side thing that would genuinely reduce it is sending fewer ACKs (stretch-ACK /
larger GRO so the return stream needs fewer TXOPs). That's a receiver behaviour rather than a
queue, and it treats the symptom. Curious whether you'd expect that to be the only real
client-side lever here, or whether I'm missing one.

btw, i'm not suffering from the CPU load issues @unfoxo is reporting, i think his issue, although
it looks the same, is different when looking closer.