hAp ac2 - large number of packet retransmissions on 2ghz-g/n and 5ghz-n/ac

[kirenida]

Hi!

I’ll try posting this question again, as I had no success the last time i asked - http://forum.mikrotik.com/t/problems-with-streaming-media-over-local-network-after-moving-to-hap-ac2/148659/1

As I thought that my problem is maybe hardware based, I sent back my first hAp and got a new unit.

The problem is back again - when I’m connected to 2ghz-g/n or 5ghz-n/ac, I’m getting ~100 packet retransmission per second when I’m running an iperf3 test.

It’s happening on 2 laptops (lenovo, asus) and 3 different smartphones (old and brand-new Android, new iphone).

I’ve tried practically everything I managed to find online and everything that was suggested in my previous post, but the problem is still here.

This https://pastebin.com/raw/hYkTzNNY is my current configuration.

It doesn’t help to restart to defaults, it doesn’t help to change channels or channel widths. I’ve tried the last 3 stable firmwares and am currently running the latest testing branch.

Can somebody just run the same test (start iperf3 server on a eth-connected machine and run the client on a wifi-connected machine) and tell me how many retransmissions they get?

I’m going slightly mad from this, as I find it unfathomable that in 2021 I can’t even send audio over my wlan without it breaking up, on a brand-new router with the latest software.

thanks!

[anav]

I think the wifi on MT is perfect and its the clients that are the problems.

[anuser]

As you already mentioned in http://forum.mikrotik.com/t/problems-with-streaming-media-over-local-network-after-moving-to-hap-ac2/148659/1

Should I just accept that for some reason, in my use case, the TP-link + OpenWRT solution is just better?

Edit: Yep, just now I setup the TP-link to act as a simple AP, connected it to the hAp and everything works perfectly. 0 retransmissions when testing with iperf3.
My conclusion: hAP ac2 has really bad builtin-wifi since it is unusable for my use case.

There is a OpenWrt port for hAP ac2, aswell: https://forum.openwrt.org/t/support-for-mikrotik-hap-ac2/23333/379
When your setup works with OpenWrt, then RourterOS is the source of the problem.

[bpwl]

Most of us know the limitations of the MT wifi implementation. But I look at it from a very technical point of view. Some claims correspond to those limitations, some don’t seem to be related.

So technically, what do you mean with “large number of packet retransmissions on 2ghz-g/n and 5ghz-n/ac” . What is your measurement point? Data stream (TCP/UDP), or 802.11 retransmits? 802.11 Software retransmits or HW retransmits? Retransmits are part of 802.11 to find the working interface rate, they are not visible to upper layers.

Packet drop is part of a routed IP network. When buffers get full, packets are dropped. TCP congestion avoidance protocols need this drops to react on the congestion. Congestion points appear always with “big pipe - small pipe” transitions. Reaction of TCP on this depends on the TCP congestion avoidance used in the end-points (Tahoe, Reno, Cubic, Compound, …)

The MT wifi implementation, knows what WMM is, but by default (out of the box) everything is only handled as “best effort”. This is not the priority needed for transmitting audio.
Even if you manage to create the firewall mangle rules to set the priority based on DSCP, the MPDU aggregation is disabled in MT by default for all except “best effort”.
If you want to transmit audio with the proper priority, some configuration work is necessary.

Wifi channels, channel width, … whatever you use… the real information is in the “Registration table”. What is actually used? What is the signal strength, CCQ, SNR, interface rate, etc etc … .

A “verbose” export is a very long text. The non-verbose output reduces this to what deviates from the default, and makes it readable. But still, the real information is probably in the life “Registration table”.

[kirenida]

So technically, what do you mean with “large number of packet retransmissions on 2ghz-g/n and 5ghz-n/ac” . What is your measurement point? Data stream (TCP/UDP), or 802.11 retransmits? 802.11 Software retransmits or HW retransmits? Retransmits are part of 802.11 to find the working interface rate, they are not visible to upper layers.

With “large number of packet retransmissions”, I mean that I get more retransmissions when I run iperf3 over 5GHz-n/ac then when running over 5GHz-a or 2GHz-b/g/n

For example:
when connected to 2ghz-b/g/n

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  67.5 MBytes  56.7 Mbits/sec  324             sender
[  5]   0.00-10.00  sec  67.3 MBytes  56.4 Mbits/sec                  receiver

5ghz-a

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  24.1 MBytes  20.2 Mbits/sec    0             sender
[  5]   0.00-10.04  sec  23.0 MBytes  19.2 Mbits/sec                  receiver

5ghz-n/ac

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  93.2 MBytes  78.2 Mbits/sec  1067             sender
[  5]   0.00-10.00  sec  93.0 MBytes  78.0 Mbits/sec                  receiver

And this is what I get when connected using the tp-link router, on 802.11bgn:

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  48.4 MBytes  40.6 Mbits/sec    0             sender
[  5]   0.00-10.02  sec  46.2 MBytes  38.7 Mbits/sec                  receiver

The MT wifi implementation, knows what WMM is, but by default (out of the box) everything is only handled as “best effort”. This is not the priority needed for transmitting audio.
Even if you manage to create the firewall mangle rules to set the priority based on DSCP, the MPDU aggregation is disabled in MT by default for all except “best effort”.
If you want to transmit audio with the proper priority, some configuration work is necessary.

Ok, I understand that. I created the mangle rule

 3    chain=prerouting action=change-dscp new-dscp=from-priority-to-high-3-bits passthrough=yes log=no log-prefix=""

And I have enabled 5, 6 and 7 in AMPDU priorities for the 5GHz interface.

What else can I configure?

Wifi channels, channel width, … whatever you use… the real information is in the “Registration table”. What is actually used? What is the signal strength, CCQ, SNR, interface rate, etc etc … .

Openwrt:

A “verbose” export is a very long text. The non-verbose output reduces this to what deviates from the default, and makes it readable. But still, the real information is probably in the life “Registration table”.

 1  R name="wlan2" mtu=1500 l2mtu=1600 mac-address=08:55:31:FF:A7:EA arp=enabled disable-running-check=no interface-type=IPQ4019 radio-name="085531FFA7EA" 
      mode=ap-bridge ssid="MikroTik5" area="" frequency-mode=regulatory-domain country=norway installation=any antenna-gain=3 frequency=5180 band=5ghz-a/n/ac 
      channel-width=20/40/80mhz-XXXX secondary-frequency="" scan-list=default wireless-protocol=802.11 rate-set=default 
      supported-rates-a/g=6Mbps,9Mbps,12Mbps,18Mbps,24Mbps,36Mbps,48Mbps,54Mbps basic-rates-a/g=6Mbps max-station-count=2007 distance=indoors tx-power-mode=default 
      vlan-mode=no-tag vlan-id=1 wds-mode=disabled wds-default-bridge=none wds-default-cost=100 wds-cost-range=50-150 wds-ignore-ssid=no 
      update-stats-interval=disabled bridge-mode=enabled default-authentication=yes default-forwarding=yes default-ap-tx-limit=0 default-client-tx-limit=0 
      wmm-support=enabled hide-ssid=no security-profile=default wps-mode=disabled station-roaming=disabled disconnect-timeout=3s on-fail-retry-time=100ms 
      preamble-mode=both compression=no allow-sharedkey=no station-bridge-clone-mac=00:00:00:00:00:00 ampdu-priorities=0,5,6,7 guard-interval=any 
      ht-supported-mcs=mcs-0,mcs-1,mcs-2,mcs-3,mcs-4,mcs-5,mcs-6,mcs-7,mcs-8,mcs-9,mcs-10,mcs-11,mcs-12,mcs-13,mcs-14,mcs-15,mcs-16,mcs-17,mcs-18,mcs-19,mcs-20,mcs-
                 21,mcs-22,mcs-23

[bpwl]

OK. Now we can talk technically.

So you retransmits are at the payload level from perf3. I still assume this is a UDP test, as normally TCP would reduce the speed to avoid lost packages (congestion avoidance)

I don’t know the region you are in, but isn’t the “a” for FCC regulated regions, and the mediocre “b” for the rest, like Europe?
Surprised a bid you mixed “a” and “b” tests. No single device near me can communicate in “a”. But maybe some indeed have both modes.

The test results from perf3 are for different setups, but your registration entry is only for one of them.
I would always need both, to come to any conclusion.

So with this partial information it is “possible” to say that: (possible is just one possibility)
In this connection chain end-to-end …

• there is no congestion up to 50Mbps.
• with higher throughput congestion arises, and climbs with the throughput. Location of the congestion, and resulting retransmits, is not known
• taking into account the large overhead of the 801.11 protocol, the wifi throughput seems to perform well. For exact numbers both perf3 and registration data is needed

For the wifi , with only one reading:

• signal seems rather good (there are more numbers/columns, you have all with detailed view)
• but why is the MT TXrate going down to 1S, while there is 2S possible as can be seen with the RXrate. Possible throughput goes down from about 360Mbps to 190Mbps)
• it can be the MT setting causing the drop or the physical position of the AP, or other transmitters (but unfortunately MT tells nothing about the channel itself. Other brands give some indication)
• why is TP-Link not going up till MCS07, or even MCS15 (2S) ? Signal too strong (-34dBm) for the receiver ???

Nothing special in the config, except for the band “b” if used, and the channel width as XXXX (you don’t know what will be used … Ceee,eCee,eeCe,or eeeC)

For WMM, this indeed should help. Check with Snooper if DSCP is set somewhere. See also WMM classes (0+3,1+2,4+5,6+7)
https://www.bintec-elmeg.com/portal/downloadcenter/dateien/workshops/current_en/ws_wlan_html_en_HTML/vowlan_infra_qos_wmm.html
I find this difficult to test. Also MT does not give numbers on aggregations used.
But actually I did not see evidence of a wifi bottleneck.(but as said, MT gives no information on CCA or the access contention process.)
(https://www.extremenetworks.com/extreme-networks-blog/what-is-a-clear-channel-assessment-cca/)

[anuser]

This actually remembers me of a test with cAP ac which has the same IP40xx chipset as the hAP ac2. The test is some years old. We put in an external wireless sniffer device which captured the traffic for some days. According to the report there were strange packet retransmissions on 5 GHz. On the screenshots you see the total data rate and the packet retransmission in %.

• For 2.4GHz

\

• For 5 GHz

[bpwl]

I have seen serious problems in this forum with the ARM hardware for wifi, cases of some years ago. (I just started late enough with MT to not been hit by that issue).

On wifi retransmits the exact numbers are in the “registration table” . Packets versus HW packets, Frames versus HW Frames.
The rate selector is somewhat controllable with the “HW retries” parameter. Default on 7, can go up to 15 to hold on to the highest rate despite retransmissions.
3 would make it more responsive in rate reduction. Lower than 3 probably is not giving decent information to the driver.
CCQ is calculated from those ratios. A failed transmission is when there is no ack received. What part of the handshake failed is unknown however.

As can be found in the wifi FAQ: https://wiki.mikrotik.com/wiki/Manual:Wireless_FAQ#What_is_hw-retries_setting.3F

[kirenida]

I decided to give uo on this issue, as I’ve spent way too much time stressing about it, and nobody needs extra stress at this time.
But, just for the sake of completeness, here is a comparison of 2.ghz performance in my case, on both the MT and TP-link.

Thank you for the comments and information, I have learned something new through this experience.

For clarity, I will be running iperf3 with the default options - “iperf3 -c ”, which uses a TCP connection, as stated in the man page:

The actual test data is sent over a separate TCP connection, as a separate flow of UDP packets, or as an inde‐
pendent SCTP connection, depending on what protocol was specified by the client.

This is the current situation at the time of writing.

TP-link:

while playing a youtube video and sending audio to the raspberry:

while running iperf3, it jumps to 135mbps a few times, but is utually on 72mbps most of the time

iperf3: (full output here)

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec   153 MBytes  42.9 Mbits/sec    1             sender
[  5]   0.00-30.03  sec   151 MBytes  42.2 Mbits/sec                  receiver

hAp AC2 - wlan1 interface reset configuration to defaults, here is the wifi situation with just the MT active

0    name="wlan1" mtu=1500 l2mtu=1600 mac-address=08:55:31:FF:A7:E9 arp=enabled disable-running-check=no interface-type=IPQ4019 radio-name="085531FFA7E9" 
      mode=ap-bridge ssid="MikroTik" area="" frequency-mode=regulatory-domain country=etsi installation=any antenna-gain=3 frequency=2412 band=2ghz-b/g/n 
      channel-width=20mhz secondary-frequency="" scan-list=default wireless-protocol=any rate-set=default supported-rates-b=1Mbps,2Mbps,5.5Mbps,11Mbps 
      supported-rates-a/g=6Mbps,9Mbps,12Mbps,18Mbps,24Mbps,36Mbps,48Mbps,54Mbps basic-rates-b=1Mbps basic-rates-a/g=6Mbps max-station-count=2007 distance=dynamic 
      tx-power-mode=default vlan-mode=no-tag vlan-id=1 wds-mode=disabled wds-default-bridge=none wds-default-cost=100 wds-cost-range=50-150 wds-ignore-ssid=no 
      update-stats-interval=disabled bridge-mode=enabled default-authentication=yes default-forwarding=yes default-ap-tx-limit=0 default-client-tx-limit=0 
      wmm-support=disabled hide-ssid=no security-profile=default wps-mode=push-button station-roaming=disabled disconnect-timeout=3s on-fail-retry-time=100ms 
      preamble-mode=both compression=no allow-sharedkey=no station-bridge-clone-mac=00:00:00:00:00:00 ampdu-priorities=0 guard-interval=any 
      ht-supported-mcs=mcs-0,mcs-1,mcs-2,mcs-3,mcs-4,mcs-5,mcs-6,mcs-7,mcs-8,mcs-9,mcs-10,mcs-11,mcs-12,mcs-13,mcs-14,mcs-15,mcs-16,mcs-17,mcs-18,mcs-19,mcs-20,mcs-
                 21,mcs-22,mcs-23 
      ht-basic-mcs=mcs-0,mcs-1,mcs-2,mcs-3,mcs-4,mcs-5,mcs-6,mcs-7 tx-chains=0,1 rx-chains=0,1 amsdu-limit=8192 amsdu-threshold=8192 tdma-period-size=2 
      nv2-queue-count=2 nv2-qos=default nv2-cell-radius=30 nv2-security=disabled nv2-preshared-key="" nv2-mode=dynamic-downlink nv2-downlink-ratio=50 
      nv2-sync-secret="" hw-retries=7 frame-lifetime=0 adaptive-noise-immunity=none hw-fragmentation-threshold=disabled hw-protection-mode=none 
      hw-protection-threshold=0 frequency-offset=0 rate-selection=advanced multicast-helper=default multicast-buffering=enabled keepalive-frames=enabled 
      skip-dfs-channels=disabled

While playing youtube video:

while running iperf3

iperf3, full output here

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec   206 MBytes  57.7 Mbits/sec  1091             sender
[  5]   0.00-30.00  sec   206 MBytes  57.6 Mbits/sec                  receiver

[bpwl]

MT wifi looks excellent. (The HW retransmits are part of the interface rate selector, and invisible (!) to iperf3.)

The Retr counter here is the TCP retransmit decided by the TCP stack and its congestion avoidance protocol used, and its congestion window, in the clients
Finding what caused these retransmits (which bottleneck or variation of throughput is causing this, why was the ACK too late or the used RTT too low?) is what is needed.
Only if this indicates that the wifi is causing this, only then is analyzing and tuning the wifi of any use. (lower hw-retries, disable highest MCS rates …)

https://www.reddit.com/r/networking/comments/fj4xtt/network_helpretransmitted_packet_count_1718k/
https://www.reddit.com/r/networking/comments/4odfvf/how_does_one_know_where_traffic_is_congested/

[kirenida]

MT wifi looks excellent. (The HW retransmits are part of the interface rate selector, and invisible (!) to iperf3.)

The Retr counter here is the TCP retransmit decided by the TCP stack and its congestion avoidance protocol used, and its congestion window, in the clients
Finding what caused these retransmits (which bottleneck or variation of throughput is causing this, why was the ACK too late or the used RTT too low?) is what is needed.
Only if this indicates that the wifi is causing this, only then is analyzing and tuning the wifi of any use. (lower hw-retries, disable highest MCS rates …)

https://www.reddit.com/r/networking/comments/fj4xtt/network_helpretransmitted_packet_count_1718k/
https://www.reddit.com/r/networking/comments/4odfvf/how_does_one_know_where_traffic_is_congested/

Ok, I understand what you are saying, and I must admit that this is a bit out of my depth at this point.

What is puzzling me now is, if we accept that there is no problem with the wifi, how is it possible that adding another network device (the tp-link) to the path packets must take to reach the destination, actually lowers the number of retransmits?

If the problem is that ACKs are not reaching my laptop in time, then my thinking is that the problem is somewhere in the link between my laptop and the MT, because when the Packet has to travel from laptop to tp-link to MT to raspberry and back, there are no problems?

Is my logic correct? Any ideas where to look? Maybe the queue type for the wireless interface? As I said before, this is a bit out of my depth, so any hints will be helpful.

Thanks!

[anuser]

Well, as mentioned before. Just install OpenWrt on hAP ac2 (https://forum.openwrt.org/t/support-for-mikrotik-hap-ac2/23333/383) and see if those large number of packet retransmissions are still there. If this is not the case, RouterOS is the source of the problem. Don´t forget to backup your license file before installing OpenWrt. Btw, yes you can change to RouterOS if you tested OpenWrt.