hAP AC LAN<->WAN gradual slow down

SergeiF · July 23, 2019, 9:46am

UPDATE: the problem is not with hAP AC.

UPDATE 2: ISP enabled fasttrack on their end and it solved the issue.

I have been very happy with my hAP AC for quite a while (few years), until last week.
Since about last week the LAN to WAN and vice-versa throughput has been gradually dropping until it hovers between 15 and 30 Mbit/s.
I have excluded cables, WAN itself and PoE.
The problem is very peculiar as if I change any settings in webfig or CLI relating to the interfaces in question (eg. flow control or ) the speed resets back to max.
The WiFi results are not conclusive, some devices are slow some are fast.
If I reboot the device the speed comes back to normal and in about 5 minutes drops back.
This is not ISP issue as testing directly leads to consistent results.

Things I checked so far:

IP is assigned to the bridge

192.168.1.254/24   192.168.1.0     bridge

WLAN and LAN interfaces are in the only one bridge.
Firewall first forward rule is fasttrack

chain=forward action=fasttrack-connection connection-state=established,related log=no log-prefix=""

The CPU load is below 5% most of the time
The SoC temp is around 42-46’C
Cooling has no effect on the slowdown.
Running version 6.45.2 (same problem with version 6.43.8)
Changing bridge protocol mode has only temporary effect (after about 5 min it slows down again).
There is nothing in the logs to indicate any problems.

Is this a case of bad hardware? What would be routerOS equivalent to Linux dmesg command?

What am I missing?

SergeiF · July 23, 2019, 10:06am

A bit of an update:
I found what is causing the issue.

A client on the network is running openvpn client that is pulling ~5Mbit/s constantly. These 5Mbit/s UDP packets somehow overload the fasttrack. I have checked and they are definitely fasttracked.

not expected	seen reply	assured	confirmed	not dying	fasttrack	srcnat	not dstnat

Somehow the 5Mbit/s will cause 10 fold drop in throughput.

The reason why any changes to the bridge or interfaces would temporarily fix the problem is due to these changes would reset the openvpn connection which would reconnect after a while and overload the fasttrack.

Now, it looks like I made the thread in wrong forum section after all. This is not a hardware issue but some RouterOS quirk.

I have to figure out why fasttrack is not “fasttracking”…

Poking at it further:

18  SAC Fs  udp  192.168.1.7:58955     XXX.XXX.XXX.XXX:1194                 2m59s         3.5Mbps   2.5Mbps   94 133 758   83 828 115

it looks like it is being fasttracked.

Another update:
disabling fasttracking does not fix the issue:

63  SAC  s  udp  192.168.1.7:57132     XXX.XXX.XXX.XXX:1194                 2m59s         3.8Mbps   3.0Mbps    1 198 734    1 143 980

slow:

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  6.38 MBytes  53.5 Mbits/sec
[  4]   1.00-2.00   sec  4.04 MBytes  33.9 Mbits/sec
[  4]   2.00-3.00   sec  2.99 MBytes  25.0 Mbits/sec
[  4]   3.00-4.00   sec  3.83 MBytes  32.2 Mbits/sec
[  4]   4.00-5.00   sec  3.84 MBytes  32.2 Mbits/sec
[  4]   5.00-6.00   sec  4.33 MBytes  36.4 Mbits/sec
[  4]   6.00-7.00   sec  3.44 MBytes  28.9 Mbits/sec
[  4]   7.00-8.00   sec  3.48 MBytes  29.2 Mbits/sec
[  4]   8.00-9.00   sec  4.27 MBytes  35.8 Mbits/sec
[  4]   9.00-10.00  sec  4.04 MBytes  33.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  40.9 MBytes  34.3 Mbits/sec  334             sender
[  4]   0.00-10.00  sec  40.8 MBytes  34.2 Mbits/sec                  receiver

34 + 5 != 100.

The thing is still slow as molasses.

I bring that connection down and voilà:

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  10.5 MBytes  88.0 Mbits/sec
[  4]   1.00-2.00   sec  11.1 MBytes  93.3 Mbits/sec
[  4]   2.00-3.00   sec  11.1 MBytes  93.4 Mbits/sec
[  4]   3.00-4.00   sec  11.1 MBytes  93.3 Mbits/sec
[  4]   4.00-5.00   sec  11.1 MBytes  93.3 Mbits/sec                  
[  4]   5.00-6.00   sec  11.1 MBytes  93.4 Mbits/sec                  
[  4]   6.00-7.00   sec  11.1 MBytes  93.5 Mbits/sec                  
[  4]   7.00-8.00   sec  11.1 MBytes  93.4 Mbits/sec                  
[  4]   8.00-9.00   sec  11.1 MBytes  93.4 Mbits/sec                  
[  4]   9.00-10.00  sec  11.1 MBytes  93.4 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   113 MBytes  94.4 Mbits/sec   21             sender
[  4]   0.00-10.00  sec   111 MBytes  93.2 Mbits/sec                  receiver

It looks like the hAP AC is not very capable router and I should look for alternatives. It cannot even do 100Mbit/s routing… I was looking at RB4011 but it comes with a crappy switch.
Suggestions?

SergeiF · July 24, 2019, 3:57am

Something makes no sense at all.

Stopping activity on the client running openvpn tunnel (1194/udp) bumps the speed back to 100Mbit/s and about 10kpps (immediately).

[admin@MikroTik] /ip settings> /interface monitor-traffic ether1
                         name:    ether1
        rx-packets-per-second:     9 603
           rx-bits-per-second:  99.2Mbps
     fp-rx-packets-per-second:     9 549
        fp-rx-bits-per-second:  98.6Mbps
          rx-drops-per-second:         0
         rx-errors-per-second:         0
        tx-packets-per-second:     5 140
           tx-bits-per-second:   3.9Mbps
     fp-tx-packets-per-second:     5 294
        fp-tx-bits-per-second:   4.0Mbps
          tx-drops-per-second:         0
    tx-queue-drops-per-second:         0
         tx-errors-per-second:         0

.

Resuming the activity on the client does not immediately slow down the traffic. In fact the traffic gradually slows down within 2 to 5 minutes to 20-30Mbit/s and 3-4kpps.

[admin@MikroTik] /ip settings> /interface monitor-traffic ether1
                         name:    ether1
        rx-packets-per-second:     3 420
           rx-bits-per-second:  27.1Mbps
     fp-rx-packets-per-second:     3 436
        fp-rx-bits-per-second:  26.5Mbps
          rx-drops-per-second:         0
         rx-errors-per-second:         0
        tx-packets-per-second:     2 121
           tx-bits-per-second:   3.0Mbps
     fp-tx-packets-per-second:     2 367
        fp-tx-bits-per-second:   3.8Mbps
          tx-drops-per-second:         0
    tx-queue-drops-per-second:         0
         tx-errors-per-second:         0

This is on the WAN port, so no unaccounted traffic.
I cannot explain this behaviour. If the device was at limit (it should not be with max of 10kpps) the ramp down should be instant and not take minutes to happen…

Anyone? Mikrotik?

PashaT · July 24, 2019, 4:32am

What is CPU and RAM usage during normal usage and slowdown?

SergeiF · July 24, 2019, 5:39am

Slowdown:

[admin@MikroTik] > /system resource print
                   uptime: 20h20m49s
              version: 6.45.2 (stable)
               build-time: Jul/17/2019 10:04:19
         factory-software: 6.34.2
              free-memory: 100.5MiB
             total-memory: 128.0MiB
                      cpu: MIPS 74Kc V5.0
                cpu-count: 1
            cpu-frequency: 720MHz
                 cpu-load: 15%
           free-hdd-space: 3320.0KiB
          total-hdd-space: 16.0MiB
  write-sect-since-reboot: 1260
         write-sect-total: 166296
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: hAP ac
                 platform: MikroTik

Normal:

[admin@MikroTik] > /system resource print
                    uptime: 20h22m40s
                    version: 6.45.2 (stable)
               build-time: Jul/17/2019 10:04:19
         factory-software: 6.34.2
              free-memory: 100.5MiB
             total-memory: 128.0MiB
                      cpu: MIPS 74Kc V5.0
                cpu-count: 1
            cpu-frequency: 720MHz
                 cpu-load: 22%
           free-hdd-space: 3320.0KiB
          total-hdd-space: 16.0MiB
  write-sect-since-reboot: 1260
         write-sect-total: 166296
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: hAP ac
                 platform: MikroTik

SergeiF · August 6, 2019, 2:21am

Something bizarre is going on…
I have changed openvpn tunnel to use TCP and voilà the speed is back to normal:

[admin@MikroTik] > /system resource print
                   uptime: 25m46s
                  version: 6.45.2 (stable)
               build-time: Jul/17/2019 10:04:19
         factory-software: 6.34.2
              free-memory: 103.2MiB
             total-memory: 128.0MiB
                      cpu: MIPS 74Kc V5.0
                cpu-count: 1
            cpu-frequency: 720MHz
                 cpu-load: 11%
           free-hdd-space: 3320.0KiB
          total-hdd-space: 16.0MiB
  write-sect-since-reboot: 203
         write-sect-total: 176445
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: hAP ac
                 platform: MikroTik

…

[admin@MikroTik] > /interface monitor-traffic ether1
                         name:    ether1
        rx-packets-per-second:     8 426
           rx-bits-per-second:  94.7Mbps
     fp-rx-packets-per-second:     7 286
        fp-rx-bits-per-second:  81.6Mbps
          rx-drops-per-second:         0
         rx-errors-per-second:         0
        tx-packets-per-second:     4 418
           tx-bits-per-second:   5.2Mbps
     fp-tx-packets-per-second:     3 807
        fp-tx-bits-per-second:   3.3Mbps
          tx-drops-per-second:         0
    tx-queue-drops-per-second:         0
         tx-errors-per-second:         0

Not sure if hAP AC is faulty or what…

Next step is factory reset and revert the config to see if some “bits” are “flipped” or what…

Update: while with Openvpn running about 5MBit/s over TCP the slowdown still happening but at lesser degree and takes a lot longer to have some effect, and occasionally recovers.
I think my router has a case of dementia.

SergeiF · August 6, 2019, 3:29am

Factory reset and reverted the config…
Edit: I have also tried to upgrade it to 6.45.3, and switching to long-term 6.44.5 both without any improvements.

The problem is still there.

So is hAP-AC a $130US junk?

It seems so. Which makes me sad as I quite like product, if only it did what it is designed for. My WAN is only 100Mbit/s, it should be able to handle it.
I don’t even know where to go next, ubiquity, pfsense on some random low power pc?
I don’t trust RB4011, looking at all the forum posts here about very basic issues, that should not been there in first place.

I am sorry for negativity .

Not sure where to go next with this thing. I could try rolling back the firmware…
For giggles I could try netgear R7000 with advanced tomato and see how that goes. I think it will do fine as I had it as 1Gbit/s gateway with CTF on doing full 1Gbit/s. Although the WiFi on the hAP-AC is stellar, and R7000 does not have PoE…

SergeiF · November 14, 2019, 5:45am

It seems I have replicated this issue with completely different machine and no OpenVPN.
It seems that when the machine utilizes about ~1-5Mbit/s of UDP it causes the hAP AC to gradually slow down. Once I kill the traffic the speed goes back up, and after restart of the UDP traffic it gradually slows down.

This problem is VERY annoying, I would really hate throwing money at the problem if there is some kind of fix.

Davis · November 18, 2019, 7:22pm

Are you sure the slowness happens in router (not ISP)?
There are some ISPs that throttle (all) the traffic heavily when detecting something they classify as P2P.

As the router is single core and CPU usage is well below 50% I would assume it is not the bottleneck (unless queues are configured!).

SergeiF · November 20, 2019, 5:25am

No queues. I was thinking configuring queues to see if it makes any difference.
100% sure it is not with ISP, as having device in another port on the ISP provided gateway gets full bandwidth (while slowdown happens on the hAP AC).

Davis · November 21, 2019, 9:55pm

ISP might implement throttling per IP address.

Much better test is:

Write down the IP address router’s WAN interface has.
Find a computer/application that causes slowness.
Connect problematic computer to the router via WIRED connection.
Close/stop the application that causes slowness.
Disconnect all other devices from the network.
Do a reliable speed measurement (e.g. SEVERAL TIMES run speedtest.net against a KNOWN server using multiple connections setting). It should show good speed.
Run the application that causes slowness.
Do a reliable speed measurement (e.g. SEVERAL TIMES run speedtest.net against the SAME server using multiple connections setting). It should show bad speed.
Close/stop the application that causes slowness.
Configure the computer’s wired network interface to have THE SAME MAC ADDRESS as the router’s WAN interface.
Connect the computer to the internet instead of the router.
Verify that computer has the same IP address as router’s WAN interface had.
Run the application that causes slowness.
Do a reliable speed measurement (e.g. SEVERAL TIMES run speedtest.net against the SAME server using multiple connections setting)…

This test doesn’t completely exclude possibility that ISP is somehow involved in the slowness, but this test is at least much better than PC in another port of ISP equipment.

BTW How you were measuring the internet speed?

Another (even better) test would be to configure another network on some port of the router and try to test bandwidth towards that network during the slowness.

SergeiF · November 23, 2019, 12:53am

I have done very similar test processes, but might repeat it to exclude all variables.
ISP is using mikrotik RB2011 as the gateway and they were doing tests during slowdown achieving the full throughput.
I have another network (going through the same NAT on hAP AC) and it also experiences slow down.
While I was getting slow down, the PC on another gateway port (not hAP AC) was consistently getting full speed.
ISP is not malicious here, they have been very cooperative.

I know exactly which application causes slow down: geth (ethereum network node/client), specifically geth discovery protocol via UDP Edit: looks like TCP is also having an effect, UDP makes it worse.
If I encapsulate the geth traffic via OpenVPN UDP the slowdown still happens. If the traffic is encapsulated in TCP there is no slowdown.
The issue is not connection tracking, as with OpenVPN UDP is only a single connection is tracked. The issue is not with fasttracking or overloading CPU, I disabled fasttrack nuked connection tracking table and still got the same slowdown.

The only thing I need to do to stop slowdown is to disable port forwarding to geth node (and wait for the sessions to die out or restart geth). The actual UDP packet stats are abysmal - it is not being flooded by UDP. It takes about 100kbit/s of geth discovery traffic to bring down the hAP AC to its knees.

I am going to dish some cash on an RB4011 (once it gets back in stock) and forget about this (hopefully , being different architecture and having way more CPU).

I can only afford so much downtime on testing before other users start complaining.

Edit: I also use iperf3 to do bandwidth testing (other side is 40+Gbit/s capable). The Ookla stuff (speedtest.net) is garbage to be honest, it completely depends on the browser performance and the traffic patterns are stupid (the payload is ABCDEF… ASCII pattern, which with some LZO can be fooled easily).

SergeiF · November 28, 2019, 3:11am

A bit of an update.

Bought RB4011.

With RB4011 the problem is still there but not as dramatic (it drops to about 40Mbit/s intermittently). So WTF is going on…

Will try “routerless” approach tonight with the geth node directly on the interwebs.

SergeiF · November 28, 2019, 5:37am

Final update:

Connected directly to the ISPs RB2011, and the slowdown was there.
All people who blamed the ISP were right, … maybe (since there is an RB2011 and AirFibre involved in between).