Community discussions

 
SergeiF
just joined
Topic Author
Posts: 23
Joined: Wed Aug 02, 2017 4:01 am

hAP AC LAN<->WAN gradual slow down

Tue Jul 23, 2019 12:46 pm

I have been very happy with my hAP AC for quite a while (few years), until last week.
Since about last week the LAN to WAN and vice-versa throughput has been gradually dropping until it hovers between 15 and 30 Mbit/s.
I have excluded cables, WAN itself and PoE.
The problem is very peculiar as if I change any settings in webfig or CLI relating to the interfaces in question (eg. flow control or ) the speed resets back to max.
The WiFi results are not conclusive, some devices are slow some are fast.
If I reboot the device the speed comes back to normal and in about 5 minutes drops back.
This is not ISP issue as testing directly leads to consistent results.

Things I checked so far:
1) IP is assigned to the bridge
192.168.1.254/24   192.168.1.0     bridge
2) WLAN and LAN interfaces are in the only one bridge.

3) Firewall first forward rule is fasttrack
chain=forward action=fasttrack-connection connection-state=established,related log=no log-prefix=""
4) The CPU load is below 5% most of the time

5) The SoC temp is around 42-46'C

6) Cooling has no effect on the slowdown.

7) Running version 6.45.2 (same problem with version 6.43.8)

8) Changing bridge protocol mode has only temporary effect (after about 5 min it slows down again).

9) There is nothing in the logs to indicate any problems.

Is this a case of bad hardware? What would be routerOS equivalent to Linux `dmesg` command?

What am I missing?
 
SergeiF
just joined
Topic Author
Posts: 23
Joined: Wed Aug 02, 2017 4:01 am

Re: hAP AC LAN<->WAN gradual slow down

Tue Jul 23, 2019 1:06 pm

A bit of an update:
I found what is causing the issue.

A client on the network is running openvpn client that is pulling ~5Mbit/s constantly. These 5Mbit/s UDP packets somehow overload the fasttrack. I have checked and they are definitely fasttracked.
not expected	seen reply	assured	confirmed	not dying	fasttrack	srcnat	not dstnat
Somehow the 5Mbit/s will cause 10 fold drop in throughput.

The reason why any changes to the bridge or interfaces would temporarily fix the problem is due to these changes would reset the openvpn connection which would reconnect after a while and overload the fasttrack.

Now, it looks like I made the thread in wrong forum section after all. This is not a hardware issue but some RouterOS quirk.

I have to figure out why fasttrack is not "fasttracking"....

Poking at it further:
18  SAC Fs  udp  192.168.1.7:58955     XXX.XXX.XXX.XXX:1194                 2m59s         3.5Mbps   2.5Mbps   94 133 758   83 828 115
it looks like it is being fasttracked.

Another update:
disabling fasttracking does not fix the issue:
63  SAC  s  udp  192.168.1.7:57132     XXX.XXX.XXX.XXX:1194                 2m59s         3.8Mbps   3.0Mbps    1 198 734    1 143 980
slow:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  6.38 MBytes  53.5 Mbits/sec
[  4]   1.00-2.00   sec  4.04 MBytes  33.9 Mbits/sec
[  4]   2.00-3.00   sec  2.99 MBytes  25.0 Mbits/sec
[  4]   3.00-4.00   sec  3.83 MBytes  32.2 Mbits/sec
[  4]   4.00-5.00   sec  3.84 MBytes  32.2 Mbits/sec
[  4]   5.00-6.00   sec  4.33 MBytes  36.4 Mbits/sec
[  4]   6.00-7.00   sec  3.44 MBytes  28.9 Mbits/sec
[  4]   7.00-8.00   sec  3.48 MBytes  29.2 Mbits/sec
[  4]   8.00-9.00   sec  4.27 MBytes  35.8 Mbits/sec
[  4]   9.00-10.00  sec  4.04 MBytes  33.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  40.9 MBytes  34.3 Mbits/sec  334             sender
[  4]   0.00-10.00  sec  40.8 MBytes  34.2 Mbits/sec                  receiver

34 + 5 != 100.


The thing is still slow as molasses.

I bring that connection down and voilà:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  10.5 MBytes  88.0 Mbits/sec
[  4]   1.00-2.00   sec  11.1 MBytes  93.3 Mbits/sec
[  4]   2.00-3.00   sec  11.1 MBytes  93.4 Mbits/sec
[  4]   3.00-4.00   sec  11.1 MBytes  93.3 Mbits/sec
[  4]   4.00-5.00   sec  11.1 MBytes  93.3 Mbits/sec                  
[  4]   5.00-6.00   sec  11.1 MBytes  93.4 Mbits/sec                  
[  4]   6.00-7.00   sec  11.1 MBytes  93.5 Mbits/sec                  
[  4]   7.00-8.00   sec  11.1 MBytes  93.4 Mbits/sec                  
[  4]   8.00-9.00   sec  11.1 MBytes  93.4 Mbits/sec                  
[  4]   9.00-10.00  sec  11.1 MBytes  93.4 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   113 MBytes  94.4 Mbits/sec   21             sender
[  4]   0.00-10.00  sec   111 MBytes  93.2 Mbits/sec                  receiver
It looks like the hAP AC is not very capable router and I should look for alternatives. It cannot even do 100Mbit/s routing... I was looking at RB4011 but it comes with a crappy switch.
Suggestions?
 
SergeiF
just joined
Topic Author
Posts: 23
Joined: Wed Aug 02, 2017 4:01 am

Re: hAP AC LAN<->WAN gradual slow down

Wed Jul 24, 2019 6:57 am

Something makes no sense at all.

Stopping activity on the client running openvpn tunnel (1194/udp) bumps the speed back to 100Mbit/s and about 10kpps (immediately).
[admin@MikroTik] /ip settings> /interface monitor-traffic ether1
                         name:    ether1
        rx-packets-per-second:     9 603
           rx-bits-per-second:  99.2Mbps
     fp-rx-packets-per-second:     9 549
        fp-rx-bits-per-second:  98.6Mbps
          rx-drops-per-second:         0
         rx-errors-per-second:         0
        tx-packets-per-second:     5 140
           tx-bits-per-second:   3.9Mbps
     fp-tx-packets-per-second:     5 294
        fp-tx-bits-per-second:   4.0Mbps
          tx-drops-per-second:         0
    tx-queue-drops-per-second:         0
         tx-errors-per-second:         0
   
.

Resuming the activity on the client does not immediately slow down the traffic. In fact the traffic gradually slows down within 2 to 5 minutes to 20-30Mbit/s and 3-4kpps.
[admin@MikroTik] /ip settings> /interface monitor-traffic ether1
                         name:    ether1
        rx-packets-per-second:     3 420
           rx-bits-per-second:  27.1Mbps
     fp-rx-packets-per-second:     3 436
        fp-rx-bits-per-second:  26.5Mbps
          rx-drops-per-second:         0
         rx-errors-per-second:         0
        tx-packets-per-second:     2 121
           tx-bits-per-second:   3.0Mbps
     fp-tx-packets-per-second:     2 367
        fp-tx-bits-per-second:   3.8Mbps
          tx-drops-per-second:         0
    tx-queue-drops-per-second:         0
         tx-errors-per-second:         0
This is on the WAN port, so no unaccounted traffic.
I cannot explain this behaviour. If the device was at limit (it should not be with max of 10kpps) the ramp down should be instant and not take minutes to happen...

Anyone? Mikrotik?
 
PashaT
just joined
Posts: 15
Joined: Sat Feb 01, 2014 1:10 am
Location: Zhytomyr, Ukraine

Re: hAP AC LAN<->WAN gradual slow down

Wed Jul 24, 2019 7:32 am

What is CPU and RAM usage during normal usage and slowdown?
 
SergeiF
just joined
Topic Author
Posts: 23
Joined: Wed Aug 02, 2017 4:01 am

Re: hAP AC LAN<->WAN gradual slow down

Wed Jul 24, 2019 8:39 am

Slowdown:
[admin@MikroTik] > /system resource print
                   uptime: 20h20m49s
              version: 6.45.2 (stable)
               build-time: Jul/17/2019 10:04:19
         factory-software: 6.34.2
              free-memory: 100.5MiB
             total-memory: 128.0MiB
                      cpu: MIPS 74Kc V5.0
                cpu-count: 1
            cpu-frequency: 720MHz
                 cpu-load: 15%
           free-hdd-space: 3320.0KiB
          total-hdd-space: 16.0MiB
  write-sect-since-reboot: 1260
         write-sect-total: 166296
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: hAP ac
                 platform: MikroTik
Normal:
[admin@MikroTik] > /system resource print
                    uptime: 20h22m40s
                    version: 6.45.2 (stable)
               build-time: Jul/17/2019 10:04:19
         factory-software: 6.34.2
              free-memory: 100.5MiB
             total-memory: 128.0MiB
                      cpu: MIPS 74Kc V5.0
                cpu-count: 1
            cpu-frequency: 720MHz
                 cpu-load: 22%
           free-hdd-space: 3320.0KiB
          total-hdd-space: 16.0MiB
  write-sect-since-reboot: 1260
         write-sect-total: 166296
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: hAP ac
                 platform: MikroTik
 
SergeiF
just joined
Topic Author
Posts: 23
Joined: Wed Aug 02, 2017 4:01 am

Re: hAP AC LAN<->WAN gradual slow down

Tue Aug 06, 2019 5:21 am

Something bizarre is going on...
I have changed openvpn tunnel to use TCP and voilà the speed is back to normal:
[admin@MikroTik] > /system resource print
                   uptime: 25m46s
                  version: 6.45.2 (stable)
               build-time: Jul/17/2019 10:04:19
         factory-software: 6.34.2
              free-memory: 103.2MiB
             total-memory: 128.0MiB
                      cpu: MIPS 74Kc V5.0
                cpu-count: 1
            cpu-frequency: 720MHz
                 cpu-load: 11%
           free-hdd-space: 3320.0KiB
          total-hdd-space: 16.0MiB
  write-sect-since-reboot: 203
         write-sect-total: 176445
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: hAP ac
                 platform: MikroTik
...
[admin@MikroTik] > /interface monitor-traffic ether1
                         name:    ether1
        rx-packets-per-second:     8 426
           rx-bits-per-second:  94.7Mbps
     fp-rx-packets-per-second:     7 286
        fp-rx-bits-per-second:  81.6Mbps
          rx-drops-per-second:         0
         rx-errors-per-second:         0
        tx-packets-per-second:     4 418
           tx-bits-per-second:   5.2Mbps
     fp-tx-packets-per-second:     3 807
        fp-tx-bits-per-second:   3.3Mbps
          tx-drops-per-second:         0
    tx-queue-drops-per-second:         0
         tx-errors-per-second:         0
Not sure if hAP AC is faulty or what...

Next step is factory reset and revert the config to see if some "bits" are "flipped" or what...

Update: while with Openvpn running about 5MBit/s over TCP the slowdown still happening but at lesser degree and takes a lot longer to have some effect, and occasionally recovers.
I think my router has a case of dementia.
 
SergeiF
just joined
Topic Author
Posts: 23
Joined: Wed Aug 02, 2017 4:01 am

Re: hAP AC LAN<->WAN gradual slow down

Tue Aug 06, 2019 6:29 am

Factory reset and reverted the config...
Edit: I have also tried to upgrade it to 6.45.3, and switching to long-term 6.44.5 both without any improvements.

The problem is still there.

So is hAP-AC a $130US junk?

It seems so. Which makes me sad as I quite like product, if only it did what it is designed for. My WAN is only 100Mbit/s, it should be able to handle it.
I don't even know where to go next, ubiquity, pfsense on some random low power pc?
I don't trust RB4011, looking at all the forum posts here about very basic issues, that should not been there in first place.

I am sorry for negativity :(.

Not sure where to go next with this thing. I could try rolling back the firmware...
For giggles I could try netgear R7000 with advanced tomato and see how that goes. I think it will do fine as I had it as 1Gbit/s gateway with CTF on doing full 1Gbit/s. Although the WiFi on the hAP-AC is stellar, and R7000 does not have PoE...

Who is online

Users browsing this forum: No registered users and 14 guests