Community discussions

MikroTik App
 
donluca
just joined
Topic Author
Posts: 7
Joined: Mon Jul 12, 2021 6:55 pm

Limiting Upload Bandwidth with hAP AC2

Mon Jul 12, 2021 8:51 pm

Context:

I come from an Ubiquiti Edgerouter Lite and the moment I opened Webfig for the first time I almost got traumatized by the sheer amount of option and configurations available.

My setup is a normal home networking setup with very few devices.

I have a Gigabit Fiber internet connection with 1Gbit down and 200Mbit up.

My current network structure is really simple:

Fiber ONT (bridge mode) -> Mikrotik -> various devices.

The issue

When connected straight into the ONT I can saturate the download bandwidth and won't have any issues.
The upload, though, is a completely different story.

Although I have 200Mbit, for reasons unknown when I saturate the upload I can reach up to 210Mbit/s and the latency and bufferbloat skyrocket to various seconds (yes, 1000ms and more).
The easy solution is to cap the upload bandwidth, I tried this with Google Chrome developer tools at first by limiting the upload bandwidth to 190Mbit/s and everything was nice and smooth.

When I tried to enable QoS to limit the upload bandwidth on my Edgerouter lite, performance collapsed: download couldn't go faster than 200Mbit and 150Mbit in upload.
This lead me to find another solution: enter the Mikrotik.

The new issue

My Mikrotik has the standard, default configuration: LAN port 1 is Internet, the other 4 (and wireless) are a hardware bridge (with hardware offloading enabled, so they are managed by the switch chip).

In order to limit the upload I had to disable Fasttrack otherwise queues wouldn't work.

Then I created a queue to limit only the upload to 180Mbit/s and to keep the bufferbloat down I used a small PFIFO queue with 5 packet queue.
I have zero packet loss and very low latency,

The download on the other hand... the performance took quite the hit, down to 400Mbit/s.
In order to get back to at least 800Mbit/s without packet loss I had to create a huge PFIFO queue with 10000 packets.
Luckily the latency and bufferbloat is still amazingly low, but I'm still unable to get the 940Mbit/s I had with Fasttrack.
On top of this, the whole box is super hot and I had to put a CPU fan in front of it to cool it down.
When I look at the Resource/CPU page on Webfig I see two of the four cores on 100% and the other two at very low values.

Question

Why is it not using all four cores?
Am I doing this right?

Are there any better solutions to get full performance on download?
I have already done a search here and it looks like I can't enable Fasttrack only for downloads and, unfortunately, it looks like I can't just assign a no-queue on the download.
 
donluca
just joined
Topic Author
Posts: 7
Joined: Mon Jul 12, 2021 6:55 pm

Re: Limiting Upload Bandwidth with hAP AC2

Tue Jul 13, 2021 3:22 pm

Replying to my own questions after doing some tests, hoping to spark some conversation and find better solutions.

The way the IPQ4019 is implemented in the hAP AC2 is that there are only two threads which can be assigned to do forwarding/routing/queueing.

I've managed to greatly improve performance by going into System Resources IRQ and manually setting CPU 2 and 3 to the IPQ4019, CPU 1 to everything I don't actually use (usb, gpiolib, etc.).

Unfortunately I can't force a CPU on the eth_tx and rx, those are read only, but they get automatically assigned to CPU 0 so that should be alright.

Finally, I overclocked the CPU to the max available and I can get full performance on download... on the second LAN port.

For some reason, I'm unable to get the same performance on another PC which is on the third LAN port and it's driving me crazy.
CPUs hit 100% and I get only 520Mbit/s in download.

I'll try today and swap ports to see if something changes.

Meanwhile, if anyone has any suggestions on improving performance that'd be greatly appreciated.

Here's my configuration
# jul/13/2021 14:17:19 by RouterOS 6.47.10
# software id = 9HVD-0AF7
#
# model = RBD52G-5HacD2HnD
# serial number = XXXXXXXXXXX
/interface bridge
add admin-mac=XXXXXXXXXX auto-mac=no comment=defconf fast-forward=no name=bridge
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip ipsec proposal
set [ find default=yes ] disabled=yes
/ip pool
add name=dhcp ranges=192.168.1.50-192.168.1.254
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge name=defconf
/queue simple
add max-limit=170M/0 name="All Bandwidth" target=bridge time=0s-1d,sun,mon,tue,wed,thu,fri,sat
/queue type
add kind=none name=no-queue
set 7 pcq-limit=500KiB pcq-total-limit=20000KiB
/routing bgp instance
set default disabled=yes
/interface bridge port
add bridge=bridge comment=defconf interface=ether2 trusted=yes
add bridge=bridge comment=defconf interface=ether3 trusted=yes
add bridge=bridge comment=defconf interface=ether4 trusted=yes
add bridge=bridge comment=defconf interface=ether5 trusted=yes
add bridge=bridge comment=defconf interface=wlan1
add bridge=bridge comment=defconf interface=wlan2
/ip neighbor discovery-settings
set discover-interface-list=none
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/ip address
add address=192.168.1.1/24 comment=defconf interface=ether2 network=192.168.1.0
/ip dhcp-client
add comment=defconf disabled=no interface=ether1 use-peer-dns=no
/ip dhcp-server network
add address=192.168.1.0/24 comment=defconf dns-server=192.168.1.3 gateway=192.168.1.1 netmask=24
/ip dns static
add address=192.168.1.1 comment=defconf name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related disabled=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip ipsec policy
set 0 disabled=yes
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set www-ssl certificate=Webfig disabled=no
set api disabled=yes
set winbox disabled=yes
set api-ssl disabled=yes
/routing bfd interface
set [ find default=yes ] disabled=yes
/system package update
set channel=long-term
/system resource irq
set 0 cpu=1
set 1 cpu=1
set 2 cpu=2
set 3 cpu=3
set 4 cpu=1
/system routerboard settings
set cpu-frequency=896MHz
/tool bandwidth-server
set enabled=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
EDIT: For some reason I disabled the fastforward on the bridge. I've rectified that now, but made no difference.
 
donluca
just joined
Topic Author
Posts: 7
Joined: Mon Jul 12, 2021 6:55 pm

Re: Limiting Upload Bandwidth with hAP AC2

Tue Jul 13, 2021 6:40 pm

Small update, swapping ports didn't change the results, so the issue is strictly in the clients.

I wonder if being logged through SSH in the clients is somehow affecting the results.

Using iperf I get full bandwidth between all the clients, so the issue is definitely in the routing because if I enable Fasttrack I get full speed.

I really wonder what the issue could be here...
 
donluca
just joined
Topic Author
Posts: 7
Joined: Mon Jul 12, 2021 6:55 pm

Re: Limiting Upload Bandwidth with hAP AC2

Fri Jul 16, 2021 4:16 pm

Did lots of more testing and unfortunately I couldn't still solve the issue of high CPU usage.

This is getting a bit irritating.

On the the test results page of the hAP AC2, MT states that in routing mode, with 25 Simple Queues, it can reach almost 2Gb of throughput: https://mikrotik.com/product/hap_ac2#fndtn-testresults

I have a SINGLE simple queue set up and I can barely reach 900Mbit/s on a single client (and, curiously, 500-600Mbit/s on others).

I wonder if I screwed up somewhere in the configuration, I might just reset to factory default and start from scratch.

After that, I can try other firmwares. Right now I'm on the 6.47.10 long term, I can see if the newer firmwares can yield better performance.

If that fails as well, I guess I'll be forced to return the hAP AC2 and look elsewhere.
 
donluca
just joined
Topic Author
Posts: 7
Joined: Mon Jul 12, 2021 6:55 pm

Re: Limiting Upload Bandwidth with hAP AC2

Fri Jul 16, 2021 6:54 pm

Just tried a reset to factory and the issue is still here.

Can anyone confirm that the hAP AC2 is not capable of pushing 1Gbit/s with Fasttrack disabled?

I'm now not even using queues, what I just did right now is the following:

• Reset to default configuration (eth0 is wan, the other 4 ports are bridged, routing is enabled from wan to bridge)
• Disabled Fasttrack
• Overclocked to 896Mhz

This is the performance in download I'm getting on various devices:

Mac Pro: 900Mbit/s
HP Microserver N40L with XigmaNAS: 680Mbit/s
Orange Pi PC 2 with Armbian + Pi-Hole: 700Mbit/s
Macbook Air with USB 3.0 Gigabit to Ethernet adapter: 800Mbit/s

All of these devices are able to hit 1Gbit/s in download with Fasttrack enabled without any issues and negligible CPU usage.

Right now there are no queues, only Fasttrack disabled.

Any help would be immensely appreciated, thanks.
 
donluca
just joined
Topic Author
Posts: 7
Joined: Mon Jul 12, 2021 6:55 pm

Re: Limiting Upload Bandwidth with hAP AC2

Sat Jul 17, 2021 10:53 pm

I think I've finally found the issue and I'll probably submit a bug report to MT ( support@mikrotik.com ).

The problem is either in the CPU scheduler or the IRQ balancer.

To be brief, what happens is that randomly all the load will be sent to Core 0 which will reach 100% with all the other cores at 0%.
Normally, this load would be split between Core 1 and Core 2.

This is seemingly random, but it can be triggered consistently with iperf3: after a first, "good" run of iperf3 between two clients which uses the whole bandwidth, the second run will use only the first core which will go to 100% and limit the bandwidth.

Manually assigning CPUs in /system resource irq doesn't seem to have any effect, nor using RPS.

This could be related to viewtopic.php?f=21&t=175735#p866216 which has the same IPQ4019 device.

EDIT: at least on 6.4.7.10 long term.

Who is online

Users browsing this forum: No registered users and 47 guests