Community discussions

MikroTik App
 
zagbur
just joined
Topic Author
Posts: 6
Joined: Sat May 21, 2022 1:12 pm

hAP ac3 gives only half the bandwidth to Linux boxes, Windows and BSD doing fine

Sat May 21, 2022 1:40 pm

Good day everyone,

I just bought hAP ac3 and did following setup:
ISP FTTH - ONT - MikroTik -(ether2)- comp A
                      |
                      \ -(ether3)- comp B                       
Between comp A and comp B I can get sustained ~940Mbps both ways with iperf3 TCP or UDP.
Uplink to the ISP is sufficiently proven to be at least 600Mbps downstream, 100Mbps upstream to some nearby speedtest servers and that's fine. It's configured with PPPoE and vlan 35.

What puzzles me is that both computers show big difference on download that depends on operating system's default configuration I can't pin-point.
Both machines show about 300-350Mbps download speed when I boot them up on recent versions of Fedora Live, OpenSUSE live, Debial live, Ubuntu live. On the other hand, started on Windows 10 or NomadBSD, I get easily all the 600Mbps. If I replace the MikroTik with some spare Archer-something I used before, I get 600 on all of the above.

So, different cables, ports, NICs, machines and the download speed seems to follow only the OS. I did try to change some OS settings like ethtool -A, different MTU, net.ipv4.tcp_congestion_control but I can't seem to identify what's that I'm missing. Any advice appreciated.
Best regards, Z.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19100
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: hAP ac3 gives only half the bandwidth to Linux boxes, Windows and BSD doing fine

Sat May 21, 2022 10:39 pm

Specific to the OS or device or cable, nothing to do with MT, its a gender neutral product. :-)
OR not very particular with which device it interacts with!!
 
zagbur
just joined
Topic Author
Posts: 6
Joined: Sat May 21, 2022 1:12 pm

Re: hAP ac3 gives only half the bandwidth to Linux boxes, Windows and BSD doing fine

Sun May 22, 2022 12:14 am

Thank you for your reply and I'm glad to hear that MT isn't part of NIC wars, but as I wrote, this problem shows up on different cables, NICs and systems and *only* when MT works as the router.

MT - Intel 1000 82541PI - Debian = 300Mbps
MT - RTL8111/8168/8411 - Debian = 300Mbps
MT - * - Ubuntu = 300Mbps
MT - * - SUSE = 300Mbps
MT - * - Fedora = 300Mbps
MT - * - BSD = 600Mbps
MT - * - Win10 = 600Mbps

but

Archer - * - * - 600Mbps

I'm happy to agree it may be some difference in network stack implementation between Linux, BSD or Windows, but it's the installation of MT that caused it to show and act up. I wrote my question here not to complain about MT ;) , but in hope that someone else found similar dependency or reaction and the one configuration variable responsible for this behaviour. It made more sense to me asking here than writing the same question of each forum dedicated to all the OS's I'm testing ;)
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 887
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: hAP ac3 gives only half the bandwidth to Linux boxes, Windows and BSD doing fine

Sun May 22, 2022 1:17 am

I'm happy to agree it may be some difference in network stack implementation between Linux, BSD or Windows, but it's the installation of MT that caused it to show and act up. I wrote my question here not to complain about MT ;) , but in hope that someone else found similar dependency or reaction and the one configuration variable responsible for this behaviour. It made more sense to me asking here than writing the same question of each forum dedicated to all the OS's I'm testing ;)
Since you are using PPPoE, my first guess would be that the Archer is automatically configuring MSS clamping, but the MikroTik is not. (I don't use pppoe, and my only MikroTik is in a lab setting, so I am not sure what quickset does as far as mss clamping when pppoe is used. In my opinion, any "setup wizard" should always enable mss clamping when the WAN ethernet interface MTU is 1500 and the pppoe interface is 1492, and the LAN is 1500.

See this for something to try (it's the TL;DR recipe)

If you want to know why, keep reading.

Do you have a switch with a mirror port you can put between the ONT and routers and capture the traffic with wireshark.

Doing that with both the Archer and the hAP ac3 and comparing to see where the timing differences were would be what I would try in your situation.

It is easy to see how mss is negotiated in the TCP three way handshake.

I like this video referenced in PacketLife’s Path MTU Discovery blog post: TCP MSS clamping – what is it and why do we need it? by Ivan Pepelnjak

There is a long deep dive on David Bombal's youtube channel (with Chris Greer wireshark examples). Network Nightmares! How TCP really works: MTU vs MSS The video description has links to other useful info.

This thread has some MikroTik info.
 
zagbur
just joined
Topic Author
Posts: 6
Joined: Sat May 21, 2022 1:12 pm

Re: hAP ac3 gives only half the bandwidth to Linux boxes, Windows and BSD doing fine

Sun May 22, 2022 8:01 pm

Thanks for the reading material. I found some new things but I can't apply them to my configuration. The rule add action=change-mss didn't improve anything, even seemed to take 30Mbps off.

I don't have anything to put between ONT and MT readily available, but since all my Win10 hosts in my LAN get full speed and Linux boxes get half, I can do some packet sniffing on MT's pppoe-out1 or vln35 interfaces to see if there's any obvious difference I can notice. Their MTU is in fact 1492, and PMTU shows that:
# tracepath -n 8.8.8.8
 1?: [LOCALHOST]                      pmtu 1500
 1:  192.168.0.1                                           0.351ms 
 1:  192.168.0.1                                           0.219ms 
 2:  192.168.0.1                                           0.211ms pmtu 1492
 
zagbur
just joined
Topic Author
Posts: 6
Joined: Sat May 21, 2022 1:12 pm

Re: hAP ac3 gives only half the bandwidth to Linux boxes, Windows and BSD doing fine

Wed May 25, 2022 1:03 pm

Got an Asus AX55 and the final results are as follows:

MT - Linux - 2 cables and ports - 2 NICs - box A and box B - 300Mb
MT - Windows - 2 cables and ports - 1 NIC - box A and box B - 600Mb
MT - NomadBSD - 2 cables and ports - 1 NIC - box B - 600Mb

Archer - * - * - * - * - 600Mb

Asus - * - * - * - * - 600Mb

I realize MT is not for beginners and/or it may be just some setting to the Linux' network stack or something, but I don't really see myself thinking about that every time I change some box here. For me the best solution is to return it and stick with Asus. Thanks for the tips anyway.
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 887
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: hAP ac3 gives only half the bandwidth to Linux boxes, Windows and BSD doing fine

Wed May 25, 2022 8:17 pm

Got an Asus AX55 and the final results are as follows:

MT - Linux - 2 cables and ports - 2 NICs - box A and box B - 300Mb
MT - Windows - 2 cables and ports - 1 NIC - box A and box B - 600Mb
MT - NomadBSD - 2 cables and ports - 1 NIC - box B - 600Mb

Archer - * - * - * - * - 600Mb

Asus - * - * - * - * - 600Mb

I realize MT is not for beginners and/or it may be just some setting to the Linux' network stack or something, but I don't really see myself thinking about that every time I change some box here. For me the best solution is to return it and stick with Asus. Thanks for the tips anyway.
What is odd is that the Archer and Asus are doing something different when Linux is being used, and that's why I suspected that mss clamping was the possible difference. If that was the difference, why wouldn't it also affect BSD and Windows? (just thinking outloud).

Were the tests done on high latency networks? The tcp window size could have an effect there. Wireshark would be the tool I would use to try to determine the cause of the observed differences.

Troubleshooting Network Latency with Wireshark

There are many youtube videos for demos of troubleshooting network performance with wireshark. Chris Greer has many.

Can you explain the "2 NICs" vs "1 NIC". I assume that means you tried Linux with 2 different types of NIC.

I can understand why you would feel that way. It sounds like you have done a reasonable amount of testing. It is too bad you don't have the ability to capture the packets to determine the cause.

You could capture on the MikroTik hAP ac3, but that would not be a valid test for performance, because it would be using resources on the hAP ac3. For debugging, on router capture can be handy, but in my opinion, using /tool/sniffer isn't good for performance measurement, just like speed test from the device itself isn't a good indication of the max routing performance. It would be good for capturing data to see if packet fragmentation is the reason for the difference in performance.

https://linux-hardware.org/?id=pci:8086-107c-8086-1376 indicates that the Intel 1000 82541PI has good linux support, and generally linux network support is good. If you look at freebsd sites you will see claims that bsd is superior, if you look at linux sites you will see claims that linux is better.

The primary advantage of the MikroTik is versatility, with with the ability to support vlans, dynamic routing protocols, capture traffic, generate traffic, and with v7.2+ has wireguard (and zerotier on ARM devices like the hap ac3) supported.

If you are just using the device as a home router and not doing anything that the standard consumer based routers will not do, the hAP ac3 may not be the best choice for your situation. And for wifi other devices may be better.

Before you send it back, can you at least do an /export hide-sensitive file=hap_not_linux_friendly_with_ppoe and post so someone else with a hap ac3 and pppoe connection can verify your results?

And how you did the testing would also be useful, e.g. were the tests done on same hardware (booting Linux and BSD from live distros loaded into ram).
 
zagbur
just joined
Topic Author
Posts: 6
Joined: Sat May 21, 2022 1:12 pm

Re: hAP ac3 gives only half the bandwidth to Linux boxes, Windows and BSD doing fine

Sat May 28, 2022 2:40 pm

I can do the export and do some other things, yes by tomorrow. Still have a few regulatory days before I need to ship it back. Things I have about are:

Box B runs some ubuntu by default and has two NICs, both tested:
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
        Subsystem: Gigabyte Technology Co., Ltd Onboard Ethernet
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Kernel driver in use: r8169

05:00.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
        Subsystem: Intel Corporation PRO/1000 GT Desktop Adapter
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 32 (63750ns min), Cache Line Size: 64 bytes
        Kernel driver in use: e1000
Additionally, I tested on this box most recent and freshly downloaded versions of NomadBSD, Suse, Fedora, Debian and Win10 I could find and boot from USB drive, so on factory defaults.

As you have guessed already, this is no professional setup - I got MT cos I had had it with Archer's interface and WiFi performance (in no particular order) and inability to do port forwarding to my liking. I would never use 95% of MT capabilities but sentiment got the better of me.

No, I'm not getting drawn into Tuxie vs Beastie, thank you, 'cos this would get OT in an instant, but my personal preference has more fangs than fins... ;)

About latency:
# tracepath -l 1500 forum.mikrotik.com
 1:  192.168.0.1                                           0.352ms
 2:  kra-bng101.neo.tpnet.pl                               3.564ms
 3:  kra-r12.tpnet.pl                                     12.118ms
 4:  poz-r11.tpnet.pl                                     40.390ms asymm  5
 5:  hbg-b2-link.ip.twelve99.net                          30.477ms asymm  9
 6:  hbg-bb3-link.ip.twelve99.net                         29.459ms asymm 10
 7:  s-bb1-link.ip.twelve99.net                           45.128ms asymm 10
 8:  riga-b3-link.ip.twelve99.net                         53.170ms asymm  9 
 9:  telialatvija-ic332270-riga-b1.ip.twelve99-cust.net   49.681ms asymm 10
10:  no reply
and 1500 ping to it also has about 55ms.

Who is online

Users browsing this forum: anav, Nospam, qatar2022 and 38 guests