Hello. I figured I would post something about how I fixed this problem after wracking my brain for half of a day and to save others the same fate.

I have FIOS, and I pay for 300/300. When I replaced my old setup (Orbi RBR50) with the hEX refresh connected directly to my Verizon ONT, I noticed that my download speed was ~300Mbps, but my upload speed was anywhere from 1-6Mbps, depending on the particular speedtest.net attempt. Plug my Orbi back in, back to 300/300. Odd. I messed with every known setting you should try to get it to work, but nothing made a difference (flow control, fasttrack, firewall rules, queues, ethernet settings, the works).

While going through the WAN link’s auto-negotiation settings and trying to force 1Gbps full duplex (even though the link was coming up as that anyway), I decided to set my link speed to 100Mbps full duplex and give that a shot. Surprise, surprise, my upload speed increased dramatically…

To make a very long story short, there seems to be a problem with the hEX refresh and Verizon’s ONT (I-211M-L in my case) where they do not, in fact, auto-negotiate the link completely/correctly, even though the hEX says the link is at 1Gbps full duplex. After doing some googling, I found a lot of other people that had this type of issue, not with the hEX refresh in particular, but with routers from other vendors like Netgear, Ubiquiti, and others.

Their collective fix? Put a small unmanaged switch between the ONT and hEX refresh to force both links to come up as 1Gbps full on their own (with the help of the switch, of course, since the ONT link only works in an auto-negotiate state). In my case, I had an old T-Mobile router from 10 years ago that I booted up, disabled WiFi/DHCP on, and just used the switchports in the back as my simple, unmanaged switch. Lo and behold, I was back to getting 300/300, but this time, it was on the hEX refresh and not just my Orbi.

Hopefully this post will help someone out in the future from pulling all of their hair out. A $15 unmanaged Netgear switch is now on order, and I’ll report back with the particular model if it works as intended for this particular issue.

So the problem is clearly with Verizon ONT ...

Just wondering:
What port did you use for uplink to ISP device ? I suppose ether1 ?
You are aware ether1 is handled completely different from the 4 other ether ports on that particular version of Hex ? (see block diagram)
When you do speedtests maxing out that connection, what CPU usage do you see on your Hex ?

I have Hex Refresh here in my home lab (to use as virtual ISP router when I need to prepare setup stuff before deploying) and I use ether2 as uplink to my main router.

If I did not make it clear enough that it wasn’t just a hEX refresh issue when I mentioned others having the same (or similar) issues with other routers, I don’t know what to say there…

That aside, yes, I am using ETH1. No, I was not aware that ETH1 is treated completely different, but I’m definitely going to look into it now and give another port a shot. I’m brand new to the world of MikroTik and these hEX routers. I come from the land of enterprise Cisco/Aruba/Palo Alto devices. In 20 years of that landscape, I’ve never run into an auto-negotiation issue with these hallmarks before, which is why this particular issue was so frustrating.

Later on in the day, when my better half doesn’t require home internet for her business, I’ll tinker around and see what else I can find out.

If I did not make it clear enough that it wasn’t just a hEX refresh issue when I mentioned others having the same (or similar) issues with other routers, I don’t know what to say there…

No, no, you made that very clear

If I did not make it clear enough that it wasn’t just a hEX refresh issue when I mentioned others having the same (or similar) issues with other routers, I don’t know what to say there…

No, no, you made that very clear

Ok, so I was able to play with all of this before the household woke up…

Results: ETH2 does in fact work considerably better when acting as the WAN port versus ETH1. Instead of 300/~3, I’m getting 300/150~. It’s still not as good as having a switch between ETH1 and the ONT though. I would assume ETH2 would improve in much of the same way with a switch in the middle of it and the ONT, however, I did not test that out this round.

Despite my best efforts, I was unable to get ETH2 to do more than 150~ upload. Swapped back to ETH1 while using my T-mobile router as a middle-man between itself and the ONT, and I’m immediately back to 300/300. As a side note, I am using the same CAT6 cabling for all of these different setups; I’m just moving the same cables around.

Thanks for the tip on swapping out the ports. Even though the manual explicitly states to use ETH1 with one’s router, I’m sure your tip will help someone else out that doesn’t have another network device to use as a middle-man.

So the problem is clearly with Verizon ONT ...

Just wondering:
What port did you use for uplink to ISP device ? I suppose ether1 ?
You are aware ether1 is handled completely different from the 4 other ether ports on that particular version of Hex ? (see block diagram)
When you do speedtests maxing out that connection, what CPU usage do you see on your Hex ?

I have Hex Refresh here in my home lab (to use as virtual ISP router when I need to prepare setup stuff before deploying) and I use ether2 as uplink to my main router.

I just realized I hadn’t answered the CPU usage question. Pulling down 300Mbps, CPU is at ~20% during the download phase of the speedtest. While uploading, CPU drops down to around 2-3% (when ETH1 is directly connected to the ONT).

Thanks so much for posting this solution! I fortunately had a 1GB switch laying around to try this out. I was about to return the hEX Refresh!

Thanks so much for posting this solution! I fortunately had a 1GB switch laying around to try this out. I was about to return the hEX Refresh!

You’re welcome! I figured only one aneurism was enough as far as this matter was concerned . I’m glad to at least help someone else with the same issue. Thankfully this is the only “major” quirk I’ve run into with the Refresh specifically, so hopefully you’re also now in the clear!

One month later and still in the clear! Thanks again! Now I have to decide if I should replace perfectly working CAPs with hAP ax ones in order to upgrade to Wifi6.

One month later and still in the clear! Thanks again! Now I have to decide if I should replace perfectly working CAPs with hAP ax ones in order to upgrade to Wifi6.

No problem! I’m glad it’s working for you. For the others out there, I winded up going with a simple 5-port TPLink switch, and it has worked perfectly so far.

I purchased two hAP AX2’s alongside the hEX refresh and have them configured using CAPsMAN on my hEX. After muddling through that learning curve, everything works great. I was using an Orbi RBR50 with one satellite before, and performance is definitely better all-around. Range isn’t as great, but I also don’t need my wifi network to reach past my house as much as the Orbi system did. I’m currently on RouterOS 7.17 all-around and I have no complaints! Definitely recommended if you’re not using WIFI6 yet and have devices that support it.

One month later and still in the clear! Thanks again! Now I have to decide if I should replace perfectly working CAPs with hAP ax ones in order to upgrade to Wifi6.

If you have cap AC devices, you can use wifi-qcom-ac driver. No need to replace them yet (unless you really want to).

I am reviving this thread, since I believe there may be an ethernet flow control bug in RouterOS itself.

My ISP is also FiOS, and I am seeing extremely similar upload speed problems with my hAP ax3. After much digging and experimenting I discovered that my hAP ax3 is receiving "Rx Pause" flow control frames from the FiOS ONT any time I run a speed test/check tool to verify my upload speed, and that particular port counter increases steadily every time I am trying to consume all the upload bandwidth.

Now, my hAP ax3 flow control settings are *definitely* set to OFF (which is also RouterOS default) for both TX and RX on ether1 (which is connected to FiOS ONT directly), which means it should simply ignore these "Rx Pause" frames coming from upstream. However, it doesn't seem to be ignoring them at all, and in fact looks like it is still pausing even though the setting is off.

I happen to have an x86 system (super old J1900 cpu, but still works fine) also running RouterOS, which is configured identically to my hAP ax3 in all regards possible. On that unit, also with flow control on it's ether1 set to OFF, I have no upload problems at all and consistently and constantly get the full upload bandwidth every time I run any speed check/test tool.

Can anyone else having this issue please check your Rx Pause counters and maybe see if the behavior looks the same? It seems to me that inserting a dumb switch between the MikroTik router and the FiOS ONT may result in the dumb switch "eating" the Rx Pause frame, so that the 'Tik never gets it and continues sending full speed...

In such cases, where the uplink (300 Mbps) is slower than the speed of the ethernet (ports) in the LAN (1 Gbps) the solution might actually be to turn on flow control on all the MikroTik router's ethernet ports on the affected path. Flow control is per default off on MikroTik devices so you can try to turn that on.

Here are some recent examples where doing so helps when traffic from a link with higher throughput rate need to be squeezed into a slower link:

viewtopic.php?t=215677#p1134411
viewtopic.php?t=215296#p1131461

I also provided some explanation in the 2nd link that might apply to your situation.

I appreciate the suggestion of turning flow control on, except that will completely mess with my QoS, so I’m not doing that.

Also, my x86 router (also with flow control off) works fine? I’d rather see the problem actually corrected rather than band-aided.

Edit: I suppose my point is that the pause frames should be properly ignored, if that is in fact the root of this particular problem.

No, the problem is because your router is ignoring the pause frames sent by the ONT. The ONT can not keep up with the 1 Gbps rate that the router is pumping to it. Before its buffers are completely filled up the ONT uses flow control (by sending pause frames) to tell the MikroTik router to slow down while it tries to process the frames in the buffers (sending to uplink).

If flow control is enabled on the ports of the MikroTik device, both the one facing the ONT as well as the port facing the client device in LAN, the router will be able to detect the pause frames and in turn also tell the sending client device (your PC) to slow down (with pause frames too). After a while, everything will be calibrated and your PC will only generate frames at the uplink rate of your internet connection instead of 1Gbps. The buffers on the ONT will not be overfilled and no frames need to be dropped.

If in contrary you disable flow control on your hAP ax³ (which is the default setting), the router ignores the pause frames, the PC keeps pumping ethernet frames at wire speed (1 Gbps) which the hAP ax³ can push to the ONT at the same speed. The ONT is unable to forward all the frames to uplink, all buffers are filled up, so it will start dropping all the excess frames. Which means massive packet loss.

Now you can read up on terms like TCP window, congestion control algorithm, bandwidth delay product, etc... you will understand that when there are packet loss, all CCAs will have to drastically reduce the sending window size, which reduces sending (upload) throughput to a crawl. You do not want packet loss with your upload!

I can plug in my MikroTik x86 router and I do not observe the same throughout issue. I can plug in an OpenWRT router and I also do not have any problem.

None of my other routers experience this “massive packet loss” you describe as a consequence when they also have flow control disabled…

The original poster could also plug in his Orbi and it also did not have a problem.

If the problem is flow control, as you state, would not all of these routers have the same problem? After all, they are also connected at 1Gb and without flow control enabled. It’s only have a flow control problem with these specific MikroTik routers? Why is it no longer a problem if you simply insert a switch in front of the MikroTik router? How can the ONT suddenly keep up with a switch in between (everything still at 1Gb, remember?) if it’s a flow control issue? If you can answer that satisfactorily I may believe you.

First, there is no reasons why you cannot enable flow control on the ethernet port when you use other QoS mechanism such as Queues, that is not related.

Second, whether flow control is enabled or not on OpenWrt depends on the chipset/network adapter. You need to add ethtool and run it with the parameter -a to see whether it's enabled or not (but only if the driver supports reporting the setting!). Some examples where it's enabled by default:

https://forum.openwrt.org/t/mt7621-mt75 ... orts/76006
https://forum.openwrt.org/t/need-help-t ... orts/98353

Same with your x86 RouterOS device, the network adapter that you installed might have hardware flow control enabled by default that RouterOS cannot even change if the driver doesn't support reporting/changing the setting. That x86 device running RouterOS is not a MikroTik device, please note that in my previous post I specifically wrote MikroTik routers and MikroTik devices.

Even the dumb switch that you put in between might know how to pause sending traffic and start filling its own buffers, that might cause frames to be dropped sooner or later, depending on that switch's own buffer size, and that difference can affect how the congestion avoidance algorithms tune the window size.

One thing that you can be sure of, is that if you see the ONT sending pause frames to the MikroTik router, and the MikroTik router does nothing to react, the ONT device WILL drop ethernet frames. To achieve high throughput with TCP and delays in the tens or hundreds ms of the internet, the sending side have to send A LOT of packets before expecting to receive a corresponding ACK response, that's the TCP window for you. Do you know what happens if just a few of those packets sent before expecting an ACK are lost? You should read more about that.

Ok, so you cannot explain it either and dance around the question with a bunch of maybe and what-if.

Enabling rx flow control will cause latency and pauses in all traffic from the router which is highly undesirable. TCP has its own congestion control mechanisms and I rather use that so I don’t have to deal with unneeded 33 ms “rx pause” latency gaps for no reason. Perhaps you should search and read on that.

Thanks for trying to help but this only convinces me more that something is wrong with RouterOS on these particular models.

Just to add a few details to what @CGGXANNX explained so clearly.

TCP congestion control is indeed smarter and more adaptive, but it only begins to work once packets are actually received by the TCP stack. If frames are dropped earlier due to buffer overflows and the router ignores incoming pause frames, TCP has no way to prevent that loss. This is where flow control can sometimes help, since it allows the upstream device to avoid dropping traffic before TCP has a chance to react.

Regarding the 33 millisecond pause you mentioned, that sounds more like a hardware or driver-specific issue. Flow control by itself does not introduce fixed delays like that. If you are consistently seeing that kind of pause, it might be worth checking offload settings, the network driver, or testing with different hardware.

So yeah, your setup might be misbehaving, but that does not mean flow control as a feature is fundamentally flawed. It all depends on the hardware and how traffic is handled in your specific case.

Regarding the 33 millisecond pause you mentioned, that sounds more like a hardware or driver-specific issue. Flow control by itself does not introduce fixed delays like that. If you are consistently seeing that kind of pause, it might be worth checking offload settings, the network driver, or testing with different hardware.

This is wrong. Pause frames contain quanta value, which is requested period of "stop" time from sender. Maximum flow control induced delay for a 1Gbps link = 33.55ms.

In any case, what you and CGGXANNX would have me believe is that nomercymahem and dogenzenji both luckily had or bought some random dumb switch which somehow can send maximum throughput on the same 1Gbps link in front of router, without loss or latency, when routerboard itself directly hooked up cannot? Meanwhile, other models/makes of routers (different ones, mind you) can be directly connected without fiddling with flow control and also not have a problem? Hmmm.

Regarding the 33 millisecond pause you mentioned, that sounds more like a hardware or driver-specific issue. Flow control by itself does not introduce fixed delays like that. If you are consistently seeing that kind of pause, it might be worth checking offload settings, the network driver, or testing with different hardware.

This is wrong. Pause frames contain quanta value, which is requested period of "stop" time from sender. Maximum flow control induced delay for a 1Gbps link = 33.55ms.

In any case, what you and CGGXANNX would have me believe is that nomercymahem and dogenzenji both luckily had or bought some random dumb switch which somehow can send maximum throughput on the same 1Gbps link in front of router, without loss or latency, when routerboard itself directly hooked up cannot? Meanwhile, other models/makes of routers (different ones, mind you) can be directly connected without fiddling with flow control and also not have a problem? Hmmm.

For me, if I hook up ETH1 on my hEX refresh to the port on my ONT, I seem to have a failure to properly auto-negotiate link speed. Even though my hEX shows as coming as 1Gb/full, it is behaving as if it is coming up as 1Gb/half. If I force it to 100/full, I get full 100/100 throughput. If I set it back to 1Gb/full or auto, I now suddenly have less than a tenth of the upload speed as I do when I force 100/full. Any other router I have in the house does not have this issue. I only have 300/300 from Verizon, so I’m not asking the ONT or my hEX to do any real work here. I believe there may be a hardware issue on the hEX refresh for sure (at least when using ETH1 to connect to the ONT). I have tried virtually every setting combination I can think of on my hEX (countless hours tinkering), and it doesn’t behave properly until I throw a switch between it and my ONT. Makes no difference if I have flow control turned on or off for my particular issue.

If anyone believe to have found a bug, in software or hardware, they should report the issue to Mikrotik directly, opening a support ticket:
https://mikrotik.com/support

The forum is only a place where users exchange their experiences, opinions and when possible try and help each other.

@codelogic; FWIW, most FTTH providers use a soft-cap with line speed set to max, meaning that buffering and traffic shaping are handled upstream in the distribution network at the POP switch.

@Larsa thanks for the feedback/input. I'm more of a lurker on the forum here but I've seen many of your posts in the past, very informative.

Given my experience with the hAP ax3 on my FiOS circuit and the shared experience by others re: hEX refresh, I am of the opinion that the MikroTik routers are not handling something properly. Other routers (or a dumb switch hooked up in front?!) work just fine without any throughput or latency problems, or fiddling with any settings which need typically need not be touched.

@nomercymayhem Thanks for jumping back into the "mayhem" Your feedback (and others) is quite telling and also points towards some sort of bug/problem for sure on at least the hEX refresh, if not RouterOS entirely.

Makes sense and I agree that the pattern might indicate something is off with how those models (or a specific ethernet PHY) handle certain flow conditions. That said, it's probably a wise idea at this point to file a bug report with support, and providing a sniffer trace would probably be a good idea too.

file a bug report with support, and providing a sniffer trace would probably be a good idea too.

Good call on the sniffer trace -- if I get some free time this week I'll see if I can generate multiple packet captures from my "good" routers vs. my hAP ax3. Would probably be very interesting to compare.