Packet loss and high latency between hEX S Refresh and CRS106 over fiber

Hi everyone,

I'm troubleshooting an issue in a client’s network and decided to reproduce it in a lab setup.
The problem seems to appear only when using the new hEX S Refresh model connected to a CRS106 via fiber.

Lab setup

Devices:

hEX S Refresh
CRS106-1C-5S (latest RouterOS and firmware)

Connection:

SFP fiber link between them

Config:

Both devices have a bridge with all ports added
Each bridge has a single IP in the same subnet
No VLANs, no firewall rules, no NAT — pure L2 bridge test
Firmware: both devices freshly upgraded to latest stable RouterOS and firmware

When pinging between the devices, I consistently get 1–2 ms latency (which is quite high for a local link) and frequent timeouts:

807 10.254.254.2 timeout
808 10.254.254.2 56 64 1ms649us
809 10.254.254.2 timeout
810 10.254.254.2 56 64 1ms629us
...
820 10.254.254.2 56 64 1ms694us
821 10.254.254.2 56 64 1ms613us
822 10.254.254.2 56 64 1ms816us

Packet loss starts small (around 5–10 %) but increases over a few minutes until pings almost constantly time out.
CPU load is low, temperature normal, and there’s no sign of a loop.

Control test

When I replace the hEX S Refresh with a hEX S PoE (also fiber-capable) using the same CRS106 and the same SFP transceivers and cables,
everything works perfectly — sub-millisecond latency (measured in microseconds), zero packet loss.

So the only variable that causes the issue is the hEX S Refresh.

Question

Has anyone experienced similar behavior between hEX S Refresh and CRS106 bridges over fiber?
Is there any known issue with hardware offload or SFP handling on the new hEX S Refresh hardware revision?

Thanks in advance for any insight!

Rysiek

Try to disable "Auto Negotiation" and use fixed speed for SFP on hEX S Refresh...

What is "hEX S PoE"? exist? at least is simply hEX S [RB760iGS]...

And also hEX S refresh [E60iUGS] is PoE, do you know?

One device is MMIPS, the other is ARM.

Which SFP modules do you use? You were careful not to write it down...

Maybe the ARM drivers are different from the MMIPS drivers...

Thanx for answer.

I’ve already tried swapping SFP modules from several manufacturers, MikroTik, TP-Link, and prolabs and the behavior is exactly the same.

There’s also no difference whether I leave Auto Negotiation on or set the SFP interface speed and duplex manually.

The problem appears only when using the hEX S Refresh (E60iUGS) with CRS106.
When I replace it with the older hEX PoE (RB960PGS) using the exact same CRS106, SFP modules, and cables, everything works perfectly — zero packet loss and normal latency.

So it seems specific to the new hardware revision (E60iUGS).

Have no idea, what the hell :face_with_spiral_eyes:

hEX PoE [RB960PGS] is mipsbe, not arm, so the driver...

Try remove all ether except SFP from the bridge..........

I’ve just tested that — removed the combo port from the bridge in my lab setup, leaving only the SFP interface.
Unfortunately, this doesn’t change anything — packet loss still appears after a few minutes.

In the real configuration, I actually need the combo port to be part of the bridge, so removing it wouldn’t be a viable solution anyway.

Of course, I could simply replace the hEX S Refresh with a hEX PoE and the problem disappears completely, but that feels more like a workaround than a proper fix.

:confused:

Which RouterOs versions are you running?

In some recent versions some speed issues on the Hex refresh (that should be common with the Hex S refresh/2025) have been reportedly fixed BUT the current 7.20.2 seems like having numerous various issues.

If I were you I would try 7.20.2 and if It behaves like you describe, create a ticket.

Since It will likely take some time, using temporarily the old Hex S seems to me a valid workaround.

Thanks!

I first tested this on RouterOS 7.18.2, and later upgraded both devices to 7.20.2.
Right after the upgrade I almost thought the issue was gone, everything looked stable for a while, but after about 30 minutes, the problem started to build up again, reaching roughly 15% packet loss.

So it does seem to persist even on 7.20.2.
Looks like I’ll have to open a support ticket for this one.

I’ve opened a support ticket for this issue: SUP-203214.
Hopefully the MikroTik team can reproduce it on their side and identify what’s happening with the hEX S Refresh (E60iUGS) SFP interface.

Generally speaking, it is unusual that something like what you describe "builds up" over time, if it is a configuration issue or a driver/Ros issue it should either work or not work immediately or in a very short time, or that would be some signs of what is causing the issue (as an example logs) accumulating data over time, the half an hour or so delay sounds strange.

Yes, I completely agree, it’s a very strange one.
That unstable, “gradually degrading” behavior is exactly what made me start troubleshooting from the hardware side in the first place.

Before deciding to reproduce the issue in a lab setup on my desk, I went through the following steps directly in the production environment:

replaced the fiber link and both patch cords,
swapped SFP modules (tried several brands),
even replaced one of the hEX S units entirely.

None of that changed anything!
What’s interesting is that the CRS106 shows this behavior only when connected to a hEX S Refresh (E60iUGS).

In other parts of the same network I have CRS106↔CRS106 and CRS106↔CRS326 fiber links, and all of those work perfectly fine, no packet loss or latency issues.
Even more curious: I have another identical hEX S Refresh connected via fiber to a CRS326, and that link also works flawlessly, no packet loss at all.

So it really seems that this issue occurs only in the specific CRS106 ↔ hEX S Refresh combination.
It’s a truly odd one.

For now, I’ve ordered two hEX PoE (RB960PGS) units to temporarily work around the problem. I have to keep the client happy and can’t afford downtime while this is being investigated.

The lab setup will remain on my desk for further testing once I receive feedback from the MikroTik support team.

Just a quick update to close the thread.

The MikroTik team was able to reproduce the issue in their lab environment and confirmed it wasn’t just my setup.
They suggested forcing the SFP interfaces to 1G-baseX mode on both devices (hEX S Refresh and CRS106) and then rebooting the CRS106.

After applying that fix, the link has been completely stable — zero packet loss, no latency spikes, even after extended uptime.

Big thanks to the MikroTik support team for investigating and providing a clear solution.
Hopefully this helps anyone who might run into the same issue in the future.

Is like what did I suggest... applied on both sides...

Which implies that you were 50% right or 50% wrong, depending on how you look at It ... :rofl:

Il senso è che io avrei provato da entrambi i lati se mi avessero suggerito qualcosa del genere, ma è un'altra storia...

The point is that I would have tried both sides if they had suggested something like that to me, but that's another story...

Sure :slightly_smiling_face:. I know, I am just kidding.

1 Like