CRS326-4C+20G+2Q+RM randomly losing 2.5GbE links to NAS — RouterOS 7.21.4

Fake_ID · September 2, 2026, 4:54pm

CRS 2.5GbE ports randomly losing link / bonding issues with NAS — RouterOS 7.21.4

Hello everyone,

I am experiencing a strange Ethernet link issue after recently upgrading the network infrastructure in a gaming/PC club, and I would appreciate some advice on where to look next.

Network topology

We recently upgraded the entire local network to 2.5/10GbE.

The main equipment is:

RB5009UG+S+IN — router/gateway
CRS309-1G-8S+IN — network core
2 × CRS326-4C+20G+2Q+RM — access switches
CRS310-8G+2S+IN — access switch

All switches are interconnected through the CRS309, which acts as the network core.

RouterOS versions:

RB5009: 7.19.6
CRS309 / CRS326 / CRS310: 7.21.4

The network is used in a gaming/PC club with a large number of gaming PCs, so there can be a significant amount of LAN traffic, especially when games and updates are being distributed from the local storage servers.

NAS configuration

There are also two NAS servers used for game storage and distribution, built with NVMe/M.2 storage.

Each NAS is connected to the switching infrastructure using:

4 × 2.5GbE Ethernet interfaces
4 × Cat7 patch cables
intended aggregate bandwidth of up to 10 Gbps per NAS

Initially, all four Ethernet interfaces on each NAS were configured as a bonded/aggregated link.

First issue — links started dropping while using bonding

Almost immediately after deploying the new equipment, during the initial testing, the setup did not remain stable for even two hours.

The Ethernet ports on the CRS connected to the NAS servers started losing link while operating as part of the bonded connection.

We temporarily removed bonding and configured the interfaces as independent 2.5GbE Ethernet links using normal auto-negotiation.

In this configuration, the ports initially appeared to work normally.

However, we noticed another strange behavior.

After rebooting either side — the CRS or the NAS — not all links would reliably come back up.

For example, out of four 2.5GbE interfaces:

sometimes all 4 links would come back;
sometimes only 2;
sometimes only 1.

As a result, we could not reliably restore the expected aggregate bandwidth of up to 10 Gbps between the NAS and the network.

Patch cables replaced

Initially, we suspected the patch cables.

All patch cables between the NAS servers and the CRS switches were replaced.

After replacing them, the problem temporarily disappeared.

We did not re-enable bonding at this point and continued running the interfaces independently with auto-negotiation enabled.

For some time, everything appeared to be stable.

The issue occurred again without any reboot or configuration change

Today, the same issue occurred again during normal operation.

This time there were:

no switch reboots;
no NAS reboots;
no cable reconnections;
no configuration changes.

The affected Ethernet ports simply lost their links during normal operation.

When I tried to manually force the expected Ethernet speed in WinBox, RouterOS reported that the selected speed was not supported.

This is particularly strange because these exact same ports had previously negotiated and operated normally at 2.5 Gbps.

Manually disabling and re-enabling the affected Ethernet interfaces on the MikroTik switch caused the links to negotiate correctly again.

After doing this, all four 2.5GbE links came back up and the expected aggregate bandwidth of up to 10 Gbps to the NAS was restored.

Current observations

So far, the behavior appears to be the following:

After initialization, the 2.5GbE links work normally.
One or several links may randomly go down.
The issue originally appeared while using bonding, but has now also occurred with bonding completely disabled.
Replacing all patch cables did not permanently solve the problem.
After rebooting either the NAS or the CRS, only some of the four 2.5GbE links may come back up.
When the problem occurs, RouterOS may report that manually selecting 2.5 Gbps is not supported.
Disabling and re-enabling the affected CRS Ethernet ports restores normal 2.5GbE operation.
Once all four links are restored, the NAS can again provide the expected aggregate bandwidth of up to approximately 10 Gbps.

Questions

Has anyone experienced similar behavior with MikroTik CRS switches when connected to 2.5GbE Ethernet devices?

Could this be related to:

a RouterOS 7.21.4 bug?
a CRS PHY/driver issue?
2.5GBASE-T auto-negotiation?
an incompatibility between the NAS Ethernet controllers and the Ethernet PHYs used by the CRS?
bonding/LACP initialization?
Energy Efficient Ethernet (EEE)?
some other Ethernet auto-negotiation issue?

The fact that simply disabling and re-enabling the affected Ethernet ports on the CRS immediately restores the links makes me suspect that this could be related to PHY initialization or auto-negotiation rather than the physical cabling itself.

What would you recommend checking, and what diagnostic information should I collect the next time the problem occurs before disabling/re-enabling or otherwise resetting the affected ports?

Any suggestions or reports of similar experiences would be greatly appreciated.

rextended · September 2, 2026, 5:14pm

Schema?
Configuration?

You have 3 CRS models, only 2 with direct 2.5G ports. What you are talking about?

You have try to change the NAS with another, or the CRSxxx with another?

Larsa · September 2, 2026, 5:20pm

This pretty much sounds more like a 2.5G PHY/autonegotiation issue than a bonding/LACP problem, especially since it also happens with bonding disabled and comes back after bouncing the port.

But the main missing part here is the ROS configuration, what NICs the NASes actually use, like the exact make/model/chipset, OS, driver/firmware versions, whether both NASes use the same NIC hardware, and so forth.

Next time, before bouncing the port, grab /interface ethernet monitor <port> once and the interface stats/logs. Comparing a dead port with a working one should tell us a lot.

Fake_ID · September 2, 2026, 7:23pm

Thanks. Let me clarify the topology, as I probably did not describe it clearly enough in the original post.

RB5009UG+S+IN — main router with two ISP connections, connected to the CRS309-1G-8S+IN via SFP+.
CRS309-1G-8S+IN — network core.
CRS326-4C+20G+2Q+RM #1 — connected to the CRS309 via SFP+. NAS #1 (production) is connected directly to this switch using 4 × 2.5GbE RJ45 links.
CRS326-4C+20G+2Q+RM #2 — connected to the CRS309 via SFP+. NAS #2 (backup) is connected directly to this switch using 2 × 2.5GbE RJ45 links.
CRS310-8G+2S+IN — also connected to the CRS309 via SFP+. It has 8 × 2.5GbE RJ45 ports, but no NAS is currently connected to it.

All remaining 2.5GbE access ports are connected to gaming PCs.

Simplified topology:

ISP #1 ─┐
        ├── RB5009 ── SFP+ ── CRS309 (CORE)
ISP #2 ─┘                     │
                              ├── SFP+ ── CRS326 #1
                              │             ├── 4 × 2.5GbE ── NAS #1 (production)
                              │             └── 2.5GbE ── Gaming PCs
                              │
                              ├── SFP+ ── CRS326 #2
                              │             ├── 2 × 2.5GbE ── NAS #2 (backup)
                              │             └── 2.5GbE ── Gaming PCs
                              │
                              └── SFP+ ── CRS310
                                            └── 2.5GbE ── Gaming PCs

The same type of issue has occurred with both NAS servers connected to two different CRS326 switches, so it does not appear to be limited to one particular CRS326 or one NAS.

However, it is much easier to catch and observe on NAS #1 because it is the production game storage server and is under continuous load during normal operation. NAS #2 is the backup server and is mostly idle, so the problem is much harder to reproduce or notice there.

We have already replaced the patch cables. We have not yet moved one of the NAS connections to the CRS310 for comparison, but that is something I can test.

Regarding the NAS NICs and software/driver versions: I will collect the exact NIC models/chipsets, OS, driver and firmware versions from both NAS servers and post them here.

And next time the issue occurs, I will not reset the affected ports immediately. I will collect:

/interface ethernet monitor <port> once

together with interface statistics and logs from both an affected port and a working port, so we can compare them before doing disable/enable.

fchk · September 2, 2026, 7:31pm

What exact type are your NAS servers?

Preconfigured systems or custom-built PCs?

I'd switch the NAS servers to single 10G SFP+ DAC connections. Lower latencies, fewer things to break, more stable.

Fake_ID · September 2, 2026, 8:54pm

The NAS servers are provided to us as a rental service by a third-party company.

The hardware itself is based on mini-PC platforms. Unfortunately, the provider does not offer a configuration with a 10G SFP+ interface. Their NAS systems are equipped with either 4 or 5 × 2.5GbE RJ45 interfaces, so a single 10G SFP+ DAC connection is currently not an option for us.

I agree that a native 10G SFP+ connection would be much simpler and would probably be my preferred solution as well.

There is also one important detail that I should have mentioned earlier:

These same NAS servers had previously been running reliably for approximately 1.5 years on CRS326-24G-2S+RM switches.

At that time, the NAS interfaces were configured in bonding and provided approximately 4 Gbps aggregate bandwidth using 4 × 1GbE links. We did not experience this kind of random physical link loss with that setup.

The problems started immediately after we upgraded the switching infrastructure and moved the NAS servers to the new CRS326-4C+20G+2Q+RM, using their 2.5GbE interfaces at 2.5 Gbps.

So, in simplified form:

Previous setup:
NAS → 4 × 1GbE → CRS326-24G-2S+RM → stable for ~1.5 years

Current setup:
NAS → 4 × 2.5GbE → CRS326-4C+20G+2Q+RM → link problems appeared almost immediately after deployment

This is one of the main reasons why I am looking at the CRS326-4C+20G+2Q+RM / 2.5GbE PHY / auto-negotiation side of the problem rather than assuming that bonding itself is the cause.

The CRS326-4C+20G+2Q+RM is also relatively new hardware and, as far as I can tell, it is not yet widely deployed in installations like ours. Because of that, there does not seem to be much real-world information available about this particular combination of hardware and 2.5GbE devices. That is essentially why I decided to ask here and see whether anyone has encountered similar behavior.

Regarding the NAS software, I unfortunately cannot provide an exact answer at the moment. It appears to be based on something like TrueNAS or another open-source/customized solution, but the provider does not disclose the exact software configuration to us.

I have already contacted the provider and specifically requested information about the Ethernet controllers/NIC chipsets used in these units. Hopefully, this will help us determine whether there could be a compatibility issue between the NAS NIC PHYs and the 2.5GbE ports on the CRS326.

Once I receive the exact NIC/controller information, I will post it here.

In any case, I would still like to understand why the physical 2.5GbE links are dropping. Even if replacing the setup with native 10G SFP+ would avoid the problem, the 2.5GbE ports should still be able to maintain a stable link during normal operation.

Fake_ID · September 3, 2026, 5:16am

Thanks everyone for the suggestions. I now have more information from the NAS provider, and I was also able to catch the problem while one of the links was actually down.

NAS / NIC details

I have now received the Ethernet controller details from the NAS provider:

Intel Ethernet Controller I226-V [8086:125c], rev 04
Linux driver: igc
Debian-based kernel: 6.1.174-1
Firmware versions reported by the provider: 2017:888d and 2014:8877
EEE is disabled
Auto-negotiation is enabled
All NICs support and advertise 2500BASE-T

Both NAS systems use the same type of Ethernet controller.

Today I was able to catch one of the links in the failed state.

The production NAS is connected to ether17-NetX through ether20-NetX on one CRS326-4C+20G+2Q+RM.

At the time of testing:

ether17-NetX — link OK, 2.5Gbps
ether18-NetX — no link
ether19-NetX — link OK, 2.5Gbps
ether20-NetX — link OK, 2.5Gbps

Most interestingly, /interface ethernet monitor ether18-NetX once showed this while the port was down:

name: ether18-NetX
status: no-link
auto-negotiation: done

supported:
    10M-baseT-half
    10M-baseT-full
    100M-baseT-half
    100M-baseT-full
    1G-baseT-full
    2.5G-baseT

advertising:
    10M-baseT-half
    10M-baseT-full
    100M-baseT-half
    100M-baseT-full
    1G-baseT-full
    2.5G-baseT

link-partner-advertising:
    10M-baseT-half
    10M-baseT-full
    100M-baseT-half
    100M-baseT-full
    1G-baseT-full
    2.5G-baseT

So RouterOS reported status: no-link, while auto-negotiation was done, and both the CRS port and the link partner were advertising 2.5G capability.

I also checked the configuration of all four NAS-facing CRS ports. They were configured identically:

auto-negotiation=yes
advertise=10M-baseT-half,10M-baseT-full,100M-baseT-half,
          100M-baseT-full,1G-baseT-full,2.5G-baseT
tx-flow-control=off
rx-flow-control=off

ether17, ether19 and ether20 had the R (running) flag, while ether18 did not.

Before touching the failed port, I also checked the hardware interface counters. There were no obvious physical-layer errors on ether18:

rx-too-short:        0
rx-too-long:         0
rx-error-events:     0
rx-fcs-error:        0
rx-fragment:         0
rx-overflow:         0
rx-jabber:           0
tx-underrun:         0
tx-collision:        0
tx-late-collision:   0

The other three NAS-facing interfaces also showed zero values for these error counters. These interfaces have carried substantial traffic; for example, ether18 has previously received about 87.5 TB.

There are some TX drop counters on the interfaces, but no FCS/error-event/fragment/jabber/collision errors, so at this point I am not assuming that the TX drops are related to the physical link failure.

Then I performed a controlled reset of only the affected CRS port:

/interface ethernet disable ether18-NetX
:delay 3s
/interface ethernet enable ether18-NetX

I did not unplug/reconnect the cable and did not change anything on the NAS side.

After waiting 10 seconds:

/interface ethernet monitor ether18-NetX once

returned:

name: ether18-NetX
status: link-ok
auto-negotiation: done
rate: 2.5Gbps
full-duplex: yes

The link partner was again advertising 2.5G, and the interface became operational at 2.5Gbps.

So in this test, the exact same:

CRS port + cable + Intel I226-V NIC

went from:

no-link

to:

link-ok / 2.5Gbps / full-duplex

solely by disabling and re-enabling the Ethernet interface on the CRS.

There is one more symptom which may be relevant: during previous failures, some interfaces recovered at 1Gbps instead of 2.5Gbps.

The NAS-side logs from Sep 02 showed repeated NIC Link is Down / NIC Link is Up events on two physical interfaces. One of them repeatedly came back at 1000Mbps, while another came back at 2500Mbps, and that interface had also previously been observed negotiating at 1000Mbps.

At one point the production NAS therefore had:

1 x 1000Mbps
3 x 2500Mbps

instead of four 2.5GbE links.

Based on the above, this currently looks more like some kind of 2.5G PHY / auto-negotiation / link initialization compatibility issue between the CRS326-4C+20G+2Q+RM and the Intel I226-V than a bonding problem.

I am not saying this proves a RouterOS or CRS326 hardware bug — it could still be an interoperability issue involving either side — but the fact that the failed link can be restored to 2.5Gbps solely by resetting the CRS Ethernet port seems significant.

If there are any additional RouterOS diagnostics that would be useful to capture while the port is in the no-link state before resetting it, please let me know. I can collect them the next time it happens.