CCR2216-1G-12XS-2XQ - 100G QSFP28 link to Cisco NCS540 breaks on RouterOS >=7.17.2 (PHY regression); blocked from upgrading due to known L3HW/OSPF kernel panic on 7.16.2

Hardware / Role

  • MikroTik CCR2216-1G-12XS-2XQ, Border Router (iBGP core + eBGP upstream, full IPv4 table, ~1.08M routes)
  • Remote peer on the 100G link: Cisco NCS540 (main upstream trunk)
  • Currently pinned to RouterOS 7.16.2 in production

Problem 1 — 100G PHY regression from 7.17 onward

Cross-tested in a lab with an identical CCR2216 unit against the same Cisco NCS540 model.

Working (100G link up):

  • 7.15.2
  • 7.15.3
  • 7.16.2 (current production)

Failing (no link, RX/TX report -40 dBm / laser appears off):

  • 7.17.2
  • 7.18.2
  • 7.19.6
  • 7.20.1
  • 7.20.7
  • 7.20.8
  • 7.21.2

Troubleshooting done on the failing versions, without success:

  • Negotiation: auto / forced 100G / off
  • FEC: fec74 / fec91 / off / auto
  • Cold boot (2 min power-off) of both the CCR2216 and the NCS540
  • rate-select high/low

This looks like a regression in the Marvell PHY driver introduced starting with 7.17.

Problem 2 — L3 Hardware Offloading not usable on 7.16.2 either

Even on the working 7.16.2, enabling L3HW offloading triggers kernel panics / memory corruption on OSPF. So today this border router runs entirely in software — CPU sustained around 53% with connection tracking already disabled.

I searched the forum before posting and found this is a well-known, heavily reported issue on its own (many CCR2216 + L3HW threads), including one on this exact version:

But per our testing above, 7.20.x is exactly where our 100G link to the NCS540 stays down, so that fix doesn't help us — we're stuck between the two bugs.

What we're looking for

  1. Any known PHY interoperability parameter for 100G against Cisco NCS540 on versions >7.17.
  2. Or: confirmation of whether there's a patch release on the 7.16.x branch that fixes the L3HW/OSPF kernel panic without touching the PHY driver.

Happy to provide supout files or more detail if useful.

Could this be an SFP problem? Do you get link if you lookback?

A few more data points on this:

  • On a single CCR2216 unit, connecting its two 100G ports to each other via a DAC cable (not looping a single port back to itself, which isn't physically possible with a DAC cable) works fine on the failing RouterOS versions.
  • I was also able to connect two separate CCR2216 units directly to each other via DAC cable successfully, and connected a CCR2216 to a CRS326 via DAC at 40Gbps - both worked fine too.
  • We tested 4 different SFP/QSFP units of the same model (Cisco-Skylane Q28QD010C00DAD6) between the Cisco NCS540 and our CCR2216 units, and all 4 fail the same way on any version above 7.16.2. That rules out a single defective transceiver - it's consistent across multiple units of the same optic.

So to summarize what we know: CCR2216 port-to-port on the same unit (DAC) works, CCR2216-to-CCR2216 (DAC) works, CCR2216-to-CRS326 (DAC) works - the only thing that fails, consistently across 4 separate optic modules, is CCR2216-to-Cisco NCS540 using this specific optic model, and only on RouterOS >7.16.2. That points pretty strongly at something in the 7.17+ handling of this particular optic's EEPROM/negotiation being incompatible specifically with how the NCS540 responds - possibly related to the "sfp - improved power control configuration for QSFP optical modules according to the EEPROM field" change I found in the 7.17 changelog.

Did you check what SFP modules are officialy supported there MikroTik wired interface compatibility - RouterOS - MikroTik Documentation ?

Do not want that there is no regression but officialy onle supported modules are compatible.

Did you open a support ticket at the official support system?

I wonder what means "happy to provide more details if useful"? Did you hide something intrestinf in your case? Supout file is needed only for raised ticket, not for forum users.

Yes, we have an open ticket - SUP-202533 - with a supout.rif from the failing version (7.20.2) and one from the working version (7.16.2) for comparison, already attached there. It's been open for 317 days now (since 24/Oct/25), with long silences in between (e.g. 9 months of nothing between our Nov/25 reply and the Aug/26 follow-up) and no fix or actionable info yet - that's exactly why we're also posting here: to get visibility, either from other users who've hit the same combination, or from someone on the MikroTik team who can move the ticket forward.

Important clarification on isolating this: the same Cisco-Skylane optical modules work fine when used between two CCR2216 units (MikroTik-to-MikroTik, same optics). They only fail specifically against the Cisco NCS540, and only on RouterOS >7.16.2. That rules out a generic "unsupported module" issue - the modules themselves link up fine on the RouterOS side, just not with this particular far-end device on the newer firmware.

On the "officially supported" point: per MikroTik's own compatibility doc, there's no restricted list - the stated policy is that any MSA-compliant module from any vendor should work, best-effort. The fact that these modules interoperate correctly with another CCR2216 already demonstrates that.

"Happy to provide more detail if useful" was just an open offer to whoever's reading this thread, not a hint that something's being withheld - the ticket has the same information as this thread (plus the supout files, which as you say belong there, not here).

First, I would stop using such old versions. I had issues with Mikrotik and Huawei at one point; switching to Huawei-compatible modules solved the problems. Mikrotik is usually more forgiving, but other brands aren't; we've seen that many times.

The modules are Cisco-compatible transceivers made by Skylane Optics (a well-known third-party transceiver manufacturer, not a Cisco product line) - part number Q28QD010C00D, 100GBase-LR4. They're programmed/coded specifically to be recognized as Cisco by the receiving equipment, with vendor lock-in compatibility as their whole selling point - not a generic/unbranded module.

For anyone who wants to check the module itself: product page - https://www.skylaneoptics.com/en/product/q28qd010c00d-qsfp28-100gbase-lr4/ and datasheet - https://assets.skylaneoptics.com/datasheets/Q28QD010C00D000_Datasheets_EN.pdf

That's what makes this puzzling: if the NCS540 were rejecting on vendor-ID, these are specifically built to pass that check. It's possible RouterOS 7.17+ changed something in how it reads/relays the EEPROM data during link negotiation (see the changelog entry I mentioned earlier - "improved power control configuration for QSFP optical modules according to the EEPROM field") in a way that affects what the NCS540 sees, even though the module itself hasn't changed. We don't have a genuine Cisco-manufactured optic on hand to test that theory further, but it's on our list if we can get one.

@MatiasMK88 could you clarify what you meant by "stop using such old versions"? It's not clear if you mean RouterOS version or the modules themselves. To be clear on RouterOS: we can't move to a newer version precisely because that's what breaks the 100G link in the first place (see the version table in the original post) - staying on 7.16.2 isn't a preference, it's the only version where this link works at all.

Yes, I was referring to the RouterOS system version. Tomorrow I'll gather the data on how I configured it on the Huawei switches to get it working and share it; it might help you, since I had a very similar problem.