Posting here an issue that plagued me until I got confirmation from mikrotik that they are aware of it, but don’t have a ETA when it might be fixed in SwOS.
This really should be in the documentation, there is nothing wrong with having “software” issues, but it is wrong to not let people know about them and run into them.
On a CRS326-24S+2Q+ running SwOS and a Q+BC0003-S+ 40 Gbps QSFP+ break-out cable to 4x10G SFP+, there is an implicit tear-down hierarchy between the first lane and the remaining lanes.
First, as a general remark, to ensure smooth connectivity, you should configure the breakout ports as follows:
- Type: 4x10G (not auto), with auto you are gambling
- Auto negotiation: OFF
- Speed: 10G, otherwise you are gambling esp. with Intel NICs
- Full Duplex: ON
- Flow Control: TX OFF/ RX OFF
The not documented flaw is that despite being advertised as independent ports, the lanes 2-4 are actually dependent on lane 1.
If lane 1 (i.e. QSFP+1.1 or QSFP+2.1) goes down, e.g. you switch off the other side, or unplug it, it will tear down the other lanes as well, i.e. QSFP+1.2, QSFP+1.3, QSFP+1.4 will immediately go link down.
You can try toggling enable/disable on these lanes, the most you get is a flap with the port negotiating, but then immediately going link down again.
What is perhaps even worse is that if you re-connect/switch on lane 1, only lane 1 will come up, lanes 2-4 will remain down. To restore connectivity, you need to manually disable lanes 2-4, apply the changes, manually enable lanes 2-4, apply the changes. Then you will have again all 4 lanes up.
This severely reduces the use-case of the breakout cable, as you have to model it as a single failure group instead of 4 independent ports (increased port density). In fact I have not found a practical use case for it at all with this bug.
The obvious recommendation from support was to switch to RouterOS, which is missing the point in my case.
Hope that this post will spare somebody some time in the future.