CRS310-8G+2S+IN: 802.3x pause to the sender isn't refreshed, and is disabled entirely by qos-hw-offloading

RouterOS
1

Setup:

a sender PC on ether5 (1G) streams 1024-byte frames at line rate through a CRS310-8G+2S+IN (98DX226S) to a receiver on sfp-sfpplus1 (10G, ConnectX-6 Dx). The receiver consumes at a fixed, deliberately limited rate, pauses the switch when it falls behind, and counts every lost frame. Both ports have rx-flow-control=on tx-flow-control=on and are hardware-offloaded bridge ports. Tested on RouterOS 7.19.6 and 7.24.4 with the same results.

My problems:

  1. The XOFF towards the sender isn't refreshed. The switch obeys the receiver's pause, but only holds the sender back while the receiver drains at least ~324 Mbit/s against a 1G sender. Below that, frames are dropped on sfp-sfpplus1 (tx-drop). The cut-off barely changes with the sender's rate. Receiver stalls are tolerated for about 33.5 ms + ~6 MB ÷ sender rate (e.g. 80–90 ms at 1G). 33.5 ms is exactly one maximum pause at 1G. This fits the switch sending one XOFF at its high-water mark and never resending it, only sending another after draining ~1.3 MB to its XON level. So pause only works for receivers draining at least ~1/3 of the sender's link speed.

  2. qos-hw-offloading=yes disables pause to the sender. With QoS offload on, the sender is never paused, and exactly the excess is dropped (66.1% at a 324M receiver). The switch still obeys the receiver's pause. It's the same whichever traffic class or use-shared-buffers setting I use. Turning offload off, plus a reboot, restores behaviour 1.

Also: I think tx-pause never counts, even when pause is clearly being sent (but my final checks got a bit lost in the testing - I'm confident this was the case on 7.19.6, but it might have been fixed).

Questions:

  1. Can the 98DX226S be made to refresh XOFF while congested, or can its XON/XOFF thresholds be tuned?
  2. What are the default XOFF and XON thresholds for 802.3x on this chip? Are they fixed, or derived from the port or queue buffer allocation? Can they be configured, e.g. to narrow the ~1.3 MB gap between XOFF and XON?
  3. Is it expected that QoS offload disables tx pause?
  4. The QoS docs list PFC and lossless buffers for the 98DX226S, but 7.24.4 has no way to configure either on this model. Is PFC usable on the CRS310, and how?
2

I'd say that the ~33.55 ms is a strong indication that the switch sends one maximum 802.3x PAUSE frame and then fails to refresh it while congestion remains. That lines up pretty well with what you're seeing.

What I wouldn't conclude yet is why it happens, especially once QoS HW offload is enabled.

Provide a full /export to rule out any configuration oddities. Then run the ROS commands below while you're seeing the problem, once with QoS off and once with QoS on.

/interface ethernet monitor [find] once
/interface bridge port print detail
/interface ethernet switch qos export
/interface ethernet print stats

Btw, just a small correction: the current QoS support table does not list PFC or lossless buffers as supported on the 98DX226S. The columns are there, but the 98DX226S row is not marked as supported.

3

Yes, apologies on my misreading of the PFC cols for the 98DX226S.

I've run those tests and captured the following logs as requested.

Each ethernet frame is made up of 256 32-bit ints which count up continuously, including across frames. This allows us to see any missing frame.

For each test, I show the logs captured both during a run and after the run has completed, as well as the diagnostics output from the generator tool and the receiver (rx) tool. These tool outputs show the data rate every second, along with reported error rates etc.

The generator is on port ether5, the receiver on port sfp-sfpplus1. ether3 is just a control port to the receiver (so not relevant). Some additional SSH data is sent over ether3.

For the run with HW QoS disabled, receiving at 500Mb/s, sending at up to 1Gb/s (the no error case, with effective back-pressure):

switch-qos-off-during-500M.txt (26.5 KB)

switch-qos-off-after-500M.txt (26.5 KB)

generator-qos-off-500M.txt (7.6 KB)

rx-qos-off-500M.txt (19.1 KB)

For the run with HW QoS disabled, receiving at 300Mb/s, sending at up to 1Gb/s (the error case, packets are dropped, but some PAUSE frames are sent by the switch and handled properly):

switch-qos-off-during-300M.txt (26.5 KB)

switch-qos-off-after-300M.txt (26.5 KB)

generator-qos-off-300M.txt (7.5 KB)

rx-qos-off-300M.txt (26.6 KB)

For the run with HW QoS enabled, receiving at 500Mb/s, sending at up to 1Gb/s (the sender never receives any PAUSE frames and sends at close to line rate, many packets lost):

switch-qos-on-during-500M.txt (26.5 KB)

switch-qos-on-after-500M.txt (26.5 KB)

generator-qos-on-500M.txt (6.0 KB)

rx-qos-on-500M.txt (26.1 KB)

4

Here's the output from \export:

forum-export.rsc (1.5 KB)

5

Good input. I'll have a closer look at this later, but first I'd remove the serial number from the export. If MikroTik IP Cloud is enabled, the serial number is part of the serial.sn.mynetname.net hostname and might be used to resolve your public IP address.

6

Good call, thanks. I also removed the MAC address. FWIW, it's not public, but thanks anyway.

7

Ok, had a look through all the files and the numbers line up pretty well.

With QoS off at 300M, the sender goes quiet for 33.6 ms every 52.4 ms. That's exactly one max pause at 1G (33.55 ms), and it never gets refreshed. So the sender starts up again while the switch is still congested, and about 279 frames get dropped each cycle (tx-drop-queue0 on sfp-sfpplus1). That works out to a hysteresis of ~1.37 MB and a cut-off around 321 Mbit/s, pretty much your ~324.

With QoS on, the sender never gets throttled at all. ether5 doesn't send anything that looks like PAUSE, and queue1 just drops the whole excess (47.75 %). Either the QoS path isn't hooked up to PAUSE, or queue1 hits its own buffer cap before XOFF kicks in. Can't tell which from the data.

tx-pause stays at 0 the whole time. The PAUSE frames seem to end up counted as 64-byte tx-multicast instead.

Btw, switch-qos-off-after-300M.txt looks like it's actually from a QoS-on run (queue1, 9M frames).

Neither case matches how the docs describe tx-flow-control (pause frames when the output is congested), so I'd open a ticket with MikroTik support and send them:

  1. A supout.rif taken while it's dropping with QoS off at 300M, and another one with QoS on. They're two separate behaviours and need a reboot in between.
  2. A short description of the setup: sender on ether5 (1G), receiver on sfp-sfpplus1 (10G, ConnectX-6 Dx), the receive rates and what you expect to happen.
  3. Your logs and the export, with the mislabeled 300M after-file renamed or left out.

Feel free to run this whenever you have a moment. TBH, you don't really need this to open the ticket, but it would confirm the issue even better. If you do, just add the output to the ticket.

  1. Run ethtool -S enp0s31f6 | grep flow_control on the sender before and after each run, if it's an I219/e1000e. That way you can see exactly how many XOFF and XON frames the sender actually gets, instead of us guessing from timing and counters.
  2. Try forcing ether5 to 100M. If it really is one pause that never gets refreshed, the stall should jump to ~335 ms and the cut-off drop to ~32 Mbit/s. If it doesn't scale like that, I've got it wrong.
  3. With QoS on, grab a few snapshots of /interface/ethernet/switch/qos/port print usage while it's dropping and check if queue1 is sitting at its cap. Just a few though, since the docs say this can mess with traffic on some chips. That should tell us which of the two QoS-on explanations it is.