Hello,

I have a CSS610-8G-2S+ connected to a CRS310-8G+2S+ via S+RJ10 on CSS610 to RJ45 port on CRS310, at 2.5Gbe via CAT6.
On the CS610 I also have a PFsense internet router connected at 1Gbe and on the CRS310 a Linux box connected at 2.5Gbe.

When I run a speedtest from the Linux box, downloading reaches line speed (1Gbe) and no TX or RX pauses are seen on either Mikrotik device.
Upload speed however, is stuck at 200-300Mbps and I’m seeing tons of TX Pauses on SFP+ (w/ S+RJ10) port on the CSS610 but no RX Pauses on the CRS310's corresponding RJ45 port.
I can reach full 1Gbe download and upload if I connect the Linux device at 1Gbe instead of 2.5Gbe.

What can I do to reach full upload speeds from the 2.5Gbe device ? Am I not configuring Flow control correctly ? Why am I only seeing TX pauses on on device and no RX pauses on the other ?

Sorry if this was solved somewhere else, I've read some similar topics but none seem to point at a solution ( viewtopic.php?t=182691 and viewtopic.php?t=183165 )
Thanks

Did you enable both tx-flow-control and rx-flow-control on all involved ports on both switches?

As far as I understand, Rx flow control only signals the other end of each physical leg that it needs to pause if port receives feedback from upstream buffer ... and as far as I understand, most switches implement Tx buffers[*]. So essentially you need both flow controls enabled on those 1Gbps interfaces (PFsense) (which gets full and consequently drops frames) and on interconnect ports on CSS610 and CRS310 and on that 2.5Gbps port connecting linux box. Because eventually it's the linux box which needs to pause in order for the buffers to get flushed in the whole chain ...

[*] This comes with an unexpected side effect: when Tx buffer of a single port becomes full, some switches send pause requests on all other ports ... even though some ports may not be communicating with the overworked port. Because switch doesn't know in advance which egress port (and thus buffer) will be used for any ingress frame. I don't know if all the switches behave the same way, but I can't imagine any other behaviour in such case. And I guess this is the reason many people warn against enabling flow control on switches ... if flow control is off, then only stream which actually causes port overload suffers (possibly other streams which are using same egress port, become victims as well).

Hey thanks for the quick and detailed reply!

I've enabled Flow control TX/RX at each port on every switch along the chain, including the 2.5G interface on the Linux box and I'm seeing no difference. Lots of TX Pauses on the CSS610 port -> CRS310 and no RX or TX Pause frames anywhere else, including the Linux box.

Rx flow control only signals the other end of each physical leg that it needs to pause

Isn't it the other way around (enabling TX flow control does the signaling)? according to the Mikrotik docs:

Isn't it the other way around (enabling TX flow control does the signaling)?

My bad.
But the point is: you need both flow controls enabled on both sides of a link or else it doesn't work.

Now, in your particular case: you're saying there are Tx pauses on CCS610 but no Rx pauses on conected CRS310 port? This means that one of devices is buggy or misconfigured somehow. The following possibilities cone to my mind:

• CSS610 thinks it's sending pause frames but in reality it doesn't
• CRS310, even if port has rx-flow control enabled, ignores/filters pause frames
• CRS310 somehow ignores configured rx-flow control (buggy configuration parser)
• CRS310 has some configuration which effectively disables flow control (without explicitly showing it)

It's hard to determine if it's the first bullet, one would have to capture frames off the wire.

One thing which you could try is perform factory reset of CRS (or even netinstall) to make sure that config is as plain as it gets and only change flow control settings ... and re-test. It wouldn't be the first time where device didn't behave according to (visible) configuration and a proper configuration reset fixed it.

So I have an extra CSS610 lying around and decided to use it instead of the CRS310. Installed two SFP+ RJ-45 transceivers on the new CSS610, one for connecting to the old CSS610 (@10G) and the other to connect the Linux box (@2.5G). Starting with fresh SwOS configs on both, only change was to enable Flow control on all connected ports.

The first iperf3 run between the Linux box (on the new CSS610) and pfsense (on the old CSS610) resulted in 100k+ TX Pauses generated on the old CSS610 (where pfsense is connected) and 2 (LOL) RX Pauses on the new one (where the Linux box is). I've repeated the test 20 more times and I can only get the downstream to generate TX 100k+ pauses, no more RX pauses registered on the uplink.

I'm starting to understand what other were referring to when they said Mikrotik devices are tricky to set up ..

So it looks like TX Pause frames generated on one end do not register on the other end of the link no matter what combination of SWOS and RouterOS I use.

In my latest tests, I have the following connection chain :
Linux @ 2.5G via RJ45 on CRS310 -> S+RJ100 @10G on CRS310 -> S+RJ100 @10G on CSS610 -> Linux box at 1G
Running a speed test between the 2 Linux boxes, I see that the CSS610 is generating 1000s of TX pauses on the 10G link to the CRS310, but there are 0 RX pauses showing up on the CRS310

I'd really like to understand why this is happening but it's been a month and the ticket I opened with Mikrotik is basically being ignored

Could you share the ticket?
It seems I have similar problems.

Its totally undeterministic, but my MS Teams calls, video streaming and even my TV channels (its over ethernet) freeze time to time. Sometimes there are no problems for 15-30 minutes, but it often happens that it freeze every other minute.

I finally got an answer from support, it looks like the S+RJ10 10GBASE-T modules don't forward TX pause frames

You do not see Rx-Pause on CRS310 device due to S+RJ10 module. Tx-Pauses transmitted by the CSS610 is received by the S+Rj10 module, but it does not forward it up to the RJ45 link.

Do you have a chance to connect the CRS310 SFP+ with CSS610 SFP+ with 10G DAC or optical SFP+ module? In this setup, flow control should work as expected, you would should be able to see Rx-Pause on CRS310.