RB5009 Slow Speed 2.5G Bug Report (as requested)

RouterOS RouterOS beta
21

I can confirm I found noticed this just this morning on 7.2.1. I have gigabit cable service and my DOCSIS modem has a 2.5G ethernet port. Tests with 2.5G ethernet enabled get me ~600Mb/s. If I remove 2.5G from the ANEG profile I get 940 all day.

I turned TX Flow control on ether1 but it’s not actually sending any TX Pause frames. I’m using a 10G SFP downstream, so looking at the block diagram I don’t see anywhere that might actually call for flow control. Besides traffic from the 10G CPU interface towards the 2.5G ether1.

However looking at /interface/ethernet/switch/print stats does show many RX Pause Frames. Meaning something is overflowing and attempting to send pause packets internally to the router.

22

The 7.3 series works well for me with the cpu-flow-control on the switch. I’m running 7.3beta37 right now, and all is well for 10G, 2.5G and 1G hosts.

Something to note is that I had a lot of problems with my CRS328 switch in RouterOS, Mikrotik has confirmed tx-flow-control does not work correctly in RouterOS. I switched to SwOS and also tried other 10G/1G switches with flow control and I have zero issues now. I can saturate any mix of link speeds without issues on my network now.

23

I can confirm running v7.2.3 on RV5009.

At 2.5Gb I get 500Mbs - 600Mbs

When I turn it down to Gigabit I get full Gigabit around 930Mbs

Any chance of a fix making it into the v7.2 branch while we wait for v7.3 to hit stable?

24

Has the issue been fixed yet?

25

I hope to be wrong but It can’t be solved due of missing of cache in the switch chip, same problem of CCR2004

26

Does the new RB5009UPr+S+IN have the same problem or is it fixed on those new boards?

27

Everything seems fine here with cpu-flow-control

28



Where is this setting located?

29

@kennethc it’s in the switch settings.

30

I’ll wait until someone receives the new RB5009upr, states their usage case (WAN port vs switched port plus connected devices), and testing methodology with numbers before assuming anything is changed in the HW besides the addition of POE. I care about single stream, switched 2.5gbe port performance for both upstream and downstream but others may care about forwarding and multiple streams from one or more source ports. None of the options I tried which are exposed in ROS made a significant change including HW flow control and CPU flow control.

The fact that MT posts performance testing results is great and I hope they continue to do so. However, I don’t feel their testing methodology covers my network very well.

31

having same problem working on rb5009 + odi DFP-34X-2C2 gpon onu device,just working on 1Gbps.
Mannul set to 2.5Gbps didn’t working,test version on v7.4.1/v7.5/v7.5rc2

32

Hi - I wanted to jump in here in hopes to help. I am slated to have one of these models deliver today. I bought one NIB from eBay and paid more than I wanted to. Below is my current network topology at home:

  1. ISP: Spectrum Gigabit
  2. Modem: Netgear Nighthawk CM2000
  3. Current Router: TP-Link ER605
  4. Switch: Cisco CBS350-16FP-2G
  5. Wireless Access Points: Cisco Business 240AC x2 (these are only here so people can have Wi-Fi on phones/tablets. Everything else is wired.)
  6. Spokes: Cisco CBS110-5T-D (These are little unmanaged 5 port switches that I have going to drops in the wall to extend out the network to multiple wired devices in different rooms)

I have a network cabinet in the basement as the “Demarc” point, where the modem/router/switch live. This is my first Mikrotik product, but I know enough about networking to be dangerous. That said, if you want to send me a DM I am happy to follow steps to get results you are looking for.

33

Piggybacking off of my last comment (I can’t figure out how to edit the first one).

I’ve tried auto-negotiation on and off. I’ve tried flow control on and off. I’ve tried advertising at JUST 1000Mbps. I can’t seem to even come close to 900mbps; high side is between 400-425, but upload seems unchanged, hovering between 35-40mbps.

I checked switch settings and CPU flow control was already enabled.

RouterOS was updated to 7.5 stable prior to being deployed. Unsure where to go at this point.

EDIT: since I’m brand new to RouterOS, I decided to completely reset and start over, following instructions on RouterOS documentation from help.mikrotik.com on a first-time configuration.
Since then, I am now pulling down between 920-924mbps with 38-40mbps upload - this is WITH 2.5g advertising enabled on WAN (note: all other advertising speeds were removed except 1000M and 2.5G. IPv6 is also disabled.). The only difference is that there was a local bridge set up this time. Not understanding what that would have to do with it, but as far as I’m concerned, I’m adopting the “if it ain’t broke don’t fix it” mindset.

34

Problem still there tried on the 5009Upr latest stable and beta with a Wireless Access Point (Netgear WAX610) and I get reduced speed and significant latency spikes compared to 1Gbps rate.

35

TLDR probably just a TCP problem the whole time. Disregard this post lol

I had a similar problem with my CRS317 (CRS317-1G-16S+RM) - bought back in april 2021.

Connected my EnGenius EWS377AP which provides 2.5G LAN on it. Got full 1Gbps throughput from my internet connection (gigabit service → gigabit modem → CRS317 → AP → laptop) and perfect 10Gbps throughput on local devices (desktop → CRS317 → local machine) but terrible 30-40 Mbps throughput on local LAN + Wifi (local machine → CRS317 → AP → laptop). It drove me absolutely batty.

Fixed the problem with the help of this thread.

  1. Booted the CRS317 into SwOS, since I saw mention of 2.5G and 5G being fixed in the release notes for some other CRSxxx device. Dunno about the latest ROS, maybe it has the same fixes.
  2. Turned on Rx AND Tx flow control for all the affected SFP+ ports
  3. In addition, my AP had STP disabled by default (configuration on the AP device). I’m not network-savvy enough to actually know what STP/RSTP truly do under the hood - but I have to assume having STP enabled on an AP is important for this use case (to span a subtree?). Then, the CRS317 detects the AP providing (doing?) STP, and sets the link to the AP as “point-to-point” instead of “edge” or “endpoint” (forgot the term). I didn’t A-B test that one, but I am guessing it’s important.

Edit, after some more experimentation:
When running iperf from wired machine → CRS317 → AP → wifi machine, I see pause frames accumulating in their respective counters for both Rx on the one CRS317 port/interface (don’t know the proper term), and Tx on the other. The throughput stays up around the proper real-world amount, matching the throughput from ISP → CRS317 → AP → wifi machine.

So flow control is almost certainly working on SwOS 2.13.

However, going from wifi machine → AP → CRS317 → wired machine is still slow, with wild swings in bandwidth, and some outright connection drops. There are no pause frames, as expected; we’re sending data from a smaller pipe (2.5G from AP → CRS317) to bigger pipe (10G from CRS317 → wired machine). So unless there’s something obviously wrong with my config that I don’t know about (which is admittedly pretty likely), I may have issues with my AP after all. I’m going to swap it out for another one and see what happens.

Edit Edit
I was using iperf3 to test bandwidth, which defaults to TCP. Saw a bunch of retries, so I switched to UDP and the problem was fixed. (see https://software.es.net/iperf/faq.html under the section: “TCP throughput drops to (almost) zero during a test, what’s going on?”) Probably some kind of fragmentation issue with my NICs/configs on client devices, and TCP freezes up/retries a bunch.

To saturate my ~866 Mbps wireless link, I needed two udp streams on two Zen2 cores, and specified the client MTU manually:
iperf3 -s -p 5201 & iperf3 -s -p 5202 && fg
iperf3 -c -T s1 -p 5201 -M 1500 -u & iperf3 -c -T s2 -p 5202 -M 1500 -u && fg

36

Any updates on this for newer rb5009?

37

FWIW, it seems to have mitigated itself in ROS 7.8, but now thruput on the SFP+ port is much lower if the 2.5GbE port is also used.

38

I have the same problem

39

Could You elaborate on this please or post some sources?
I’ve been trying to use RB5009 with 2.5GBit GPON SFP+ stick (Leox LXT-010S-H) and whilst everything should check out, i seem to be only getting about 1.5Gbit/s so it sounds about right.

40

Actually ROS 7.9(?) fixed the problem I’d seen in 7.8 . I get full speed on both SFP+ and Ether1 now.