Slow download speed over SFP+ module

Hardware MikroTik hardware general
1

I have a CRS309-1G-8S+ on RouterOS 7.23.2 with the default configuration. The router is connected to the upstream router via a 10G SR transceiver and downstream via a 10Gtek (ASF-10G-T) 10GBase-T RJ-45 SFP+ transceivers.

ISSUE: Connecting to the router's 1G RJ-45 ethernet port, I get around 950Mbps upload and download. Connecting to the 10Gtek 10GBase-T SFP+ module , I get about 950Mbps upload but depending on the device, I get 7Mbps to 250Mbps download.

I changed the following with no success:

  • Interface SFP Rate Select: High to Low
  • Interface Ethernet TX and RX Flow Controls to AUTO, ON and OFF
  • Enabled and Disabled Auto negotiation
  • Removed each of the advertised rates one after the other
  • Replaced the 10Gtek module with another one in case it was defective

Nothing is working! I'm worried that ordering a Mikrotik 10GBase-T SFP+ module and waiting several days for the module to arrive, it will still not resolve the issue and it will be several days later.

Is there anything else that I should try before ordering and waiting days for a Mikrotik 10GBase-T SFP+ module?

Thank you!

2

Would be good to clarify how you tested: from, to, protocol, routed or switched traffic, ... as it matters a lot.

For starters, strange that 10g adapter, outputs only 1G speed, even for upload.
what are the relevant os's reporting regarding interface negotiation / speed setting?

3

crs309 is not a router

4

I get about 950Mbps upload but depending on the device, I get 7Mbps to 250Mbps download.

Sounds like packet stream/chunks coming from 10Gpbs have higher rate that can be output using 1Gbps port, so some packet must be dropped.

If its routed setup, try to add output buffering to downstream port, so spikes are smoothed out:

/queue type add kind=codel name=ethernet-aqm
/queue interface set ether1 queue=ethernet-aqm

Replace sfp1 with port actual port name.

5

Here a few more details about the infrastructure and tests that may be relevant.

  • 3Gbps FTTH internet connected to a Mikrotik X86
  • The Mikrotik X86 has a WAN port and LAN port used as a trunk with 10VLAN and a native LAN. It's connected to a CRS310.
  • The CRS310 is used as a switch where all ports are Trunks for the 10 VLAN and the native LAN
  • A 2.5G (cheap Chinese) managed switch is connected to the CRS310 via a 10G fiber cable.
  • 2.5G clients connected to the (cheap Chinese) 2.5G managed switch get near 2.5G speed when doing speedtest.net and fast.com.
  • 1G Windows clients connected to the (cheap Chinese) 2.5G switch get near 1G speed when doing speedtest.net and fast.com

I replaced the cheap Chinese switch with an CRS309 and immediately noticed the speed difference.

  • To eliminate as many variables as possible, I did a System/Reset Configuration on the CRS309 so it has the default configuration and only switching native LAN traffic.
  • A 1G Windows client connected to the CRS309's 1G ethernet port gets near 1G speed for both upload and download
  • A 1G Windows client (with negotiated 1G speed) connected to the CR309's SFP+ port with a 10GBe transceiver get near 1G speed for upload but about only 250Gbps download.
6

If I read correctly, I won't be able to use L3 Hardware Offloading if set up a /queue because the queue is managed by the CPU. Does that makes sense?

7

yes codel is software, but you can use hardware QOS

8

According to updated setup description L2 switching is done

A 2.5G (cheap Chinese) managed switch is connected to the CRS310 via a 10G fiber cable.

2.5G clients connected to the (cheap Chinese) 2.5G managed switch get near 2.5G speed when doing speedtest.net and fast.com.

so recommendation to try setup output que on CPU to smooth out spikes is not relevant.

If I read correctly, I won't be able to use L3 Hardware Offloading if set up a /queue because the queue is managed by the CPU. Does that makes sense?

Se above, switching are L2 and in most cases are HW offloaded. Regarding L3 HW offload case does not know, need to consult manual.

btw, what happens when CRS310 and CRS309 link is set to 2.5Gbe as when Chinese switch is used?

9

Your problem is that the 10GbE copper sfp always negotiates at 10G on the sfp (crs) side, even when it's connected at 1GbE on the copper side. This means that the buffers on the switch are ineffective, because from the perspective of the switch it's free to push way more than 1Gbps towards that port. (And the sfp-copper sfp doesn't itself have adequate buffers.)

Generally this is a setup that is not recommended, for this exact reason. In general, it will never work fully correctly.

You can try setting flow control on the given port to enabled. If the copper sfp emits the appropriate pause frames then this will help a lot. If it doesn't, you may attempt to find a module that does.

Another option is to use a 1GbE only copper sfp. This also has certain problems.

Playing with software queues will not help.

10

you can get 2.5g base T sfp modules. (eg. SF.COM for support, Aliexpress for cheap)
which will connect at a backend speed of 2.5G so might be good.