Page 1 of 1

CRS317 10G -> 1G traffic slow, everything else fine

Posted: Wed Dec 20, 2017 11:37 am
by gmstack
Traffic from 10Gbit machines to other 10Gbit machines is fine - gets ~9Gbit.
Traffic from 1Gbit machines to other 1Gbit machines is fine - gets ~950mbit.
Traffic from 1Gbit machines to 10Gbit machines is fine - gets ~950mbit.
But...
Traffic from 10Gbit machines to 1Gbit machines has a very high retry count and sits at about 150mbit.

Currently I've got all the 1Gbit machines on a separate subnet and the CRS317 is routing them to the 10G machines - and this is faster, at ~950mbit.

Happens on RouterOS or SwOS - and I've tried the latest RouterOS (6.41rc66) and 6.40.5, both have the issue. SwOS shows the issue too - the 2.3p it ships with I can't get to work, but 2.6 has the issue too. I've also tried with the 1G side being on a SFP-RJ45 converter, the 1G side being 1000BaseLX and the front panel gigabit port. All still do the same thing.

I've looked around and I see a similar thread here with issues on the CSS326.
In the changelogs I also see:
*) crs326 - fixed packet processing speed on switch chip if individual port link speed differs;
*) crs326 - improved transmit performance from SFP+ to Ethernet ports;
I'm wondering if the CRS317 has a related issue, because that sounds like what I see...

iperf3 example is below. 172.16.1.20 is a 10Gbit server, 172.16.1.19 is a 1Gbit machine.
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.16.1.19, port 55059
[  5] local 172.16.1.20 port 5201 connected to 172.16.1.19 port 55060
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   111 MBytes   931 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   940 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   940 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   940 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   940 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-5.00   sec   569 MBytes   955 Mbits/sec                  receiver
iperf3: the client has terminated
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.16.1.19, port 55062
[  5] local 172.16.1.20 port 5201 connected to 172.16.1.19 port 55063
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  5]   0.00-1.00   sec  17.1 MBytes   143 Mbits/sec  282   12.7 KBytes
[  5]   1.00-2.00   sec  17.3 MBytes   145 Mbits/sec  266   22.6 KBytes
[  5]   2.00-3.00   sec  17.5 MBytes   147 Mbits/sec  264   14.1 KBytes
[  5]   3.00-4.00   sec  17.2 MBytes   144 Mbits/sec  282   14.1 KBytes
[  5]   4.00-5.00   sec  17.3 MBytes   145 Mbits/sec  251   12.7 KBytes
[  5]   5.00-6.00   sec  17.4 MBytes   146 Mbits/sec  250   12.7 KBytes
[  5]   6.00-7.00   sec  17.8 MBytes   149 Mbits/sec  282   15.6 KBytes
[  5]   7.00-8.00   sec  17.2 MBytes   144 Mbits/sec  295   12.7 KBytes
[  5]   8.00-9.00   sec  17.3 MBytes   145 Mbits/sec  268   12.7 KBytes
[  5]   9.00-10.00  sec  17.8 MBytes   149 Mbits/sec  292   22.6 KBytes
[  5]  10.00-10.01  sec   255 KBytes   247 Mbits/sec    4   14.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.01  sec   174 MBytes   146 Mbits/sec  2736             sender
[  5]   0.00-10.01  sec  0.00 Bytes  0.00 bits/sec                  receiver

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Thu Dec 21, 2017 6:19 pm
by dksoft
That's a known problem.

RouterOS 6.41rc66 and the upcoming SwOS 2.7 do not fix it on my CRS326/CSS326.
You can downgrade to SwOS 2.3 or 2.4. This will restore performance but might put your router into an endless reboot loop if SFP+ is inserted at boot.

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Sun Dec 24, 2017 12:52 pm
by Lakis
wtf waiting 2 months still
performance in 6.41 still not fixed on crs317 + tons of bugs
no fix 10gbit sfp tx rx redings
no fix PSU reading...

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Thu Jan 18, 2018 10:22 am
by gmstack
Seems partially fixed in 6.42rc9
Connecting to host 172.16.1.20, port 5201
Reverse mode, remote host 172.16.1.20 is sending
[  5] local 172.16.1.19 port 55268 connected to 172.16.1.20 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  78.5 MBytes   659 Mbits/sec                  
[  5]   1.00-2.00   sec  75.9 MBytes   636 Mbits/sec                  
[  5]   2.00-3.00   sec  77.8 MBytes   653 Mbits/sec                  
[  5]   3.00-4.00   sec  79.3 MBytes   666 Mbits/sec                  
[  5]   4.00-5.00   sec  77.7 MBytes   652 Mbits/sec                  
[  5]   5.00-6.00   sec  76.9 MBytes   645 Mbits/sec                  
[  5]   6.00-7.00   sec  77.2 MBytes   647 Mbits/sec                  
[  5]   7.00-8.00   sec  75.9 MBytes   637 Mbits/sec                  
[  5]   8.00-9.00   sec  77.0 MBytes   646 Mbits/sec                  
[  5]   9.00-10.00  sec  76.5 MBytes   642 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   774 MBytes   649 Mbits/sec  6065             sender
[  5]   0.00-10.00  sec   773 MBytes   648 Mbits/sec                  receiver

iperf Done.

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Fri Jan 19, 2018 12:05 pm
by dksoft
Seems partially fixed in 6.42rc9
I can not confirm that. Read speed went up from 25MByte/s to around 50MByte/s with 6.42rc* but it's still far away from 1GBit/s.
Write speed is at full speed.

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Fri Jan 19, 2018 12:15 pm
by ste
SFP+RJ10 still reports up/down link state without a cable connected.

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Sat Feb 03, 2018 6:29 pm
by geoffc123
same issue here, here is my setup

ccr1036 to crs317 --> crs317 to Crs212 10s 1s+

crs317 is slow download for my customers that have a 1g sfp, so i had no choice to 10g trunk into a crs212 10s 1s+

the crs212 needs reboots every few weeks and the 317 does not have good download speeds on 1g sfp, ( download speed very slow for the customers)

so basically i have 9 customers that have a 1gig fiber and i have no reliable way of servicing them because the 212 always needs reboots, ive replaced it 3 times and same issue, and the 317 has an issue with traffic going from sfp+ to sfp 1g ( Slow )

If this dowsnt get fixed im going to have to start looking for alternative solutions other than mikrotik.

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Sat Mar 31, 2018 3:23 pm
by netflow
Have not such devices yet, however interested by any findings.
Have you tried using RED queues https://wiki.mikrotik.com/wiki/Manual:Queue#RED ?
Good resource for common problems listed here: https://fasterdata.es.net/network-tunin ... ze-issues/

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Tue Apr 03, 2018 7:54 am
by hooyao
SFP+RJ10 still reports up/down link state without a cable connected.
Hi I have 2 S+RJ10 on css326, they are connected to a NAS and a workstation. these 2 modules seems extremely unstable. They disconnects all the time, avg once a day. I found it in my workstation log. and sometimes the NIC says it's connected, but no data sent/received. the cable is CAT6A. and I have put an fan besides the 2 modules inside the switch.

do you experience the similar issue?

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Mon Aug 13, 2018 5:53 pm
by hooyao
Have not such devices yet, however interested by any findings.
Have you tried using RED queues https://wiki.mikrotik.com/wiki/Manual:Queue#RED ?
Good resource for common problems listed here: https://fasterdata.es.net/network-tunin ... ze-issues/
I own a CRS326 and 2 s+rj10 10G base-t module, this switch suffer the same 10g -> 1g issue, I tried your suggested approach
enable ip filtering in bridge
create a mangle rule to mark 10g -> 1g packets
create a red/pcq/... queue with max limit 1000M
it speeds up the data transfer speed from 200M to 300M, but cpu usage hits 80%

then i noticed that just enabling ip filtering and disable hardware offload for ports can achieve the same result.

can you provide more hints?

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Thu Aug 16, 2018 1:32 am
by lapsio
Might be related:
viewtopic.php?f=3&t=138082

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Thu Aug 16, 2018 1:37 am
by lapsio
SFP+RJ10 still reports up/down link state without a cable connected.
Still happens in August...

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Thu Aug 30, 2018 5:58 pm
by hooyao
I just receive reply from Mikrotik
I want to inform you that we are aware of 10G to 1G performance problem on CRS3xx devices and currently are working on new software fixes for better buffer allocation.

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Wed Jan 09, 2019 6:28 am
by awonglk
Any replies from Mikrotik support since Aug 2018?

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Wed Jan 09, 2019 6:35 am
by ste
Any replies from Mikrotik support since Aug 2018?
Search the changelogs. Problem is solved.

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Sat Jan 12, 2019 5:43 am
by awonglk
Says “crs3xx - improved data transmission between 10G and 1G ports;”

Anyone able to measure how much of an improvement that is?

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Sat Jan 26, 2019 8:49 pm
by ToBeFrank
I can confirm it is fixed in 6.43.8.

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Mon Jan 28, 2019 1:20 pm
by marekm
I can confirm it is fixed in 6.43.8.
What is the switch buffer size? It might be good enough for LAN use, but still too small for real bursty traffic from the Internet. The buffer should be approximately sized to hold amount of data corresponding to the average RTT (round trip time), and in a switch this is limited by hardware (high-speed RAM inside the switch chip, not the much larger RAM available to RouterOS). Some switches specify the size (TP-Link T1700G-28TQ packet buffer memory: 1.5 MB) and it's fairly small (12 ms worth of data at 1 Gbps - small compared to average RTT over the Internet) so it also matters how it is allocated dynamically between ports.

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Mon Jan 28, 2019 4:29 pm
by mkx
I can confirm it is fixed in 6.43.8.
What is the switch buffer size? It might be good enough for LAN use, but still too small for real bursty traffic from the Internet. The buffer should be approximately sized to hold amount of data corresponding to the average RTT (round trip time), and in a switch this is limited by hardware (high-speed RAM inside the switch chip, not the much larger RAM available to RouterOS). Some switches specify the size (TP-Link T1700G-28TQ packet buffer memory: 1.5 MB) and it's fairly small (12 ms worth of data at 1 Gbps - small compared to average RTT over the Internet) so it also matters how it is allocated dynamically between ports.
The buffer size (RTT times speed) you're mentioning is too large (and is usually referred to as buffer-bloat). Needed buffer size in switch is typically smaller, a few frames (jumbo if needed) per switch. If there's a congestion on single port, flow control needs to kick-in.

The buffer size you're mentioning is typical TCP window size ... which is allocated and managed by IP stack on TCP connection endpoints.

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Mon Jan 28, 2019 5:08 pm
by marekm
The buffer size (RTT times speed) you're mentioning is too large (and is usually referred to as buffer-bloat). Needed buffer size in switch is typically smaller, a few frames (jumbo if needed) per switch. If there's a congestion on single port, flow control needs to kick-in.

The buffer size you're mentioning is typical TCP window size ... which is allocated and managed by IP stack on TCP connection endpoints.
The buffer needs to cope with bursty traffic from not one, but many simultaneous TCP connections - I know this from own experience as a small local ISP (a few hundreds of customers), who had trouble with different upstream ISPs on three separate occasions over a few years. Buffer sizes were left at factory defaults in their switch (1Gb port ->100Mb port) or licensed radio links (1Gb port -> 150Mb radio, 1Gb port -> 300Mb radio), and larger size (once the issue was discovered and I convinced them to make this change) helped a lot. Otherwise there was packet loss (and my customers complaining about poor speedtest results, on a new larger and supposedly better upstream connection) when the link capacity was only half-utilized. For example, NEC iPasolink 200/400 have default queue size of 64 KB, maximum is 1 MB. My complaints to the upstreams were initially rejected as they couldn't see the problem when testing with their network tester device (which generated packets at regular intervals, not as bursty as real traffic from the Internet).

Flow control has its own issues, the other device needs to have enough buffer size for it to work as you effectively move the queue there, and can bring down large part of a network if done wrong (bad device continuously flooding a port with pause frames). My point is the buffer size should be specified, and tunable (NEC got it right in their radio, but the operator must also read the docs to make use of this setting). Not every port needs so much, but for real traffic from the Internet it's a must, as it's really bursty. While I could shape upload traffic at my end to reduce bursts, I had no such control over the other (upstream ISP's) end for download traffic.

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Fri Apr 26, 2019 9:23 pm
by geoffc123
Seriously Mikrotik team! When do you plan on fixing this shit, I have so much of your 10g switches ( all models ) collecting dust!

It doesn’t work 10g to 1g

On any switches

1g port gets 30mbps download speed
If I negotiate it to 100base-t it does almost 100mbps

Until mikrotik can fix this issue I don’t recommend any crs series switch unless all ports are negotiated at the same speed

Re: CRS317 10G -> 1G traffic slow, everything else fine

Posted: Fri Apr 26, 2019 9:36 pm
by lapsio
Seriously Mikrotik team! When do you plan on fixing this shit, I have so much of your 10g switches ( all models ) collecting dust!

It doesn’t work 10g to 1g

On any switches

1g port gets 30mbps download speed
If I negotiate it to 100base-t it does almost 100mbps

Until mikrotik can fix this issue I don’t recommend any crs series switch unless all ports are negotiated at the same speed
What is exact scenario when this bug happens? I'm using CRS326 and two CRS317 (with 4 ports in 1G mode) and I never experienced downgraded 1G performance. Does it only occur when there's connection between 10G server and 1G device that goes through CRS switches? Or does it occur when simply SFP+ ports are used as trunks? Because I use SFP+ ports as trunks everywhere and my 1G ports (both 1G SFP modules in SFP+ ports in CRS317 and normal copper ports in CRS326) operate at 1G. That said I always access 1G <-> 1G on both ends, just VLANs span through SFP+ trunks. I never tried 10G server <-> 1G device. I also always use LACP on trunks with minimum 2 SFP+ interfaces, never used just plain single 10G uplink between switches.