Community discussions

MikroTik App
 
gmstack
just joined
Topic Author
Posts: 4
Joined: Wed Aug 16, 2017 12:01 pm

CRS317 10G -> 1G traffic slow, everything else fine

Wed Dec 20, 2017 11:37 am

Traffic from 10Gbit machines to other 10Gbit machines is fine - gets ~9Gbit.
Traffic from 1Gbit machines to other 1Gbit machines is fine - gets ~950mbit.
Traffic from 1Gbit machines to 10Gbit machines is fine - gets ~950mbit.
But...
Traffic from 10Gbit machines to 1Gbit machines has a very high retry count and sits at about 150mbit.

Currently I've got all the 1Gbit machines on a separate subnet and the CRS317 is routing them to the 10G machines - and this is faster, at ~950mbit.

Happens on RouterOS or SwOS - and I've tried the latest RouterOS (6.41rc66) and 6.40.5, both have the issue. SwOS shows the issue too - the 2.3p it ships with I can't get to work, but 2.6 has the issue too. I've also tried with the 1G side being on a SFP-RJ45 converter, the 1G side being 1000BaseLX and the front panel gigabit port. All still do the same thing.

I've looked around and I see a similar thread here with issues on the CSS326.
In the changelogs I also see:
*) crs326 - fixed packet processing speed on switch chip if individual port link speed differs;
*) crs326 - improved transmit performance from SFP+ to Ethernet ports;
I'm wondering if the CRS317 has a related issue, because that sounds like what I see...

iperf3 example is below. 172.16.1.20 is a 10Gbit server, 172.16.1.19 is a 1Gbit machine.
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.16.1.19, port 55059
[  5] local 172.16.1.20 port 5201 connected to 172.16.1.19 port 55060
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   111 MBytes   931 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   940 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   940 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   940 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   940 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-5.00   sec   569 MBytes   955 Mbits/sec                  receiver
iperf3: the client has terminated
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.16.1.19, port 55062
[  5] local 172.16.1.20 port 5201 connected to 172.16.1.19 port 55063
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  5]   0.00-1.00   sec  17.1 MBytes   143 Mbits/sec  282   12.7 KBytes
[  5]   1.00-2.00   sec  17.3 MBytes   145 Mbits/sec  266   22.6 KBytes
[  5]   2.00-3.00   sec  17.5 MBytes   147 Mbits/sec  264   14.1 KBytes
[  5]   3.00-4.00   sec  17.2 MBytes   144 Mbits/sec  282   14.1 KBytes
[  5]   4.00-5.00   sec  17.3 MBytes   145 Mbits/sec  251   12.7 KBytes
[  5]   5.00-6.00   sec  17.4 MBytes   146 Mbits/sec  250   12.7 KBytes
[  5]   6.00-7.00   sec  17.8 MBytes   149 Mbits/sec  282   15.6 KBytes
[  5]   7.00-8.00   sec  17.2 MBytes   144 Mbits/sec  295   12.7 KBytes
[  5]   8.00-9.00   sec  17.3 MBytes   145 Mbits/sec  268   12.7 KBytes
[  5]   9.00-10.00  sec  17.8 MBytes   149 Mbits/sec  292   22.6 KBytes
[  5]  10.00-10.01  sec   255 KBytes   247 Mbits/sec    4   14.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.01  sec   174 MBytes   146 Mbits/sec  2736             sender
[  5]   0.00-10.01  sec  0.00 Bytes  0.00 bits/sec                  receiver
 
dksoft
Member Candidate
Member Candidate
Posts: 148
Joined: Thu Dec 06, 2012 8:56 am
Location: Germany

Re: CRS317 10G -> 1G traffic slow, everything else fine

Thu Dec 21, 2017 6:19 pm

That's a known problem.

RouterOS 6.41rc66 and the upcoming SwOS 2.7 do not fix it on my CRS326/CSS326.
You can downgrade to SwOS 2.3 or 2.4. This will restore performance but might put your router into an endless reboot loop if SFP+ is inserted at boot.
 
Lakis
Forum Veteran
Forum Veteran
Posts: 703
Joined: Wed Sep 23, 2009 7:52 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Sun Dec 24, 2017 12:52 pm

wtf waiting 2 months still
performance in 6.41 still not fixed on crs317 + tons of bugs
no fix 10gbit sfp tx rx redings
no fix PSU reading...
 
gmstack
just joined
Topic Author
Posts: 4
Joined: Wed Aug 16, 2017 12:01 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Thu Jan 18, 2018 10:22 am

Seems partially fixed in 6.42rc9
Connecting to host 172.16.1.20, port 5201
Reverse mode, remote host 172.16.1.20 is sending
[  5] local 172.16.1.19 port 55268 connected to 172.16.1.20 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  78.5 MBytes   659 Mbits/sec                  
[  5]   1.00-2.00   sec  75.9 MBytes   636 Mbits/sec                  
[  5]   2.00-3.00   sec  77.8 MBytes   653 Mbits/sec                  
[  5]   3.00-4.00   sec  79.3 MBytes   666 Mbits/sec                  
[  5]   4.00-5.00   sec  77.7 MBytes   652 Mbits/sec                  
[  5]   5.00-6.00   sec  76.9 MBytes   645 Mbits/sec                  
[  5]   6.00-7.00   sec  77.2 MBytes   647 Mbits/sec                  
[  5]   7.00-8.00   sec  75.9 MBytes   637 Mbits/sec                  
[  5]   8.00-9.00   sec  77.0 MBytes   646 Mbits/sec                  
[  5]   9.00-10.00  sec  76.5 MBytes   642 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   774 MBytes   649 Mbits/sec  6065             sender
[  5]   0.00-10.00  sec   773 MBytes   648 Mbits/sec                  receiver

iperf Done.
 
dksoft
Member Candidate
Member Candidate
Posts: 148
Joined: Thu Dec 06, 2012 8:56 am
Location: Germany

Re: CRS317 10G -> 1G traffic slow, everything else fine

Fri Jan 19, 2018 12:05 pm

Seems partially fixed in 6.42rc9
I can not confirm that. Read speed went up from 25MByte/s to around 50MByte/s with 6.42rc* but it's still far away from 1GBit/s.
Write speed is at full speed.
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Fri Jan 19, 2018 12:15 pm

SFP+RJ10 still reports up/down link state without a cable connected.
 
geoffc123
just joined
Posts: 3
Joined: Sat Feb 03, 2018 6:22 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Sat Feb 03, 2018 6:29 pm

same issue here, here is my setup

ccr1036 to crs317 --> crs317 to Crs212 10s 1s+

crs317 is slow download for my customers that have a 1g sfp, so i had no choice to 10g trunk into a crs212 10s 1s+

the crs212 needs reboots every few weeks and the 317 does not have good download speeds on 1g sfp, ( download speed very slow for the customers)

so basically i have 9 customers that have a 1gig fiber and i have no reliable way of servicing them because the 212 always needs reboots, ive replaced it 3 times and same issue, and the 317 has an issue with traffic going from sfp+ to sfp 1g ( Slow )

If this dowsnt get fixed im going to have to start looking for alternative solutions other than mikrotik.
 
netflow
Frequent Visitor
Frequent Visitor
Posts: 94
Joined: Sat Oct 01, 2016 3:53 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Sat Mar 31, 2018 3:23 pm

Have not such devices yet, however interested by any findings.
Have you tried using RED queues https://wiki.mikrotik.com/wiki/Manual:Queue#RED ?
Good resource for common problems listed here: https://fasterdata.es.net/network-tunin ... ze-issues/
 
hooyao
newbie
Posts: 37
Joined: Mon Feb 20, 2017 6:11 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Tue Apr 03, 2018 7:54 am

SFP+RJ10 still reports up/down link state without a cable connected.
Hi I have 2 S+RJ10 on css326, they are connected to a NAS and a workstation. these 2 modules seems extremely unstable. They disconnects all the time, avg once a day. I found it in my workstation log. and sometimes the NIC says it's connected, but no data sent/received. the cable is CAT6A. and I have put an fan besides the 2 modules inside the switch.

do you experience the similar issue?
 
hooyao
newbie
Posts: 37
Joined: Mon Feb 20, 2017 6:11 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Mon Aug 13, 2018 5:53 pm

Have not such devices yet, however interested by any findings.
Have you tried using RED queues https://wiki.mikrotik.com/wiki/Manual:Queue#RED ?
Good resource for common problems listed here: https://fasterdata.es.net/network-tunin ... ze-issues/
I own a CRS326 and 2 s+rj10 10G base-t module, this switch suffer the same 10g -> 1g issue, I tried your suggested approach
enable ip filtering in bridge
create a mangle rule to mark 10g -> 1g packets
create a red/pcq/... queue with max limit 1000M
it speeds up the data transfer speed from 200M to 300M, but cpu usage hits 80%

then i noticed that just enabling ip filtering and disable hardware offload for ports can achieve the same result.

can you provide more hints?
 
User avatar
lapsio
Long time Member
Long time Member
Posts: 514
Joined: Wed Feb 24, 2016 5:19 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Thu Aug 16, 2018 1:32 am

Might be related:
viewtopic.php?f=3&t=138082
 
User avatar
lapsio
Long time Member
Long time Member
Posts: 514
Joined: Wed Feb 24, 2016 5:19 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Thu Aug 16, 2018 1:37 am

SFP+RJ10 still reports up/down link state without a cable connected.
Still happens in August...
 
hooyao
newbie
Posts: 37
Joined: Mon Feb 20, 2017 6:11 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Thu Aug 30, 2018 5:58 pm

I just receive reply from Mikrotik
I want to inform you that we are aware of 10G to 1G performance problem on CRS3xx devices and currently are working on new software fixes for better buffer allocation.
 
awonglk
newbie
Posts: 33
Joined: Sat Oct 31, 2015 3:43 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Wed Jan 09, 2019 6:28 am

Any replies from Mikrotik support since Aug 2018?
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Wed Jan 09, 2019 6:35 am

Any replies from Mikrotik support since Aug 2018?
Search the changelogs. Problem is solved.
 
awonglk
newbie
Posts: 33
Joined: Sat Oct 31, 2015 3:43 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Sat Jan 12, 2019 5:43 am

Says “crs3xx - improved data transmission between 10G and 1G ports;”

Anyone able to measure how much of an improvement that is?
 
ToBeFrank
newbie
Posts: 33
Joined: Mon Dec 18, 2017 7:31 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Sat Jan 26, 2019 8:49 pm

I can confirm it is fixed in 6.43.8.
 
marekm
Member
Member
Posts: 379
Joined: Tue Feb 01, 2011 11:27 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Mon Jan 28, 2019 1:20 pm

I can confirm it is fixed in 6.43.8.
What is the switch buffer size? It might be good enough for LAN use, but still too small for real bursty traffic from the Internet. The buffer should be approximately sized to hold amount of data corresponding to the average RTT (round trip time), and in a switch this is limited by hardware (high-speed RAM inside the switch chip, not the much larger RAM available to RouterOS). Some switches specify the size (TP-Link T1700G-28TQ packet buffer memory: 1.5 MB) and it's fairly small (12 ms worth of data at 1 Gbps - small compared to average RTT over the Internet) so it also matters how it is allocated dynamically between ports.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11433
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Mon Jan 28, 2019 4:29 pm

I can confirm it is fixed in 6.43.8.
What is the switch buffer size? It might be good enough for LAN use, but still too small for real bursty traffic from the Internet. The buffer should be approximately sized to hold amount of data corresponding to the average RTT (round trip time), and in a switch this is limited by hardware (high-speed RAM inside the switch chip, not the much larger RAM available to RouterOS). Some switches specify the size (TP-Link T1700G-28TQ packet buffer memory: 1.5 MB) and it's fairly small (12 ms worth of data at 1 Gbps - small compared to average RTT over the Internet) so it also matters how it is allocated dynamically between ports.
The buffer size (RTT times speed) you're mentioning is too large (and is usually referred to as buffer-bloat). Needed buffer size in switch is typically smaller, a few frames (jumbo if needed) per switch. If there's a congestion on single port, flow control needs to kick-in.

The buffer size you're mentioning is typical TCP window size ... which is allocated and managed by IP stack on TCP connection endpoints.
 
marekm
Member
Member
Posts: 379
Joined: Tue Feb 01, 2011 11:27 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Mon Jan 28, 2019 5:08 pm

The buffer size (RTT times speed) you're mentioning is too large (and is usually referred to as buffer-bloat). Needed buffer size in switch is typically smaller, a few frames (jumbo if needed) per switch. If there's a congestion on single port, flow control needs to kick-in.

The buffer size you're mentioning is typical TCP window size ... which is allocated and managed by IP stack on TCP connection endpoints.
The buffer needs to cope with bursty traffic from not one, but many simultaneous TCP connections - I know this from own experience as a small local ISP (a few hundreds of customers), who had trouble with different upstream ISPs on three separate occasions over a few years. Buffer sizes were left at factory defaults in their switch (1Gb port ->100Mb port) or licensed radio links (1Gb port -> 150Mb radio, 1Gb port -> 300Mb radio), and larger size (once the issue was discovered and I convinced them to make this change) helped a lot. Otherwise there was packet loss (and my customers complaining about poor speedtest results, on a new larger and supposedly better upstream connection) when the link capacity was only half-utilized. For example, NEC iPasolink 200/400 have default queue size of 64 KB, maximum is 1 MB. My complaints to the upstreams were initially rejected as they couldn't see the problem when testing with their network tester device (which generated packets at regular intervals, not as bursty as real traffic from the Internet).

Flow control has its own issues, the other device needs to have enough buffer size for it to work as you effectively move the queue there, and can bring down large part of a network if done wrong (bad device continuously flooding a port with pause frames). My point is the buffer size should be specified, and tunable (NEC got it right in their radio, but the operator must also read the docs to make use of this setting). Not every port needs so much, but for real traffic from the Internet it's a must, as it's really bursty. While I could shape upload traffic at my end to reduce bursts, I had no such control over the other (upstream ISP's) end for download traffic.
 
geoffc123
just joined
Posts: 3
Joined: Sat Feb 03, 2018 6:22 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Fri Apr 26, 2019 9:23 pm

Seriously Mikrotik team! When do you plan on fixing this shit, I have so much of your 10g switches ( all models ) collecting dust!

It doesn’t work 10g to 1g

On any switches

1g port gets 30mbps download speed
If I negotiate it to 100base-t it does almost 100mbps

Until mikrotik can fix this issue I don’t recommend any crs series switch unless all ports are negotiated at the same speed
 
User avatar
lapsio
Long time Member
Long time Member
Posts: 514
Joined: Wed Feb 24, 2016 5:19 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Fri Apr 26, 2019 9:36 pm

Seriously Mikrotik team! When do you plan on fixing this shit, I have so much of your 10g switches ( all models ) collecting dust!

It doesn’t work 10g to 1g

On any switches

1g port gets 30mbps download speed
If I negotiate it to 100base-t it does almost 100mbps

Until mikrotik can fix this issue I don’t recommend any crs series switch unless all ports are negotiated at the same speed
What is exact scenario when this bug happens? I'm using CRS326 and two CRS317 (with 4 ports in 1G mode) and I never experienced downgraded 1G performance. Does it only occur when there's connection between 10G server and 1G device that goes through CRS switches? Or does it occur when simply SFP+ ports are used as trunks? Because I use SFP+ ports as trunks everywhere and my 1G ports (both 1G SFP modules in SFP+ ports in CRS317 and normal copper ports in CRS326) operate at 1G. That said I always access 1G <-> 1G on both ends, just VLANs span through SFP+ trunks. I never tried 10G server <-> 1G device. I also always use LACP on trunks with minimum 2 SFP+ interfaces, never used just plain single 10G uplink between switches.
 
michel687
just joined
Posts: 1
Joined: Sun Jan 07, 2024 4:34 pm

Re: CRS317 10G -> 1G traffic slow, everything else fine

Sun Jan 07, 2024 4:41 pm

I have the exact same problem as OP stated :

CRS309-1G-8S+ , bridge mode.

switch 10G -> 10gBaseT module (negociated speed 1g/2.5G).

Lots of retransmit when 10g is transmitting to 10gbaseT module to a AP.

When I connect the AP into a switch connected with SFP(1gbps) DAC Cable to the same Mikrotik, RX/TX = Good.

Flow Control is off on all ports, not using VLANs yet.... what could I be missing?

Who is online

Users browsing this forum: Fl3tch and 31 guests