CCR/CRS Slow Upload-Speed on single TCP-Connection using SFP

RouterOS General
1

Hi All,

hope this is the correct sub-forum for my thread.
I’m having problems with single TCP-Connection uploads via SFP-WAN, tested it on two different Mikrotik devices (CCR1009-8G-1S-1S+, RouterOS 6.37.1 and CRS125-24G-1S-IN, RouterOS 6.36).

I do have a symmetrical 1GBit/s internet connection via BiDi Fiber and I tested with two different transceivers:

  • Flexoptix S.B1312.10.DL-MSA01
  • TP-Link TL-SM321B

I’m getting the same results with both transceivers.
Debugging this issue was quite hard, but my neighbour is connected to the same ISP on the same POP, just with different hardware.

We tested this using multiple methods:

FTP-Upload to a specific speedtest.tele2.net Server (fra36-speedtest-1.tele2.net):

  • My upload speed: ~2MB/s
  • My neighbours upload speed: ~30MB/s

Traceroutes to this server are exactly the same from both our connections.

iperf to our ISPs server:

  • my speed: ~300-400Mbit/s
  • my neighbours speed: ~900Mbit/s

Traceroutes to this server are exactly the same from both our connections.

iperf to another external server:

  • my speed: ~70Mbit/s
  • my neighbours speed: ~600Mbit/s

Traceroutes to this server are exactly the same from both our connections.

We tested it using IPv4 and IPv6 with the same results.

Speedtest:
speedtest.net results show ~950Mbit/s in both directions, as they are using multiple connections.

Furthermore I asked two friends to download a 2GB file via HTTP/HTTPS from one of my servers. When using one single connection for download (e.g. downloading with their browser), they could download the file with 10-16MBit/s. When using jDownloader with 20 concurrent connections on the same file, they maxed out their 100MBit/s download speed.


Today I bought a TP-Link MC220L Media-Converter, created a new Bridge-Interface on the CCR1009 (sfp1+ether7), changed FW-Rules/TrafficFlow/DHCP etc. from sfp1 to the new Bridge-Interface, removed the SFP transceiver from sfp1 and connected the Media-Converter via ether7. Now everything is working as expected. I get the same upload speeds as my neighbour, my friends can download with full speed on their 100MBit/s connection via a single HTTP-Connection - and I’m kind of speechless.

I really do not have any idea what the problem could be. There isn’t much you can configure on the SFP interface. I left the MTU-Settings untouched (actual: 1500, L2 MTU 1580), no bandwidth limit, SFP Rate Select: high (tried both), Flow Control: off (tried both), Auto Negotiation: on, Loop Protection: default.


I hope someone has an idea what the problem with the SFP-Interface could be? Or if something else could cause these problems?


Thanks a lot in advance,

j

2

Hello

Seems like no one has an idea what the problem could be? I diffed my configs (before and after bridge change with media converter). I don’t see any relevant changes.

1c1
< # oct/07/2016 20:04:53 by RouterOS 6.37.1
---
> # oct/10/2016 20:29:54 by RouterOS 6.37.1
6a7
> add name=WAN
15a17
> set [ find default-name=sfp1 ] advertise=1000M-full
55c57
< add address-pool=dhcp_pool1 disabled=no interface=ether7 name=server1
---
> add address-pool=dhcp_pool1 interface=ether7 name=server1
77a80,81
> add bridge=WAN interface=sfp1
> add bridge=WAN interface=ether7
93c97,98
< add address=192.168.100.1/24 interface=ether7 network=192.168.100.0
---
> add address=192.168.100.1/24 disabled=yes interface=ether7 network=\
>     192.168.100.0
99c104
<     interface=sfp1
---
>     interface=WAN
170c175
<     srcnat,dstnat in-interface=sfp1
---
>     srcnat,dstnat in-interface=WAN
172c177
<     in-interface=sfp1
---
>     in-interface=WAN
174c179
<     in-interface=sfp1
---
>     in-interface=WAN
181,182d185
< add action=accept chain=forward comment="ALLOW ETHER3" in-interface=ether7
< add action=accept chain=input in-interface=ether7
190c193
<     dst-address=XXX.XXX.XXX.XXX/29 dst-port=33434-33534 in-interface=sfp1 \
---
>     dst-address=XXX.XXX.XXX.XXX/29 dst-port=33434-33534 in-interface=WAN \
193c196
<     sfp1 protocol=icmp
---
>     WAN protocol=icmp
197,198c200
<     XXX.XXX.XXX.YYY dst-port=9966,9977,9988,2010 in-interface=sfp1 protocol=\
<     udp
---
>     XXX.XXX.XXX.YYY dst-port=9966,9977,9988,2010 in-interface=WAN protocol=udp
200c202
<     in-interface=sfp1 protocol=tcp
---
>     in-interface=WAN protocol=tcp
202c204
<     in-interface=sfp1 protocol=tcp
---
>     in-interface=WAN protocol=tcp
204c206
<     dst-port=32768-65535 in-interface=sfp1 protocol=tcp
---
>     dst-port=32768-65535 in-interface=WAN protocol=tcp
206c208
<     27015-27016 in-interface=sfp1 protocol=tcp
---
>     27015-27016 in-interface=WAN protocol=tcp
208c210
<     27015-27016 in-interface=sfp1 protocol=udp
---
>     27015-27016 in-interface=WAN protocol=udp
210c212
<     in-interface=sfp1 protocol=tcp
---
>     in-interface=WAN protocol=tcp
212c214
<     in-interface=sfp1 protocol=tcp
---
>     in-interface=WAN protocol=tcp
214c216
<     in-interface=sfp1 protocol=tcp
---
>     in-interface=WAN protocol=tcp
216c218
<     dst-port=80,443,25,21,20,465,143,993,5232 in-interface=sfp1 protocol=tcp
---
>     dst-port=80,443,25,21,20,465,143,993,5232 in-interface=WAN protocol=tcp
218c220
<     32768-65535 in-interface=sfp1 protocol=tcp
---
>     32768-65535 in-interface=WAN protocol=tcp
221c223
<     in-interface=sfp1
---
>     in-interface=WAN
223c225
<     in-interface=sfp1
---
>     in-interface=WAN
226c228
<     XXX.XXX.XXX.ZZZ in-interface=sfp1 src-address-list=SDNS
---
>     XXX.XXX.XXX.ZZZ in-interface=WAN src-address-list=SDNS
230c232
<     in-interface=sfp1 protocol=udp
---
>     in-interface=WAN protocol=udp
232c234
<     in-interface=sfp1 protocol=tcp
---
>     in-interface=WAN protocol=tcp
234c236
<     32768-65535 in-interface=sfp1 protocol=tcp
---
>     32768-65535 in-interface=WAN protocol=tcp
244c246
<     DMZ-Bridge out-interface=sfp1
---
>     DMZ-Bridge out-interface=WAN
248c250
<     dst-address=XXX.XXX.XXX.XXX/29 in-interface=sfp1
---
>     dst-address=XXX.XXX.XXX.XXX/29 in-interface=WAN
280c282
< add action=masquerade chain=srcnat out-interface=sfp1 src-address=\
---
> add action=masquerade chain=srcnat out-interface=WAN src-address=\
282c284
< add action=masquerade chain=srcnat out-interface=sfp1 src-address=10.0.1.0/24
---
> add action=masquerade chain=srcnat out-interface=WAN src-address=10.0.1.0/24
285c287
< add action=masquerade chain=srcnat out-interface=sfp1 src-address=\
---
> add action=masquerade chain=srcnat out-interface=WAN src-address=\
289c291
< add action=masquerade chain=srcnat comment="Guest VPN" out-interface=sfp1 \
---
> add action=masquerade chain=srcnat comment="Guest VPN" out-interface=WAN \
313c315
<     in-interface=sfp1 protocol=tcp to-addresses=192.168.0.249 to-ports=32400
---
>     in-interface=WAN protocol=tcp to-addresses=192.168.0.249 to-ports=32400
358c360
< add distance=10 gateway=192.168.0.20 routing-mark=DE-VPN
---
> add disabled=yes distance=10 gateway=192.168.0.20 routing-mark=DE-VPN
360c362
< add distance=11 gateway=192.168.0.21 routing-mark=CH-VPN
---
> add disabled=yes distance=11 gateway=192.168.0.21 routing-mark=CH-VPN
375c377
< set active-flow-timeout=5m enabled=yes interfaces=sfp1,DMZ
---
> set active-flow-timeout=5m enabled=yes interfaces=WAN,DMZ
383c385
< add address=XXXX:XXXX:XXXX::1/128 advertise=no interface=sfp1
---
> add address=XXXX:XXXX:XXXX::1/128 advertise=no interface=WAN
393c395
< add add-default-route=yes interface=sfp1 pool-name=BBBB pool-prefix-length=\
---
> add add-default-route=yes interface=WAN pool-name=BBBB pool-prefix-length=\
422c424
<     XXXX:XXXX:XXXX:10::AAAA/128 dst-port=9966,9977,9988 in-interface=sfp1 \
---
>     XXXX:XXXX:XXXX:10::AAAA/128 dst-port=9966,9977,9988 in-interface=WAN \
425c427
<     dst-port=30034 in-interface=sfp1 protocol=tcp
---
>     dst-port=30034 in-interface=WAN protocol=tcp
427c429
<     dst-port=10012 in-interface=sfp1 protocol=tcp
---
>     dst-port=10012 in-interface=WAN protocol=tcp
429c431
<     dst-port=8080 in-interface=sfp1 protocol=tcp
---
>     dst-port=8080 in-interface=WAN protocol=tcp
431c433
<     dst-port=25565 in-interface=sfp1 protocol=tcp
---
>     dst-port=25565 in-interface=WAN protocol=tcp
434c436
<     in-interface=sfp1 protocol=tcp
---
>     in-interface=WAN protocol=tcp
436c438
<     dst-port=32768-65535 in-interface=sfp1 protocol=tcp
---
>     dst-port=32768-65535 in-interface=WAN protocol=tcp
444c446
<     XXXX:XXXX:XXXX:10::BBBB/128 in-interface=sfp1 src-address-list=SDNS
---
>     XXXX:XXXX:XXXX:10::BBBB/128 in-interface=WAN src-address-list=SDNS
448c450
<     dst-port=53 in-interface=sfp1 protocol=udp
---
>     dst-port=53 in-interface=WAN protocol=udp
450c452
<     dst-port=53,80,443 in-interface=sfp1 protocol=tcp
---
>     dst-port=53,80,443 in-interface=WAN protocol=tcp
452c454
<     dst-port=32768-65535 in-interface=sfp1 protocol=tcp
---
>     dst-port=32768-65535 in-interface=WAN protocol=tcp

Any suggestions? Or should I directly contact Mikrotik Support?

Thanks!

3

Hi

I don’t know but I’m kind of wacthing similar symptons like you on my ccr 1036 with SFP wan port (S+31DLC10D). We have an Internet Provider with 1,5 Gbps but when we make some speedtest the results are quite rare. DS 9 to 10Mbps and US is around 100Mbps. This is not a conclusion I know, but there’s no relation between having major ds than us. And I’m suspecting the CCR and now reading this, suspecting from the SFP.

I’m writing some conclusions for the forums

Thanks

4

Hi cabo81

Thanks for your feedback. I’m not sure if I understand you correctly, but it seems you’re having problems with your downstream rather than your upstream?
I didn’t see any problems in speedtest.net speedtests. I was just able to reproduce it with iperf (TCP) and FTP. “Regular” speedtests like speedtest.net do use multiple connections and my problems were gone with multiple TCP upstream connections.

Do you have some servers near your location to test with iperf or FTP? If not, please tell me where you’re located, I do have some VPS in the US, Canada, Germany and Switzerland where I could start an iperf server.


Thanks a lot

j

5

i also have a CCR1036 with the 2 sfp+ ports and have notice odd behavior with tx/rx numbers and transceivers. Additionally i have seen issues with IPv4 between TCP/UDP. I seem to have less issues with IPv6 and TCP/UDP. I will try and find a previous post of mine with additional details. I can say that i have tested the CHR and with enough resources for tengb that testing has been smoother.

6

Anyone else observing this problem?

7

Have a CCR1036-8G-2S+ , we have a 500mbps symetric connection.
Download speed is fine, but upload is less then 50%. I can’t say for certain it is the sfp and not the ISP’s fault at this point. on v6.37.1 SFP module S-31DLC20D

8

I have the same sitituation here, both on my CCR1009, i have full download and less then 50% upload speed, sometimes 25%, with one single tcp connection. Also i have ipip tunnel between my CCR routers, remote end gets iptv and cannot watch hd channels because of upload problem. My customer with 10mbps connection cannot watch hd because single tcp connection is between 2 -3 mbps, but watching sd channels is ok which is between 1.3 - 1.5 mbps, i will order TP-Link media converter and give it a try

9

Thanks for your feedback! Did you already tried with the media converter?