Community discussions

 
JonathanYoung2161
just joined
Topic Author
Posts: 21
Joined: Fri Apr 05, 2019 4:46 am
Location: Iowa

v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 2:56 am

I original posted finding over at RouterOS>General forum viewtopic.php?f=2&t=153474&p=757800#p757800 but I believe I have tracked it down to a OS/Firmware issues so am re-posting the my finding here. I hope this does not violate a forum rule. but suspect Ill get better response here.

Issues present across 2 separate CRS-317-1G-16S+
I do not believe this is a hardware issue. This looks to be a Firmware issue.

Image
https://photos.app.goo.gl/K7pXRZgPTb2Adrse7

will follow up with older firmware results and I have ordered in a MikroTik branded SFP AOC for more testing
Last edited by JonathanYoung2161 on Sun Nov 03, 2019 2:58 am, edited 1 time in total.
JonathanYoung2161
 
JonathanYoung2161
just joined
Topic Author
Posts: 21
Joined: Fri Apr 05, 2019 4:46 am
Location: Iowa

Re: v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 2:57 am

JonathanYoung2161
 
JonathanYoung2161
just joined
Topic Author
Posts: 21
Joined: Fri Apr 05, 2019 4:46 am
Location: Iowa

Re: v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 3:18 am

Found some of the port will go full speed if I bounce the port with speed trouble and on some this has no effect. After a reboot the issues return.
JonathanYoung2161
 
JonathanYoung2161
just joined
Topic Author
Posts: 21
Joined: Fri Apr 05, 2019 4:46 am
Location: Iowa

Re: v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 4:36 am

tested with 6.44.6 and 6.45.7, and 7.0B3 same results

tested ports on CRS-24P-4S-RM across 3 different units

Image
https://photos.app.goo.gl/Evc6G41fNwat3yqE7
JonathanYoung2161
 
JonathanYoung2161
just joined
Topic Author
Posts: 21
Joined: Fri Apr 05, 2019 4:46 am
Location: Iowa

Re: v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 5:00 am

Equipment I original picked up for 10G network upgrade:
x1 CCR1016-12S-1S+
x1 CRS317-1G-16S+
x2 CRS328-24P-4S+
x1 RB4011iGS+5HacQ2HnD

Equipment I picked up To verify and Trouble shoot 10G issues:
x1 CRS317-1G-16S+
x2 CRS328-24P-4S+

+ 2600$+ in hardware and shipping
+ all the SFP+ DACs and AOCs
+ over 100 hour troubleshooting researching 10G issues and corresponding with MikroTik.
+ more then 6 months late I still don't have a 10G network that is stable or fully functional.

The average person surely would have gave up already. Come on MikroTik lets work together and find a solution!!!

tested with:
2x Aquantia AQtion SFP+ NIC, 10G Ethernet Network Adapter
4x AOC 10Gtek AOC-S1S1-003
3x CAB 10Gtek CAB-10GSFP-P1M

I have ordered 2x MikroTik SFP+ 5m Active Optics Direct Attach Cable to continue testing I'm sure we will see the same results
JonathanYoung2161
 
mducharme
Trainer
Trainer
Posts: 868
Joined: Tue Jul 19, 2016 6:45 pm

Re: v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 6:48 am

Share your configs please, /export hide-sensitive

Also this shouldn't be in the SwOS forum since I imagine you are running RouterOS and not SwOS?
 
JonathanYoung2161
just joined
Topic Author
Posts: 21
Joined: Fri Apr 05, 2019 4:46 am
Location: Iowa

Re: v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 6:53 am

# nov/02/2019 23:52:20 by RouterOS 6.45.7
# software id = MLNL-08DF
#
# model = CRS317-1G-16S+
# serial number = 955C0A96A156
/interface bridge
add admin-mac=74:4D:28:08:B6:58 auto-mac=no comment=defconf name=bridge
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
add bridge=bridge comment=defconf interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
add bridge=bridge comment=defconf interface=sfp-sfpplus9
add bridge=bridge comment=defconf interface=sfp-sfpplus10
add bridge=bridge comment=defconf interface=sfp-sfpplus11
add bridge=bridge comment=defconf interface=sfp-sfpplus12
add bridge=bridge comment=defconf interface=sfp-sfpplus13
add bridge=bridge comment=defconf interface=sfp-sfpplus14
add bridge=bridge comment=defconf interface=sfp-sfpplus15
add bridge=bridge comment=defconf interface=sfp-sfpplus16
/interface list member
add interface=ether1 list=LAN
add interface=sfp-sfpplus1 list=LAN
add interface=sfp-sfpplus2 list=LAN
add interface=sfp-sfpplus3 list=LAN
add interface=sfp-sfpplus4 list=LAN
add interface=sfp-sfpplus5 list=LAN
add interface=sfp-sfpplus6 list=LAN
add interface=sfp-sfpplus7 list=LAN
add interface=sfp-sfpplus8 list=LAN
add interface=sfp-sfpplus9 list=LAN
add interface=sfp-sfpplus10 list=LAN
add interface=sfp-sfpplus11 list=LAN
add interface=sfp-sfpplus12 list=LAN
add interface=sfp-sfpplus13 list=LAN
add interface=sfp-sfpplus14 list=LAN
add interface=sfp-sfpplus15 list=LAN
add interface=sfp-sfpplus16 list=LAN
/ip dhcp-client
add dhcp-options=hostname,clientid disabled=no interface=bridge
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=America/Chicago
/system routerboard settings
set boot-os=router-os
JonathanYoung2161
 
mducharme
Trainer
Trainer
Posts: 868
Joined: Tue Jul 19, 2016 6:45 pm

Re: v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 6:59 am

I see you are just using the default exactly pretty much. But this switch is connected to a router, yes? What about the router config?

To test the switch properly you should be doing testing without going through a router - do a speed test between two computers going through the switch with iperf3. That way you aren't possibly introducing other unrelated problems that are botching your test.

6.45.7 is new, and I would generally suggest people use the long-term release in most cases instead of the "stable" for production environments.
 
mducharme
Trainer
Trainer
Posts: 868
Joined: Tue Jul 19, 2016 6:45 pm

Re: v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 7:03 am

Also I hope that you are updating your firmware too, not only the RouterOS version. If you are only updating RouterOS you might be behind on the firmware updates.
 
JonathanYoung2161
just joined
Topic Author
Posts: 21
Joined: Fri Apr 05, 2019 4:46 am
Location: Iowa

Re: v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 7:07 am

the routers:
# nov/03/2019 00:06:18 by RouterOS 6.45.7
# software id = PE2R-6G34
#
# model = CCR1016-12S-1S+
# serial number = 91660A145255
/interface bridge
add name=bridge1
/interface list
add name=WAN
add name=LAN
add name=Guests
add name=Tor
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/ip pool
add name=dhcp_pool1 ranges=10.0.1.2-10.0.1.254
/ip dhcp-server
add address-pool=dhcp_pool1 bootp-lease-time=lease-time bootp-support=dynamic disabled=no interface=bridge1 name=dhcp1
/tool user-manager customer
set admin access=own-routers,own-users,own-profiles,own-limits,config-payment-gw
/dude
set data-directory=disk1/dude-data enabled=yes
/interface bridge port
add bridge=bridge1 interface=sfp3
add bridge=bridge1 interface=sfp4
add bridge=bridge1 interface=sfp5
add bridge=bridge1 interface=sfp6
add bridge=bridge1 interface=sfp7
add bridge=bridge1 interface=sfp8
add bridge=bridge1 interface=sfp9
add bridge=bridge1 interface=sfp10
add bridge=bridge1 interface=sfp11
add bridge=bridge1 interface=sfp12
add bridge=bridge1 interface=sfpplus1
/interface detect-internet
set detect-interface-list=WAN internet-interface-list=WAN lan-interface-list=LAN wan-interface-list=WAN
/interface list member
add interface=sfpplus1 list=LAN
add interface=sfp1 list=WAN
add interface=sfp2 list=WAN
add interface=sfp3 list=LAN
add interface=sfp4 list=LAN
add interface=sfp5 list=LAN
add interface=sfp6 list=LAN
add interface=sfp7 list=LAN
add interface=sfp8 list=LAN
add interface=sfp9 list=LAN
add interface=sfp10 list=LAN
add interface=sfp11 list=LAN
add interface=sfp12 list=LAN
add interface=bridge1 list=LAN
/ip address
add address=10.0.1.1/24 comment=defconf interface=bridge1 network=10.0.1.0
/ip dhcp-client
add dhcp-options=hostname,clientid disabled=no interface=bridge1
add dhcp-options=hostname,clientid disabled=no
add dhcp-options=hostname,clientid disabled=no
add dhcp-options=hostname,clientid disabled=no
/ip dhcp-server lease
add address=10.0.1.2 client-id=1:74:4d:28:8:b6:58 mac-address=74:4D:28:08:B6:58 server=dhcp1
add address=10.0.1.3 client-id=1:b8:69:f4:26:32:80 mac-address=B8:69:F4:26:32:80 server=dhcp1
/ip dhcp-server network
add address=10.0.1.0/24 dns-server=8.8.8.8 gateway=10.0.1.1 netmask=24
/ip dns
set servers=8.8.8.8
/ip firewall nat
add action=masquerade chain=srcnat out-interface=sfp1 out-interface-list=WAN src-address=10.0.1.0/24
/ip route
add distance=1 gateway=10.0.1.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=America/Chicago
/system routerboard settings
set auto-upgrade=yes
/system upgrade upgrade-package-source
add address=10.0.1.1 user=admin
/tool user-manager database
set db-path=user-manager
JonathanYoung2161
 
JonathanYoung2161
just joined
Topic Author
Posts: 21
Joined: Fri Apr 05, 2019 4:46 am
Location: Iowa

Re: v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 7:10 am

I'm happy to perform any test to continue troubleshooting these issues. I have done point to point file transfers before with similar results some ports perform well with high throughput and low ping and others with low through put and high loss
JonathanYoung2161
 
mducharme
Trainer
Trainer
Posts: 868
Joined: Tue Jul 19, 2016 6:45 pm

Re: v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 7:15 am

So first thing, make sure your firmware is upgraded, not only the RouterOS version, otherwise you may be troubleshooting phantom bugs that should have been fixed.

Second if everything is updated is to unplug the switch from everything and do port to port speed tests with only two computers plugged in on 10Gbps links, using iperf3.
 
JonathanYoung2161
just joined
Topic Author
Posts: 21
Joined: Fri Apr 05, 2019 4:46 am
Location: Iowa

Re: v6.45.7 SFP+ speed and package issues

Sun Nov 03, 2019 7:21 am

All MiktoTik equipment we are using has v6.45.7 as the OS and the Boards firmware updated and we tested with 6.44.6 and 7.0B3 all came back with the same results and same issues. Also all equipment was factor reset with minimum changes to make the network usable.

That said tomorrow we will test with iperf3 and report our finding.
JonathanYoung2161
 
mducharme
Trainer
Trainer
Posts: 868
Joined: Tue Jul 19, 2016 6:45 pm

Re: v6.45.7 SFP+ speed and package issues

Mon Nov 04, 2019 12:04 am

All MiktoTik equipment we are using has v6.45.7 as the OS and the Boards firmware updated and we tested with 6.44.6 and 7.0B3 all came back with the same results and same issues. Also all equipment was factor reset with minimum changes to make the network usable.

That said tomorrow we will test with iperf3 and report our finding.
OK. One other question, how is the switch connected to the router? What port on the router is going to what port on the switch? Are you using a DAC cable for that or something else and if so, what?
 
JonathanYoung2161
just joined
Topic Author
Posts: 21
Joined: Fri Apr 05, 2019 4:46 am
Location: Iowa

Re: v6.45.7 SFP+ speed and package issues

Mon Nov 04, 2019 12:12 am

CCRs is connected via the SFP + port connected to the CRS SFP + port 1 as configured as a Lan bridge. And all their Network equipment connects off of the 10-gig switch through SFP + ports. We have multiple aoc's and dacs test results don't vary depending on what we use both come back the same results. Normal scenario we use all AOC or all DAC and don't mix and match on the network not that should have any impact. I have not tried running the tests off of a SFP+ to 10gb ethernet yet as the test didn't change between AOC and the DAC I didn't bother SFP+ to ether net.
JonathanYoung2161
 
mducharme
Trainer
Trainer
Posts: 868
Joined: Tue Jul 19, 2016 6:45 pm

Re: v6.45.7 SFP+ speed and package issues

Mon Nov 04, 2019 12:21 am

CCRs is connected via the SFP + port connected to the CRS SFP + port 1 as configured as a Lan bridge. And all their Network equipment connects off of the 10-gig switch through SFP + ports. We have multiple aoc's and dacs test results don't vary depending on what we use both come back the same results. Normal scenario we use all AOC or all DAC and don't mix and match on the network not that should have any impact. I have not tried running the tests off of a SFP+ to 10gb ethernet yet as the test didn't change between AOC and the DAC I didn't bother SFP+ to ether net.
OK, was just making sure that you were not trying to plug into one of the regular SFP (instead of SFP+) ports on the CCR.

You'll have to do the iperf3 tests on the switch by itself. I use this switch model at the core of an ISP doing about 1Gbps traffic and do not have loss or any performance issues.
 
JonathanYoung2161
just joined
Topic Author
Posts: 21
Joined: Fri Apr 05, 2019 4:46 am
Location: Iowa

Re: v6.45.7 SFP+ speed and package issues

Mon Nov 04, 2019 12:28 am

Will get those test results soon
JonathanYoung2161
 
JonathanYoung2161
just joined
Topic Author
Posts: 21
Joined: Fri Apr 05, 2019 4:46 am
Location: Iowa

Re: v6.45.7 SFP+ speed and package issues  [SOLVED]

Sun Nov 10, 2019 7:26 pm

Issue resolved!

Found it to be the Aquantia AQtion SFP+ NIC cards.

Suspect a compatibility issue with either the SFPs that are Mikrotik compatible or between the Mikrotik and Aquantia.

Mind blown...

Thanks for all the help and every ones time!
JonathanYoung2161
 
llag
newbie
Posts: 34
Joined: Sat Aug 04, 2018 12:12 am

Re: v6.45.7 SFP+ speed and package issues

Tue Nov 12, 2019 1:04 pm

Issue resolved!

Found it to be the Aquantia AQtion SFP+ NIC cards.

Suspect a compatibility issue with either the SFPs that are Mikrotik compatible or between the Mikrotik and Aquantia.

Mind blown...

Thanks for all the help and every ones time!
This issue could be caused by the CRS317 being picky about what it connects to. When connected to the CRS328 my CRS317 the link is flapping with lots of FCS and RX errors. Are the links to your CRS328 stable (check the link down counter for the interface to the CRS328)?
Are the Aquantia's stable when you connect them to the CRS328 SFP+ ports?

Who is online

Users browsing this forum: No registered users and 0 guests