RB5009 SFP+ 10G Optical & Copper DAC w/Intel X520-DA1

Hardware MikroTik hardware general
1 
 8 RS name="sfp-sfpplus1" default-name="sfp-sfpplus1" mtu=1500 l2mtu=1514 mac-address=2C:C8:1B:FF:XX:XX
      orig-mac-address=2C:C8:1B:FF:8F:65 arp=enabled arp-timeout=auto loop-protect=default 
      loop-protect-status=off loop-protect-send-interval=5s loop-protect-disable-time=5m auto-negotiation=no 
      advertise="" full-duplex=yes tx-flow-control=off rx-flow-control=off speed=10Gbps 
      bandwidth=unlimited/unlimited switch=switch1 sfp-rate-select=high sfp-shutdown-temperature=95C 
> interface/ethernet/monitor sfp-sfpplus1 
                               name: sfp-sfpplus1
                             status: link-ok
                               rate: 10Gbps
                        full-duplex: yes
                    tx-flow-control: no
                    rx-flow-control: no
                 sfp-module-present: yes
                        sfp-rx-loss: no
                       sfp-tx-fault: no
                           sfp-type: SFP-or-SFP+
                 sfp-connector-type: optical-pigtail
  sfp-link-length-copper-active-om4: 3m
                    sfp-vendor-name: AVAGO
             sfp-vendor-part-number: AFBR-7CER03Z-BD1
                  sfp-vendor-serial: XXX
             sfp-manufacturing-date: 13-08-22
                     sfp-wavelength: 850nm
                    eeprom-checksum: good

And on the client side:

$ sudo ethtool ens2
Settings for ens2:
Supported ports: /[ FIBRE /]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Auto-negotiation: off
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: umbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
$ sudo ethtool -m ens2
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x0b (Optical pigtail)
Transceiver codes : 0x10 0x00 0x00 0x00 0x40 0x40 0x08 0x40 0x00
Transceiver type : 10G Ethernet: 10G Base-SR
Transceiver type : FC: short distance (S)
Transceiver type : FC: Shortwave laser w/o OFC (SN)
Transceiver type : FC: Multimode, 62.5um (M6)
Transceiver type : FC: 800 MBytes/sec
Encoding : 0x06 (64B/66B)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 0m
Length (62.5um) : 0m
Length (Copper) : 3m
Length (OM3) : 0m
Laser wavelength : 850nm
Vendor name : AVAGO
Vendor OUI : 00:17:6a
Vendor PN : AFBR-7CER03Z-BD1
Vendor rev :
Option values : 0x00 0x12
Option : RX_LOS implemented
Option : TX_DISABLE implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : XX
Date code : 130822

When ixgbe loads/reloads (using sudo modprobe -r ixgbe && sudo modprobe ixgbe allow_unsupported_sfp=1 debug=16) - I get:

[ 6002.933547] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 6002.933549] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 6003.100831] ixgbe 0000:04:00.0: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16 XDP Queue count = 0
[ 6003.101235] ixgbe 0000:04:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[ 6003.101563] ixgbe 0000:04:00.0: MAC: 2, PHY: 16, SFP+: 5, PBA No: H29295-003
[ 6003.101565] ixgbe 0000:04:00.0: 90:e2:ba:bd:XX:XX
[ 6003.105440] ixgbe 0000:04:00.0: Intel(R) 10 Gigabit Network Connection
[ 6003.105649] libphy: ixgbe-mdio: probed
[ 6003.106393] ixgbe 0000:04:00.0 ens2: renamed from eth0
[ 6003.286464] ixgbe 0000:04:00.0: registered PHC device on ens2
[ 6003.473490] ixgbe 0000:04:00.0 ens2: detected SFP+: 5
[ 6003.745516] ixgbe 0000:04:00.0 ens2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 6004.645635] ixgbe 0000:04:00.0 ens2: NIC Link is Down

Unfortunately I don’t have many other bits of 10G hardware to go replacing things, but I’m a bit at a loss. This is on 7.1rc6 apart from disabling autoneg wasnt too sure what to do. I tried forcing it to 1GB but the other side (linux won’t let me):

$ sudo ethtool -s ens2 speed 1000 duplex full autoneg off
netlink error: link settings update failed
netlink error: Invalid argument

Any ideas what could be wrong here? It seems everything should be working?

2

same problem with you, I have some 10g optical device, but x520-da1 is the only pci-e device. so i can not locate the problem either.
Huawei router + 10g Optical Module————rb5009 + 10g Optical Module work.
Huawei switch + 10g optical module————rb5009 + 10g Optical Module work.
x520-da1 in windows computer+10g optical module————rb5009 + 10g Optical Module NOT work.(no matter set to 10g Full-Duplex or not)
x520-da1 in windows computer+1g optical module————rb5009 + 1g Optical Module work.
x520-da1 in ubuntu 21 computer+10g optical module————rb5009 + 10g Optical Module NOT work.(it’s always set to 10g Full-Duplex or not) same nic link is down with you
x520-da1 in ubuntu 21 computer+1g optical module————rb5009 + 1g Optical Module work.

so i think the problem is not optical module, Im tying to get another card like mcx411a to do more test

3

mcx311 is working good……but it doesn’t support sriov……so im still working on my x520-da1.
I tried to use dac to connect.still not working

4

Have you tried changing the sfp-rate-select setting? Apparently with some transceivers that setting matters.

5

I tried both low & high, makes no difference. I also got a copper pigtail but behaviour is exactly the same:

[571235.413215] ixgbe 0000:04:00.0 ens2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[571235.523187] ixgbe 0000:04:00.0 ens2: NIC Link is Down
[571238.933228] ixgbe 0000:04:00.0 ens2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[571239.043198] ixgbe 0000:04:00.0 ens2: NIC Link is Down
[571347.182964] ixgbe 0000:04:00.0 ens2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[571347.292951] ixgbe 0000:04:00.0 ens2: NIC Link is Down
[571376.552867] ixgbe 0000:04:00.0 ens2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[571377.454217] ixgbe 0000:04:00.0 ens2: NIC Link is Down



% sudo ethtool -m ens2
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x21 (Copper pigtail)
Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00 0x00
Transceiver type : Passive Cable
Encoding : 0x00 (unspecified)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 0m
Length (62.5um) : 0m
Length (Copper) : 2m
Length (OM3) : 0m
Passive Cu cmplnce. : 0x30 (unknown) [SFF-8472 rev10.4 only]
Vendor name : Arista Networks
Vendor OUI : 00:1c:73
Vendor PN : SFP-10G-CU1.5M
Vendor rev :
Option values : 0x00 0x00
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : ARR210408XX
Date code : 210408

6

hi,

i’ve same problem with auto negotiation activated. Firmware 7.1.1.
Proxmox server with X520DA1.
2m DAC cable.
Auto Negotiation → incomplete.
Test manual config 1Gb, without sucess.

Regards.

7

I too am having the same problem, same setup. I’m only trying with a Copper DAC, but the link won’t stay up. Anyone make any progress with this?

8

I have same issues, RB5009+ CRS328 connected by 3m-long Mikrotik sfp+ DAC. And link between RB5009 and CRS328 is up and down from time to time.
The auto negotiation also shows “incomplete”.

9

I have the same setup and the same issue: RB5009+ (7.1.1) connected to CRS328 (6.49.2/7.1.1) using XS+DA0003. “Incomplete” auto neg status and link is flipping multiple times a day.

10

Does anyone else have any information on this? Is there expected to be a software fix for this? Is this simply just incompatible hardware? Does anyone have any good alternatives to using a 10G DAC with a X520-DA1?

11

I have sent email to mikrotik support to report this issue and got a ticket #. but so far no reply from mikrotik.

12

sfpworking.png
Works for me with a pair of UBNT UF-MM-1G (between RB5009 and a RB260GSP). MikroTik’s own S+DA0001, appears to be incompatible

13

Today I tested my friend’s RB5009+ and it has the same issue - incomplete auto neg and port flipping. So, or it is some hardware issue (I hope not) or just software.

14

Interesting behaviour. I have an RB5009 running 7.1 connected to a CRS328 running 7.1 (although it also worked when running 6.48) connected via a Mikrotik DA0001. It’s never had a connectivity issue, so I wonder what’s different for you folks?

UPDATE: I checked mine in more detail and there’s no flapping issues but it does say Autonegotiation incomplete with a link speed of 10Gbs.

15

Do a constant ping to the RB5009 from the CRS side of the network. I thought SFP was running stable on my rb5009 but although the logs don’t show port flapping, occasionally something happens which causes the link to drop/reset and about 3 consecutive pings are dropped (Windows ping with default parameters)

16

Same issue using intel xxv710-da2(the sfp28 version) and rb5009
I tried dac cable and aoc cable(both two are oem cable), both side recognized sfp+ module, with same issue(up and down and up and down …)

Work fine with same cable connected to rb4011 and crs305
now I use crs305 to ‘bridge’ the connection of the intel xxv710 card and rb5009

hope it will fix in future

17

I had another go with a different X520-DA2 card and the behaviour is identical.

Has someone opened a support ticket with mikrotik about this?

The link even worked for long enough for me to get an IP before dying when i enabled flow control and autonegociation.

Mar 10 10:40:32 xyz dhcpcd[395]: ens2f1: IAID 9f:6f:82:32
Mar 10 10:40:32 xyz kernel: ixgbe 0000:04:00.1 ens2f1: detected SFP+: 4
Mar 10 10:40:33 xyz dhcpcd[395]: ens2f1: soliciting an IPv6 router
Mar 10 10:40:33 xyz kernel: ixgbe 0000:04:00.1 ens2f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Mar 10 10:40:33 xyz kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ens2f1: link becomes ready
Mar 10 10:40:33 xyz dhcpcd[395]: ens2f1: soliciting a DHCP lease
Mar 10 10:40:33 xyz dhcpcd[395]: ens2f1: offered 192.168.88.15 from 192.168.88.1
Mar 10 10:40:33 xyz dhcpcd[395]: ens2f1: probing address 192.168.88.15/24
Mar 10 10:40:34 xyz dhcpcd[395]: ens2f1: carrier lost
Mar 10 10:40:34 xyz kernel: ixgbe 0000:04:00.1 ens2f1: NIC Link is Down

18

I am having the same issue on a CRS326-24S+2Q+RM running RouterOS v6.49.6. The ports consistently flap towards the X520 Cards. Running 5m DAC cables and not sure what else to try here.

19

So I should have mentioned this issue on the RB5009 is fixed since 7.3.x, obviously I have no idea on the CRS326.