Page 1 of 3

Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Thu Nov 15, 2018 6:47 pm
by andersa
Hi

I am running 17 CRS328-24P-4S+ connected to 2 CRS317-1G-16S+ using this SFP S+32LC10D. The 328 boxes are running RouterOS 6.43.2 and the CRS317 boxes are running 6.43.4. I am experiencing that all the ports on the CRS317 are going up and down with anything between minutes to hours apart. It is almost no traffic on the network. I am totally in the dark here, do any of you have any idea what this could be?

Example from log:
17:02:12 interface,info sfp-sfpplus2 link down
17:04:07 interface,info sfp-sfpplus2 link up (speed 10G, full duplex)
17:09:04 interface,info sfp-sfpplus8 link down
17:09:05 interface,info sfp-sfpplus8 link up (speed 10G, full duplex)
17:18:00 interface,info sfp-sfpplus3 link down
17:18:01 interface,info sfp-sfpplus3 link up (speed 10G, full duplex)
17:24:13 interface,info sfp-sfpplus4 link down
17:24:15 interface,info sfp-sfpplus4 link up (speed 10G, full duplex)
17:25:04 interface,info sfp-sfpplus16 link down
17:25:05 interface,info sfp-sfpplus16 link up (speed 10G, full duplex)
17:34:00 interface,info sfp-sfpplus4 link down
17:34:01 interface,info sfp-sfpplus4 link up (speed 10G, full duplex)

Interface stats from one of the CRS328:
/interface ethernet print stats from=sfp-sfpplus2
name: sfp-sfpplus2
driver-rx-byte: 4 096 343 318
driver-rx-packet: 44 864 695
driver-tx-byte: 1 460 265 180
driver-tx-packet: 9 040 331
rx-bytes: 4 277 853 088
rx-too-short: 0
rx-too-long: 0
rx-unicast: 9 695 367
rx-broadcast: 31 908 257
rx-pause: 0
rx-multicast: 3 273 516
rx-error-events: 239
rx-fcs-error: 0
rx-fragment: 0
rx-overflow: 0
rx-jabber: 0
tx-bytes: 1 498 564 543
tx-unicast: 9 049 910
tx-broadcast: 231
tx-pause: 0
tx-multicast: 4 463
tx-underrun: 0
tx-collision: 0
tx-late-collision: 0
tx-rx-64: 31 287 843
tx-rx-65-127: 4 828 178
tx-rx-128-255: 17 182 394
tx-rx-256-511: 179 280
tx-rx-512-1023: 436 533
tx-rx-1024-1518: 17 622
tx-queue0-packet: 9 037 205
tx-queue1-packet: 0
tx-queue2-packet: 0
tx-queue3-packet: 0
tx-queue4-packet: 0
tx-queue5-packet: 0
tx-queue6-packet: 0
tx-queue7-packet: 0

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Tue Dec 04, 2018 9:19 pm
by keefe007
Try 6.43.7 and make sure routerboot is also updated.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Tue Dec 04, 2018 10:38 pm
by andersa
Sure, i dont see anything about the 317s in the changelog, but i'll try. Will give a new status in about 24h, will see if anything has changed.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Wed Dec 05, 2018 9:30 am
by andersa
Try 6.43.7 and make sure routerboot is also updated.
Sad to say, but this made no difference.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Wed Dec 05, 2018 4:30 pm
by keefe007
I am also seeing some link downs on CRS328 switches but the firmware update made them significantly less for me.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Mon Dec 10, 2018 10:01 pm
by keefe007
Did you find a fix yet?

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Tue Dec 11, 2018 9:47 am
by andersa
Unfortunately not. I sendt them the support files at 22. Nov and did a follow up on 4. Des. Still no reply. Would be great if MT would have a look at this.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Tue Dec 11, 2018 10:59 am
by mducharme
From the MikroTik site it looks to me like the S+32LC10D is specifically meant to have the S+23LC10D module on the other end. Are you using the same module on both ends?

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Tue Dec 11, 2018 11:21 am
by Elliot
I have the SAME EXACT problem. Mikrotik needs to fix this. No problem with 1Gbps SFP but with OWM SFP+ it's hell. I think this could be combination of software and hardware problem with SFP+ modules.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Tue Dec 11, 2018 11:54 am
by andersa
I have the SAME EXACT problem. Mikrotik needs to fix this. No problem with 1Gbps SFP but with OWM SFP+ it's hell. I think this could be combination of software and hardware problem with SFP+ modules.
I agree with you assumtion that this is an edge case between the SFP units and the switch. But strangely it only occurs on one of our two CRS317 in the installation.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Tue Dec 11, 2018 11:55 am
by andersa
From the MikroTik site it looks to me like the S+32LC10D is specifically meant to have the S+23LC10D module on the other end. Are you using the same module on both ends?
I am using the paired SFP+ from Mikrotik S+2332LC10D. So yes.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Tue Dec 11, 2018 6:51 pm
by keefe007
What's the fix Mikrotik?

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Sat Dec 22, 2018 6:42 pm
by cowgirl
Push - she problem on CRS328 here.... waiting for a fix

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Sat Dec 22, 2018 6:49 pm
by sff
We had this same problem appear on one of our 317s. I am not 100% sure when it started, but it was observed on 6.43.4. The upgrade to 6.43.7 fixed it, but ONLY after I did a full factory reset and reloaded the configuration by hand (it wasn't complex).

We also observed it for a brief time on several of the 328s. Switching from using SFP+1 to SFP+2 for the uplink fixed it.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Fri Dec 28, 2018 6:37 pm
by wotoni
We are experiencing the same thing. Running 6.42.7 with DAC Cable from CRS317 to CRS328-4C-20S-4S+RM.

I had to go to a SFP 1G on both units for the flapping to stop. I had tried SFP+ DAC and SFP+ single mode fiber with the same issue.

We have 8 CRS212 plugged into the CRS317 via DAC Cable with no issue. Only the 328 SFP+ ports appear to have the issue.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Wed Feb 06, 2019 11:21 am
by AVA
Exactly the same problem here. Links between CRS317 and CRS328-4C-20S-4S+ are flapping. I have 3 pairs of 317 <-> 328 and all have the same problem. I tried different SFP/DAC. The latest firmware (6.43.8) does not help. No problem with 1GbE links....... :(

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Wed Feb 06, 2019 1:05 pm
by andersa
I sendt MT a service dump in november - still no answer. Could i bother you guys that are experiencing the same problem with sending MT an email urging them to have a look at the issue - or at least give some feedback. I am experiencing total radio silence.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Wed Feb 06, 2019 2:52 pm
by jprietove
Are any of you using port ether1 or just sfp-plus ports? If you are, remember that "The new Cloud Router Switch 317-1G-16S+RM is a rack-mountable manageable switch with Layer3 features, it has 16 SFP+ ports for high performance 10GbE connectivity and a 1GbE copper port for management. "

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Wed Feb 06, 2019 3:04 pm
by andersa
Are any of you using port ether1 or just sfp-plus ports? If you are, remember that "The new Cloud Router Switch 317-1G-16S+RM is a rack-mountable manageable switch with Layer3 features, it has 16 SFP+ ports for high performance 10GbE connectivity and a 1GbE copper port for management. "
I can only speak for myself - and i am only using the SFP+ ports.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Wed Feb 06, 2019 4:35 pm
by AVA
I think the problem is the CRS328. We use the CRS317 as Core with downlinks to different CA devices, some with 1GbE and some with 10GbE. The only affected links are those between CRS317 and CRS328 with 10GbE.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Thu Feb 07, 2019 4:29 pm
by sebg
exact same issue here, same symptoms, same workaround (force 1G link), it would be great if Mikrotik took this issue seriously...

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Thu Feb 07, 2019 8:20 pm
by Redmor
I ever had bad experiences with CRSs port flapping and I don't use them anymore, both ethernets and SFPs.
Sometimes on CCRs ports go totally down, disabling and enabling them solves the problem.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Fri Feb 08, 2019 9:57 am
by sebg
Sometimes on CCRs ports go totally down, disabling and enabling them solves the problem.
In production context ? :)

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Mon Feb 18, 2019 5:08 pm
by AVA
I've wrote to Mikrotik Support ~2 weeks ago and also our Distributor said he would contact Mikrotik but till now no reaction. :cry:

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Mon Feb 25, 2019 11:50 am
by AVA
Hi,

I got an answer from support. They could reproduce the flapping issue and are working on a solution.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Fri Mar 01, 2019 8:01 pm
by RFG
Same problem over here with CRS328-4C-20S-4S+RM on Sfp+ ports. I'm connected to a Dell X1052 with S+DA0003.
Let's wait for the correction.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Tue Mar 05, 2019 5:48 am
by K0IBM
I will add my agreement to this thread. I have a CRS317-1G-16S+ with 4 CRS328-24P-4S+ and 3 CSS326-24G-2S+RM All are using the Maxxwave MW-SX+MM-US Modules. I have the same port flaping on all the CRS328 ports and the CRS317 reboots on a regular basis. The links to the CSS326's seem to be stable. I too am interested in this fix.

Steve Cearley
Steve@cearley.us
Wellshire Presbyterian Church.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Tue Mar 05, 2019 12:56 pm
by inatec
hi,

same for us. We fighting with the same problem on our CRS317-1G-16S+ (viewtopic.php?f=2&t=146044). We thougth, it might be a configuration issue. I replaced cable, SFP+ modules (Allnet / Ubiquitiy) but nothing helped. Now I replaced the both ports (core-03 and core-04) with the 10G copper SFP ports and its seems now stable.
We replaced in the first time also one switched, which helped (core-lan-01). But now we have the same issue between core-lan-03 and core-lan-04.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Thu Mar 07, 2019 6:26 pm
by Stick1970
i mean the same problem
port flapping on CRS328-24P-4S+ and CRS317-1G-16S+
software version 6.44
I tried different SFP modules.
Are there any solutions to this problem?

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Fri Mar 08, 2019 8:21 am
by coelliale
Me too..
317 - Ccr 10Gb mk Dac ok
317 - 326 10Gb mk S+85DLC03D ok
317 - 328 10Gb mk Dac flapping
The strange thing is that for over one month all worked fine, after that, without any change, the flapping began.
Forcing the Dac to 1Gbps stop the flapping immediatly, now we are monitoring it, but we need 10Gb connections.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Fri Mar 08, 2019 8:34 am
by roggles
hello folks,

we have multiple CRS317-1G-16S+ and CRS328-24P+4S+ with the same problem.
All running RouterOS and firmware 6.44.
They all connected togehter with Mikrotik DAC S+DA0001.
After switching to RouterOS and firmware to 6.45beta6 (Testing) don't solve the flapping port problem.

Last try....
Finally we switched on the CRS317-1G-16S+ to SwitchOS 2.9.

So the CRS328-24P+4S+ with RouterOS works with the CRS317-1G-16S+ with SwitchOS for 24h without problems...


So i think it's a problem on the CRS317-1G-16S+ with RouterOS.

regards

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Fri Mar 08, 2019 9:06 am
by inatec
hi,
Me too..
317 - Ccr 10Gb mk Dac ok
317 - 326 10Gb mk S+85DLC03D ok
317 - 328 10Gb mk Dac flapping
The strange thing is that for over one month all worked fine, after that, without any change, the flapping began.
Forcing the Dac to 1Gbps stop the flapping immediatly, now we are monitoring it, but we need 10Gb connections.
since, not all ports flapping, just one or two, I assume that there is a hardware failure. But we have to wait, until MikroTik have something so say.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Fri Mar 08, 2019 10:33 pm
by hapi
hmm, same problem

CRS317-1G-16S+ vs CRS326-24G-2S+ - DAC

today CRS317 stop traffic and in same time CRS326 stop working port connecting to CRS317.
MKv6.44

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Mon Mar 11, 2019 10:08 am
by roggles
hello folks,

update:

After we switched on the CRS317-1G-16S+ to SwitchOS 2.9
5 days without port flapping. :)

regards

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Mon Mar 11, 2019 10:13 am
by andersa
hello folks,

update:

After we switched on the CRS317-1G-16S+ to SwitchOS 2.9
5 days without port flapping. :)

regards
This is good news and indicative of software issues. Raising the probability of a fix. I have however met nothing but radio silence from MT lately, have any of you guys gotten any info from them?

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Mon Mar 25, 2019 8:17 pm
by keefe007
What's the stats of the fix?

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Thu Mar 28, 2019 8:43 am
by coelliale
After we switched on the CRS317-1G-16S+ to SwitchOS 2.9
5 days without port flapping. :)
Hi roggles, still stable with SwitchOs?

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Thu Mar 28, 2019 11:01 am
by roggles
yes, since there are no log entries

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Thu Mar 28, 2019 11:26 am
by hapi
CRS317 replaced with UBNT EdgeSwitch 16XG. Problem solved. I will not wait for the device to stop 1Gbit traffic.

I won't wait for it to ever be fixed. Similar to flapping on RB3011 and the like.



MIkrotik -1000.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Wed Apr 03, 2019 10:51 am
by maxpower
Have the same problem with UF-SM-10G & CRS317-1G-16S+ with 6.42.12.
Any RoOS version with fix?

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Wed Apr 03, 2019 11:35 am
by Guntis
Have the same problem with UF-SM-10G & CRS317-1G-16S+ with 6.42.12.
Any RoOS version with fix?
Could you please send us email, about your setup and problem, to support@mikrotik.com?

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Wed Apr 03, 2019 11:43 am
by andersa
Have the same problem with UF-SM-10G & CRS317-1G-16S+ with 6.42.12.
Any RoOS version with fix?
Could you please send us email, about your setup and problem, to support@mikrotik.com?
Hi! How Is the fix coming along?

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Wed Apr 03, 2019 12:45 pm
by maxpower
Could you please send us email, about your setup and problem, to support@mikrotik.com?
Ticket#2019040322003846

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Wed Apr 03, 2019 12:57 pm
by andersa
Could you please send us email, about your setup and problem, to support@mikrotik.com?
Ticket#2019040322003846
Where do i get additional information about tickets? I see no links in the emails i have gotten.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Tue Apr 09, 2019 5:29 pm
by asadsayyad123
Severe port flapping on 10G links, works with auto-negotiation turned off and set to 1g.

CRS317-1G-16S+ <<>> CCR1009-8G-1S-1S+, Flapping at 10G, Works stable at 1G

CRS317-1G-16S+ <<>> CRS328-4C-20S-4S+, Flapping at 10G, Works stable at 1G

CRS317-1G-16S+ <<>> CRS212-1G-10S-1S+, Works Stable at 10G.

CRS328-4C-20S-4S+<<>> CRS328-4C-20S-4S+, Works Stable at 10G.




So, it all boils down to ROS BUG in CRS317.

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Sun Apr 14, 2019 6:09 pm
by maxpower
Does anyone have port flapping problem with 10G link between CRS317-1G-16S+ <<>> CRS317-1G-16S+ ?
In our case it starts to happen after about a week after reboot and it happens on 2 of 3 ports. We tried to interchange modules used on stable link - but on other link it also flaps.

We have flapping on 1 & 5 ports, but no flapping on 9 port. Coincidence?

In SwitchOS you dont have log entries and the link is stable also? May be in SwOS it just does not log the short term failures?

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Sun Apr 14, 2019 6:21 pm
by coelliale
Mikrotik support tell us that the problem is well known and that they don't have ETA for the resolution.
Their official workarround is to downgrade links to 1Gb or use switchos on 317
I hope that someone take care about this terrible bug!!

Alessandro

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Sun Apr 14, 2019 7:26 pm
by neutronlaser
file bug

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Mon Apr 15, 2019 6:18 am
by asadsayyad123
Hello,

SwitchOS is one big pile of shit with no real featues, as it was intended for the general audience.

When i add SFP media converters or non-mikrotik or unmanaged switches on the other end it does not pick the link up.

But ROS does, hence we cannot use SWOS at all (As a last resort even if it a very crap OS .).

Re: Severe port flapping on CRS328-24P-4S+ and CRS317-1G-16S+

Posted: Mon Apr 15, 2019 9:13 am
by kosyot
Hello,

SwitchOS is one big pile of shit with no real featues, as it was intended for the general audience.

When i add SFP media converters or non-mikrotik or unmanaged switches on the other end it does not pick the link up.

But ROS does, hence we cannot use SWOS at all (As a last resort even if it a very crap OS .).
You are absolutely right about SwOS, but what can do with many introduced MiktoTik devices with only SwOS ?
It's not normal to produce a lot of hardware with no software capable to handle it...
I still believe, that MiktoTik will get attention on this and will produce real working OS.
btw I keep all my with SwOS 2.7