Community discussions

MUM Europe 2020
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3090
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

fcs error on new CCR1009

Sun Oct 04, 2015 4:12 pm

Yesterday replaced a rb1100AH running v.6.30.2 to a brand new CCR1009-8G-1S-1S+ router.

Looked in the log today and found it filled with fcs errors?
Screenshot 2015-10-04 15.03.14.png
All ethernet cables are almost new and connect to new routers and on the rb1100AH showed no errors and were Gigabit connected (auto negotiation on)

Now these failures?
I installed the CCR with 6.32.2 but reverted back to 6.30.2 as the most stable version but the problem didn't disappear....

1. It can't be the cables, they were all fine on the rb1100AH and basically all new...
2. It can't be the software? 6.32.2 or 6.30.4 makes no difference. And most of my other routers now work fine with either one of these two versions...

I ran 2 ping tests with 100ms timeout to adjacent router and that is fine. 1st is 100ms and 50 bytes packet size, 2nd is 100ms but 1500 packet size. On the latter I see an occasional time out and I see 1% package loss...
The stats of the interface is also showing the error. 2100 errors in 25 mins uptime....

Do I have a failing CCR? Where is this error coming from?
You do not have the required permissions to view the files attached to this post.
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3090
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: fcs error on new CCR1009

Tue Oct 06, 2015 11:19 am

Nobody? Mikrotik? support Ticket#2015100466000162 send last week thursday, still no answer?
Screenshot 2015-10-06 10.15.20.png
Traffic seems to flow normally. When run a ping over the link no delays, no package losses.....
Router on the other end shops no issues on same connection.
You do not have the required permissions to view the files attached to this post.
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1164
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: fcs error on new CCR1009

Tue Oct 06, 2015 7:37 pm

Three things cause FCS errors typically, with the first two being the most likely.

1) Physical cabling issues

2) Speed / duplex mismatch or errors

3) Bugs

Steps to resolve.

1) Change the cable with a known good one on the link you are getting FCS and reset counters.

2) Hard code speed and duplex to match on both sides of the link.

3) Move to the bugfix version - 6.30.4 as it is the most stable version for ROS right now in general
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 1761
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: fcs error on new CCR1009

Tue Oct 06, 2015 7:50 pm

be sure you have updated the firmware (not routeros)

put a manageable switch as a bypass on that link to see if switch receives the same errors
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3090
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: fcs error on new CCR1009

Tue Oct 06, 2015 8:14 pm

Three things cause FCS errors typically, with the first two being the most likely.

1) Physical cabling issues

2) Speed / duplex mismatch or errors

3) Bugs

Steps to resolve.

1) Change the cable with a known good one on the link you are getting FCS and reset counters.

2) Hard code speed and duplex to match on both sides of the link.

3) Move to the bugfix version - 6.30.4 as it is the most stable version for ROS right now in general
1). After 1st install router showed fcs errors on 6 out of the 8 gigabit connections. Later most have gone into a very occasional error, but this port 7 keeps going on.
This router served as replacement for slower rb1100AH but that one didn't have a single fcs error on the same cables...
2). Since ports are gigabit auto setting has to be followed or it simply doesn't work. (MT and internet documentation depict it has to be 'auto'!)
3.) Already reverted back to 6.30.4 to no avail, and up to 6.32.2 again and latest firmware is running.
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1164
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: fcs error on new CCR1009

Wed Oct 07, 2015 5:35 pm

Really sounds like you may have a bad board out of the box....few comments below

1) If you can swap the 1100 back in and the FCS goes away, then i would agree cabling is prob not it

2) This is true only on copper (and you can set 100M Full on each side just for testing) SFP ports can be hard coded to 1000/Full as well.

Examples here:
[admin@IPA-LAB-CCR1009-01] > interface ethernet monitor 1
              name: ether2
            status: link-ok
  auto-negotiation: disabled
              rate: 100Mbps
       full-duplex: yes
   tx-flow-control: no
   rx-flow-control: no
   
 [admin@IPA-LAB-CCR1009-01] > interface ethernet monitor 9
                    name: sfp1
                  status: link-ok
        auto-negotiation: disabled
                    rate: 1Gbps
             full-duplex: yes
         tx-flow-control: no
         rx-flow-control: no
      sfp-module-present: yes
             sfp-rx-lose: no
            sfp-tx-fault: no
      sfp-connector-type: LC
     sfp-link-length-9um: 20000m
         sfp-vendor-name: Mikrotik
  sfp-vendor-part-number: S-31DLC20D
       sfp-vendor-serial: MT40813H3283
  sfp-manufacturing-date: 14-08-13
          sfp-wavelength: 1310nm
            sfp-rx-power: -7dBm
3) In our lab we have a CCR1009 connected to MikroTik, Cisco and HP network equipment on copper and fiber without issue....unfortunately you probably got a bad CCR.

There was a batch of bad CCR1009s early on and it's possible you got one that's been on the shelf for a while as all the new CCR1009s shipping seem to be pretty solid.

http://forum.mikrotik.com/viewtopic.php?t=88173
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
gabrielpike
Frequent Visitor
Frequent Visitor
Posts: 84
Joined: Thu Apr 17, 2014 4:17 pm

Re: fcs error on new CCR1009

Wed Oct 07, 2015 6:49 pm

Is this router plugged in to an airfiber? I have a CCR1036 that spews this error no matter what when an airfiber is attached. Almost same scenario replaced an 1100AHx2 with the CCR1036.
Gabriel Pike
MTCNA
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3090
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: fcs error on new CCR1009

Thu Oct 08, 2015 11:53 am

Nailed the issue for 95%:

All links / ethernet cables leave from CCR1009 over some 15 meter of ftp cable (all well grounded on both ends....) to a tower cabinet.
In the cabinet a 24V PoE battery fed midspan power inserter with lighting/surge protector is fit. From here some 8-10 meter of ftp (all well grounded on both ends...) run up the tower to connect to the radios. Mostly Netmetals.

The midspan PoE injector is from Cyberteam, Poland. This is a managed Gigabit version. This is actually the 3rd box that give me problems. One just died, the second had no more webserver working and became useless because of that en this last one after 2 months again fell without web server control. So my suspicion went to this device. Last night I replaced it to a brand new unmanaged gigabit net-protector and almost all fcs error disappeared from the logs.....
So this is the 3rd managed netprotector heading for the bin (with 2 unmanaged going the same before..... poor quality stuff....)

I still suspect the last remaining errors, now only on one link, are due to this new netprotector. But to be honest there are not a lot 24V multiple port passive PoE midspan injectors in gigabit version with IP management on the market. The nearest solution I can find are Netonix WISP switch but I need to convert it into 1:1 port to port switch only since the several routes have different dhcp-servers and hence these 'pipe's need to stay separated.

To reply to some of your comments;
1) If you can swap the 1100 back in and the FCS goes away, then i would agree cabling is prob not it
I am 100% its not the cabling. It also happened on almost every port, and they all have Gb connections.
2) This is true only on copper (and you can set 100M Full on each side just for testing) SFP ports can be hard coded to 1000/Full as well.
Since the midspan PoE insert and the fact that CCR sits as router in a crosspoint of links, 1:1 SFP connections are not possible. Also because we don't have the tools to make fibre cables nor can I buy prefab SFP with lengths of more than 5 meters against affordable prices....
3) In our lab we have a CCR1009 connected to MikroTik, Cisco and HP network equipment on copper and fiber without issue....unfortunately you probably got a bad CCR.

There was a batch of bad CCR1009s early on and it's possible you got one that's been on the shelf for a while as all the new CCR1009s shipping seem to be pretty solid.
I agree, I am now more looking at another midspan PoE solution.
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3090
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: fcs error on new CCR1009

Thu Oct 08, 2015 11:56 am

I replace the midspan PoE netprotector last night at 1 am.
I opened the log of the CCR this morning at 10 am. During this 9 hours it ran it shows only 4 fsc error on one and the same link.
But while posting on this forum in just 20 mins I had another 5 errors of that same port....... weird...
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3090
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: fcs error on new CCR1009

Thu Oct 08, 2015 3:31 pm

Update:
This is what I send to support (some grammar edits...):
Ok,

Last night at around 1 am we replaced the gigabit netprotector (Poe midspan injector) for another more simple one (Also giga, also Cyberteam, but no remote access, no control whatsoever) brand new out of the box.
Immediately after the change the fcs errors didn't come back. Well, instead of every 30 seconds until this morning 10 am when I logged in again the log only showed 4 fcs errors in total since the change.... good!
.............. You would think.................... =>

While looking over a remote winbox session in this CCR1009's log this morning I saw the errors coming back! First every 10 mins, then every 5 mins, every 3-4 mins and now they are back to every 2 mins.......
How is this possible? Overnight the problem vanished just to come back in the morning....? Very, very weird....

I don't know what to do now. Cannot go to 100mbps setting on each end since this is my main backhaul that at times has more traffic than 100mbps. Fast Ethernet will create a bottleneck again. (I spend a lot of time and money and energy into reforming a previously duo bonded link with double antenna sets etc. high in towers into a 'ac' single 40Mhz wide link to get up to 300Mbps over this new link.... (tested up to 280M!)

I also cannot fit in the gigabit power inserters from Mikrotik. Because the poe insert is in the middle of a cable. Both ends need to be at least 10 meters.....
I don't seem to be able to find any passive poe midspan inserters that work on 24V battery power and have gigabit port and are remotely controllable......

So, main question; Until I find any solution, how bad is it these errors are there? They fill the log, but do they have any other harm? I don't seem to notice problems (yet?). Ping on long and short package see no losses and ping times are good and traffic flows....
But out of the 12 NetMetals installed over the last weeks, all running with latest ROS and firmware in bridge mode (fastpath enabled) I have 3 of them that just stop passing traffic once a 2 or 3 weeks....... only a power reboot brings these back in working order again. Can this have anything to do with some fcs error too? (Since the Netmetals don't show these errors in the logs I never looked for them....)
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1164
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: fcs error on new CCR1009

Thu Oct 08, 2015 5:41 pm

While it doesn't sound like the cabling itself is the issue, there does appear to be a physical issue in the copper path (which is the type of error that FCS is designed to indicate) possibly with your midspan PoE.

I would probably do two things to isolate this given the info you provided:

1) Swap in a new CCR1009 and see if the FCS still show up.

2) Take the CCR1009 you removed and set it up as a test to a netmetal (on a port that was getting FCS) without the midspan PoE injection device you mentioned. If it stays clean then you know where the problem is.

Good luck and let us know :-)
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
BChenault
just joined
Posts: 1
Joined: Tue Jun 16, 2015 7:11 pm
Location: College Station, TX
Contact:

Re: fcs error on new CCR1009

Tue Mar 29, 2016 9:12 pm

Howdy folks,

I am wondering if anyone was able to figure out what was causing these issues? I too have been having the same problem with an fcs error on a couple CCRs both 1009 and 1016. One router has an AF24 from those other guys :roll: and one of them has a newly deployed Mimosa B5. While this does not seem to be causing any issues with throughput performance (getting over 1gb compined throughput when pushing the link) it does clog up the system log which is keeping me from being able to see when other events are happening since I get an FCS error alert every 30 sec on the dot. Any suggestions or help would be greatly appreciated! Thanks!
BC
WISP Manager
CEO Etc.
College Station, TX
MTCNA, MTCRE, MTCWE
 
lambert
Long time Member
Long time Member
Posts: 537
Joined: Fri Jul 23, 2010 1:09 am

Re: fcs error on new CCR1009

Wed Mar 30, 2016 8:24 am

FCS on the same second of every minute, whether that's every 30 seconds or every 60 seconds, may be an AirFiber thing. The AF developers may have identified the issue and be fixing it in the next firmware.

I see most FCS errors on port 8 of my CCR1009 routers. That is the PoE in port. I have a hypothesis that grounding issues at the tower site and / or just static build up on the cable may cause link issues, perhaps because the power connections to the port might make for a better ground loop path. That may just happen to be where I like to put my highest bandwidth radios.

On one router, a CCR1009-8G-1S-1S+, we were seeing a lot of FCS errors on ether8, which connects to an AF24. The port would also drop and re-negotiate at 100Mbps during the warmest part of the day. That hurts when you normally run 500+Mbps. There are 3 other AF devices plugged into this router in ether7, ether6, and ether5. We only had issues, which couldn't be resolved by fixing the cabling, with this one. We moved the AF24 from ether8 to ether4 and have not had any further issues, yet. The board temperature has been hitting the same highs as were associated with the renegotiation issues with ether8.

I see occasional FCS errors on other ports, but the most common occurrences are on ether8 on the CCR1009s (the originals and the PCs). Maybe I'm just lucky, I don't have a large sample set, only about 6 devices.

Who is online

Users browsing this forum: kez, Sob and 90 guests