Really now MT...

A CCR1036-8G-2S+, with about 180 days uptime.  Mysteriously just starts flapping all ports (Ethernet + SFP), and on reboot the device is just dead?  Reboots, shows Booting Router -> Loading Kernel, and then just reboots, over and over?

You call this rubbish enterprise ready?  This will be the last CCR I purchase, ever.  A D-Link ADSL router is more stable than your "flagship" enterprise equipment.  I guess at the end of the day, you get what you pay for yes.

Let me rather go and buy that Cisco ASR like I should have done right from the start.

I'm sorry to hear that. 
It is probably possible to resolve this situation. Can you give us more info about the circumstances? 

It probably is possible to re-install with netinstall yes... There's just a few problems with doing that... 

1) The router is some 9500km away from me,
2) Netinstall is Windows only, there's no Linux equivalents (such as bootp/tftp, like 99% of all other SANE network equipment vendors support),
3) Now I have to pay over EUR200 to ship the routers (both ways) to reinstall with Netinstall because I don't have (and never will have) Windows of X-Windows servers in a datacenter,
4) It still doesn't change the fact that the router just point blank, out of the blue, started flapping *all* ports,
5) It still doesn't change the fact that the router just point blank, refused to reboot successfully

Your equipment, is not *reliable*.

EDIT: From 3 CCR's purchased, 1 has been RMAed already, 2nd one now died.  2/3 failure rate - rather concerning to say the least.

Any device can die, even expensive Cisco/Juniper kit. We have over 50 CCR's in our network for the last three years, not one has died. The nice thing is that if one were to die, it will not cost an arm and leg to replace.

Yes,

Not within 180 days of purchasing it.  And not 2 out of 3 devices purchased either (of which one has already been RMA'ed too).

Frankly, whilst I'm sure it does (and can) happen, being a CCNA and CCNP and working with Cisco my entire life basically, I have *not once* had a Cisco device die on me personally.

Yes,

Not within 180 days of purchasing it.  And not 2 out of 3 devices purchased either (of which one has already been RMA'ed too).

Frankly, whilst I'm sure it does (and can) happen, being a CCNA and CCNP and working with Cisco my entire life basically, I have *not once* had a Cisco device die on me personally.

Sorry to hear that very strange. We actually just had a nexus fibre switch die last week, half the ports were dead. 

Only three devices doesn't sound like statistically significant...
Have you bought them more or less at the same time? Are all of them of the same model? Faulty batch, perhaps?

Only three devices doesn't sound like statistically significant...
Have you bought them more or less at the same time? Are all of them of the same model? Faulty batch, perhaps?

That's your opinion and you're entitled to it.
These forums are FULL of complaints about ethernet ports just starting to flap.  Across a LOT of different type of devices too (I just had a RB912 a few days ago, also just randomly flapping ethernet ports but that one thankfully still had it's software and rebooted successfully). CCR's are also riddled with the one problem after the next regarding their SFP(+) ports, again, just have a look on this very forum.
The last time I upgraded software on a CCR, I uploaded the packages, rebooted, and the router never came back again.  After putting a console on it, no software on the router at all.  So even a SOFTWARE UPDATE on a CCR, is buggy...  Cisco (for example), you upload two images - the new one and the existing old one.  If the new one fails to load, the device falls back and loads the old image, simple.  Hell, if the packages aren't uploaded to the MT devices correctly, a SIMPLE MD5 checksum is enough for MT to dump the package file and not attempt to update the router, but nooooo.  Then there's more complicated solutions too where NAND storage could be made to two partitions (one being for recovery software), again, MT can't be bothered.
MT is frankly more interested in dumping the one feature after the next onto ROS, instead of FIXING problems (and simple things, I'm not even talking about complicated things).  Then I'm not even going to START about "netinstall" being your ONLY recourse to recover a router either.

  Cisco (for example), you upload two images - the new one and the existing old one.  If the new one fails to load, the device falls back and loads the old image, simple.  Hell, if the packages aren't uploaded to the MT devices correctly, a SIMPLE MD5 checksum is enough for MT to dump the package file and not attempt to update the router, but nooooo.  Then there's more complicated solutions too where NAND storage could be made to two partitions (one being for recovery software), again, MT can't be bothered.

i spent my 20 years of experience building BF networks with Cisco and Juniper. My last 10 years were infected with RouterOS. there are good sides, there are bad sides. the cisco world you look for is long time over. their big boxes run IOS-XR, which is more like the "windows of Internet", literally takes 20+ minutes to boot, and upgrades and SMUs are kind of nightmares in many cases, you may need to re-boot the box twice, remove conflicting SMUs. even Huawei manages that better. JunOS is far better on the ISSU area, basically their MX960 (and all MX2+ series boxes) were the first one i experienced to do ISSU as expected.
coming back to IOS-XR: the "single binary on the flash is decompressed to ram and booted" times are long over. basically IOS-XR uses the same concept as routeros, there is an actual filesystem on the flash (ok, not on linear NAND but say an SSD) and there's the filesystem the router will boot from. and updates modify that. and there are actual routers (3-4 year old asr9k boxes with RSP4G) that are not physically capable of hosting 2 different images. at some time it was a viable option to do Z/X-modem and upload stuff even through RomMon, but today's images are in the 100x of megabytes and even over 1GB. and those are not enterprise but service provider devices. and they seem to have the weirdest issues - like not being able to do packet fragmentation on PPPoE sessions, or simply discarding to big packages with DF bit set w/o sending the ICMP unreachable message. and it takes more than 1 year to get this fixed.
regarding the ASR1k, we have some in our network, not too many, but enough. they are pretty solid boxes. but the prices associated to them are way in another league. you can buy a CCR1072 for merely 3k USD, whereas you'll be charged 10kUSD only for IOS-XE software. and then come the licenses (right to use, and scalability). you may not use all 1G/10G ports available on the box, and so on. i just finished 2 asr1002-FX configuration. just the bare metal with 4x10GE ports and 4xGE ports enabled on it, it's a 2-slot pizza box, crypto module for up to 10G throughput, up to 16k users PPPoE termination and IPSEC licenses. guess what: the IPSEC box w/o PPPoE goes for 136kUSD and the PPPoE w/o IPSec for 153k USD. even if we get say 50% discount, it will be way more than 10x the price of the CCR, whereas the features are more or less in par. all this without support or access to software updates - that will cost additional 4-8% of the list price.
i guess the lessons are the following: distant devices shall have 2 partitions, and console access. and you shall upgrade one at a time.
to my best knowledge, routerOS validates the packages found on the disk before updating the base OS with them.
however i totally agree with you on that windows-ish toolkit: i hate it as well. you can get the device up&asking for the "net install" image, but you don't have access to that directly. you could (theoretically) extract it from the netinstall binary, but i would call it rather tinkering than actual troubleshooting. probably mikrotik doesn't want to hand out scripts which reveal how license management is done. i don't know, and i don't care. i just don't like this windows-only stuff.
i'd rather have a "netinstall image" feature set for say any (or a specific) mikrotik device with USB slot (say map2n) and deploy it as last resort to those locations to act as a boot server. cause getting transparent L2 connectivity is not easy from 1000+ kms. and this is required for netinstall.
OTOH, it just fell into my mind, and probably it would be faster/easier than ship the box back and forth.
start netinstall on your PC which is connected to the internet. 
forward TFTP port on the outside of your router to that PC. set up CCR to use DHCP as boot protocol.
next to the dead CCR on any device (say a mikrotik with dhcp package) start dhcp server and set option 66 to the router's IP address and option 67 to the boot filename (/image?) and don't forget to send the gateway so the net-booted CCR will be able to access the internet. (if it is connected with it's net-bootable-if to the network).
it should be able to load the required stuff from your PC all over the internet and show up in netInstall.
or you could ask someone to plug any mikrotik device to the CCR's boot port and netinstall it over an eoip tunnel

Yes,

Not within 180 days of purchasing it.  And not 2 out of 3 devices purchased either (of which one has already been RMA'ed too).

Frankly, whilst I'm sure it does (and can) happen, being a CCNA and CCNP and working with Cisco my entire life basically, I have *not once* had a Cisco device die on me personall

It happens all the time with Cisco if you work on very large networks (10,000+ routers/switches/firewalls/etc) . In the last year I've had to deal with failures (hardware and software) on Cisco Nexus 2K, 3K, 5K, 7K , Cisco ASR 1002/1006 as well as Cisco 6500VSS. 4500XVSS and 2960/2960X. That's just the stuff I remember...i'm sure there are more.

It's also worth noting that we have clients running CCRs in multi-billion dollar enterprises that have had no issues and great uptime/stability.

it's impossible to account for all of the hardware and software issues you could come across with any given vendor, which is why as a network engineer and designer, it is better to assume that all hardware will fail at some point and design the right HA strategy 

It happens enough in the ASR series, Cisco has a howto guide on diagnosing crashes

http://www.cisco.com/c/en/us/support/do ... crash.html

You can argue all you want...

The facts remains... 
- For YEARS NOW Mikrotik suffers (and continues to suffer) from random, mysterious port flaps.
- The CCRs has NUMEROUS issues, which aren't getting fixed and/or better.

I couldn't care less on what works, and what doesn't work, and this vendor vs that vendor.  *MIKROTIK* (which is what hardware we are discussing here), is NOT enterprise ready.  I simply don't buy it that a "flagship product" just randomly "looses" it's software image, or that a "flagship product" just randomly starts flapping ports.  If Mikrotik expects me to pay thousands of $ for a CCR, with a shelf life of a few months, then sorry, that's not buying hardware - that's wasting money.

The majority of the Internet is built on Cisco, Lucent, Brocade, etc. for a reason....  Finished.  

Couldn't care less about the rest - I won't ever buy a CCR again, and I will advise everyone that asks me, strongly against buying a CCR too.

Ok. Bad personal experience. It can be. You just have lost the reason to keep your networks running on mikrotik. OK. So it can be. Never mind. Just use further whatever make of routers you still believe in. But this can always happen in the future even with Cisco. My habit is to keep a backup device on the site when it is critical and out of my direct access. In such cases I can easily instruct someone who can swap the devices fastly and there is no significant outage. Just hint for you...

The result of ( budget * flexibility * uptime * stability ) is all about.
If you can afford more expensive solutions then go for it. But even than you cannot be sure that devices are stable enough to not fail.

The majority of the Internet is built on .... for a reason

Maybe core net but "the core" is not "majority".

This starts to sound more and more the usual " give me all the Cisco,Juniper "benefits", but for 1/10 or 1/20 the price" topic that we have time after time here. i will not go into discussion, is it wrong or right, it is just the way it is. and from where i sit, every next year is an overall improvement.

On critical sites i usually deploy 3x devices ( we do buy them from different distributors ) - 1st online now, 2nd backup device, that is online and just need to trigger some scripts to initiate instead of 1st. and 3rd, that is not even attached to electricity. , but with all the configuration and cables attached.

worst case scenario - someone need to go there and unplug/plug power cords.

YES, it is 3 times more expensive to have 3 devices, but it is still several times cheaper that have one device from competition. So CCR can be "enterprise ready" simply your solution isn't

Thanks, again, three CCRs, 1 already failed, another failed now (after unplug/plug the power cable as you so nicely put it - it doesn't boot up again!! PS: Not the first time either - multiple devices)

The point is, these devices being 'flagship' devices, aren't stable at all.  If I can't even TRUST a Mikrotik (that does MULTIPLE gbps in traffic) to come up again after a simple reboot, HOW ON EARTH can you trust it for mission critical services?

I mean really now.  Seriously.   A SIMPLE REBOOT!!!!  I guess the standard of 'reliability' is just lower when it comes to Mikrotik then, because if THIS is acceptable to Mikrotik and to a community that uses Mikrotik, then sorry.  It's COMPLETELY unacceptable that a device won't even reboot (and that there's NO way to recover it either - except for a crappy netinstall that runs on Windows only).

Anways... /care.

Failure of multiple CCRs are not normal, you should contact your distributor to make sure these devices are not from bad batch and get replacement or RMA.

BTW Netinstall works fine on Linux, too.

Things to think about - You have RMA'd a router at this location before? Maybe it's the location? Do you have stats on it? Do you monitor the power or temp? 
Personally, every router I manage "calls home" every minute to report it's health. Also, why didn't you have a second partition?

That said... If you have had issues for years with Mikrotik, why do you keep using them?

You can argue all you want...

The facts remains... 
- For YEARS NOW Mikrotik suffers (and continues to suffer) from random, mysterious port flaps.
- The CCRs has NUMEROUS issues, which aren't getting fixed and/or better.

I couldn't care less on what works, and what doesn't work, and this vendor vs that vendor.  *MIKROTIK* (which is what hardware we are discussing here), is NOT enterprise ready.  I simply don't buy it that a "flagship product" just randomly "looses" it's software image, or that a "flagship product" just randomly starts flapping ports.  If Mikrotik expects me to pay thousands of $ for a CCR, with a shelf life of a few months, then sorry, that's not buying hardware - that's wasting money.

The majority of the Internet is built on Cisco, Lucent, Brocade, etc. for a reason....  Finished.  

Couldn't care less about the rest - I won't ever buy a CCR again, and I will advise everyone that asks me, strongly against buying a CCR too.

It's not an argument...what I presented are facts. Doesn't mean I never have an issue with CCRs, but it's not any more frequent than Cisco or any other vendor. 
If you're relying on the inherent stability of hardware / software on a singe platform to keep critical services online, then you need to rethink the design which includes power and physical layer concerns as well as the network hardware. 

It's COMPLETELY unacceptable ....

Please give us reasons or the only one reason why you have bought CCR if you trust other brands as core devices ?
Why have you choosen NNN dollars over 3 x, 4x, 5 x times more expensive devices ?
Why haven't you bought backup device if the location is so far from your office ? It seems to be not so clever move.

Remeber: I do not suggest that cheap could mean less quality.

Why haven't you bought backup device if the location is so far from your office ? It seems to be not so clever move.

I agree with this 100%. Every router that is more than 3 hours from me, has a second, identical router in a box already configured enough to get online.

@savage:

I can understand your frustration, but it sounds like you have moved away from being constructive, to ranting.

We too have seen numerous failures on the CCR1036's.   The commonalities are:

- Memory Errors
- Power Supply Failure/Leaking Capacitors

It is my suspicion that the memory errors may be related to leaking capacitors and poor ripple control in the PSU. 

It would be interesting to see the results of you running a memory test on the affected router.

@savage:I can understand your frustration, but it sounds like you have moved away from being constructive, to ranting.

You word that as is if he JUST started to rant mate When I have something that breaks, the 1st thing I do is go to the community forums and list of bunch of stuff that is of no use to product developer whatsoever. It always works. I don't know why companies even have support contacts.

i think you are looking gin the wrong direction

3 devices failed on your premises, can be a signal of a problem at the site itself, electric problem etc

if a device is critical you must have a backup or redundancy design period.

good luck with your issue

Are you just complaining or did you actually report this to MikroTik support? Forum is for users but if you want to get something fixed, then you should go through support.
But yes, could not agree more - use backups. Also you have to look to the problem from different sides - router is not only device in equation. Power source, cables, etc.

you can't blame vendor for and start calling their ger "unlreliable" just because one of units malfunctioned.
there is no "absolutely reliable" hardware of any kind. and if you looking for aerospace/military-reliability-grade networking gear - rugged "by design" and Heavily tested after, but it would raise price-tage around 100x sometimes.
excuse of absence x-windows to not using winbox over WINE - soudns funny. never saw network administrators w/o consistent, but complete workstations, running pure-cli/ssh management routines Only and for good reasons(there mission-critical stuff that depends on in each installation, even if you can and would do 90% things thru clie/console).
so for me its sounds not only (reasonable)emotional but purposed over-exaggeration.
9500km, ha. and you not emplaced "backup router" to be ready for (eventual, regardless brand or model)failure ? what nonsense!!
(9.5k km away from tech personell and have no tech robust/resilent/relobale stategy/plan of using/deploying gear/infrastructure ? no spare units, parts... that was not "enterprise". that was a Joke actually and very bad one)
bottom line: CCR's is FAR more "enterprise-ready" than many of competitors. one of reasons why we're diched almost all ER8 from your installations for example.
CISCO/Juniper/Lucent/Ericcson/HP - had quite different price-tag and proirties. so hardly they are comptetitor to.
as for causes(other than production faults) - its may be poor electric grid performance(level,stability and both kinds of noise in it), termal issue and most complex and underrated thing - EMI impact on gear.
so if you run it under say TV tower or in industrial compound (with MW's-worth electric consumption and emission from production machney) or in unusual OTH/locations in Earth, known for such problems - you would b ready to counter it.
in my experience - its more frequently about "unreliable reseller problem", where some of them - persistently and on purpose - try repeately re-sell units that know to be faulty w/o repair or with basic repair by them(no RMA whatsoever) than something more mysterios, then power issues then come others. but production - always can and would make defective units sometimes.

My biggest grind with RouterOS and CCR routers is the routing engine. 

Why an earth are we still running BGP on one core of the router? our core router has 150 BGP Peers and its just screwed after a month of uptime mainly because the router cant add any routes into the routing table anymore. Which in turn screws OSPF up. 

Seriously we have been promised loads of feature upgrades to RouterOS. Whats the point of having a CCR1072 if it cant be used to its full potential?? 

Ive never seen the port flap issues on the CCR's but then we never get more than 30-40 days uptime out the core routers...  

Its not really an enterprise reliable worthy product. i have to agree... 

tile chips originally made to embrace scalability of "Zero overhead linux" of tilera itself, where Majority of stuff - runs in user-space(eg alike "microkernel-based" OS'es, just made off mainstream Linux, just like SEL4, but with emphasis on userspace, rather than medium rings of).
so my guess until ROS7 CCR's wouldn't scale really as many hoped. not yet.
but they far superior anyway to closest similar things.
bulk/uniform Linux - s... on multicore setups, let alone many-core chips. so both DragonflyBSD and Tilera distribution or any kind of microkernel Linuxes(there was 4x actively developed. aside proprietary forks of them,developed inhouse, like qualcomm-made and alikes)

Cisco? The DLink switch have a two image on board! Its a big problem install second flash chip of RB(redudance power not problem at all)? And create alert if user configure log all package to the disk in 300 Mb/s networking?


Have a good day!

I've had a similar issue you had in that the CCR goes into a boot loop. You may want to shut it down for a while and than turn it back on and look at system -> health and look at the voltage. If the voltage keeps dropping this is a PSU issue and a good chance to do some modding for better thermals. I was one of those that purchased the CCR1036-8G-2S+ when it came out and it had power issues like the USB not being able to supply a stable amount of power (raspberry pi on usb froze but on other routers USB had no issues).

SInce replacing the PSU with a better one i have not had any power issues.  If you have to replace the PSU it supports a large range of voltages but the suggested is 24V PSU with 4A output. I use one with 5A output and by having the PSU outside i can go with silent cooling and still get good thermals. I did a full maintainance of the CCR and took pictures so i could post it later on the modifications i did to get a CCR1036 thats even quieter than a ubiquiti edgerouter pro.

All about risk management.

Four letters... VRRP.

Sent from my SM-G920I using Tapatalk

and damage control(if you failed 'risk management')skills aswell, yep.
generally (not only in IT) "management" eg Planning thing - do 80% of time, and only Rest(time, manpower/money,etc) - will come into actual Actions.

All about risk management.

Four letters... VRRP.

Sent from my SM-G920I using Tapatalk

VRRP is great for edge routers, but not for core routers.  MT really needs to implement a different type of clustering that supports state synchronization for seamless failover without dropping connections.  Then, it'll be a good core router.  Not suitable for mission-critical applications as is.

All about risk management.

Four letters... VRRP.

Sent from my SM-G920I using Tapatalk

VRRP is great for edge routers, but not for core routers.  MT really needs to implement a different type of clustering that supports state synchronization for seamless failover without dropping connections.  Then, it'll be a good core router.  Not suitable for mission-critical applications as is.

I don't think you know what a core router does. Core routers run routing protocols and mpls. Redundancy is done via these.

Now I agree a firewall should have such a mechanism, but mikrotik are routers.

No, redundancy is done by the access routers (which are connected to the core with dynamic routing protocols)

I don't think you know what a core router does. Core routers run routing protocols and mpls. Redundancy is done via these. 

Now I agree a firewall should have such a mechanism, but mikrotik are routers.

So, I think you and I are thinking of two different types of networks.  You're thinking Enterprise WAN networks, where you're right, MPLS and routing protocols would do everything you needed.  But in a corporate LAN environment, you're probably not running MPLS on your internal LAN.  Possibly at your edge where you may have a connection to a leased private WAN to optimize traffic flow for lower bandwidth availability to your satellite offices, but certainly not at the core.

All about risk management.

Four letters... VRRP.

Sent from my SM-G920I using Tapatalk

VRRP is great for edge routers, but not for core routers.  MT really needs to implement a different type of clustering that supports state synchronization for seamless failover without dropping connections.  Then, it'll be a good core router.  Not suitable for mission-critical applications as is.

While I would love to see this, it's one of the hardest tricks to pull off in netowrk hardware/software development. Cisco has been working on this for more than a decade in the form of VSS and it still doesn't always fail cleanly without dropping connections. Cisco Nexus on the other hand uses HSRP, VRRP and GLBP as the primary redundancy mechanism for a Data Center core and it is proabbaly at the heart of 1/3 of all Enterprise and Data Center networks.