I'm sorry for the length of this post but details are important in this question. What I am about to describe is very strange but I have checked it repeatedly:

I have a link: RB493G===RB433AH<-----/------>RB433AH===RB493G. The 493s are router PPPoE concentrators and the 433s are mounted with XR-5 radios from Ubiquiti.

Everything running beautifully for more than a year with upgrades through 4.xx.

A month ago I pushed out RouterOS 5.5 to all of them. The link went down. Had to drive to the sites and power cycle the units to get the links to come back up.

A week later the link drops and I drive to the sites and *all four* RBs had done what appeared to be a shutdown at the *same* time. Now then I log to a syslog server. They were not shutdown because I get a different signature in my logs when I shut one down. They just stopped talking on all their ports and the lights were OFF. Power cycle and they come back up.

Of course that is weird but it gets even more interesting... So for the last week they have been turning themselves off (well they have the same physical appearance as one that has been shutdown). Change all the passwords because someone is messing with me? Not. Changed the passwords. It is not a compromised password because it continues.

Couple days ago I downloaded Rev 4.17 to all four. All goes well for a day and then the link is down again and the units have the appearance of being shutdown. Today I go out there and reset the configuration of all of them and reconfigure with a backup.

And I discover that they are shutting themselves down, not rebooting, shutting themselves down. I thought that clearing the config and reloading it would probably fix whatever happened during the upgrade process. It didn't. I have watched one of them shut it self down repeatedly when it was not connected to the network. I have watched them do a shutdown after a reset configuration. I have watched them shutdown when I told them to reboot from Winbox and I have watched them simply shutdown for no apparent reason at all. The only way I could duplicate the problem would be to download the RouterOS sources and fix them to cause this problem and then install them or perhaps to discover a flaw in the IP stack that would let me craft a packet that would cause a shutdown. The Linux IP stack is pretty mature and I'd be very surprised to hear that someone has discovered such a flaw but even that would not explain the one that shuts itself down even when off the network.

Clearly we are going to replace them but I need to know if anyone else has seen anything like this happening with these particular units? Is there possibly some magic combination of firmware upgrades that might lead to anything like this?

Thanks in advance, and I wish I could get to another of the training sessions.

Mike Erskine
(You remember me I'm CALEA guy that stuck antennas in a foam cup when you were testing 802.11n cards )

What if this is not a software issue. What power supplies are you using? How many volts? Do a visual inspection of the devices, they all look fine, no damaged parts?

What is this is not a software issue. What power supplies are you using? How many volts? Do a visual inspection of the devices, they all look fine, no damaged parts?

Thanks Normis, I will check the power supplies and get back on this.

What if this is not a software issue. What power supplies are you using? How many volts? Do a visual inspection of the devices, they all look fine, no damaged parts?

Ok..

The power supplies are all 18v, 1.1 amp, 20 watt. The devices look fine. I have removed one from it's enclosure and it looks just fine
To be honest normis, I'm not sure what the hell I am seeing at all. I think this will take a bit more study on my part and I will keep this thread alive if anymore information becomes available.

Meanwhile, unless you get additional reports that RB493Gs are shutting themselves down "auto-magically", you should probably ignore this.

For example one of the units that was shutting itself down the other evening sat on my desk for a couple of hours today powered, happily pinging my desktop and never shut itself down again. The truly weird thing is I saw it do a shutdown when it *WAS NOT* connected to the network the other evening.

Obviously there is something here I don't GROK. I'll get back to you.

Hope all is well with you folks.

Mike

do you have any cards plugged into the router, and what configuration you have on it?

check if wifi modules are not damage, on one of devices try to reset configuration and see if router still reboots at any conditions.

do you have any cards plugged into the router, and what configuration you have on it?

check if wifi modules are not damage, on one of devices try to reset configuration and see if router still reboots at any conditions.

A little more on this then Janisk;

I have an RB450 (ROS 5.5) on my desk that I use for NAT. I just set a watchdog in it and gave it an address to watch that it could not reach. I set the start ping time to 00:01:00 so it will reboot more often. It rebooted itself several times and then shut itself off. I power cycled it and it shut itself off again within about five minutes. Power cycled it and since then it has rebooted itself a dozen times and not shut itself down.

Right beside that I have an RB433G that was upgraded to ROS 5.5 then downgraded to ROS 4.17. I reset the configuration on it and then set the watchdog timer to an address it could not reach. Within a few minutes it shut itself down. Since then it has been happily rebooting for about ten minutes.

Yesterday we were working on one of our core towers and had to take it down for about 20 minutes. While it was down three other routerboards at three different locations shut themselves down. They are all on a 5 minute watchdog. There are six towers that depend upon the tower we were working on. All of them have Mikrotik PPPoE servers. Like I said, three shut themselves down and three behaved normally.

We believe the issue is tied to the watch dog timer or the reboot code. I think if you want to try to duplicate it, you should set up a few units running firmware which has been upgraded from 4.xx to 5.5. Then with a scratch config, enable the watchdog on an address they can not get to. Set your timer to 00:01:00 for the start of ping time. Then just walk away and come back in half an hour and you might see it happen. What is so odd is that I've watched this one reboot itself for half an hour now without another problem. Very odd.

An unrelated issue (I think) is that ethernet port negotiation in the more recent versions of firmware does not seem to be as robust as it was in versions prior to 4.11. We are trying to address that issue with 24 volt power supplies.

I just disabled the watchdog on my desk RB and left the IP address in the setup. I did not power cycle the unit. It just rebooted itself. That should clear the watchdog but it does not. You have to uncheck the box AND remove the IP address in the watchdog window to get the unit to stop rebooting on the timer. What is the purpose of the checkbox in the watchdog window in winbox?

I believe there is something wrong with the watchdog code or the reboot code.

Is there a source code package of the ROS distribution of Linux? I'd like to look at that code if that is possible. It might be a problem in the code that is called when a reboot is done rather than the watchdog timer itself as there are a rare few times I have sworn I did a reboot and had to drive out to a site to power up a unit.

Thanks
Mike

thanks for detailed information about the problem. I will check on this. About source code - please write to support with a request. you will be given details how to acquire it.

first part of answer:
tela b
as described in manual:
there are 2 things in watchdog configuration, one is internal check, that is enabled/disabled by setting watchdog timer to yes/no. it is not connected with address check in no way. It checks RouterOS responsiveness all the time. With time interval and check-address you set up external watchdog, that pings some address and reboots router if it fails. So if you want to disable check-address you have to nothing

first part of answer:
tela b
as described in manual:
there are 2 things in watchdog configuration, one is internal check, that is enabled/disabled by setting watchdog timer to yes/no. it is not connected with address check in no way. It checks RouterOS responsiveness all the time. With time interval and check-address you set up external watchdog, that pings some address and reboots router if it fails. So if you want to disable check-address you have to nothing

Code: Select all

/system watchdog set watch-address=""

Thanks I see I need to read more carefully.

small mistake:

one of these would be ones to unset the set value of software watchdog for hardware watchdog just set watchdog-timer=no

I have been gone for a couple of weeks. Have now returned. We continued to experience this problem on several routers while I was gone.

Today I am turning off all watchdog timers on the routers which have ever shut themselves down. I will get back with the results of that change in a few days.

I have configured the routers that have been exhibiting this problem thus: Thanks
Mike

...
Today I am turning off all watchdog timers on the routers which have ever shut themselves down. I will get back with the results of that change in a few days.
...

Well, I turned off watchdog on those devices which were shutting them selves down, and yes, they were powering themselves down.

Not a single one has done that since. Whatever is causing the problem, it is the watchdog timers that are causing the shutdowns.

Hope that helps you folks discover the root of the problem.

Bottom line for me is to turn off the watchdogs on the rest of the network. Thanks for the help guys.

Mike