Community discussions

MikroTik App
 
marisspringis
just joined
Topic Author
Posts: 15
Joined: Wed Dec 12, 2018 2:17 pm

RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jun 03, 2019 1:18 pm

Hi everyone,
since the last week of may, strange thing has happened to Router Boards which i manage.
issue - Router simply bricks, what i mean by that is - you cannot connect to router in any way (stays on logging in and nothing more happens), APs that are connected lose all config from dude, SNMP stops working and so on.
in the same time, from computers which are connected to switch, internet is working. also i can ping that router
this can be resolved only by hard reset (take power cable off/on)
issue has happened only on RB4011 and RB1100AHx4 Dude edition
Router Os - 6.44.3
previously this has never happened.
these RB are in different countries.

so far this has happened only once but with every RB4011 we have and one RB1100 AHx4
does anyone else has seen this?
 
User avatar
ccardenas
Trainer
Trainer
Posts: 5
Joined: Thu May 15, 2014 9:04 pm
Location: Spain

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jun 03, 2019 1:48 pm

Hello! Do you have bridges in your network implementation? How many hosts are passing traffic among these "random bricking" devices? Could you provide us more info? So we can understand your problem and we'll be able to help you better.

We have a similar issue, and we are suspecting about the bridge host table size and (possible) memory exhaustion problem. It only happens in new arm devices (RB4011 and RB1100x4).

Symptoms are loss of connectivity, manageability and it's impossible to access the device in any way, but it keeps working as a switch. After a reboot (unplug power cable/ replug) all begin to work fine and we can see a lot log lines like: snmp, warning timeout while waiting for program XX (where xx is a variable two digit number)

Image

Regards
 
marisspringis
just joined
Topic Author
Posts: 15
Joined: Wed Dec 12, 2018 2:17 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jun 03, 2019 2:43 pm

Hi ccardenas,
yes, we have 3 bridges in RB4011 and in RB1100AHx4.
hosts in RB4011 most of the time are - 30-40
in RB1100 not more than 5 connected directly, it is used as dude server for monitoring.

one more thing to point that problem is in these RB4011 is that we have a lot of RB2011 with the same config, and they work perfectly, without any problems. also ROS version in all ar is the 6.44.3

symptoms are identical to yours.
 
marcin21
Member Candidate
Member Candidate
Posts: 202
Joined: Tue May 04, 2010 4:50 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jun 03, 2019 4:32 pm

Itr seems that cpu is getting exhausted over time.
this particular ARM based WAP60g is 149d up.
You do not have the required permissions to view the files attached to this post.
 
User avatar
ccardenas
Trainer
Trainer
Posts: 5
Joined: Thu May 15, 2014 9:04 pm
Location: Spain

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jun 03, 2019 5:45 pm

one more thing to point that problem is in these RB4011 is that we have a lot of RB2011 with the same config, and they work perfectly, without any problems. also ROS version in all ar is the 6.44.3
Hello!! Yes, totally true. RB2011 and RB1100Hx2 in the same place, in the same network, in the same situation and nothing happens, they never block. We've opened a support ticket to Mikrotik and they told us to plug a serial cable and wait the device to block, then try to access it via console and make a supout, but we have a couple of them with cables attached and now they never block! :cry:

Other devices within the network keep blocking randomly. In the meanwhile we have scheduled a reboot (lame solution, but it saves the day) at nights a couple of times a week until we find the real problem, but it seems that some process inside the routerboard hangs or collapses the memory, making the another processes fall in cascade and block the access to the device.

If someone is experiencing the same problem, please share with us, maybe we can find a hint in the meanwhile, until I can get a good supout file and send to Mikrotik support.

Regards!!
Last edited by ccardenas on Mon Jun 03, 2019 7:27 pm, edited 1 time in total.
 
Dude2048
Member Candidate
Member Candidate
Posts: 113
Joined: Thu Sep 01, 2016 4:04 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jun 03, 2019 6:38 pm

I have a RB1100ahx4 Dude edition which has the same behavior. What happens is that the memory hogs and the device will become inaccessible. I have a script that reboots the device when 70% is used. During the times that it is inaccessible I tried to make a supout, via console, but that didn't work.
 
User avatar
kehrlein
just joined
Posts: 9
Joined: Tue Jul 09, 2019 1:35 am
Location: Munich, Germany
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Dec 05, 2019 12:24 pm

Same issue here with 3 different RB1100AHx4. The situations happens with < 7 days uptime.
During the incident, the devices don't accept new SSH, Winbox, SNMP or PPTP connections to the router itself.
Also logins via CLI aren't possible (serial connection is possible; login doesn't work). We tried to have a running serial connection to the affected devices. If the issue occurs, we are able to type commands into the cli, but generating the supout or doing real actions (e.g. initiate a reboot) doesn't work.
Other traffic goes through the router smoothly. The issue can temporary be fixed by powering off an on the device. Temporary solution is a scheduled reboot.
 
meshnet
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Tue Jun 01, 2004 6:57 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Dec 05, 2019 9:05 pm

Starting to see this issue also on 4011s. 6.44.1 on the last one it happened to.
Power cycle restores all functionality..
SNMP polling becomes very intermittent right before this happens, pointing to the CPU issues..
Only bridge configured on the devices is an empty bridge for loopback.
 
Dude2048
Member Candidate
Member Candidate
Posts: 113
Joined: Thu Sep 01, 2016 4:04 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Dec 06, 2019 12:01 pm

Same issue here with 3 different RB1100AHx4. The situations happens with < 7 days uptime.
During the incident, the devices don't accept new SSH, Winbox, SNMP or PPTP connections to the router itself.
Also logins via CLI aren't possible (serial connection is possible; login doesn't work). We tried to have a running serial connection to the affected devices. If the issue occurs, we are able to type commands into the cli, but generating the supout or doing real actions (e.g. initiate a reboot) doesn't work.
Other traffic goes through the router smoothly. The issue can temporary be fixed by powering off an on the device. Temporary solution is a scheduled reboot.


What version are you using.
 
quackyo
Member Candidate
Member Candidate
Posts: 125
Joined: Mon Nov 16, 2015 10:14 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Dec 06, 2019 1:36 pm

Does everybody experiencing this run Dude on the device?
I have had the exact same issue on my RB1100AHx4 shortly after setting up Dude. Happened 3 or 4 times over a month or two before i connected the dots. Disabled Dude server and no hickup since (6-8 months ago...).
 
User avatar
kehrlein
just joined
Posts: 9
Joined: Tue Jul 09, 2019 1:35 am
Location: Munich, Germany
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Dec 06, 2019 1:38 pm

Same issue here with 3 different RB1100AHx4. The situations happens with < 7 days uptime.
During the incident, the devices don't accept new SSH, Winbox, SNMP or PPTP connections to the router itself.
Also logins via CLI aren't possible (serial connection is possible; login doesn't work). We tried to have a running serial connection to the affected devices. If the issue occurs, we are able to type commands into the cli, but generating the supout or doing real actions (e.g. initiate a reboot) doesn't work.
Other traffic goes through the router smoothly. The issue can temporary be fixed by powering off an on the device. Temporary solution is a scheduled reboot.
What version are you using.

Latest bad experience was with 6.45.7.
Time for experience with 6.46 was too short.

After some mails with the MikroTik support during the last month, I finally got this answer:
Unfortunately, this problem seems to be caused by a hardware issue.
Please contact the seller and return the router for warranty repairs, if the router is still covered by it. You can refer to this ticket number - SUP-3012.
So now I am talking to the reseller about refund.
 
User avatar
kehrlein
just joined
Posts: 9
Joined: Tue Jul 09, 2019 1:35 am
Location: Munich, Germany
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Dec 06, 2019 1:40 pm

Does everybody experiencing this run Dude on the device?
I have had the exact same issue on my RB1100AHx4 shortly after setting up Dude. Happened 3 or 4 times over a month or two before i connected the dots. Disabled Dude server and no hickup since (6-8 months ago...).
I had the issues on several devices without running the Dude.
 
Bolle
just joined
Posts: 5
Joined: Fri Aug 23, 2019 9:42 pm
Location: Germany

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Dec 06, 2019 2:49 pm

Hi !

Same here with RB4011iGS+5HacQ2HnD und RB1100Dx4.

Both devices ´freezes´ several times a week.
The RB1100 sometimes two or three times a day.

I tryied with ROS 6.45.7 and 6.46beta59.
 
marcin21
Member Candidate
Member Candidate
Posts: 202
Joined: Tue May 04, 2010 4:50 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Dec 16, 2019 1:38 pm

Anyone has found a solution to this problem ?
Maybe SNMP disable?

I've got problematic 4011 and since few days it started to die.
 
aoakeley
newbie
Posts: 49
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Dec 21, 2019 3:18 pm

I have two 1100AHx4 with this issue, and an open case with Mikrotik (Ticket#2019100722004559). I have just been able to connect in with a serial cable and generate a supout file.

I am not running Dude on them.
SNMP is enabled though.
 
FreeVoip
just joined
Posts: 4
Joined: Tue Oct 15, 2019 5:33 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Dec 21, 2019 4:49 pm

Hello!.

Same problem here!!!!! viewtopic.php?f=2&t=154859&sid=9cf5b071 ... 23dad9a667

I don't know what else to do. It already happened to me 4 times in the week.
There is no high CPU consumption, no RAM consumption or anything strange.

It is a very big problem for me.
 
FreeVoip
just joined
Posts: 4
Joined: Tue Oct 15, 2019 5:33 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Dec 21, 2019 4:57 pm

It is incredible that since June this was reported and nobody from Mikrotik did anything.

In my case they suggested connecting via serial cable but I am 800 km from my node. It is impossible for me to do that. It is easier to restart it but it is not very serious for an internet provider to cut the service every day at the same time. Needless to say, if it "hangs" at 10 am I have to wait until the next day to access again.
 
marcin21
Member Candidate
Member Candidate
Posts: 202
Joined: Tue May 04, 2010 4:50 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Dec 21, 2019 8:31 pm

since last problem, I 've got winbox session opened on screen a this problemtic 4011 is up and running for 5d+
I wonder if it has something in common with
viewtopic.php?f=3&t=142298
 
aoakeley
newbie
Posts: 49
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Dec 23, 2019 3:34 am

It is incredible that since June this was reported and nobody from Mikrotik did anything.
I suspect that they did not have enough information to replicate the issue and develop a fix. Like you many of my devices were a long way away, and they asked to have a serial cable plugged in. This was difficult, but I now have one connected and have sent a supout to support.

With a few more people reporting the issue now, hopefully they will be able to find a fix.
 
marcin21
Member Candidate
Member Candidate
Posts: 202
Joined: Tue May 04, 2010 4:50 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Dec 23, 2019 11:59 pm

I wouldn't be so optimistic regarding fast tracking the problem down.
it seems that very similar issue with rb4011
viewtopic.php?f=3&t=142298
was reported in dec'18, and in april'19 first response from Mikrotik staff that they recognise problem and working on solution,
and it seems that they haven't solved it till now.
maybe there is some serious flaw in hardware, as we see it above
Unfortunately, this problem seems to be caused by a hardware issue.
Please contact the seller and return the router for warranty repairs, if the router is still covered by it. You can refer to this ticket number - SUP-3012.
either the way i've got my winbox opened on my problematic 4011 and 7d12h uptime. maybe winbox opened does the trick? I doubt it but whatever, when it works.
Merry Xmas :)
 
luciansilviu
just joined
Posts: 1
Joined: Fri Dec 27, 2019 4:10 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Dec 27, 2019 4:14 pm

Hi!

Just a quick "me too". I have had this issue with two boards RB4011iGS+ running on 6.45.4.
 
reapster
just joined
Posts: 2
Joined: Mon Jun 10, 2013 8:01 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Dec 28, 2019 8:03 pm

Hey there,

We also see this on our RB1100AHx4 (Currently 6.45.7, and inaccessible), every few weeks it exhibits this behaviour. Cannot access via Winbox/Telnet/SSH, but routing remains fine (thankfully!). This unit does almost no traffic (< 10Mbps), and has no special configuration, its not even doing SNMP. About 10 firewall rules, usual masquerade NAT and I think a DHCP server, very basic setup.

All functionality returns after a power cycle, sadly not a feasible long term solution though. We have quite a few Mikrotik units deployed, but will have to start moving away from Mikrotik if there's no real solution to this.
 
aoakeley
newbie
Posts: 49
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Jan 02, 2020 2:57 pm

We have quite a few MikroTik units deployed, but will have to start moving away from Mikrotik if there's no real solution to this.
We have hundreds (thousands??) of MikroTik units in the field, and only 2 exhibiting this issue. So don't throw the baby out with the bath water.

After obtaining supout.rif files by console cable, MikroTik have organised warranty replacement for the two 1100 units exhibiting this issue. Can't ask for more than that.
We have not seen this issue with any 4011 units.

Andy
 
allevot
just joined
Posts: 7
Joined: Fri Oct 19, 2018 2:44 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Jan 03, 2020 3:18 pm

Hi,

We have several RB4011 and exactly the same happens to us. Randomly some MikroTik is no longer accessible by management, and in Dude appears the services "memory, disk and cpu" down, but the ping goes well.
The services work correctly, but there is no way to log in to the equipment and the only way to be able to have management is to restart it.

For more info we have VRRP, Bridge, Tunnel GRE, ACLs, BGP, Queues, SNMP and traffic is 20-50Mbps.

Regards,
 
aoakeley
newbie
Posts: 49
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Jan 04, 2020 2:56 am

We have several RB4011 and exactly the same happens to us. Randomly some MikroTik is no longer accessible by management, and in Dude appears the services "memory, disk and cpu" down,

Regards,
- Use a console cable to get a supout.rif file while the router is in the non responsive state (might as well do this when you lodge the ticket, as they will ask you to do this anyway). To generate the suport.rif you will probably need to have the console cable connected and be logged on before the router craps out.
- Log a ticket with support@mikrotik.com.au
- if there are any autosupout.rif in the file list send these also

Good luck.
 
allevot
just joined
Posts: 7
Joined: Fri Oct 19, 2018 2:44 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Jan 09, 2020 10:43 am

Hi,

We are unable to make a supout.rif because we are unable to log in into the device in any way.
Someone from MikroTik support can help us? It's a big problem for our company.

Thanks,
 
r00t
Member
Member
Posts: 313
Joined: Tue Nov 28, 2017 2:14 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Jan 09, 2020 6:51 pm

If router is still running, you could try to generate supout by watchdog script that:
1) checks if router can ping some IP that should normally be reachable at all times (gateway, some other LAN device)
2) if not (interfaces are frozen, down, IP not accessible) generate supout
3) reboot the router
Run it every minute and hope it works when the issue happens.
But if router is not even accessible by console cable, it's likely completely frozen (CPU/ROS) and above script would not run anyway...

Some more debugging options for ROS would be great, like redirecting kernel syslog to console port at all times.
 
User avatar
inteq
Member Candidate
Member Candidate
Posts: 183
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Jan 10, 2020 4:31 pm

Only one RB4011 (without WiFi) out of 12 crashed once with some process stuck.
None of RB1100AHx4 or RB1100AHx4 Dude Edition out of 19 crashed so far.
Also, bricking can happen to Mikrotiks, but it did not happen to me (yet) and if a power reset fixes it, it did not happen to you (yet).
 
User avatar
jdejansb
Frequent Visitor
Frequent Visitor
Posts: 69
Joined: Thu Jul 13, 2006 1:35 pm
Location: Srbija
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Jan 11, 2020 12:07 am

... and I was just about buying a 4011 . . . should I do that or not :? .. it's for a main network gateway, PPPoE, dude, . . .
 
aoakeley
newbie
Posts: 49
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jan 13, 2020 5:18 am

... and I was just about buying a 4011 . . . should I do that or not :? .. it's for a main network gateway, PPPoE, dude, . . .
I have plenty running that have no issue. I think the devices with this issue are very few.
 
tquibell
just joined
Posts: 2
Joined: Fri Jan 17, 2020 6:17 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Jan 18, 2020 7:41 am

Same Problem as OP I think.

Not using any fancy routing. Only doing basic Firewall/NAT/Mangle and DHCP Server and Client. Some static routes, etc. Nothing else.

RouterOS Long Term 6.44.3 would cause RB4011 to randomly lock up. First few times a reboot did the trick, uptimes averaged about 60 days between lockups. The router would remain visible in Winbox but otherwise I could not communicate with it in any fashion. IP addresses would become 0.0.0.0 etc. Terminal unresponsive. Have not confirmed whether the built in serial console was also unresponsive. Finally on the last go-round the board bricked.

On boot up was able to see that the board was hanging on loading kernel. Used Netinstall to wipe the NAND and reflash the Firmware and RouterOS. Voila. Moved to Stable 6.46.2 to see if this thing will survive with a different version.

Will report back after some light stress testing and a few weeks of uptime (hopefully alot more).
 
Mikhalich
just joined
Posts: 6
Joined: Mon Feb 05, 2018 10:16 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jan 20, 2020 7:05 pm

Have same problem on RB4011 without WiFi.
System stops forward client packets but I can connect via winbox to WAN-port.
System runs script make supout.rif every 2-3 minutes but can not finish it.
Device did not execute any commands via console but respond on command via winbox.
Tools-Profile shows 100% load 1 of 4 core and 77% unclassified load.
Uptime was about 60 days. ROS 6.46.1

After reboot via winbox system runs normally.
Last edited by Mikhalich on Mon Jan 20, 2020 7:12 pm, edited 1 time in total.
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 266
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Jan 25, 2020 7:12 pm

Hello
I have a 4011 that had the same behaviour described. I replaced it with another 4011, same problem!
Ethernet ports simply stop responding, then in the logs there were "waiting for progam xx" , sometimes 20, sometimes other number as described.
I tried with watchdog to one ip to reboot automatically, with no help.

I solved completely planning a system reboot at 05:30:00 every day.
Then with the latest 6.44.6, every few day.
zero issues since then :)
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
omidkosari
Trainer
Trainer
Posts: 634
Joined: Fri Sep 01, 2006 4:18 pm
Location: Iran , Karaj
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Feb 01, 2020 3:05 pm

Hello,

I want to share my experience with same issue. This bug exists for years with several different device types like RBxx ,CCR1009 , CCR1036 , x86 Server , etc. I had several tickets with mikrotik support but unfortunately they were not able to find the problem . The only common feature in all of those routers was PPPoE concentrator (pppoe server).

The only current workaround is to schedule reboot the router everyday. Even with rebooting everyday the problem may appear randomly each few weeks.
MTCNA , MTCRE, MTCWE, Mikrotik Certified Trainer
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 266
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Feb 01, 2020 6:20 pm

In my setup there were not any pppoe services, just plain routing with simple queues.
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
omidkosari
Trainer
Trainer
Posts: 634
Joined: Fri Sep 01, 2006 4:18 pm
Location: Iran , Karaj
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Feb 02, 2020 8:56 am

We have to find a common point. After reading this topic I am suspicious about two new points.

1-Do you have this issue on a device which is not monitored by Dude ?
Please also take a look at following topic viewtopic.php?f=2&t=114365

2-Do you have this issue on a device which is not using Queues?
MTCNA , MTCRE, MTCWE, Mikrotik Certified Trainer
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 266
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Feb 02, 2020 9:31 am

Hello. All my devices are monitored by dude. And have queues. I have one without queues and it works flawlessy.
Last edited by Maggiore81 on Sun Feb 02, 2020 5:07 pm, edited 1 time in total.
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
omidkosari
Trainer
Trainer
Posts: 634
Joined: Fri Sep 01, 2006 4:18 pm
Location: Iran , Karaj
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Feb 02, 2020 9:58 am

In my devices like yours, all of the buggy devices have queues. Almost each pppoe user which connects to router will have dynamic queue from radius server. I think the issue happens when a specific dynamic queue will change (add or remove) on the router. The main problem to solving this issue is that , the issue is not reproducible by mikrotik staff.

Please describe your router configs and your thoughts about the situation which the issue may happen.
MTCNA , MTCRE, MTCWE, Mikrotik Certified Trainer
 
Bolle
just joined
Posts: 5
Joined: Fri Aug 23, 2019 9:42 pm
Location: Germany

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Feb 02, 2020 10:44 am

We have to find a common point. After reading this topic I am suspicious about two new points.

1-Do you have this issue on a device which is not monitored by Dude ?
Please also take a look at following topic viewtopic.php?f=2&t=114365

2-Do you have this issue on a device which is not using Queues?

I have a RB 4011 monitored by Dude, and a 1100DX4 wich is the Dude Server.
On both devices are only the default Interface Queues.
ROS 6.46.2
And both devices freezes
 
marcin21
Member Candidate
Member Candidate
Posts: 202
Joined: Tue May 04, 2010 4:50 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Feb 02, 2020 4:15 pm

my 4011 has no pppoe-server, nor dynamic queues and yet still stalls.
although it does lot of nat, close to 1gbps, also BGP, and it is monitored by dude and snmp.
 
User avatar
inteq
Member Candidate
Member Candidate
Posts: 183
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Feb 02, 2020 6:46 pm

No queues or pppoe server/client.
Not even NAT. Just routing.
2nd time this one crashes and reboots
mt.png
You do not have the required permissions to view the files attached to this post.
 
Bolle
just joined
Posts: 5
Joined: Fri Aug 23, 2019 9:42 pm
Location: Germany

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Feb 03, 2020 5:07 am

today the RB4011 freezes after 11 hours.
No ROS management via Dude-server, no telnet, no Winbox.
but the WLAN is still running...
Dude recorded a cpu spike at 20% cpu usage, normal is 2% - 4%.

gtx Bolle
 
AlenDV
just joined
Posts: 6
Joined: Sat May 21, 2016 2:36 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Feb 03, 2020 5:09 pm

Hi everyone,

Our RB1100AHx4 running with ROS 6.44.6 yesterday froze in a remote Data Center, so we had to power recycle it, since then all works fine. The router is in use for several months, this seem to be the second or third time it's failing this way. Before RB1100AHx2 was in use with the same config - no issues in several years.

I disabled relatively newly added options, particularly standard queues (did not remove the rules, just disabled) and SNMP - most probable suspects. Will see if that helps.

P.S. Also updated ROS to 6.45.8.
 
allevot
just joined
Posts: 7
Joined: Fri Oct 19, 2018 2:44 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 07, 2020 10:31 am

Hi!

Since one month ago, no MikroTik have bricked. I have 50 of them and they still with the same RouterOS 6.44.6.
The only thing that I changed on all of them its the gateway of every route. Before the change, the gateway had the interface and now they have the IP of the next hop.

I'm not sure this is the solution but it's fine at the moment.

Best regards,
 
seho
newbie
Posts: 36
Joined: Fri Aug 09, 2013 11:05 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 07, 2020 2:36 pm

I also got a report from customer that is running two RB1100AHx4 as VPN concentrators (L2TP over IPSec+ IPSec IKEv2) in a VRRP Setup.

The VRRP Master device stopped accepting VPN connections, but unfortunately the VRRP has continued working - it stayed master device and the fail-over to the slave device wasn't performed. They restarted the master device and then everything has worked again.

SSH and Winbox access were not working was reported.

I want to build at least a workaround for them maybe by using a script, that at least the slave device will automatically take over. Maybe by increasing the VRRP interface priority of the slave device.

The problem looks to me like the posted Problem. But currently I got no idea how to detect that failure in a reliable way.

Can someone who was able to connect to the device using the serial port give a hint how to detect that the problem has occured?

From what I read:
1) SSH not working
2) Winbox not working

Is the device responding to ICMP pings?
Responding to ARP-Requests?

Every hint is appreciated.

It also thinkable for me to run the detection on a external application. So really every hint is welcome, not only the one which were detectable using scripting.

Thanks in advance.

Kind Regards,
Sebastian
 
netflow
Frequent Visitor
Frequent Visitor
Posts: 85
Joined: Sat Oct 01, 2016 3:53 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Feb 08, 2020 12:25 am

Well you just gave the solution:
The VRRP Master device stopped accepting VPN connections
So just try to open a VPN connection :D
 
oukltdtech
just joined
Posts: 2
Joined: Fri Jan 27, 2017 11:39 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Feb 17, 2020 2:55 pm

We had a brand new RB4011 stop responding last week after around 6 hours of putting it in. Winbox sat at "Logging In".

It had a couple of bridges on it we had set so we could monitor traffic going through the ports. Other than that, just an IP. It was updated to 6.46.3 before we put it in production.

The unit still bridged traffic and so didn't take our site down however it needed a hard reboot to fix.

Awaiting response from Miktorik Support.
 
allevot
just joined
Posts: 7
Joined: Fri Oct 19, 2018 2:44 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Feb 24, 2020 5:35 pm

I have configured Watchdog, but when MikroTik was bricked, the watchdog don't work.
Is very stressful, we have Rb4011 in differents sites...

Staff MikroTik please, can you find a solution for this problem?

Thanks a lot.
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 266
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Feb 24, 2020 5:44 pm

Schedule a nightly reboot each day.
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
allevot
just joined
Posts: 7
Joined: Fri Oct 19, 2018 2:44 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Feb 24, 2020 5:50 pm

There are services 24*7, i can't reboot it every day...
 
Filip92
just joined
Posts: 7
Joined: Fri Dec 30, 2016 6:28 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Feb 26, 2020 12:19 am

I have the same problem.

Device: RB1100AHx4 Dude Edition
Configuration:
- running dude server with ~320 routerboards, ~390 services (ping)
- routing 11 static routes
- firewall 7 filter rules, 14 NAT rules, ~1k connections
- no queues
- running FTP server for auto mikrotik's backup
- 3 bridges and ~ 550 hosts in network


The device was work good, and around after half year this issue appeared...

I had a serial connection, but it was also suspended. In my console window i was see:
LOOPER: read_raw read failed: EOF
died with signal
Image

I have Logging rules: critical, error and info - action: disk. In log file is nothing special, only:
Feb/25/2020 11:03:48 system,info,account user dude logged in from xx.xx.xx.247 via dude
Feb/25/2020 11:04:25 system,info,account user dude logged out from xx.xx.xx.247 via dude
----- BRICK --------
Feb/25/2020 05:28:12 system,error,critical router was rebooted without proper shutdown
Last edited by Filip92 on Fri Feb 28, 2020 7:02 pm, edited 2 times in total.
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 266
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Feb 26, 2020 8:35 am

There are services 24*7, i can't reboot it every day...

I see. But if you prefer bricking... it is better a 30 sec outage for reboot in the night....
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
inteq
Member Candidate
Member Candidate
Posts: 183
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Feb 26, 2020 9:05 am

There are services 24*7, i can't reboot it every day...

I see. But if you prefer bricking... it is better a 30 sec outage for reboot in the night....
We prefer a fix. Rebooting every night ain't one. I hope you don't do that to your users.
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 266
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Feb 26, 2020 9:12 am

I do. At 04 am ti 04.00.30 is not an issue...
I know that is not a fix but works. Eventually I replaced the 4011 with a ccr1009.
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
omidkosari
Trainer
Trainer
Posts: 634
Joined: Fri Sep 01, 2006 4:18 pm
Location: Iran , Karaj
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Feb 27, 2020 10:15 am

We prefer a fix. Rebooting every night ain't one. I hope you don't do that to your users.


We all prefer to solve this problem. Our reports and complains to mikrotik staff did not solve the problem so we try to find a workaround.
MTCNA , MTCRE, MTCWE, Mikrotik Certified Trainer
 
seho
newbie
Posts: 36
Joined: Fri Aug 09, 2013 11:05 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Feb 27, 2020 12:49 pm

I got some supouts from our customer which has reported the problem with non responding device. One with a device running in non error state and one in error state.

Because of an NDA i'm currently not allowed to share the supout with MikroTik.

I had a look to the supout.rif locally on my pc using Mikrotik RIF decoder (https://github.com/farseeker/go-mikrotik-rif)

Unfortunately i'm pretty unexperienced interpreting all the stuff written out.

But theres something looking very suspicious to me is the "== SECTION profile", the device got a very high unclassified part in the cpu usage.
When I look at the same entries of device which is not in failure state the values are much lower.
== SECTION profile
NAME                    CPU        USAGE
total                                 0%

total                                 0%

total                                 0%

firewall                            1.2%
management                          3.1%
traffic-accou...                    2.6%
unclassified                       21.5%
total                              28.4%

NAME                    CPU        USAGE


management                1         0.5%
traffic-accou...          1           3%
unclassified              1        16.5%
cpu1                                 20%

firewall                  0           0%
management                0          10%
traffic-accou...          0          23%
unclassified              0       146.5%
cpu0                              179.5%
management                1         0.5%
traffic-accou...          1           3%
unclassified              1        16.5%
cpu1                                 20%
firewall                  2        25.5%
networking                2         0.5%
management                2        14.5%
unclassified              2           6%
cpu2                               46.5%
firewall                  3           3%
management                3           3%
unclassified              3           0%
cpu3                                  6%
Could someone who encountered the same problem, and have a supout of the device in failure, please have a look at a supout.rif in failure state? And report back if the section at their failure device has a similar high "unclassified" cpu usage.

Maybe this has nothing to do with the problem at all. I read that the unclassified could came from encryption.
 
allevot
just joined
Posts: 7
Joined: Fri Oct 19, 2018 2:44 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Feb 27, 2020 4:59 pm

One of my devices has an autouport.rif, it was created on 24-02-2020, the same day Mikrotik bricked.
The Supout.rif Reader shows this:
MEMORY
MemTotal:        1034864 kB
MemFree:          966468 kB
Buffers:               0 kB
Cached:            27688 kB
SwapCached:            0 kB
Active:            10596 kB
Inactive:          22656 kB
Active(anon):       5664 kB
Inactive(anon):       64 kB
Active(file):       4932 kB
Inactive(file):    22592 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          5636 kB
Mapped:             5508 kB
Shmem:               160 kB
Slab:              22976 kB
SReclaimable:       3192 kB
SUnreclaim:        19784 kB
KernelStack:         672 kB
PageTables:          412 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      517432 kB
Committed_AS:      11020 kB
VmallocTotal:    1024000 kB
VmallocUsed:        6696 kB
VmallocChunk:     973848 kB
CPU
Processor	: ARMv7 Processor rev 4 (v7l)
processor	: 0
BogoMIPS	: 2793.47

processor	: 1
BogoMIPS	: 2793.47

processor	: 2
BogoMIPS	: 2793.47

processor	: 3
BogoMIPS	: 2793.47

Features	: swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc0f
CPU revision	: 4

Hardware	: AnnapurnaLabs Alpine (Device Tree)
Revision	: 0000
Serial		: 0000000000000000
 
angriukas
Frequent Visitor
Frequent Visitor
Posts: 81
Joined: Fri Nov 22, 2013 9:20 am
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 28, 2020 1:31 pm

10 days ago installed two RB4011iGS+ devices.
Firewall, 3 bridges, CAPsMAN with two SSID's, EoIP between both devices, few queues.
No bricks, no reboots, no issues at all.
RoS and firmware: 6.45.8
 
User avatar
fposavec
newbie
Posts: 40
Joined: Thu Apr 21, 2011 12:59 pm
Location: Čakovec
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 28, 2020 1:52 pm

[XXX@YYY] > system resource print 
             uptime: 35w2d21h45m51s
            version: 6.44.3 (stable)
         build-time: Apr/23/2019 12:37:03
   factory-software: 6.43
        free-memory: 904.0MiB
       total-memory: 1024.0MiB
                cpu: ARMv7
          cpu-count: 4
      cpu-frequency: 1400MHz
           cpu-load: 9%
     free-hdd-space: 418.4MiB
    total-hdd-space: 512.3MiB
  architecture-name: arm
         board-name: RB4011iGS+
           platform: MikroTik
no issue at all.
BGP routing, NAT, DHCP...
 
Overide
just joined
Posts: 8
Joined: Tue Jul 03, 2018 5:00 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 28, 2020 5:49 pm

We've been dealing with this issue since we began deploying 4011's

We use virtual bridges, pppoe, DCHP and ospf. not mush else. this issue has never occurred for us on any other hardware platform.
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1189
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 28, 2020 6:09 pm

We've run into this with a few clients that run the 4011 and have used the following as a workaround while we work on a ticket with MikroTik. It seems to be more stable in the few we have tested.

Set the CPU frequency to 1200mhz
system routerboard settings set cpu-frequency=1200
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
ingdaka
Member
Member
Posts: 335
Joined: Thu Aug 30, 2012 3:06 pm
Location: Albania
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 28, 2020 8:13 pm

I have 3 of them and I've not faced this problem never! Uptime more than 1 year running without any problem with normal 700Mbps to 1G of traffic!
Ilir Daka
Electronic & Network Engineer
E-mail: ilirdaka@live.com
Mob: +355692982151
WhatsApp: +355692982151
Mikrotik Official Consultant
CCNA | Fortinet NSE3 | MTCRE | MTCSE | MTCWE | RIPE NCC Certified Professional
 
anschluss
just joined
Posts: 7
Joined: Fri Mar 30, 2018 3:46 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Mar 02, 2020 4:09 pm

We've run into this with a few clients that run the 4011 and have used the following as a workaround while we work on a ticket with MikroTik. It seems to be more stable in the few we have tested.

Set the CPU frequency to 1200mhz
system routerboard settings set cpu-frequency=1200
Thanks very much for this helpful hint! The RB4011 has been acting up along the lines described in this thread, with intervals of running seemingly normal for weeks, then days, then hours. Until I downclocked the CPU, the device started jettisoning all routing functions even after just a minute's connection, such as a phone call, a VPN or remote desktop connection.
 
User avatar
cpbruton
just joined
Posts: 12
Joined: Fri Jun 07, 2013 4:23 am
Location: Pasadena, California, USA
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Mar 02, 2020 10:10 pm

I have encountered this issue with two different RB1100AHx4s. Both routers became inaccessible after a few weeks deployed. Forwarding and OSPF continued to work fine so we had no interruption in our network traffic, but could not log into the routers via web, ssh, mac-telnet, or direct console connection. A physical power cycle restored access.

Both routers have watchdog enabled but only one of them generated a partial autosupout.rif file, with the following message at the top:
--- nv::message --------
died with signal 6 on Sun Feb 23 15:02:12 2020
Both were running 6.45.8 (long-term). I downgraded to 6.44.6 before finding this thread, but it sounds like people have encountered this on older versions too.

I will try the suggestion to reduce CPU frequency to 1200MHz.
 
aoakeley
newbie
Posts: 49
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Tue Mar 03, 2020 4:03 pm


After obtaining supout.rif files by console cable, MikroTik have organised warranty replacement for the two 1100 units exhibiting this issue.
And yet another 1100AHX4 with this issue.
The previous 2 units which were replaced under warranty have not had a recurrence of the issue.
 
User avatar
omidkosari
Trainer
Trainer
Posts: 634
Joined: Fri Sep 01, 2006 4:18 pm
Location: Iran , Karaj
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Apr 02, 2020 6:40 pm

Hello

I see that your router did a reboot because of Kernel Watchdog.

For debugging you should turn off watchdog and test again.
"/system watchdog set watchdog-timer=no"

Connect now to your router over serial console, make sure that you have accessed RouterOS command line interface and leave console running.

Now router either:

1) might be stuck (freeze). If that happens, then you have to generate a supout file on the router through a serial console. Now you can reboot the device.

2) might be stuck (freeze) and become unavailable over a serial console. If that happens, then reboot the router and then generate a supout file.

3) will reboot. If that happens, then after reboot manually generate a supout file.

Send supout file and full serial console output (within text file) to us for investigation.

Before that, I suggest that you upgrade to the latest "stable" version (if there is an actual bug, then it might be already fixed).

Best Regards


This is a reply from mikrotik support. Has anyone done this ?
MTCNA , MTCRE, MTCWE, Mikrotik Certified Trainer
 
aoakeley
newbie
Posts: 49
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Apr 03, 2020 10:00 am


This is a reply from mikrotik support. Has anyone done this ?
Yes. see my previous posts
 
jebz
Member Candidate
Member Candidate
Posts: 270
Joined: Sun May 01, 2011 12:03 pm
Location: Australia

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Apr 03, 2020 2:36 pm

And yet another 1100AHX4 with this issue.
The previous 2 units which were replaced under warranty have not had a recurrence of the issue.
.
Have you had any feedback on what the fault was?
 
aoakeley
newbie
Posts: 49
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Apr 06, 2020 11:05 am

Have you had any feedback on what the fault was?
.
No. No feedback. Have replaced 3 under warranty now.

And I have one of the ones that was previously replaced, and had been running fine since end of Jan now inaccessible again. So either there is a systemic issue in this model or there is something in my config screwing them up. I am really not doing anything complex though, they are all just NAT routers for branch offices. I have hundreds of routers with very similar config and the only ones I am seeing issues with are the 1100AHx4 Models.

See screenshot of as many authentication attempts as I could get on one screen, yet DNS forwarding is still working as are general routing duties.
You do not have the required permissions to view the files attached to this post.
 
jebz
Member Candidate
Member Candidate
Posts: 270
Joined: Sun May 01, 2011 12:03 pm
Location: Australia

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Apr 06, 2020 3:40 pm

See screenshot of as many authentication attempts as I could get on one screen, yet DNS forwarding is still working as are general routing duties.
I've experienced issues with connecting with Winbox on occasions. In a few cases Webfig was still available so I disabled Winbox access and re-enable Winbox which restored Winbox access. On one occasion neither was accessible although routing was still operational. This required a reboot to regain access.
 
CheshirCa
just joined
Posts: 2
Joined: Thu Mar 21, 2019 1:13 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Apr 09, 2020 2:44 am

That night, one 1100AHx4 hung. Winbox stuck at "logging in". Webfig - cannot authenticate, wrong user or password.
Work as PPTP VPN server and DUDE server
 
akarag
just joined
Posts: 7
Joined: Mon Aug 04, 2014 5:46 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Apr 12, 2020 10:48 am


SSH and Winbox access were not working was reported.

I want to build at least a workaround for them maybe by using a script, that at least the slave device will automatically take over. Maybe by increasing the VRRP interface priority of the slave device.

The problem looks to me like the posted Problem. But currently I got no idea how to detect that failure in a reliable way.

Can someone who was able to connect to the device using the serial port give a hint how to detect that the problem has occured?

From what I read:
1) SSH not working
2) Winbox not working


Is the device responding to ICMP pings?
Responding to ARP-Requests?

Every hint is appreciated.

It also thinkable for me to run the detection on a external application. So really every hint is welcome, not only the one which were detectable using scripting.

Thanks in advance.

Kind Regards,
Sebastian
Hello, I would like to add my case as well regarding the well known problem with RB1100. We have several RB1100 and we have seen that 2 of them have similar behavior as the one described in this thread. Regarding the behavior, in every case the routing continued to work without any downtime, however the SNMP, winbox access, SSH access, even MAC based access was not possible, Regarding the access, seems that there is a common process that controls the access and this process failed. However, i am not sure regarding the SNMP if has some common points with the login process.
Currently we are at the latest version 6.46.5.Some of these routers support critical systems in a vrrp configuration while others support backbone. We depend on Mikrotik to deal with this annoying issue. So far we have confirmed that unclassified processes in Tool/profile have increased before the random brick of the device.

If the problem persists, would you recommend another Mikrotik device, meybe in the series of CCRs ? Are there any reports of the same problem in CCR series?
 
aoakeley
newbie
Posts: 49
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Apr 16, 2020 11:42 am

Currently we are at the latest version 6.46.5.Some of these routers support critical systems in a vrrp configuration while others support backbone. We depend on Mikrotik to deal with this annoying issue. So far we have confirmed that unclassified processes in Tool/profile have increased before the random brick of the device.

That's interesting. I will keep my eyes open for that. How many processes are you talking about before it crashes? I have just checked a few different routers (CCR and MIPSBE and 1100 and CHR) and they all have a handful of unclassified processes.

If the problem persists, would you recommend another Mikrotik device, meybe in the series of CCRs ? Are there any reports of the same problem in CCR series?

I have not seen this issue on any other Mikrotik device.
 
akarag
just joined
Posts: 7
Joined: Mon Aug 04, 2014 5:46 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Apr 24, 2020 9:51 am

Ooops it did it again!

Once again one of our main RB1100 routers that support critical business functions, was bricked in terms of access to winbox. We cannot login to the router, MAC discovery is not working. However, routing is working. What we observed is that SNMP is not working, however netflow information is transmitted to the NMS.

It is getting really annoying issue for us.

Is there any fix for this issue for RB1100AHx4 or should we consider move to another appliance or vendor?

As soon as I have access to winbox, I will create a supout and send it to mikrotik for further investigation.However, this will be accomplished after a router restart through power disconnection
 
User avatar
inteq
Member Candidate
Member Candidate
Posts: 183
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Apr 24, 2020 8:36 pm

And again on a RB4011iGS+RM
crash.png
You do not have the required permissions to view the files attached to this post.
 
nje431
newbie
Posts: 47
Joined: Tue Sep 10, 2013 5:17 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Apr 25, 2020 1:41 am

We are having the same problems as the OP. I got the same response from support that it's failed hardware. We bought 50 of the RB1100AHx4 and have deployed about 13 so far with 5 units each having done a management lockout. So I'm having a hard time believing a hardware failure. It's a pretty dismal failure rate if it is. Thank goodness for managed PDUs.

I'm trying different configuration changes, but so far nothing good to report.

From what I gather here, it doesn't sound like there is a magic software version yet. The one post I saw that piqued my interest was reducing the clock rate. For those that have tried it, what were the results?

Thanks
 
anschluss
just joined
Posts: 7
Joined: Fri Mar 30, 2018 3:46 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Apr 25, 2020 4:26 am

From what I gather here, it doesn't sound like there is a magic software version yet. The one post I saw that piqued my interest was reducing the clock rate. For those that have tried it, what were the results?
No more problems since reducing the CPU freq to 1200 Mhz; running for two months now.
 
nje431
newbie
Posts: 47
Joined: Tue Sep 10, 2013 5:17 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Apr 25, 2020 5:18 pm

No more problems since reducing the CPU freq to 1200 Mhz; running for two months now.
That sounds promising. Thanks! I just hope I didn't jinx you :)
 
Overide
just joined
Posts: 8
Joined: Tue Jul 03, 2018 5:00 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Apr 29, 2020 12:04 am

1200mhz tested and has not fixed the issue for us. So far no clue what triggers the issue
 
msatter
Forum Guru
Forum Guru
Posts: 1626
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Apr 29, 2020 12:45 am

Looking for a long time at the 4011 and I afraid that it will stay with looking at and not buying.
One RB4011 and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2-500/600 Mb/s.
Running:
RouterOS 6.47rcX / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
thsun
just joined
Posts: 12
Joined: Mon Mar 26, 2012 10:22 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Apr 29, 2020 4:34 am

I have many of RB4011 and RB1100 AHx4, some of them encountered the same issue in this week. No matter use 6.46, 6.46.5 and 6.45.8.
Currently, I scheduled an on/off snmp every 6 hours and also set cpu to 1200Mhz. Hope it help.

/system schedule
add interval=6h name=ResetSNMP on-event="{\r\
\n/snmp\r\
\nset enabled=no;\r\
\n:delay 3s;\r\
\nset enabled=yes;\r\
\n:log warning \"Reset SNMP to prevent ROS encounter timeout while waiting program XX\"\r\
\n}" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=apr/28/2020 start-time=09:20:04
 
wojo
newbie
Posts: 37
Joined: Tue Aug 21, 2018 4:37 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Apr 29, 2020 7:15 am

This concerns me but running a RB4011iGS+ right now at stock frequency with no issue for months under Gigabit load. Will be watching this thread for updates.
 
sarada
just joined
Posts: 7
Joined: Mon Mar 12, 2018 2:26 pm
Location: Hungary

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Apr 29, 2020 9:49 am

Hi

Maybe the NTP package is not enabled?

For me, too, due to the ntp package, the ARM architecture cAP ac was restarted randomly.
Once I turned off the ntp package, it has been stable ever since.

Link that can be resolved: viewtopic.php?t=153988
 
User avatar
inteq
Member Candidate
Member Candidate
Posts: 183
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri May 08, 2020 6:53 am

Setup Dude to monitor CPU on all 4011s
So far only two have issues but tired of this.
Starting to replace all 4011. Not worth the trouble.
cpu.png
You do not have the required permissions to view the files attached to this post.
 
maloi3390
just joined
Posts: 4
Joined: Thu Apr 19, 2018 11:03 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri May 08, 2020 9:48 am

Hello everyone! I have two RB1100DX4 and both devices are brick every week, max uptime was 30 days. Support says to send device to warranty, but I don't believe in fix this problem...
Last edited by maloi3390 on Fri May 08, 2020 2:54 pm, edited 1 time in total.
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 266
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri May 08, 2020 10:22 am

As I told you, I just scheduled a nightly reboot.
I cant believe that there are people that cant reboot the router nightly till the fix is out.
I understand that is not a solution but it is a full working fix.
today went out the new longterm!
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
nmt1900
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Feb 01, 2017 12:36 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri May 08, 2020 11:08 am

Now I understand why I did not end up finding this thread before.

I am having same kind of issues with 2 devices - one is RB1100x4 and another is RB1100Dx4 - and these are only ones plagued with problems because these are only ones updated to 6.45.8 from 6.44.6 (only long-term releases in use for customer locations, but even this strategy is backfiring sometimes).

Both suffer from "unclassified" processes taking over one CPU core and sometimes loss of all static routing thus no remote access except via IPsec tunnel before its' death.

It is hard to believe this being strictly hardware issue if it appeared only after upgrade to 6.45.x version - but then again my issue might be something else - only common denominators being 6.45.x version and 1100x4 - which shares same CPU with 4011 series. Go figure...
 
Filip92
just joined
Posts: 7
Joined: Fri Dec 30, 2016 6:28 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri May 08, 2020 11:39 am

I have RB1100AHx4 Dude Edition with Factory firmware 6.38.4, currently working on version 6.42.12 (long-term). This issue began after a year of impeccable work... Maybe this CPU has a special feature that breaks under warranty.

PS. My device is running from 14 days with 1200MHz and still working
 
maloi3390
just joined
Posts: 4
Joined: Thu Apr 19, 2018 11:03 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri May 08, 2020 2:36 pm

At now I try to replace 1100Dx4 to CHR because work with this issue 24/7 impossible. I want to be change all service and users to CHR and than try to test 1100Dx4 with RoS7beta5. I believe that this broblem with the brick device fix in future in new RoS version.
 
nje431
newbie
Posts: 47
Joined: Tue Sep 10, 2013 5:17 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon May 25, 2020 9:48 pm

I may have found something. While it's a bit too early to know for sure, the fix is simple enough and safe to try.

First, what didn't work. I tried dropping the CPU clock, but that didn't work for me. Then I disabled the SNTP client, but that just changed how the failure presented itself from random management lockouts to random reboots.

Last weekend, following up on a tip in another thread, I started testing Ethernet speed/duplex settings on the 5 routers that have given me the most trouble. I had set a few to a mix of fixed 100/full and auto 1000/full only ports; some to auto 100/full only and auto 1000/full only ports; and one to a mix of fixed 100/full and default autonegotiate settings. Both the first 2 groups have been stable over the last week, while the 3rd had a random reboot.

Yesterday, while researching the Ethernet chipset for these routers, I came across the manufacturer's sales sheet. It explicitly states that 1000/half is not supported, yet the defaults in the ROS autonegotiate settings is to enable advertising that combo. Since the chipset doesn't support that, it's an unnecessary setting. I went through all 28 of our deployed 1100's yesterday evening and turned 1000/half off on all ports. This morning when checking the logs, the system was unusually stable overnight. Not even any of the EoIP drop-outs that I sometimes see.

Anyway, I'd be curious to see what others results are when they disable 1000m/half.

Cheers
 
msatter
Forum Guru
Forum Guru
Posts: 1626
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon May 25, 2020 10:19 pm

If you look in the status tab you will see that 1000 half is not advertised by the 4011. However it is a test worth to see if also disabling in the settings brings a improvement or a solution.
One RB4011 and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2-500/600 Mb/s.
Running:
RouterOS 6.47rcX / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
nje431
newbie
Posts: 47
Joined: Tue Sep 10, 2013 5:17 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon May 25, 2020 10:32 pm

Yes, I noticed that. My suspicion is that ROS is asking the chip to do something it's not capable of, and creating an out of bounds condition. That in turn is causing unexpected behaviour. That's my current working theory anyway.
 
jebz
Member Candidate
Member Candidate
Posts: 270
Joined: Sun May 01, 2011 12:03 pm
Location: Australia

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Tue May 26, 2020 2:46 am

I have RB1100AHx4 Dude Edition with Factory firmware 6.38.4, currently working on version 6.42.12 (long-term). This issue began after a year of impeccable work... Maybe this CPU has a special feature that breaks under warranty.
Open a case with Mikrotik and send a Supout.rif . It's best to have it fixed under warranty.
 
nje431
newbie
Posts: 47
Joined: Tue Sep 10, 2013 5:17 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Tue May 26, 2020 3:09 pm

Actually, I did. A couple times. That answer is always the same. You have bad hardware, return it to the seller for warranty (not Mikrotik, like most manufacturers would do). We couldn't get the Amazon seller to even respond to one with a bad power supply out of the box. Plus with so many failing this way, that would indicate a real quality control problem. No, I suspect this problem looks like a hardware failure because the hardware is being ask to do what it's not designed to do. But that's just a guess at this point.

Who is online

Users browsing this forum: ahmedali44, sohel07 and 258 guests