Community discussions

MikroTik App
 
marisspringis
just joined
Topic Author
Posts: 15
Joined: Wed Dec 12, 2018 2:17 pm

RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jun 03, 2019 1:18 pm

Hi everyone,
since the last week of may, strange thing has happened to Router Boards which i manage.
issue - Router simply bricks, what i mean by that is - you cannot connect to router in any way (stays on logging in and nothing more happens), APs that are connected lose all config from dude, SNMP stops working and so on.
in the same time, from computers which are connected to switch, internet is working. also i can ping that router
this can be resolved only by hard reset (take power cable off/on)
issue has happened only on RB4011 and RB1100AHx4 Dude edition
Router Os - 6.44.3
previously this has never happened.
these RB are in different countries.

so far this has happened only once but with every RB4011 we have and one RB1100 AHx4
does anyone else has seen this?
 
User avatar
ccardenas
Trainer
Trainer
Posts: 5
Joined: Thu May 15, 2014 9:04 pm
Location: Spain

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jun 03, 2019 1:48 pm

Hello! Do you have bridges in your network implementation? How many hosts are passing traffic among these "random bricking" devices? Could you provide us more info? So we can understand your problem and we'll be able to help you better.

We have a similar issue, and we are suspecting about the bridge host table size and (possible) memory exhaustion problem. It only happens in new arm devices (RB4011 and RB1100x4).

Symptoms are loss of connectivity, manageability and it's impossible to access the device in any way, but it keeps working as a switch. After a reboot (unplug power cable/ replug) all begin to work fine and we can see a lot log lines like: snmp, warning timeout while waiting for program XX (where xx is a variable two digit number)

Image

Regards
 
marisspringis
just joined
Topic Author
Posts: 15
Joined: Wed Dec 12, 2018 2:17 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jun 03, 2019 2:43 pm

Hi ccardenas,
yes, we have 3 bridges in RB4011 and in RB1100AHx4.
hosts in RB4011 most of the time are - 30-40
in RB1100 not more than 5 connected directly, it is used as dude server for monitoring.

one more thing to point that problem is in these RB4011 is that we have a lot of RB2011 with the same config, and they work perfectly, without any problems. also ROS version in all ar is the 6.44.3

symptoms are identical to yours.
 
marcin21
Member Candidate
Member Candidate
Posts: 201
Joined: Tue May 04, 2010 4:50 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jun 03, 2019 4:32 pm

Itr seems that cpu is getting exhausted over time.
this particular ARM based WAP60g is 149d up.
You do not have the required permissions to view the files attached to this post.
 
User avatar
ccardenas
Trainer
Trainer
Posts: 5
Joined: Thu May 15, 2014 9:04 pm
Location: Spain

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jun 03, 2019 5:45 pm

one more thing to point that problem is in these RB4011 is that we have a lot of RB2011 with the same config, and they work perfectly, without any problems. also ROS version in all ar is the 6.44.3
Hello!! Yes, totally true. RB2011 and RB1100Hx2 in the same place, in the same network, in the same situation and nothing happens, they never block. We've opened a support ticket to Mikrotik and they told us to plug a serial cable and wait the device to block, then try to access it via console and make a supout, but we have a couple of them with cables attached and now they never block! :cry:

Other devices within the network keep blocking randomly. In the meanwhile we have scheduled a reboot (lame solution, but it saves the day) at nights a couple of times a week until we find the real problem, but it seems that some process inside the routerboard hangs or collapses the memory, making the another processes fall in cascade and block the access to the device.

If someone is experiencing the same problem, please share with us, maybe we can find a hint in the meanwhile, until I can get a good supout file and send to Mikrotik support.

Regards!!
Last edited by ccardenas on Mon Jun 03, 2019 7:27 pm, edited 1 time in total.
 
Dude2048
Member Candidate
Member Candidate
Posts: 110
Joined: Thu Sep 01, 2016 4:04 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jun 03, 2019 6:38 pm

I have a RB1100ahx4 Dude edition which has the same behavior. What happens is that the memory hogs and the device will become inaccessible. I have a script that reboots the device when 70% is used. During the times that it is inaccessible I tried to make a supout, via console, but that didn't work.
 
User avatar
kehrlein
just joined
Posts: 9
Joined: Tue Jul 09, 2019 1:35 am
Location: Munich, Germany
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Dec 05, 2019 12:24 pm

Same issue here with 3 different RB1100AHx4. The situations happens with < 7 days uptime.
During the incident, the devices don't accept new SSH, Winbox, SNMP or PPTP connections to the router itself.
Also logins via CLI aren't possible (serial connection is possible; login doesn't work). We tried to have a running serial connection to the affected devices. If the issue occurs, we are able to type commands into the cli, but generating the supout or doing real actions (e.g. initiate a reboot) doesn't work.
Other traffic goes through the router smoothly. The issue can temporary be fixed by powering off an on the device. Temporary solution is a scheduled reboot.
 
meshnet
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Tue Jun 01, 2004 6:57 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Dec 05, 2019 9:05 pm

Starting to see this issue also on 4011s. 6.44.1 on the last one it happened to.
Power cycle restores all functionality..
SNMP polling becomes very intermittent right before this happens, pointing to the CPU issues..
Only bridge configured on the devices is an empty bridge for loopback.
 
Dude2048
Member Candidate
Member Candidate
Posts: 110
Joined: Thu Sep 01, 2016 4:04 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Dec 06, 2019 12:01 pm

Same issue here with 3 different RB1100AHx4. The situations happens with < 7 days uptime.
During the incident, the devices don't accept new SSH, Winbox, SNMP or PPTP connections to the router itself.
Also logins via CLI aren't possible (serial connection is possible; login doesn't work). We tried to have a running serial connection to the affected devices. If the issue occurs, we are able to type commands into the cli, but generating the supout or doing real actions (e.g. initiate a reboot) doesn't work.
Other traffic goes through the router smoothly. The issue can temporary be fixed by powering off an on the device. Temporary solution is a scheduled reboot.


What version are you using.
 
quackyo
Member Candidate
Member Candidate
Posts: 125
Joined: Mon Nov 16, 2015 10:14 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Dec 06, 2019 1:36 pm

Does everybody experiencing this run Dude on the device?
I have had the exact same issue on my RB1100AHx4 shortly after setting up Dude. Happened 3 or 4 times over a month or two before i connected the dots. Disabled Dude server and no hickup since (6-8 months ago...).
 
User avatar
kehrlein
just joined
Posts: 9
Joined: Tue Jul 09, 2019 1:35 am
Location: Munich, Germany
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Dec 06, 2019 1:38 pm

Same issue here with 3 different RB1100AHx4. The situations happens with < 7 days uptime.
During the incident, the devices don't accept new SSH, Winbox, SNMP or PPTP connections to the router itself.
Also logins via CLI aren't possible (serial connection is possible; login doesn't work). We tried to have a running serial connection to the affected devices. If the issue occurs, we are able to type commands into the cli, but generating the supout or doing real actions (e.g. initiate a reboot) doesn't work.
Other traffic goes through the router smoothly. The issue can temporary be fixed by powering off an on the device. Temporary solution is a scheduled reboot.
What version are you using.

Latest bad experience was with 6.45.7.
Time for experience with 6.46 was too short.

After some mails with the MikroTik support during the last month, I finally got this answer:
Unfortunately, this problem seems to be caused by a hardware issue.
Please contact the seller and return the router for warranty repairs, if the router is still covered by it. You can refer to this ticket number - SUP-3012.
So now I am talking to the reseller about refund.
 
User avatar
kehrlein
just joined
Posts: 9
Joined: Tue Jul 09, 2019 1:35 am
Location: Munich, Germany
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Dec 06, 2019 1:40 pm

Does everybody experiencing this run Dude on the device?
I have had the exact same issue on my RB1100AHx4 shortly after setting up Dude. Happened 3 or 4 times over a month or two before i connected the dots. Disabled Dude server and no hickup since (6-8 months ago...).
I had the issues on several devices without running the Dude.
 
Bolle
just joined
Posts: 5
Joined: Fri Aug 23, 2019 9:42 pm
Location: Germany

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Dec 06, 2019 2:49 pm

Hi !

Same here with RB4011iGS+5HacQ2HnD und RB1100Dx4.

Both devices ´freezes´ several times a week.
The RB1100 sometimes two or three times a day.

I tryied with ROS 6.45.7 and 6.46beta59.
 
marcin21
Member Candidate
Member Candidate
Posts: 201
Joined: Tue May 04, 2010 4:50 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Dec 16, 2019 1:38 pm

Anyone has found a solution to this problem ?
Maybe SNMP disable?

I've got problematic 4011 and since few days it started to die.
 
aoakeley
newbie
Posts: 47
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Dec 21, 2019 3:18 pm

I have two 1100AHx4 with this issue, and an open case with Mikrotik (Ticket#2019100722004559). I have just been able to connect in with a serial cable and generate a supout file.

I am not running Dude on them.
SNMP is enabled though.
 
FreeVoip
just joined
Posts: 4
Joined: Tue Oct 15, 2019 5:33 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Dec 21, 2019 4:49 pm

Hello!.

Same problem here!!!!! viewtopic.php?f=2&t=154859&sid=9cf5b071 ... 23dad9a667

I don't know what else to do. It already happened to me 4 times in the week.
There is no high CPU consumption, no RAM consumption or anything strange.

It is a very big problem for me.
 
FreeVoip
just joined
Posts: 4
Joined: Tue Oct 15, 2019 5:33 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Dec 21, 2019 4:57 pm

It is incredible that since June this was reported and nobody from Mikrotik did anything.

In my case they suggested connecting via serial cable but I am 800 km from my node. It is impossible for me to do that. It is easier to restart it but it is not very serious for an internet provider to cut the service every day at the same time. Needless to say, if it "hangs" at 10 am I have to wait until the next day to access again.
 
marcin21
Member Candidate
Member Candidate
Posts: 201
Joined: Tue May 04, 2010 4:50 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Dec 21, 2019 8:31 pm

since last problem, I 've got winbox session opened on screen a this problemtic 4011 is up and running for 5d+
I wonder if it has something in common with
viewtopic.php?f=3&t=142298
 
aoakeley
newbie
Posts: 47
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Dec 23, 2019 3:34 am

It is incredible that since June this was reported and nobody from Mikrotik did anything.
I suspect that they did not have enough information to replicate the issue and develop a fix. Like you many of my devices were a long way away, and they asked to have a serial cable plugged in. This was difficult, but I now have one connected and have sent a supout to support.

With a few more people reporting the issue now, hopefully they will be able to find a fix.
 
marcin21
Member Candidate
Member Candidate
Posts: 201
Joined: Tue May 04, 2010 4:50 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Dec 23, 2019 11:59 pm

I wouldn't be so optimistic regarding fast tracking the problem down.
it seems that very similar issue with rb4011
viewtopic.php?f=3&t=142298
was reported in dec'18, and in april'19 first response from Mikrotik staff that they recognise problem and working on solution,
and it seems that they haven't solved it till now.
maybe there is some serious flaw in hardware, as we see it above
Unfortunately, this problem seems to be caused by a hardware issue.
Please contact the seller and return the router for warranty repairs, if the router is still covered by it. You can refer to this ticket number - SUP-3012.
either the way i've got my winbox opened on my problematic 4011 and 7d12h uptime. maybe winbox opened does the trick? I doubt it but whatever, when it works.
Merry Xmas :)
 
luciansilviu
just joined
Posts: 1
Joined: Fri Dec 27, 2019 4:10 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Dec 27, 2019 4:14 pm

Hi!

Just a quick "me too". I have had this issue with two boards RB4011iGS+ running on 6.45.4.
 
reapster
just joined
Posts: 2
Joined: Mon Jun 10, 2013 8:01 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Dec 28, 2019 8:03 pm

Hey there,

We also see this on our RB1100AHx4 (Currently 6.45.7, and inaccessible), every few weeks it exhibits this behaviour. Cannot access via Winbox/Telnet/SSH, but routing remains fine (thankfully!). This unit does almost no traffic (< 10Mbps), and has no special configuration, its not even doing SNMP. About 10 firewall rules, usual masquerade NAT and I think a DHCP server, very basic setup.

All functionality returns after a power cycle, sadly not a feasible long term solution though. We have quite a few Mikrotik units deployed, but will have to start moving away from Mikrotik if there's no real solution to this.
 
aoakeley
newbie
Posts: 47
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Jan 02, 2020 2:57 pm

We have quite a few MikroTik units deployed, but will have to start moving away from Mikrotik if there's no real solution to this.
We have hundreds (thousands??) of MikroTik units in the field, and only 2 exhibiting this issue. So don't throw the baby out with the bath water.

After obtaining supout.rif files by console cable, MikroTik have organised warranty replacement for the two 1100 units exhibiting this issue. Can't ask for more than that.
We have not seen this issue with any 4011 units.

Andy
 
allevot
just joined
Posts: 7
Joined: Fri Oct 19, 2018 2:44 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Jan 03, 2020 3:18 pm

Hi,

We have several RB4011 and exactly the same happens to us. Randomly some MikroTik is no longer accessible by management, and in Dude appears the services "memory, disk and cpu" down, but the ping goes well.
The services work correctly, but there is no way to log in to the equipment and the only way to be able to have management is to restart it.

For more info we have VRRP, Bridge, Tunnel GRE, ACLs, BGP, Queues, SNMP and traffic is 20-50Mbps.

Regards,
 
aoakeley
newbie
Posts: 47
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Jan 04, 2020 2:56 am

We have several RB4011 and exactly the same happens to us. Randomly some MikroTik is no longer accessible by management, and in Dude appears the services "memory, disk and cpu" down,

Regards,
- Use a console cable to get a supout.rif file while the router is in the non responsive state (might as well do this when you lodge the ticket, as they will ask you to do this anyway). To generate the suport.rif you will probably need to have the console cable connected and be logged on before the router craps out.
- Log a ticket with support@mikrotik.com.au
- if there are any autosupout.rif in the file list send these also

Good luck.
 
allevot
just joined
Posts: 7
Joined: Fri Oct 19, 2018 2:44 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Jan 09, 2020 10:43 am

Hi,

We are unable to make a supout.rif because we are unable to log in into the device in any way.
Someone from MikroTik support can help us? It's a big problem for our company.

Thanks,
 
r00t
Member Candidate
Member Candidate
Posts: 299
Joined: Tue Nov 28, 2017 2:14 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Jan 09, 2020 6:51 pm

If router is still running, you could try to generate supout by watchdog script that:
1) checks if router can ping some IP that should normally be reachable at all times (gateway, some other LAN device)
2) if not (interfaces are frozen, down, IP not accessible) generate supout
3) reboot the router
Run it every minute and hope it works when the issue happens.
But if router is not even accessible by console cable, it's likely completely frozen (CPU/ROS) and above script would not run anyway...

Some more debugging options for ROS would be great, like redirecting kernel syslog to console port at all times.
 
User avatar
inteq
Member Candidate
Member Candidate
Posts: 176
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Jan 10, 2020 4:31 pm

Only one RB4011 (without WiFi) out of 12 crashed once with some process stuck.
None of RB1100AHx4 or RB1100AHx4 Dude Edition out of 19 crashed so far.
Also, bricking can happen to Mikrotiks, but it did not happen to me (yet) and if a power reset fixes it, it did not happen to you (yet).
 
User avatar
jdejansb
Frequent Visitor
Frequent Visitor
Posts: 69
Joined: Thu Jul 13, 2006 1:35 pm
Location: Srbija
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Jan 11, 2020 12:07 am

... and I was just about buying a 4011 . . . should I do that or not :? .. it's for a main network gateway, PPPoE, dude, . . .
 
aoakeley
newbie
Posts: 47
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jan 13, 2020 5:18 am

... and I was just about buying a 4011 . . . should I do that or not :? .. it's for a main network gateway, PPPoE, dude, . . .
I have plenty running that have no issue. I think the devices with this issue are very few.
 
tquibell
just joined
Posts: 2
Joined: Fri Jan 17, 2020 6:17 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Jan 18, 2020 7:41 am

Same Problem as OP I think.

Not using any fancy routing. Only doing basic Firewall/NAT/Mangle and DHCP Server and Client. Some static routes, etc. Nothing else.

RouterOS Long Term 6.44.3 would cause RB4011 to randomly lock up. First few times a reboot did the trick, uptimes averaged about 60 days between lockups. The router would remain visible in Winbox but otherwise I could not communicate with it in any fashion. IP addresses would become 0.0.0.0 etc. Terminal unresponsive. Have not confirmed whether the built in serial console was also unresponsive. Finally on the last go-round the board bricked.

On boot up was able to see that the board was hanging on loading kernel. Used Netinstall to wipe the NAND and reflash the Firmware and RouterOS. Voila. Moved to Stable 6.46.2 to see if this thing will survive with a different version.

Will report back after some light stress testing and a few weeks of uptime (hopefully alot more).
 
Mikhalich
just joined
Posts: 2
Joined: Mon Feb 05, 2018 10:16 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Jan 20, 2020 7:05 pm

Have same problem on RB4011 without WiFi.
System stops forward client packets but I can connect via winbox to WAN-port.
System runs script make supout.rif every 2-3 minutes but can not finish it.
Device did not execute any commands via console but respond on command via winbox.
Tools-Profile shows 100% load 1 of 4 core and 77% unclassified load.
Uptime was about 60 days. ROS 6.46.1

After reboot via winbox system runs normally.
Last edited by Mikhalich on Mon Jan 20, 2020 7:12 pm, edited 1 time in total.
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 252
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Jan 25, 2020 7:12 pm

Hello
I have a 4011 that had the same behaviour described. I replaced it with another 4011, same problem!
Ethernet ports simply stop responding, then in the logs there were "waiting for progam xx" , sometimes 20, sometimes other number as described.
I tried with watchdog to one ip to reboot automatically, with no help.

I solved completely planning a system reboot at 05:30:00 every day.
Then with the latest 6.44.6, every few day.
zero issues since then :)
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
omidkosari
Trainer
Trainer
Posts: 633
Joined: Fri Sep 01, 2006 4:18 pm
Location: Iran , Karaj
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Feb 01, 2020 3:05 pm

Hello,

I want to share my experience with same issue. This bug exists for years with several different device types like RBxx ,CCR1009 , CCR1036 , x86 Server , etc. I had several tickets with mikrotik support but unfortunately they were not able to find the problem . The only common feature in all of those routers was PPPoE concentrator (pppoe server).

The only current workaround is to schedule reboot the router everyday. Even with rebooting everyday the problem may appear randomly each few weeks.
MTCNA , MTCRE, MTCWE, Mikrotik Certified Trainer
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 252
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Feb 01, 2020 6:20 pm

In my setup there were not any pppoe services, just plain routing with simple queues.
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
omidkosari
Trainer
Trainer
Posts: 633
Joined: Fri Sep 01, 2006 4:18 pm
Location: Iran , Karaj
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Feb 02, 2020 8:56 am

We have to find a common point. After reading this topic I am suspicious about two new points.

1-Do you have this issue on a device which is not monitored by Dude ?
Please also take a look at following topic viewtopic.php?f=2&t=114365

2-Do you have this issue on a device which is not using Queues?
MTCNA , MTCRE, MTCWE, Mikrotik Certified Trainer
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 252
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Feb 02, 2020 9:31 am

Hello. All my devices are monitored by dude. And have queues. I have one without queues and it works flawlessy.
Last edited by Maggiore81 on Sun Feb 02, 2020 5:07 pm, edited 1 time in total.
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
omidkosari
Trainer
Trainer
Posts: 633
Joined: Fri Sep 01, 2006 4:18 pm
Location: Iran , Karaj
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Feb 02, 2020 9:58 am

In my devices like yours, all of the buggy devices have queues. Almost each pppoe user which connects to router will have dynamic queue from radius server. I think the issue happens when a specific dynamic queue will change (add or remove) on the router. The main problem to solving this issue is that , the issue is not reproducible by mikrotik staff.

Please describe your router configs and your thoughts about the situation which the issue may happen.
MTCNA , MTCRE, MTCWE, Mikrotik Certified Trainer
 
Bolle
just joined
Posts: 5
Joined: Fri Aug 23, 2019 9:42 pm
Location: Germany

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Feb 02, 2020 10:44 am

We have to find a common point. After reading this topic I am suspicious about two new points.

1-Do you have this issue on a device which is not monitored by Dude ?
Please also take a look at following topic viewtopic.php?f=2&t=114365

2-Do you have this issue on a device which is not using Queues?

I have a RB 4011 monitored by Dude, and a 1100DX4 wich is the Dude Server.
On both devices are only the default Interface Queues.
ROS 6.46.2
And both devices freezes
 
marcin21
Member Candidate
Member Candidate
Posts: 201
Joined: Tue May 04, 2010 4:50 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Feb 02, 2020 4:15 pm

my 4011 has no pppoe-server, nor dynamic queues and yet still stalls.
although it does lot of nat, close to 1gbps, also BGP, and it is monitored by dude and snmp.
 
User avatar
inteq
Member Candidate
Member Candidate
Posts: 176
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sun Feb 02, 2020 6:46 pm

No queues or pppoe server/client.
Not even NAT. Just routing.
2nd time this one crashes and reboots
mt.png
You do not have the required permissions to view the files attached to this post.
 
Bolle
just joined
Posts: 5
Joined: Fri Aug 23, 2019 9:42 pm
Location: Germany

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Feb 03, 2020 5:07 am

today the RB4011 freezes after 11 hours.
No ROS management via Dude-server, no telnet, no Winbox.
but the WLAN is still running...
Dude recorded a cpu spike at 20% cpu usage, normal is 2% - 4%.

gtx Bolle
 
AlenDV
just joined
Posts: 6
Joined: Sat May 21, 2016 2:36 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Feb 03, 2020 5:09 pm

Hi everyone,

Our RB1100AHx4 running with ROS 6.44.6 yesterday froze in a remote Data Center, so we had to power recycle it, since then all works fine. The router is in use for several months, this seem to be the second or third time it's failing this way. Before RB1100AHx2 was in use with the same config - no issues in several years.

I disabled relatively newly added options, particularly standard queues (did not remove the rules, just disabled) and SNMP - most probable suspects. Will see if that helps.

P.S. Also updated ROS to 6.45.8.
 
allevot
just joined
Posts: 7
Joined: Fri Oct 19, 2018 2:44 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 07, 2020 10:31 am

Hi!

Since one month ago, no MikroTik have bricked. I have 50 of them and they still with the same RouterOS 6.44.6.
The only thing that I changed on all of them its the gateway of every route. Before the change, the gateway had the interface and now they have the IP of the next hop.

I'm not sure this is the solution but it's fine at the moment.

Best regards,
 
seho
newbie
Posts: 36
Joined: Fri Aug 09, 2013 11:05 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 07, 2020 2:36 pm

I also got a report from customer that is running two RB1100AHx4 as VPN concentrators (L2TP over IPSec+ IPSec IKEv2) in a VRRP Setup.

The VRRP Master device stopped accepting VPN connections, but unfortunately the VRRP has continued working - it stayed master device and the fail-over to the slave device wasn't performed. They restarted the master device and then everything has worked again.

SSH and Winbox access were not working was reported.

I want to build at least a workaround for them maybe by using a script, that at least the slave device will automatically take over. Maybe by increasing the VRRP interface priority of the slave device.

The problem looks to me like the posted Problem. But currently I got no idea how to detect that failure in a reliable way.

Can someone who was able to connect to the device using the serial port give a hint how to detect that the problem has occured?

From what I read:
1) SSH not working
2) Winbox not working

Is the device responding to ICMP pings?
Responding to ARP-Requests?

Every hint is appreciated.

It also thinkable for me to run the detection on a external application. So really every hint is welcome, not only the one which were detectable using scripting.

Thanks in advance.

Kind Regards,
Sebastian
 
netflow
Frequent Visitor
Frequent Visitor
Posts: 78
Joined: Sat Oct 01, 2016 3:53 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Sat Feb 08, 2020 12:25 am

Well you just gave the solution:
The VRRP Master device stopped accepting VPN connections
So just try to open a VPN connection :D
 
oukltdtech
just joined
Posts: 2
Joined: Fri Jan 27, 2017 11:39 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Feb 17, 2020 2:55 pm

We had a brand new RB4011 stop responding last week after around 6 hours of putting it in. Winbox sat at "Logging In".

It had a couple of bridges on it we had set so we could monitor traffic going through the ports. Other than that, just an IP. It was updated to 6.46.3 before we put it in production.

The unit still bridged traffic and so didn't take our site down however it needed a hard reboot to fix.

Awaiting response from Miktorik Support.
 
allevot
just joined
Posts: 7
Joined: Fri Oct 19, 2018 2:44 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Feb 24, 2020 5:35 pm

I have configured Watchdog, but when MikroTik was bricked, the watchdog don't work.
Is very stressful, we have Rb4011 in differents sites...

Staff MikroTik please, can you find a solution for this problem?

Thanks a lot.
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 252
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Feb 24, 2020 5:44 pm

Schedule a nightly reboot each day.
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
allevot
just joined
Posts: 7
Joined: Fri Oct 19, 2018 2:44 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Feb 24, 2020 5:50 pm

There are services 24*7, i can't reboot it every day...
 
Filip92
just joined
Posts: 6
Joined: Fri Dec 30, 2016 6:28 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Feb 26, 2020 12:19 am

I have the same problem.

Device: RB1100AHx4 Dude Edition
Configuration:
- running dude server with ~320 routerboards, ~390 services (ping)
- routing 11 static routes
- firewall 7 filter rules, 14 NAT rules, ~1k connections
- no queues
- running FTP server for auto mikrotik's backup
- 3 bridges and ~ 550 hosts in network


The device was work good, and around after half year this issue appeared...

I had a serial connection, but it was also suspended. In my console window i was see:
LOOPER: read_raw read failed: EOF
died with signal
Image

I have Logging rules: critical, error and info - action: disk. In log file is nothing special, only:
Feb/25/2020 11:03:48 system,info,account user dude logged in from xx.xx.xx.247 via dude
Feb/25/2020 11:04:25 system,info,account user dude logged out from xx.xx.xx.247 via dude
----- BRICK --------
Feb/25/2020 05:28:12 system,error,critical router was rebooted without proper shutdown
Last edited by Filip92 on Fri Feb 28, 2020 7:02 pm, edited 2 times in total.
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 252
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Feb 26, 2020 8:35 am

There are services 24*7, i can't reboot it every day...

I see. But if you prefer bricking... it is better a 30 sec outage for reboot in the night....
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
inteq
Member Candidate
Member Candidate
Posts: 176
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Feb 26, 2020 9:05 am

There are services 24*7, i can't reboot it every day...

I see. But if you prefer bricking... it is better a 30 sec outage for reboot in the night....
We prefer a fix. Rebooting every night ain't one. I hope you don't do that to your users.
 
User avatar
Maggiore81
Member Candidate
Member Candidate
Posts: 252
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Wed Feb 26, 2020 9:12 am

I do. At 04 am ti 04.00.30 is not an issue...
I know that is not a fix but works. Eventually I replaced the 4011 with a ccr1009.
Dott. Elia Spadoni
---
Network Administrator,
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
omidkosari
Trainer
Trainer
Posts: 633
Joined: Fri Sep 01, 2006 4:18 pm
Location: Iran , Karaj
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Feb 27, 2020 10:15 am

We prefer a fix. Rebooting every night ain't one. I hope you don't do that to your users.


We all prefer to solve this problem. Our reports and complains to mikrotik staff did not solve the problem so we try to find a workaround.
MTCNA , MTCRE, MTCWE, Mikrotik Certified Trainer
 
seho
newbie
Posts: 36
Joined: Fri Aug 09, 2013 11:05 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Feb 27, 2020 12:49 pm

I got some supouts from our customer which has reported the problem with non responding device. One with a device running in non error state and one in error state.

Because of an NDA i'm currently not allowed to share the supout with MikroTik.

I had a look to the supout.rif locally on my pc using Mikrotik RIF decoder (https://github.com/farseeker/go-mikrotik-rif)

Unfortunately i'm pretty unexperienced interpreting all the stuff written out.

But theres something looking very suspicious to me is the "== SECTION profile", the device got a very high unclassified part in the cpu usage.
When I look at the same entries of device which is not in failure state the values are much lower.
== SECTION profile
NAME                    CPU        USAGE
total                                 0%

total                                 0%

total                                 0%

firewall                            1.2%
management                          3.1%
traffic-accou...                    2.6%
unclassified                       21.5%
total                              28.4%

NAME                    CPU        USAGE


management                1         0.5%
traffic-accou...          1           3%
unclassified              1        16.5%
cpu1                                 20%

firewall                  0           0%
management                0          10%
traffic-accou...          0          23%
unclassified              0       146.5%
cpu0                              179.5%
management                1         0.5%
traffic-accou...          1           3%
unclassified              1        16.5%
cpu1                                 20%
firewall                  2        25.5%
networking                2         0.5%
management                2        14.5%
unclassified              2           6%
cpu2                               46.5%
firewall                  3           3%
management                3           3%
unclassified              3           0%
cpu3                                  6%
Could someone who encountered the same problem, and have a supout of the device in failure, please have a look at a supout.rif in failure state? And report back if the section at their failure device has a similar high "unclassified" cpu usage.

Maybe this has nothing to do with the problem at all. I read that the unclassified could came from encryption.
 
allevot
just joined
Posts: 7
Joined: Fri Oct 19, 2018 2:44 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Feb 27, 2020 4:59 pm

One of my devices has an autouport.rif, it was created on 24-02-2020, the same day Mikrotik bricked.
The Supout.rif Reader shows this:
MEMORY
MemTotal:        1034864 kB
MemFree:          966468 kB
Buffers:               0 kB
Cached:            27688 kB
SwapCached:            0 kB
Active:            10596 kB
Inactive:          22656 kB
Active(anon):       5664 kB
Inactive(anon):       64 kB
Active(file):       4932 kB
Inactive(file):    22592 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          5636 kB
Mapped:             5508 kB
Shmem:               160 kB
Slab:              22976 kB
SReclaimable:       3192 kB
SUnreclaim:        19784 kB
KernelStack:         672 kB
PageTables:          412 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      517432 kB
Committed_AS:      11020 kB
VmallocTotal:    1024000 kB
VmallocUsed:        6696 kB
VmallocChunk:     973848 kB
CPU
Processor	: ARMv7 Processor rev 4 (v7l)
processor	: 0
BogoMIPS	: 2793.47

processor	: 1
BogoMIPS	: 2793.47

processor	: 2
BogoMIPS	: 2793.47

processor	: 3
BogoMIPS	: 2793.47

Features	: swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc0f
CPU revision	: 4

Hardware	: AnnapurnaLabs Alpine (Device Tree)
Revision	: 0000
Serial		: 0000000000000000
 
angriukas
Frequent Visitor
Frequent Visitor
Posts: 76
Joined: Fri Nov 22, 2013 9:20 am
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 28, 2020 1:31 pm

10 days ago installed two RB4011iGS+ devices.
Firewall, 3 bridges, CAPsMAN with two SSID's, EoIP between both devices, few queues.
No bricks, no reboots, no issues at all.
RoS and firmware: 6.45.8
 
User avatar
fposavec
newbie
Posts: 40
Joined: Thu Apr 21, 2011 12:59 pm
Location: Čakovec
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 28, 2020 1:52 pm

[XXX@YYY] > system resource print 
             uptime: 35w2d21h45m51s
            version: 6.44.3 (stable)
         build-time: Apr/23/2019 12:37:03
   factory-software: 6.43
        free-memory: 904.0MiB
       total-memory: 1024.0MiB
                cpu: ARMv7
          cpu-count: 4
      cpu-frequency: 1400MHz
           cpu-load: 9%
     free-hdd-space: 418.4MiB
    total-hdd-space: 512.3MiB
  architecture-name: arm
         board-name: RB4011iGS+
           platform: MikroTik
no issue at all.
BGP routing, NAT, DHCP...
 
Overide
just joined
Posts: 6
Joined: Tue Jul 03, 2018 5:00 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 28, 2020 5:49 pm

We've been dealing with this issue since we began deploying 4011's

We use virtual bridges, pppoe, DCHP and ospf. not mush else. this issue has never occurred for us on any other hardware platform.
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1183
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 28, 2020 6:09 pm

We've run into this with a few clients that run the 4011 and have used the following as a workaround while we work on a ticket with MikroTik. It seems to be more stable in the few we have tested.

Set the CPU frequency to 1200mhz
system routerboard settings set cpu-frequency=1200
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
ingdaka
Member Candidate
Member Candidate
Posts: 289
Joined: Thu Aug 30, 2012 3:06 pm
Location: Albania
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Feb 28, 2020 8:13 pm

I have 3 of them and I've not faced this problem never! Uptime more than 1 year running without any problem with normal 700Mbps to 1G of traffic!
Ilir Daka
Electronic & Network Engineer
E-mail: ilirdaka@live.com
Mob: +355692982151
WhatsApp: +355692982151
Mikrotik Official Consultant
CCNA | Fortinet NSE3 | MTCRE | MTCSE | MTCWE
 
anschluss
just joined
Posts: 6
Joined: Fri Mar 30, 2018 3:46 pm

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Mar 02, 2020 4:09 pm

We've run into this with a few clients that run the 4011 and have used the following as a workaround while we work on a ticket with MikroTik. It seems to be more stable in the few we have tested.

Set the CPU frequency to 1200mhz
system routerboard settings set cpu-frequency=1200
Thanks very much for this helpful hint! The RB4011 has been acting up along the lines described in this thread, with intervals of running seemingly normal for weeks, then days, then hours. Until I downclocked the CPU, the device started jettisoning all routing functions even after just a minute's connection, such as a phone call, a VPN or remote desktop connection.
 
User avatar
cpbruton
just joined
Posts: 12
Joined: Fri Jun 07, 2013 4:23 am
Location: Pasadena, California, USA
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Mon Mar 02, 2020 10:10 pm

I have encountered this issue with two different RB1100AHx4s. Both routers became inaccessible after a few weeks deployed. Forwarding and OSPF continued to work fine so we had no interruption in our network traffic, but could not log into the routers via web, ssh, mac-telnet, or direct console connection. A physical power cycle restored access.

Both routers have watchdog enabled but only one of them generated a partial autosupout.rif file, with the following message at the top:
--- nv::message --------
died with signal 6 on Sun Feb 23 15:02:12 2020
Both were running 6.45.8 (long-term). I downgraded to 6.44.6 before finding this thread, but it sounds like people have encountered this on older versions too.

I will try the suggestion to reduce CPU frequency to 1200MHz.
 
aoakeley
newbie
Posts: 47
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Tue Mar 03, 2020 4:03 pm


After obtaining supout.rif files by console cable, MikroTik have organised warranty replacement for the two 1100 units exhibiting this issue.
And yet another 1100AHX4 with this issue.
The previous 2 units which were replaced under warranty have not had a recurrence of the issue.
 
User avatar
omidkosari
Trainer
Trainer
Posts: 633
Joined: Fri Sep 01, 2006 4:18 pm
Location: Iran , Karaj
Contact:

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Thu Apr 02, 2020 6:40 pm

Hello

I see that your router did a reboot because of Kernel Watchdog.

For debugging you should turn off watchdog and test again.
"/system watchdog set watchdog-timer=no"

Connect now to your router over serial console, make sure that you have accessed RouterOS command line interface and leave console running.

Now router either:

1) might be stuck (freeze). If that happens, then you have to generate a supout file on the router through a serial console. Now you can reboot the device.

2) might be stuck (freeze) and become unavailable over a serial console. If that happens, then reboot the router and then generate a supout file.

3) will reboot. If that happens, then after reboot manually generate a supout file.

Send supout file and full serial console output (within text file) to us for investigation.

Before that, I suggest that you upgrade to the latest "stable" version (if there is an actual bug, then it might be already fixed).

Best Regards


This is a reply from mikrotik support. Has anyone done this ?
MTCNA , MTCRE, MTCWE, Mikrotik Certified Trainer
 
aoakeley
newbie
Posts: 47
Joined: Mon May 21, 2012 11:45 am

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Apr 03, 2020 10:00 am


This is a reply from mikrotik support. Has anyone done this ?
Yes. see my previous posts
 
jebz
Member Candidate
Member Candidate
Posts: 260
Joined: Sun May 01, 2011 12:03 pm
Location: Australia

Re: RB4011 and RB1100 AHx4 "bricks" randomly

Fri Apr 03, 2020 2:36 pm

And yet another 1100AHX4 with this issue.
The previous 2 units which were replaced under warranty have not had a recurrence of the issue.
.
Have you had any feedback on what the fault was?

Who is online

Users browsing this forum: MSN [Bot], sistelec, volkirik and 141 guests