Community discussions

 
Michaelcrapse
just joined
Topic Author
Posts: 19
Joined: Wed Oct 25, 2017 9:57 pm

100 % CPU on some Routerboards

Tue Oct 08, 2019 10:24 pm

I submitted this issue to support@mikrotik.com i think 3 times now, and have never heard back from them on the issue.

After running a firmware update, in fact any firmware past 6.42.x causes multiple devices to lock up, and causes the DHCP server to not start, the CPU is at 100% and barely passes any traffic. The logs show unable to start a program. Note: only 4 out of hundreds so far, but it's still devices that should be working well.

The issue at this point is that Sonar cannot connect to these devices because they're a lower firmware version not using the new authentication methods. Which i'm assuming is the root of this problem. I wonder if mikrotik can help us fix these or get them RMAd.
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5829
Joined: Mon Jun 08, 2015 12:09 pm

Re: 100 % CPU on some Routerboards

Tue Oct 08, 2019 11:49 pm

Have you already tried to use netinstall to format and reinstall them to the current version with blank configuration, then re-configure them to your needs?
Don't use backup/restore. At most do a /export first and then use that to re-configure them manually or by pasting sections of the /export that you first have examined for any strange things.

When there is old firmware on it and it misbehaves, it most likely has been hacked and there now is malware running on it.
 
mistry7
Forum Guru
Forum Guru
Posts: 1314
Joined: Tue Oct 13, 2009 11:57 am
Location: Germany

Re: 100 % CPU on some Routerboards

Wed Oct 09, 2019 6:09 am

We saw this on old 300 MHz RBs and sometimes on 400MHz RB411
 
Michaelcrapse
just joined
Topic Author
Posts: 19
Joined: Wed Oct 25, 2017 9:57 pm

Re: 100 % CPU on some Routerboards

Wed Oct 09, 2019 6:58 pm

Have you already tried to use netinstall to format and reinstall them to the current version with blank configuration, then re-configure them to your needs?
Don't use backup/restore. At most do a /export first and then use that to re-configure them manually or by pasting sections of the /export that you first have examined for any strange things.

When there is old firmware on it and it misbehaves, it most likely has been hacked and there now is malware running on it.
Yes, just tried this now. Issue seems to persist.
These units were definitely not hacked. No possible way for that to occur. Access lists etc. and access monitoring logs show no unauthorized accesses.
 
Michaelcrapse
just joined
Topic Author
Posts: 19
Joined: Wed Oct 25, 2017 9:57 pm

Re: 100 % CPU on some Routerboards

Wed Oct 09, 2019 6:59 pm

We saw this on old 300 MHz RBs and sometimes on 400MHz RB411
This has happened on powerbox pros so far, and rb850gx2
 
User avatar
doneware
Trainer
Trainer
Posts: 520
Joined: Mon Oct 08, 2012 8:39 pm
Location: Hungary

Re: 100 % CPU on some Routerboards

Thu Oct 10, 2019 1:06 am

make sure there's nothing attached to the serial console if the device has one.
this was a very hard issue to track down: we had devices with long serial cables connected to the router and the other end of the cable (~10m long) wasn't connected to anything. it picked up some EM noise and constantly trickled the serial port thus the serial console.
serial console is an interrupted device in RouterOS, so it will bully the cpu all the time. do a /sys resource irq print to see what you have there:
[admin@hgw] > /sys resource irq print 
Flags: ro - read-only 
 #    IRQ USERS                                          CPU ACTIVE-CPU         COUNT
 0      3 usb1                                          auto          0            65
 1      4 switch0                                       auto          0 3 179 276 774
 2     16 beeper                                        auto          0           797
 3 ro  19 serial                                        auto                       18
 4    112 ts                                            auto          0             0
the values next to serial should be low. if it is high and increasing rapidly, you have the same issue as we had.
then you just need to disconnect the cable...
#TR0359
 
User avatar
krafg
Member
Member
Posts: 311
Joined: Sun Jun 28, 2015 7:36 pm

Re: 100 % CPU on some Routerboards

Thu Oct 10, 2019 1:21 pm

I also had problems with DHCP using ROS 6.42.x. I updated to 6.43 and the problem was solved.

Regards.
Image
 
pe1chl
Forum Guru
Forum Guru
Posts: 5829
Joined: Mon Jun 08, 2015 12:09 pm

Re: 100 % CPU on some Routerboards

Thu Oct 10, 2019 2:19 pm

make sure there's nothing attached to the serial console if the device has one.
this was a very hard issue to track down: we had devices with long serial cables connected to the router and the other end of the cable (~10m long) wasn't connected to anything. it picked up some EM noise and constantly trickled the serial port thus the serial console.
From the days we worked with Unix systems and serial terminals I remember well that a long RS232 cable can sometimes "echo" everything.
There is capacitance between the RX and TX lead and this is enough to couple everything sent to the port back to the RX, especially at high baudrates.
When this occurs, it can happen that an intiial banner is received back and treated as a response to the login prompt, this results in an error message, and that again is used as the next login.
In such cases you should see log messages about invalid logins.
 
Michaelcrapse
just joined
Topic Author
Posts: 19
Joined: Wed Oct 25, 2017 9:57 pm

Re: 100 % CPU on some Routerboards

Tue Oct 15, 2019 10:57 pm

I also had problems with DHCP using ROS 6.42.x. I updated to 6.43 and the problem was solved.

Regards.
My issue is that I cannot upgrade past 6.42.x without crippling these devices.
 
Michaelcrapse
just joined
Topic Author
Posts: 19
Joined: Wed Oct 25, 2017 9:57 pm

Re: 100 % CPU on some Routerboards

Tue Oct 15, 2019 10:59 pm

make sure there's nothing attached to the serial console if the device has one.

the values next to serial should be low. if it is high and increasing rapidly, you have the same issue as we had.
then you just need to disconnect the cable...
Thanks for the answer, we don't have serial cables connected, and the serial doesn't show up in our profiler, so it's not the issue

Who is online

Users browsing this forum: No registered users and 5 guests