Community discussions

MikroTik App
 
MARK47
just joined
Topic Author
Posts: 6
Joined: Tue May 31, 2022 7:06 am

CCR1072 only one core 100% CPU usage

Tue May 31, 2022 7:57 am

One of the processor cores is constantly under high load, in the tools profile this core is used by the management process and it doesn't depend on the amount of traffic.
Total CPU load 10-50%.
The network is working fine, but management of mikrotik is difficult, commands are executed for a long time, sometimes it doesn't respond to snmp.
I read about similar problems and often the cause was SNMP, but I tried to disable SNMP and there was no result. The message sometimes appears in the logs "snmp/warning timeout while waiting for program 20"
What is configured on CCR1072 (RouterOS 6.49.5):
3984 NAT rules (src-nat 1:1)
8 filters (fasttrack connection enabled)
1 mangle
290 vlans
313 ip addresses on mikrotik's interfaces
317 ip routes (21 via ospf)
5 OSPF neighbors
1 bonding
NO scripts
NO queues
NO BGP
NO Hotspot
NO tunnels and encryption (IPSec/L2TP etc.)
What I tried and it didn't solve the problem:
- disabled mangle rule
- added "no track" RAW rules for clients who don't use NAT
- disabled dhcp-relays
- disabled SNMP
- disabled filters rules
- MAC-Server, MAC-WinBox only from ether1
- discovery interface: none
- disabled all ip-services
- disabled DNS allow remote requests, filtered 53/UDP
- disabled LCD
What other process can use "management" like this?
You do not have the required permissions to view the files attached to this post.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 3005
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CCR1072 only one core 100% CPU usage

Tue May 31, 2022 5:05 pm

to reduce management usage first suggestion is:

in a bussy router i suggest to have "good practices" in winbox usage

winbox is a powerfull tool but if you aks to many element or too much info to be displayed that will be exhausting for the router

try to avoid to "show" winbox sections with many elements, if you need to display some heavy window once you do that you need this close all winbox internal windows and tabs and close any winbox session on that router, always start winbox with a empty window

sometimes takes upto 30 minutes for management usage to lower after you display busy elements on winbox

i think beyond single most busy core problem you have a very busy machine for that ammount of traffic in fast-track

in "normal" conditions ccr1072 moves that traffic in fast-track with single digit cpu usage

try profile selecting CPU - > total to spot other load contributors

and off course consider posting your config (anonymize sensible info)

to spot possible fix/optimize opportunities
profile-total.png
You do not have the required permissions to view the files attached to this post.
 
MARK47
just joined
Topic Author
Posts: 6
Joined: Tue May 31, 2022 7:06 am

Re: CCR1072 only one core 100% CPU usage

Wed Jun 01, 2022 11:39 am

to reduce management usage first suggestion is:

in a bussy router i suggest to have "good practices" in winbox usage

winbox is a powerfull tool but if you aks to many element or too much info to be displayed that will be exhausting for the router

try to avoid to "show" winbox sections with many elements, if you need to display some heavy window once you do that you need this close all winbox internal windows and tabs and close any winbox session on that router, always start winbox with a empty window

sometimes takes upto 30 minutes for management usage to lower after you display busy elements on winbox

i think beyond single most busy core problem you have a very busy machine for that ammount of traffic in fast-track

in "normal" conditions ccr1072 moves that traffic in fast-track with single digit cpu usage

try profile selecting CPU - > total to spot other load contributors

and off course consider posting your config (anonymize sensible info)

to spot possible fix/optimize opportunities

profile-total.png
I tried disabling winbox (in ip services) and logging in via telnet/ssh and still saw one core at 100%.
I have an assumption that such a large load on management is due to counters on the interfaces
You do not have the required permissions to view the files attached to this post.
 
BabyTeanofred
just joined
Posts: 2
Joined: Sat Apr 30, 2022 3:03 pm

Re: CCR1072 only one core 100% CPU usage

Wed Jun 01, 2022 1:06 pm

I run X86 routeros with ~10000 queues for 2500 customers (4 queues per customer for different priorities). I'm using a fairly old HP dual xeon 2x6 core server (ebay special), and a intel 10gb nic. Customers push just under 3Gb of traffic during peak times and the CPU load never goes much above 25%. I used to have a CCR but I was having load issues, especially when gathering data from the queues using the API. X86 absolutely smokes the tile CPU. Be aware that X86 routers is 32bit so any ram above 2GB isn't used. Not a problem when just doing stuff like queues and basic routing though.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 3005
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CCR1072 only one core 100% CPU usage

Wed Jun 01, 2022 5:00 pm

Maybe You need to consider optimize such a large NAT rule set
 
MARK47
just joined
Topic Author
Posts: 6
Joined: Tue May 31, 2022 7:06 am

Re: CCR1072 only one core 100% CPU usage

Thu Jun 23, 2022 11:23 am

Continuation of the story about the high load of one CPU core. I did a lot of work to remove the NAT rules, now there is no NAT and connection tracking is disables. But this didn't solve the problem, even at the most unloaded time, when the total CPU load is ~10% and 3Gbps of traffic - all the same, IP lists, interfaces, filter rules are not loading in winbox and telnet, the router becomes unmanageable, although traffic passes normally, there is no packet loss
You do not have the required permissions to view the files attached to this post.
 
User avatar
rushlife
Member Candidate
Member Candidate
Posts: 245
Joined: Thu Nov 05, 2015 12:30 pm

Re: CCR1072 only one core 100% CPU usage

Thu Jun 23, 2022 11:44 am

What about temperatures ? It is not just thermal throttling ?
 
MARK47
just joined
Topic Author
Posts: 6
Joined: Tue May 31, 2022 7:06 am

Re: CCR1072 only one core 100% CPU usage

Thu Jun 23, 2022 11:50 am

CPU temperature 46 С
You do not have the required permissions to view the files attached to this post.
 
MARK47
just joined
Topic Author
Posts: 6
Joined: Tue May 31, 2022 7:06 am

Re: CCR1072 only one core 100% CPU usage

Thu Jun 23, 2022 11:59 am

Another thing I noticed, for some reason there are some running processes in System Script Jobs, but scripts are not used on this router. If I delete these processes, then they appear again
You do not have the required permissions to view the files attached to this post.
 
Kindis
Member
Member
Posts: 434
Joined: Tue Nov 01, 2011 6:54 pm
Location: Sweden

Re: CCR1072 only one core 100% CPU usage

Thu Jun 23, 2022 12:44 pm

At this point I would do a full export of the config, review it in full so you don't have any bad stuff in there.
Then netinstall to the same version you have, secure the router and import the same config. My guess at this point it will be quicker and hopefully works if there are issues with the install.
Also do a full backup just in case.
 
MARK47
just joined
Topic Author
Posts: 6
Joined: Tue May 31, 2022 7:06 am

Re: CCR1072 only one core 100% CPU usage

Thu Jun 23, 2022 12:52 pm

At this point I would do a full export of the config, review it in full so you don't have any bad stuff in there.
Then netinstall to the same version you have, secure the router and import the same config. My guess at this point it will be quicker and hopefully works if there are issues with the install.
Also do a full backup just in case.
Unfortunately, exporting the config doesn't work either, just as the supout.rif is not created, but a backup is working.
Since now this router handles a lot of traffic, I can't do experiments with rebooting
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 3005
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CCR1072 only one core 100% CPU usage

Thu Jun 23, 2022 6:33 pm

check
system scheduler
for suspicious automated tasks

also check
system users
check
ip services
for services enabled, disable what you can and change default port for which you need
 
ubikrotik
Member Candidate
Member Candidate
Posts: 158
Joined: Wed May 25, 2016 3:56 am

Re: CCR1072 only one core 100% CPU usage

Sat Aug 12, 2023 5:34 am

@MARK47 did you fix your issue? We are experencing the same problem running 6.45.9 on a rb4011. for a long moment cpu is low then one cpu goes crazy and management process too.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 564
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR1072 only one core 100% CPU usage

Sat Aug 12, 2023 7:41 am

Did you try to upgrade to v7 ?
 
ubikrotik
Member Candidate
Member Candidate
Posts: 158
Joined: Wed May 25, 2016 3:56 am

Re: CCR1072 only one core 100% CPU usage

Sat Aug 12, 2023 5:21 pm

not ready to dive into ros7 for now, I upgraded to the last LTS ros6 instead.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 564
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR1072 only one core 100% CPU usage

Sun Aug 13, 2023 12:17 pm

I think you may have a sort of malaware on it if you have some scheduled tasks.
you should try to enter in console and then make an export, maybe only at the scripts part, so we can have a look.
You should definitely disconnect it from the network, maybe in the night, then examinate the router.
remove or disable all scheduled tasks, logins, then upgrade to the latest release.

Who is online

Users browsing this forum: PBondurant and 74 guests