Community discussions

MUM Europe 2020
 
meetriks2
newbie
Topic Author
Posts: 26
Joined: Wed May 07, 2014 12:18 am

CPU load v6 much higher than v5 same router same traffic

Wed May 07, 2014 12:19 am

I will try to keep a long story short. It all started with the following patron with routeros v6.7
Image
In Green the uptime the other colors are the load of each core.
(notice at first I only monitored the first core add mid-march the other cores)

Conclusion, the system gets busier when the uptimes gets higher.


Let’s look at the traffic patrons during that period.
Image
Image
Not the same patron, doesn’t grow and goes low again like the cpu load.

If I magnify the memory use vs uptime during the same period I get this
Image
Same like cpu load. I know it's not a lot of memory difference, but same patron.


To solve the issue I tried some more v6 version:
Image
Strange things, I want to understand why the cpu load rises and the traffic doesn’t. Downgrade to v5.26 show the old trends I always say.


Image
Long term trend view… (please ignore the DDOS period)


The router is running about 8 bgp peers, 2 ospf peers, 300k bgp routes, 50 ospf routes, no conntrack, x86, 20 forward firewall rules. I can’t help that I think something is going wrong. I understand that v5 and v6 report a different value for the same traffic but it should be linked to the load / use and should not build up when the uptime becomes more. The problem is not really an issue since the router can handle the load.

I have had routertable corrupting in v5.xx and thats why i want to run v6..

I will try keep you all informed how things go. Any tips are more then welcome.
 
meetriks2
newbie
Topic Author
Posts: 26
Joined: Wed May 07, 2014 12:18 am

Re: CPU load v6 much higher than v5 same router same traffic

Thu May 08, 2014 10:47 am

Hi

As you can see runing again v5.26 is using a lot less CPU.
It's now running 10 day this version.

Image

Cheers!
 
meetriks2
newbie
Topic Author
Posts: 26
Joined: Wed May 07, 2014 12:18 am

Re: CPU load v6 much higher than v5 same router same traffic

Sat May 17, 2014 8:18 am

Hi All,

I just upgraded to 6.13, it looks okay. (just like v6.7)
Will keep it running until load builds up again, or hopefull everthing is fine now.

Image
As you can see no significant load change, it looks less busy, which is normal since it is outside office hours.

Here a graph of the 5.26 running for 20 days
Image
As you can see 5.26 uses less the other cores but is very smooth with the load

I will keep you all posted. If v6.13 get busy I will show some more winbox screenshot.

Cheers!
 
jarda
Forum Guru
Forum Guru
Posts: 7602
Joined: Mon Oct 22, 2012 4:46 pm

Re: CPU load v6 much higher than v5 same router same traffic

Sat May 17, 2014 9:14 am

As I realized the upgrades are not always as clean as they look. Export config, remove config, netinstall latest, remove config again and import the saved original config could help with many weird things.
 
meetriks2
newbie
Topic Author
Posts: 26
Joined: Wed May 07, 2014 12:18 am

Re: CPU load v6 much higher than v5 same router same traffic

Mon Jun 02, 2014 7:56 pm

As I realized the upgrades are not always as clean as they look. Export config, remove config, netinstall latest, remove config again and import the saved original config could help with many weird things.
You are right, I should test that. I was hoping that it was not needed and solved with the latest release (v6.13).
But it seems not.

Thanks for your advice.

Cheers,
Harry
 
meetriks2
newbie
Topic Author
Posts: 26
Joined: Wed May 07, 2014 12:18 am

Re: CPU load v6 much higher than v5 same router same traffic

Mon Jun 02, 2014 7:57 pm

Hi all,

Well it seems v6.13 suffers from the same problem as v6.7.
Load is become higher. I will let it run for a couple of days / weeks depending on the drops I see.

Image
Load is better devide over de cores, but last 15 hours is become high again


Image
Memory is also still increasing but this is not my problem. Plenty of memory available

Image
Checked the number of packets and througput nothing to explain the trend deviation

I will do more research this week and try to post some winbox screenshots / details.

Cheers,
Harry
 
phuang3
newbie
Posts: 40
Joined: Thu Apr 21, 2011 3:39 am

Re: CPU load v6 much higher than v5 same router same traffic

Wed Jun 04, 2014 7:07 am

I also notice that V6 has higher CPU usage. Besides, hardware switch performance is also affected. On V5.X, I can get over 100MB/sec between different LANs. With V6.x, the highest throughput is around 70~80MB/sec. (By moving huge file between Windows shares)
 
User avatar
tomaskir
Trainer
Trainer
Posts: 1122
Joined: Sat Sep 24, 2011 2:32 pm
Location: Slovakia

Re: CPU load v6 much higher than v5 same router same traffic

Thu Jun 05, 2014 1:48 pm

1) you should probably contact the official MikroTik support, this seems like an issue for them to investigate.
2) what do you use to generate these graphs, particularly the pps one is very nice :)
Unimus - configuration management, automation and backup solution
Mass Config Push, network-wide RouterOS upgrades, and more!
 
meetriks2
newbie
Topic Author
Posts: 26
Joined: Wed May 07, 2014 12:18 am

Re: CPU load v6 much higher than v5 same router same traffic

Thu Jun 05, 2014 1:54 pm

Hi All,

Check this

Image
It still rising as you can see. Traffic is not different. (max is 400% for this graph)

Image
This is the sum of all cores deviced by the sum of the core, so 100% means full cpu load.

Lets check winbox. A couple of times per minut is will show like this
Image

Notice the following:

- CPU graph is different than what I get with SNMP.
- A single core is 100%
- Profile shows that routing is wasting the power
- See the low mbit and packet a second. Notice de RX Drops.

My planned actions:
- wipe the router and do a full reinstall v6.13 to prevent upgrade problems
- I should at least reboot the router soon due to drop count which rarely increases with v5.x but with v6.x quite easy.


@Kreacher, I understand a new OS higher load on same hardware doing the same. But my problem is that it increase after some weeks for unknown reasons
Hardware used: NA-330 http://www.axiomtek.com/Products/ViewPr ... p?view=913, Not bad, not super, it works for nice price.

Anyone some advice?

Cheers!
Last edited by meetriks2 on Fri Jul 04, 2014 1:20 pm, edited 1 time in total.
 
meetriks2
newbie
Topic Author
Posts: 26
Joined: Wed May 07, 2014 12:18 am

Re: CPU load v6 much higher than v5 same router same traffic

Thu Jun 05, 2014 2:08 pm

1) you should probably contact the official MikroTik support, this seems like an issue for them to investigate.
2) what do you use to generate these graphs, particularly the pps one is very nice :)

I did contact support but the short answers was its still 80% idle. So no problem.
Well I decided to make a public forum post with all the fancy details and try to contact support later on.
My ticket number is 2014042766000357.

I will send them an e-mail to check this post.

About the graphs, i use SNMP and Zabbix to make the graphs.

Cheers!
 
meetriks2
newbie
Topic Author
Posts: 26
Joined: Wed May 07, 2014 12:18 am

Re: CPU load v6 much higher than v5 same router same traffic

Mon Jun 16, 2014 12:28 pm

Hi All,

I got responses from Mikrotik support, the important part:
V5 had routing table corruption. To fix this as workaround we introduced frequent routing table refresh and also
recalculation on each route update in v6. Unfortunately there is nothing much we can do
to improve this behavior in current implementation. Even if routing sits at 100% cpu usage on one core it should not affect overall
performance or packet forwarding.

We are also working on new much faster implementation for RouterOS version 7,
which will fix high cpu usage, and other remported problems that cannot be fixed
in v6.
Well that's it for me. I have not a unknown problem, but is known behavior and should not impact performance for now.

Intersesting they became a litte more idle again and will probably rise again within two weeks.

Image

Image

This is my final post for this issue.
Thanks for you help.

Cheers!
 
meetriks2
newbie
Topic Author
Posts: 26
Joined: Wed May 07, 2014 12:18 am

Re: CPU load v6 much higher than v5 same router same traffic

Sat Aug 15, 2015 8:29 am

Hi all,

The patron is still the same and i'm running v6.19 for many months.
Version 6.30 introduces the route cache disable option.
What's new in 6.30 (2015-Jul-08 09:07):
*) possibility to disable route cache - improves DDOS attack handling performance up to 2x (note that ipv4 fastpath depends on route cache);
With route cache disabled no fast path. Maybe i get better performance since I run on x86 I have no fast path.

Searching around a bit I found a changelog with the process making my router busy:
What's new in 6.0rc14 (2013-Apr-24 11:52):
*) route - automatically repair FIB inconsistencies
Anyone with some information on route cache disabled?
Without route cache there are no fib inconsistencies possible?
That should free up one core checking all routes. :D
Twice the performance for DDOS. What about normal traffic?

Cheers!
 
meetriks
just joined
Posts: 18
Joined: Sun Apr 27, 2014 9:41 pm

Re: CPU load v6 much higher than v5 same router same traffic

Wed Oct 21, 2015 11:30 pm

Anyone any experience with disabling route-cache?

Cheers,
Harry

Who is online

Users browsing this forum: Deengeano, marting and 87 guests