Community discussions

MikroTik App
 
terintamel
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sun Sep 05, 2010 3:09 am

Replaced 5yr old x86 with CHR (less than impressed)

Sat Apr 20, 2019 6:23 pm

I just replace my Linktechs Powerrouter V3 (https://www.streakwave.com/itemdesc.asp?ic=PRV3) bought in 2014 with the Vengeance as the Powerrouter has major interface issues after the upgrade to ROS 6.x from 5.x and the mfg says these routers are not supported above 5.x. Basically the sfp interfaces would sometimes just stop passing data and the only fix was to reboot the router. Very bad as this is my border router for our ISP.

Well with the PowerRouter I would max out at around 40% CPU usage which was pushing around 1.4Gbps of traffic. This was an 4 core router (3.4ghz older I-7 running hyperthreading so ROS saw it as an 8 core).

With the Vengeance running the exact same config I seem to have higher CPU usage at lower traffic levels. Is this because of the lack of hyperthreading on the CHR router or something else? When I did an internal UDP Btest to 127.0.0.1 on the CHR prior to deployment it maxed out at 27Gbps so I know it can push some data.
 
terintamel
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sun Sep 05, 2010 3:09 am

Re: Replaced 5yr old x86 with CHR (less than impressed)

Sat Apr 20, 2019 6:27 pm

After having the vengeance in for the last 11 hours I will say I am not as impressed with the performance as I thought I would be. While I expect is will be more stable than my older router it still seems to have much higher CPU usage than my 5 year old router with the same config.

Is there something I have setup poorly at the VMware level that could explain this or could my ROS configuration just not work well on CHR?

Attached is my CPU usage prior and post Vengeance install. I am hitting periods of extended 50+% usage during non peak hours where I usually would only hit 40-50% usage during peak hours.
Capture.PNG
Also attached is the traffic usage both before and after the swap. I have two BGP peers getting full tables.
Capture2.PNG
You do not have the required permissions to view the files attached to this post.
 
terintamel
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sun Sep 05, 2010 3:09 am

Re: Replaced 5yr old x86 with CHR (less than impressed)

Sat Apr 20, 2019 6:29 pm

I went with the Vengeance (x86 CHR) due to my success with an x86 ROS router before. Also I went with it over the CCR1036 as I needed at least 3 sfp+ ports and I went with it over the CCR1072 due to wanting the faster BGP performance as I take 2 full tables. Now I wonder if that was a mistake.
 
terintamel
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sun Sep 05, 2010 3:09 am

Re: Replaced 5yr old x86 with CHR (less than impressed)

Sat Apr 20, 2019 6:44 pm

Looking at the CPU profile in Winbox it seems the every few minutes one of the 4 core goes from 20-30% usage to 100% usage. This corresponds with a sudden increase in the routing process. Not sure the cause as there was no sudden spike in traffic.
You do not have the required permissions to view the files attached to this post.
 
terintamel
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sun Sep 05, 2010 3:09 am

Re: Replaced 5yr old x86 with CHR (less than impressed)

Sat Apr 20, 2019 6:47 pm

So it looks like most of the time 3 out of the 4 cores show little usage on the Routing process. Is the routing process multi-threaded? What could cause one core to be randomly taking on almost all of the routing duty?

Routing process
Capture8.PNG
Routing process just a few seconds later with no major jump in passed bandwidth.
Capture7.PNG
You do not have the required permissions to view the files attached to this post.
 
terintamel
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sun Sep 05, 2010 3:09 am

Re: Replaced 5yr old x86 with CHR (less than impressed)

Sat Apr 20, 2019 6:54 pm

Capture10.PNG
Capture11.PNG
You do not have the required permissions to view the files attached to this post.
 
terintamel
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sun Sep 05, 2010 3:09 am

Re: Replaced 5yr old x86 with CHR (less than impressed)

Sat Apr 20, 2019 10:25 pm

I'm sorry to say I have not been so disappointed with an ~$3000 router before. Here I am in the afternoon pushing around less data and using 20-30% more CPU power to do it. Is this really the best this flagship router can do? I mean I am pushing no more than 1Gbps of internet traffic and the CPU is running between 50-70%. Before the older router would be no more than 30% average with some jumps to 40-45% on occasion.

So either my 5 year old I7 4 core router (running as 8 w/hyperthreading) on bare metal is better hardware than the Vengeance, or CHR w/ESXI is just this bad?
 
InoX
Forum Guru
Forum Guru
Posts: 1966
Joined: Tue Jan 09, 2007 6:44 pm

Re: Replaced 5yr old x86 with CHR (less than impressed)

Sat Apr 20, 2019 11:45 pm

I remember that Mikrotik recomends disable the HT, don't remember why but lower than real cpu usage might be the reason. More cores, less total cpu usage...
Last edited by InoX on Sun Apr 21, 2019 12:02 am, edited 2 times in total.
 
terintamel
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sun Sep 05, 2010 3:09 am

Re: Replaced 5yr old x86 with CHR (less than impressed)

Sat Apr 20, 2019 11:51 pm

Sorry. I just re-read and realized I did not explain.
https://www.balticnetworks.com/maxxwave ... ption.html
 
Samot
Member Candidate
Member Candidate
Posts: 113
Joined: Sat Nov 25, 2017 10:01 pm

Re: Replaced 5yr old x86 with CHR (less than impressed)

Sun Apr 21, 2019 7:52 pm

OK so did you install X86 or did you install CHR? There is no X86 CHR, they are not the same. X86 is meant for physical hardware and binds itself to the HDD much like the ROS appliances. Follows the same levels of licensing and limitations of that licensing. CHR is a completely different beast. It's meant for virtual machines and nothing physical. It doesn't bind to the HDD and it can be moved between VMs if needed. It also has completely different licensing and levels.

So are you saying you bought a physical machine and just put CHR on it as the OS?
 
terintamel
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sun Sep 05, 2010 3:09 am

Re: Replaced 5yr old x86 with CHR (less than impressed)

Sun Apr 21, 2019 8:01 pm

Bought a physical machine that came with CHR running on it on top of Esxi 6.5. Sorry for not being clear.

It replaced a 5yr old x86 Mikrotik server.

Product Specifications:
• CPU: LGA1151 socket for 7th Generation Intel® Core™ i7 processor
• Processor: Kaby Lake 7th Gen. i7-7700. Speed: 3.6GHz/8M
• Chipset: Intel® C236
• RAM: 32GB 2 x 16GB DDR4-2400
• SSD: 500GB mSata Samsung Evo
• Ethernet: Eight 10/100/1000 Auto MDI-X (4 bypass)
• Expansion: PCI-E 8x, MiniPCI slot
• USB: 2 x 3.0 ports on front
• Power: 1U redundant 280 W power supply (1+1)
• Operating System: VMware OSXi 6.X (free edition) / MikroTik RouterOS CHR P10 (10Gig) as a VM
 
Samot
Member Candidate
Member Candidate
Posts: 113
Joined: Sat Nov 25, 2017 10:01 pm

Re: Replaced 5yr old x86 with CHR (less than impressed)

Sun Apr 21, 2019 9:04 pm

OK, that makes much more sense. Here check out this post, viewtopic.php?t=138549

The basic result is turn off hyperthreading on the host system and the guest VMs will improve CPU performance. All the reasoning is detailed in that post.
 
terintamel
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sun Sep 05, 2010 3:09 am

Re: Replaced 5yr old x86 with CHR (less than impressed)

Wed Apr 24, 2019 3:20 am

Hyperthreading is off and the box came that way from BalticNetworks. However they setup the CHR to use 4 virtual cores, but the host only has 4 physical cores. Would I get better performance if I set the CHR to use 3 cores which would leave 1 core always open to the hypervisor than having the CHR use all 4 cores?

Also I have questions about CHR CPU usage vs Hypervisor usage. So if I graph the hypervisor CPU usage it pretty much keeps a steady 60-70% at peak hours and 40-50% at non peak hours, but the CHR shows average CPU usage of 40-60% in peak hours and as low a 15-20% at off peak hours. Which CPU % should I pay the most attention too? Hypervisor or CHR (winbox)? Should I watch Hypervisor and CHR (exsi reported usage) instead?
 
User avatar
Steveocee
Forum Guru
Forum Guru
Posts: 1120
Joined: Tue Jul 21, 2015 10:09 pm
Location: UK
Contact:

Re: Replaced 5yr old x86 with CHR (less than impressed)

Tue Apr 30, 2019 10:57 am

Maybe a conversation with Baltic networks as to what they'd expect to see?
If there are fundamental issues with the machine or the way it is set up, all you are doing is wasting your time trying to chase a potentially non existent problem around a user forum.
 
hedele
Member
Member
Posts: 338
Joined: Tue Feb 24, 2009 11:23 pm

Re: Replaced 5yr old x86 with CHR (less than impressed)

Tue Apr 30, 2019 1:04 pm

To be honest, I'm not that surprised by your results. First of all, VMware is less performant for CHR installations than Hyper-V, so this machine is not using the optimal hypervisor. Also the CPU in this new router is essentially the same as the one in your old router (Intel Core i7 with 4 physical cores), so it's probably got at best around 15-20% more raw performance compared to your old router but now additionally has to handle the hypervisor and virtual networking overhead which seems to lead to a net loss of performance for you.

The periodic 100% on one core load is caused by the BGP table scan which is run in set intervals and is normal behaviour if you have one or more full bgp tables on the router. This is less impactful on Hyper-V than on VMware if i remember the presentation about CHR hypervisors correctly.

So basically, I get the impression that you just bought the wrong tool for the job (or were consulted poorly).

Who is online

Users browsing this forum: No registered users and 9 guests