Community discussions

MUM Europe 2020
 
vklimovs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 57
Joined: Fri Dec 16, 2005 5:37 pm

Maximum bandwidth with Quad Core and PCI-E

Wed Nov 04, 2009 4:51 pm

We are using two 3 Ghz Quad Core machines with 2 Gb of RAM as our border routers. Each machine has two interfaces, which are PCI-E Intel Gigabit cards. CPU Load never exceeds 30%. However we are unable to get more than about 650 Mbit/s full duplex on the links. We are unsure if this is our issue or issue on the side of upstream provider. So this post it to confirm that it is possible to push more than 650 Mbit full duplex through a machine like described. :) Any replies appreciated.
 
vklimovs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 57
Joined: Fri Dec 16, 2005 5:37 pm

Re: Maximum bandwidth with Quad Core and PCI-E

Wed Nov 04, 2009 11:10 pm

OK, not a single person is using that setup? If you are using any other setup for high-bandwidth routing which is it?
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Wed Nov 04, 2009 11:13 pm

quad core? I think, 30% means that one core is used fully (25%), and that process is limiting the bandwidth.

MikroTik plans to add multicore optimizations in v5 or v6 of ROS
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Wed Nov 04, 2009 11:15 pm

for now, I think, you should move some QoS/Firewalling to another router(s)
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
chapex
Member Candidate
Member Candidate
Posts: 137
Joined: Wed May 30, 2007 1:23 am

Re: Maximum bandwidth with Quad Core and PCI-E

Wed Nov 04, 2009 11:37 pm

quad core? I think, 30% means that one core is used fully (25%), and that process is limiting the bandwidth.

MikroTik plans to add multicore optimizations in v5 or v6 of ROS
v5 or v6 :cry: :cry: :cry: :cry: :cry:

i
Ubnet datacenter: bandwith transport
 
vklimovs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 57
Joined: Fri Dec 16, 2005 5:37 pm

Re: Maximum bandwidth with Quad Core and PCI-E

Wed Nov 04, 2009 11:39 pm

Thank you for the reply. There is no QOS on these routers and firewalling is very rudimental. But here
http://www.mikrotikrouter.com/speedtests.asp they claim to do almost 6 Gbps (okay, that is in one direction on one card, for us it would be 1.5 Gbps both directions on two cards - still far more than enough)? And RB1000 claims to do 400,000 pps which is also more than enough in our case as well?
 
changeip
Forum Guru
Forum Guru
Posts: 3804
Joined: Fri May 28, 2004 5:22 pm

Re: Maximum bandwidth with Quad Core and PCI-E

Wed Nov 04, 2009 11:47 pm

p4 style supermicro, intel pcie nics, full gigabit both ways:

http://h1x.com/mt/gigabit/gigabit.html
and
http://h1x.com/mt/gigabit/gigabit2.html

Turn off connection-tracking and see if that helps. Also disable USB and any other onboard devices you are not using. The above was done with 2.9.24.
Colo and Wholesale Bandwidth Available! Sales at SanDiegoBroadband dot com
 
vklimovs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 57
Joined: Fri Dec 16, 2005 5:37 pm

Re: Maximum bandwidth with Quad Core and PCI-E

Thu Nov 05, 2009 12:09 am

Thank you for the input. This is however only half of what is needed - as we need full gigabit both ways on two cards, not just one.
 
vklimovs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 57
Joined: Fri Dec 16, 2005 5:37 pm

Re: Maximum bandwidth with Quad Core and PCI-E

Thu Nov 05, 2009 12:29 am

quad core? I think, 30% means that one core is used fully (25%), and that process is limiting the bandwidth.

MikroTik plans to add multicore optimizations in v5 or v6 of ROS
I started to look into that suggestion and amazingly it seems that this is true. Router caps exactly when CPU reaches 25%. However starting some other activity (e.g. disable/enable all rules in firewall - even though there are not many) increases CPU usage beyond that point, which means it happens on another core. However all routing is done on one core.
 
vklimovs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 57
Joined: Fri Dec 16, 2005 5:37 pm

Re: Maximum bandwidth with Quad Core and PCI-E

Thu Nov 05, 2009 12:39 pm

We got RB1000 for a test today. Put it in place of one of the border routers, disabled conntrack, empty firewall, queues, etc. The device was only able to pull around 55000 pps or 350 mbps, less than our Quad-Core machines and far less than the specified 400000 pps. Screenshot is attached. I am not sure what we can do now - there some 3.6 ghz CPUs out there, maybe we should try one of them.
You do not have the required permissions to view the files attached to this post.
 
vklimovs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 57
Joined: Fri Dec 16, 2005 5:37 pm

Re: Maximum bandwidth with Quad Core and PCI-E

Thu Nov 05, 2009 2:13 pm

Can somebody confirm or deny my findings?
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Thu Nov 05, 2009 2:23 pm

far less than the specified 400000 pps
I think, it's more bandwidth limitation, not pps. on one of our core router we have ~800Mbps, 100kpps. but when with ~400 Mbps load I run bandwidth test with small udp packets to this router, I can easily get ~200kpps. just haven't tested more =)
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
vklimovs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 57
Joined: Fri Dec 16, 2005 5:37 pm

Re: Maximum bandwidth with Quad Core and PCI-E

Thu Nov 05, 2009 2:38 pm

far less than the specified 400000 pps
I think, it's more bandwidth limitation, not pps. on one of our core router we have ~800Mbps, 100kpps. but when with ~400 Mbps load I run bandwidth test with small udp packets to this router, I can easily get ~200kpps. just haven't tested more =)
What are the specs of that router? What functionality is enabled on it?
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Thu Nov 05, 2009 2:53 pm

Core2Duo, integrated Intel NIC, a few firewalling (conntrac is enabled)
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
gmsmstr
Trainer
Trainer
Posts: 940
Joined: Fri Jun 04, 2004 2:22 am
Location: St. Louis, MO
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Thu Nov 05, 2009 7:18 pm

There is a lot more to it than just simple CPU usage. PCI buss design and limitiations, IRQs, testing application, etc. So just buy putting some hardware togeather, don't mean its a good design. There is lots more to it than just build it and it will work!
Dennis Burgess, MCTCE, MTCNA, MCTCTE, MTCWE, MTCNIE, A+, N+, MCP, MTCSE Mikrotik Certified Consultant / Trainer
Need Mikrotik Support: http://www.linktechs.net -- Link Technologies, Inc.
-- Author of "Learn RouterOS: Second Edition"
 
vklimovs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 57
Joined: Fri Dec 16, 2005 5:37 pm

Re: Maximum bandwidth with Quad Core and PCI-E

Thu Nov 05, 2009 7:32 pm

We user Intel server motherboards (SH3200), Quad core 3ghz CPU's, Intel PCI-E NIC's. What could possibly be wrong here hardware wise?
 
XTLMeth
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Mon Sep 07, 2009 7:10 am

Re: Maximum bandwidth with Quad Core and PCI-E

Thu Nov 05, 2009 11:53 pm

There is nothing wrong with what you have setup however things like which network adapter or which motherboard can make a difference. The OS underneath is linux and unless mikrotik has changed anything in the drivers for performance you're going to find that different cards/drivers will have better performance. Some drivers for the motherboard will be better then others. A server cpu may be better for what you're doing. Instead of using a core2duo you could use a single core xeon or something that will give you more power in a single cpu
 
vklimovs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 57
Joined: Fri Dec 16, 2005 5:37 pm

Re: Maximum bandwidth with Quad Core and PCI-E

Fri Nov 06, 2009 12:19 am

So, to rephrase you, we could get more out of 3.6 Ghz P4 (if I'am not mistaken 3.6 Ghz is the fastest one can get)?
 
XTLMeth
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Mon Sep 07, 2009 7:10 am

Re: Maximum bandwidth with Quad Core and PCI-E

Fri Nov 06, 2009 9:03 am

In short yes. If you are having issues with maxing out your cpu then getting a faster cpu will help you but it may not help you enough. I think you need to look at the bigger picture and that is you need to figure out where you bottlenecks are. It could be one of many things like cpu, memory, motherboard. If I were you I would ditch the P4 and go with a much beefier processor like Xeon or the newer i5 or i7 processors and in the process you'll get a performance boost also because you'll probably be running a faster motherboard with faster ram.
 
User avatar
martini
Member Candidate
Member Candidate
Posts: 296
Joined: Tue Dec 21, 2004 12:13 am

Re: Maximum bandwidth with Quad Core and PCI-E

Fri Nov 06, 2009 9:29 am

Try to run more than one BT on the same router, open 4-6 console terminal and run Bw test in each console, whith this test on ethernet you can see maximum speed for your router.
 
kostil
Frequent Visitor
Frequent Visitor
Posts: 74
Joined: Wed Jun 18, 2008 6:01 pm
Location: Moscow region, Russia

Re: Maximum bandwidth with Quad Core and PCI-E

Sat Feb 27, 2010 3:05 pm

Core2Duo, integrated Intel NIC, a few firewalling (conntrac is enabled)
Could you tell you conntrac configuration? Do you use NAT on mikrotik?
 
danix
newbie
Posts: 25
Joined: Mon Feb 22, 2010 12:29 am
Location: Romania
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Sat Feb 27, 2010 5:44 pm

hello, i have one question if i have: Bandwith 30 Mb > RB 450G > switches with management > clients ; do you think this configuration support more than 100-200 clients without crashes ? ( RB 450G with firewall, bandwith limited over mac, dhcp server, and other configuration). you guys think it will work ?
Thank you and sorry for my bad english.
This is linux-land. In the quiet night you can hear windows machines rebooting!
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Sat Feb 27, 2010 6:14 pm

Core2Duo, integrated Intel NIC, a few firewalling (conntrac is enabled)
Could you tell you conntrac configuration? Do you use NAT on mikrotik?
no NAT at all
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
kostil
Frequent Visitor
Frequent Visitor
Posts: 74
Joined: Wed Jun 18, 2008 6:01 pm
Location: Moscow region, Russia

Re: Maximum bandwidth with Quad Core and PCI-E

Sun Feb 28, 2010 10:13 pm

Do you use conntrack for traffic flow?
 
dog
Member Candidate
Member Candidate
Posts: 186
Joined: Wed Aug 12, 2009 3:37 pm
Location: Germany

Re: Maximum bandwidth with Quad Core and PCI-E

Sun Feb 28, 2010 10:47 pm

650 Mbit/s full duplex on the links
That is the value I've seen in all my (netio-)tests with cheap Realtek or Yukon chipsets under Linux.
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1825
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Mon Mar 01, 2010 6:12 am

There was an extremely interesting presentation on multi 10gigabit routing using Linux based routers given at the 2010 New Zealand Network Operators Group conference.

The presentation covers chipsets, architectures, advantages of NUMA, Kernel status in regards to routing throughput.

Most of this is relevant to RouterOS on x86 and is clearly presented by two highly knowledgeable researchers.

It is the first video on http://www.onfarm.net.nz/nznog.html



Regards,



Andrew
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Mon Mar 01, 2010 2:20 pm

Do you use conntrack for traffic flow?
conntrack is enabled, and we are collecting NetFlow information from our routers

I hope, I'll soon test LACP on 2-port GigE Intel card - will see if it can bring the throughput on higher level =)
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
kostil
Frequent Visitor
Frequent Visitor
Posts: 74
Joined: Wed Jun 18, 2008 6:01 pm
Location: Moscow region, Russia

Re: Maximum bandwidth with Quad Core and PCI-E

Mon Mar 01, 2010 2:46 pm

Do you use ospf on your NAS?
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1825
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Mon Mar 01, 2010 4:32 pm

I did not realise that LACP is supported by RouterOS...
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Mon Mar 01, 2010 8:01 pm

Do you use ospf on your NAS?
sure
I did not realise that LACP is supported by RouterOS...
but it is, although my previous try (early v3 and some old Zyxel switch) was unsuccessful =)
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
kostil
Frequent Visitor
Frequent Visitor
Posts: 74
Joined: Wed Jun 18, 2008 6:01 pm
Location: Moscow region, Russia

Re: Maximum bandwidth with Quad Core and PCI-E

Mon Mar 01, 2010 8:49 pm

What configuration do you use? Do you using NSSA like describe here or like on mikrotik wiki and manual using networks and passive ospf for users connects?
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Wed Mar 10, 2010 11:32 pm

omg, just saw it! direct routing, no nat, a few firewall rules (one is matching against address list with ~1000 items), NetFlow:
You do not have the required permissions to view the files attached to this post.
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1825
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Wed Mar 10, 2010 11:39 pm

Chupaka, I am confused ? What have you just seen?

Looks like some nice throughput....
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Thu Mar 11, 2010 12:24 am

exactly - that router have only one NIC =)
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1825
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: Maximum bandwidth with Quad Core and PCI-E

Thu Mar 11, 2010 1:02 am

damn, thats impressive.
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA

Who is online

Users browsing this forum: JohnNL and 90 guests