We are using two 3 Ghz Quad Core machines with 2 Gb of RAM as our border routers. Each machine has two interfaces, which are PCI-E Intel Gigabit cards. CPU Load never exceeds 30%. However we are unable to get more than about 650 Mbit/s full duplex on the links. We are unsure if this is our issue or issue on the side of upstream provider. So this post it to confirm that it is possible to push more than 650 Mbit full duplex through a machine like described. 
Any replies appreciated.
OK, not a single person is using that setup? If you are using any other setup for high-bandwidth routing which is it?
quad core? I think, 30% means that one core is used fully (25%), and that process is limiting the bandwidth.
MikroTik plans to add multicore optimizations in v5 or v6 of ROS
for now, I think, you should move some QoS/Firewalling to another router(s)
v5 or v6 
i
Thank you for the reply. There is no QOS on these routers and firewalling is very rudimental. But here
http://www.mikrotikrouter.com/speedtests.asp they claim to do almost 6 Gbps (okay, that is in one direction on one card, for us it would be 1.5 Gbps both directions on two cards - still far more than enough)? And RB1000 claims to do 400,000 pps which is also more than enough in our case as well?
p4 style supermicro, intel pcie nics, full gigabit both ways:
http://h1x.com/mt/gigabit/gigabit.html
and
http://h1x.com/mt/gigabit/gigabit2.html
Turn off connection-tracking and see if that helps. Also disable USB and any other onboard devices you are not using. The above was done with 2.9.24.
Thank you for the input. This is however only half of what is needed - as we need full gigabit both ways on two cards, not just one.
I started to look into that suggestion and amazingly it seems that this is true. Router caps exactly when CPU reaches 25%. However starting some other activity (e.g. disable/enable all rules in firewall - even though there are not many) increases CPU usage beyond that point, which means it happens on another core. However all routing is done on one core.
We got RB1000 for a test today. Put it in place of one of the border routers, disabled conntrack, empty firewall, queues, etc. The device was only able to pull around 55000 pps or 350 mbps, less than our Quad-Core machines and far less than the specified 400000 pps. Screenshot is attached. I am not sure what we can do now - there some 3.6 ghz CPUs out there, maybe we should try one of them.
Can somebody confirm or deny my findings?
I think, it’s more bandwidth limitation, not pps. on one of our core router we have ~800Mbps, 100kpps. but when with ~400 Mbps load I run bandwidth test with small udp packets to this router, I can easily get ~200kpps. just haven’t tested more =)
What are the specs of that router? What functionality is enabled on it?
Core2Duo, integrated Intel NIC, a few firewalling (conntrac is enabled)
There is a lot more to it than just simple CPU usage. PCI buss design and limitiations, IRQs, testing application, etc. So just buy putting some hardware togeather, don’t mean its a good design. There is lots more to it than just build it and it will work!
We user Intel server motherboards (SH3200), Quad core 3ghz CPU’s, Intel PCI-E NIC’s. What could possibly be wrong here hardware wise?
There is nothing wrong with what you have setup however things like which network adapter or which motherboard can make a difference. The OS underneath is linux and unless mikrotik has changed anything in the drivers for performance you’re going to find that different cards/drivers will have better performance. Some drivers for the motherboard will be better then others. A server cpu may be better for what you’re doing. Instead of using a core2duo you could use a single core xeon or something that will give you more power in a single cpu
So, to rephrase you, we could get more out of 3.6 Ghz P4 (if I’am not mistaken 3.6 Ghz is the fastest one can get)?
In short yes. If you are having issues with maxing out your cpu then getting a faster cpu will help you but it may not help you enough. I think you need to look at the bigger picture and that is you need to figure out where you bottlenecks are. It could be one of many things like cpu, memory, motherboard. If I were you I would ditch the P4 and go with a much beefier processor like Xeon or the newer i5 or i7 processors and in the process you’ll get a performance boost also because you’ll probably be running a faster motherboard with faster ram.
Try to run more than one BT on the same router, open 4-6 console terminal and run Bw test in each console, whith this test on ethernet you can see maximum speed for your router.