Is anyone using MikroTik with 10Gbps connections? We’re looking at possibly using Intel based 10Gbps SFP+ cards from HotLava. Is there some sort of theoretical maximum of bandwidth that Mikrotik can route or is hardware the limiting factor?
I would think it would be hardware, along with whether or not RouterOS currently supports that card or 10g at all.
The problem with 10 Gbps is that even if the system is able to use one processor core for each Network card queue there is not a lot of power dedicated to this stream.
Today (and since about 12 years) processors have only about 3 Ghz frequency, this is the limitation. As soon as you will try to do advanced filtering or tunneling (IPsec, Level 7 filtering…) bandwith will slow down.
Ask Intel to release new 100 GHz procesors, and wired speed routing at 10 Gbps will be possible with Router OS. But i’m afraid they prefer to sell DSP or FPGA chips for this market segment, instead of killing it releasing a 100 GHz x86 processor.
Today only dedicated FPGA or DSP based hardware can process 10 Gbps or more at full wire wire speed. Routerboard hardware is limited to 3.2 Gbps (ideal case) for the higher end model.
X86 seems not a very powerfull platform with Router OS, as it seems users cannot get more than 800 - 900 Mbps total throughput.
So i will say that it’s difficult to use linux only based solutions for carrier class routing / filtering at 10 Gbps or more regardless if you are using Mikrotik or similar linux based solution without hardware acceleration.
If you need a router with two or more 10 Gbps ports, there are good chances you will get packet loss as soon as the trafic rise in the Gbps and more level. This is not tolerable for today carrier class routing, with VoIP and Video trafic.
http://www.iis.se/docs/10G-OS-router_2_.pdf
ROS v5 has some improvements in multicore area - you should try it on 10G =)
is SMP really going to help with a single card pulling 10gbit? I cam imagine it would help with multiple 1gbit interface where it can load balance the work but even with TCP offloading done on the card it will be putting quite a strain on the CPU
well, also RPS is going to help 
I don’t have the hardware to test this, but i would be curious to know if it’s possible to simply transfer 10 Gbps of TCP trafic between two 10 Gbps ports and Router OS on X86 hardware.
I think as well that SMP can help for multiple 1 Gbps ports, but 10 Gbps seems really fast for 3 GHz processor core.
10 Gbps is about 1.6 GBps. This mean that one core at 3.2 GHz has only 2 clock cycle to manage 1 Byte of trafic…
I know this is a very bad approximate, because X86 processors do have SSE instruction set, and there are some helps like DMA PCI bus transfers, but this simple calculus show that it will be never possible to do advanced filtering and routing with 3.2 GHz non network optimized processors at 10 Gbps.
If someone tested this it would be fair to share it on the forum and give the model of the hardware tester set used to check packet loss.
I’m interested as well in jitter and latency results at 10 Gbps on X86 hardware.
any cheap 10Gbit cards you can recommend for testing? Maybe there is something new and popular we have missed
Seriously did I wake up and miss the years where everyone in the world got FTTH or something? Who around here is seriously pushing enough traffic to need a 10gbit NIC that is going to trust some x86 box to do it’s work?
Still, why not prepare for the future ? Any recommended cards we could add support for ?
Normis are you say you can route 10gbit thru a ROS device?
Have the interface up is one thing, being able to use it is another. Is MT looking at doing custom ASIC’s for routing and packet inspection now?
Anyway, take a swing at Intel’s 82598EB. Its Copper based but the Intel cards already have linux drivers that try and spread the load a bit (SMP) I’d like to see the lab you use to generate 10gbit of traffic!
There is no problem for the 10 Gbps test LAB. I would recommand the new EXFO “low cost” FTB-1 tester with the Copper module FTB-860G.
You’ll need a second unit with a FTB-860GL module for loopback.
It can support 10 GigE LAN or WAN by software option.
It is a wire speed 10 Gbps Ethernet tester. You will need two units to do a real bidirectionnal test.
EXFO do as weel a 100 Gbps tester. I’ve seen it pushing 100 Gbps of trafic without packet loss. So 10 Gbps is not a problem at all for modern testers.
The good thing with EXFO is that they support EtherSAM testing, so it is possible to test QOS directly with multiple streams at the same time.
It is really better than a simple RFC2544 test.
Whats the price bracket on those things, Seem them around downunder and the decent ones are $10k+ for gigbit testing.
I cant imagine MT dropping $20k to test 10gbit cards that about 5 clients are going to use lol
No i think the FTB-1 is under 10 000 $. And this kind of hardware can be rent by week.
Wire speed Gigabit Ethernet testers can be found for 1500 - 3000 $ on the second hand market. I bought one a few monthes ago for 800 €. (i know i was a bit lucky).
I have one main unit and one reflector unit, so i can measure round trip time and packet loss at wire speed, but only at Gbps speed.
There is no real second hand market yet for 10 Gbps testers, so renting or buying new ones are the only options except if you are very lucky in finding one second hand.
I think that the market for 10 Gbps and Router OS is really higher than 5 clients. Mediumly sized providers (1000 - 30 000 clients) can be interested in lowering their cost using Router OS (or alternatives) if they can have wire speed at 10 Gbps without packet loss and a reliable BGP engine in the router.
As soon as the number of clients rise at something higher than 4000, Internet trafic can rise higher than 1 Gbps. So you need at least one 10 Gbps link with at least one tier one or tier two provider to avoid saturations.
This is for France, in countries where “You Tube” is used by everyone, trafic can be 10 times higher…
And this will rise fastly because FTTH deployement is planned in most european countries (except ireland :=( )
Do not forget that big providers are using today 40 Gbps links, and are testing 100 Gbps links for tomorrow use in a couple years.
If They can have wirespeed 10gbit, Which means being able to push 10gbit of routing or packet marks or even plain old PPPoE traffic.
I get your point that the traffic is coming but I really doubt ROS is going to find a hardware platform that’s going to allow that. The CPU’s simply don’t allow for it and SMP isn’t going to help too much. You need co-processors to offload the highly parallel operations to do for things like routing or packet inspection.
Getting the link up is only the first part, no point in that if you cant push thru the router. But hey MT here’s a challenge, Prove me wrong, Show us 10gbit of traffic running thru one of your routers 
It will be possible in a near futur, because FPGA chips do allow this for a few $ since at least six years.
We simply need to wait that programmers learn HDL language coding, and translate Linux network code inside those high speed logic and parallel processing circuits to allow 10 Gbps wirespeed or even 100 Gbps wirespeed.
Actually, Small and talentuous companies like Mikrotik are using ready made network processors or switching circuits, this is the limitation.
But as soon as they’ll be able to program themself FPGA chips for marking, filtering and routing they will be able to enter in competion with Cisco, Juniper and other Brocade or similar manufacturers, at a fraction of their cost.
Good points but still it’s a heavy investment no matter how you look at it either in time or in money and if you’re handling 10gbit of traffic you’re really past MT’s level.
After asking to EXFO, i can confirm that the FTB-1 Ethernet tester with 10 Gbps copper module 860GL is under 10 000 €.
IPv6 testing is possible as an option.
The 860G and 860GL copper modules have only one 10 Gbps port. So you’ll need two testers to test a router bandwith.
For 1 Gbps testing, it’s possible to use the 860G module, it has two 1 Gbps Ethernet copper ports, so the device under test can be connected between those two ports.
But for 1 Gbps testing there are cheaper alternatives on the second hand market.
The FTB-1 can be rent in europe. Can be a cheaper alternative to test a device before going to production or during development.
Interesting to note I guess in our corner of the world these things have a bigger markup