Getting a 10 Gbps Connection - router / switch options?

RouterOS Beginner Basics
1

A local ISP is laying down fiber and my hookup date is in a week. Apparently, the new line is going to be “up to” 10 Gbps symmetrical with no caps for $50/mo EDIT: $40/mo.

I know that is insane amount of potential bandwidth, but it appears that there aren’t many non-enterprise level solutions for capitalizing on my connection. I work at a tech company, but sysadmin I am not

I know ASUS is coming out with their new AXE16000 which has a 10Gbps RJ45 port, but are there other solutions that are readily available?

2

https://mikrotik.com/product/rb4011igs_rm has 10G SFP+ interface, and power to suit. https://mikrotik.com/product/rb4011igs_5hacq2hnd_in same + wireless.

3

…i’d actually doubt that a RB4011 will be capable of running/saturate a 10G WAN (incl. NAT) link…it can serve around 6+ Gbps, I think…maybe 7Gbps with overclocking the CPU a bit.
Keep in mind, that on a fiber ISP connection, the link will often have to be established va ppoe in a tagged VLAN…ppoe is single threaded.

I also doubt that a consumer grade “thingy” like an ASUS will be up to that task.

I’d look into a CCR2004 … stil cheap IMHO : https://mikrotik.com/product/ccr2004_16g_2splus#fndtn-testresults

4

While the RB4011 and its quasi-successor the RB5009 have an SFP+ port capable of 10 Gbit/sec and a fair bit of CPU power behind it, there’s only the one. Worse, in the case of the RB4011, it’s tied to the CPU, not to the switch chip so any single-threaded test is likely to choke down to 1-2 Gbit/sec.

The architecture of the RB5009 differs, putting the switch chip in the middle, which should help out considerably in real-world applications. Yet, since all of the other ports are gigabit, no single client can get more than that without playing bonding games, which is an imperfect solution to this problem; it’s easily possible to have a LAG group devolve to single-link performance.

No, for this sort of application, I’d have to recommend something with quite a bit more grunt than either of those two fine products.

If you can live with individual stations having only 1G uplinks, needing multiple stations to soak up the 10G pipe, I’d start my product research with the CCR1009-7G-1C-1S+PC. That’s the smallest and cheapest unit in the CCR1009 line, good for high-end home use. The key spec to me is those nine cores, allowing you to do heavy firewalling, queueing, VPN, and other applications on each of those 1G ports simultaneously. A good rule of thumb is 1 GHz per gigabit, so even though nine cores might seem overkill, you really do need it unless you’re expecting the unit to do more switching than routing.

The next unit up the line worth looking at is the CCR2004. It costs about the same as the higher end of the CCR1009 line, but it gets you twelve SFP+ ports. You’ll have to add the cost of SFP+ modules to this, but in exchange you get the option to push 10G from a single port through the fiber link. (Or try, anyway; saturating 10G with a single link isn’t easy!)

If you have only a single 10G LAN downlink from the router, as to an internal CRS328 or CRS312, it might be worth considering the CCR1036 because it’s got even more CPU grunt, and it’ll save you on the SFP+ modules, with all the 1G ports it provides. Definitely not my first choice, but you might have a CCR1036-sized hole in your life.

5

Assuming you will do srcNAT towards 10GB WAN, I would recommend to also consider RB5009.
It has a SFP+ too and about 30% more CPU power compared to RB4011, but there is no WiFi version (yet?).

NAT is handled by the CPU and the RB5009 is capable of NAT routing about 5GB to/from WAN, depending on amount FW rules and queues apllied.
This sounds like wasting some of the 10GB WAN, but outside artficial speed tests you will hardly achieve more than 5GB anyway, probably a bit more if you run many (hundreds) of parallel connections
As others in this thread have mentioned, if you really want a router capable of routing 10GB at wire speed including NAT, FW and queues, you will have to do for something in the lhigh end home / lower enterprise range. Substantially adding to the price tag.

Is it good enough to have 1x2.5GB and 7x1GB in parallel for internet clients? Or needs one client to be able to use the whole WAN bandwith on a single connection?
If yes, you either put a switch on RB5009 SFP+ (router on a stick) or go for a CCR/CRS model with more than one 10GB port.

RB5009 also has a better switch chip with full L2 hw offload support for VLAN filtering, (R)STP, IGMP and DHCP snooping allowing for full wire speed bridging among all 8 ether ports.
RB4011 got some L2 hw offload support with ROS7, but it is still limited (no IGMP/DHCP snooping) and only works in 2 groups ether1-5 and ether6-10 (2 different switch chips).

At the other hand RB4011 runs very stable on ROS 6.49.2, while RB5009 requires ROS 7 and still has some serious issues (ether1 2,5GB links, IPv6, queues) to be sorted out with future ROS releases. Same as with RB4011 back when it was new.

6

If this is worse depends on the usage scenarios. Having the SFP+ directly attached to the RB4011 CPU makes it very good for router on a stick applications. The RB4011 is capable to move much more than 1-2GB/s btw. VLANs on SFP+ (in my tests ca. 5GB with FW/queues and about 9GB without FW but with fasttrack).

If the plan is to combine switching and routing in the same box, RB5009 is the better choice. If an external switch is used anyway, the RB4011 has it’s place as a very capable router, especially for IPsec as it has a very wide HW crypto support for different cyphers, also quite exotic ones.

7

Single-threaded, as I qualified it, or do you have to get all four cores working to achieve it, as I expect?

8

A single TCP connection is always handled by one thread to avoid packet reordering hampering throughput.
In my RB4011 experience, single TCP srcNAT connections max out at 1-3 GB, depending on FW and queues.

But in most real life applications, there are typically hundreds to thousands of parallel connections towards WAN by different LAN clients.
And RB4011 nicely schedules parallel connection threads among available CPU cores achieving 5-9GB in total.

There might be execeptions, but typical Office/Home WAN usage is more about handling many parallel connections with acceptable latency than a single one using the full band width.
A single Windows box running Teams and Outlook keeps 10-20 open connections to the MS cloud all the time. Loading a complex homepage triggers dozends of parallel connections.

9

So we agree, then.

The point of the post you yeah-butted was that the OP can’t expect to run a single iperf3 client across the router, all set up with the good strong filtering RouterOS allows, and expect to fill the 10G fiber upstream.

If he buys an RB4011 anyway, at least now it’ll be with his expectations properly set.

10

Agreed.
But often I see setups and bechmarks concentrating on single connection performance. Forgetting that is not a realistic use case for most installations were efficient handling of many parallel connections is much more important.

To avoid confusion by our discussion, my recommendation would be:

If not filling the 10GB fibre with a single connection is acceptable:

  • RB4011 for router on a stick or if WiFi is required


  • RB5009 if full featured L2 hw offload (switching) is required

If filling full 10GB WAN capacity also for single connection and with full firewalling is required

  • one of the CRS or CCR models, depending on required set of interfaces.