I have RB5009UG+S+IN router and CRS309-1G-8S+IN 10Gb/sec switch.
I have setup a docker container on the RB5009 router with iperf3.
However, I am still only getting a maximum of around 3.00 Gbits/sec when connecting from my windows 10Gb/sec network card with iperf3 in dos window.
It was my understanding that CRS309-1G-8S+IN can switch at 10Gb/sec on ALL ports, and RB5009UG+S+IN router can handle 10Gb/sec across its SFP+ port.
I have replaced the fibre card on my windows PC, I have replaced the fibre DAC from windows PC to the switch, and I have replaced the short DAC from router to switch - still no luck.
Please can someone advise what the problem might be?
According to my understainding of official test results for RB5009 (and many other long-time forum members’ understanding as well) it can route in real life at around 3Gbps (give or take). Running a (CPU intensive) app inside container on same router, competing for same CPU resources, means that your use case is more on “… or take” side of stated number.
So I guess what you see is what your RB is capable of in your usage scenario.
I second what @mkx said, as that echoes my experience. You’ve hit the limit of what the quad-core CPUs’ can handle on RB5009, RB4011, and CCR2004. The ports are fine, and if you’re able to come up with a scenario where you’re just switching or bridging, then more of it hits the hardware. But speed testing, especially in a container, is definitely CPU-bound.
Mine is capable of better than that with an RB4011, and an RB5009 should do a smidge better still. See the benchmark results at the bottom of the linked documentation.
Partly that’s the low-overhead nature of the setup, but also it’s careful setup of the test. The most significant factors are using a number of parallel test threads (-P) equal to the core count, “reverse” (-R) mode, and use of iperf3 3.16, which improves handling of parallel threads considerably over earlier versions.
It was my understanding that CRS309-1G-8S+IN can switch at 10Gb/sec on ALL ports
It can…if you run “iperf3 -s” on a sufficiently dialed-in server connected to one of those 10G ports and run “iperf3 -c” on an equally dialed-in client machine, so that the CRS309 runs in pure switch-chip mode, not running anything through the CPU. I’ve gotten 9.4 Gbit/sec results across a CRS328 that way.
When you put the iperf3 container on either of these inexpensive RouterOS boxes instead, iperf3 must do everything using that same CPU, by necessity, and there’s only so much it can do with a few ARM cores in the Cortex A class.
First rule of troubleshooting: test one thing at a time. You’ve got several unnecessary complexities in this setup:
Two fiber links to the Home Server in the lower left corner of the network diagram. Unplug one to give the switch exactly one option to choose when forwarding traffic. (I’m not telling you that link aggregation is bad, merely that we’re testing single-link performance here. If you want a LAGG test, you’d do that separately after getting single-link working, and then with a second host that is also set up for dual-link.)
ZeroTier. Turn that off. You don’t need packets going up to the Internet and back down merely to get across the house due to mistaken routing.
“Fiber DAC”, really, across whole-house distances? Even an active DAC cable might be limited to 10m; passive, less. Unless you’re living in an open-plan shoebox apartment and can therefore do straight-line cable runs, you’ve either chosen the wrong tech or your diagram is wrong. The proper choice here is SFP+ transceivers with dumb fiber cabling. I’d go with OM3 or OM4 MMF.
Then with all of that fixed, you might still have problems, as “Server” and “Studio PC” tells me nothing about how capable those are of sourcing/sinking 10G of traffic continually. By “dialed-in” above, I mean you need quality NICs, good cabling, and a CPU powerful enough to drive it all.
Yes, CPU, and for the same reason as our ARM comments above.
Two fiber links to the Home Server in the lower left corner of the network diagram. Unplug one to give the switch exactly one option to choose when forwarding traffic. (I’m not telling you that link aggregation is bad, merely that we’re testing single-link performance here. If you want a LAGG test, you’d do that separately after getting single-link working, and then with a second host that is also set up for dual-link.)
Hi, that is actually not the case now I just have one fibre link, need to update the diagram
ZeroTier. Turn that off. You don’t need packets going up to the Internet and back down merely to get across the house due to mistaken routing.
I did actually turn it off for the purpose of the tests.
“Fiber DAC”, really, across whole-house distances? Even an active DAC cable might be limited to 10m; passive, less. Unless you’re living in an open-plan shoebox apartment and can therefore do straight-line cable runs, you’ve either chosen the wrong tech or your diagram is wrong. The proper choice here is SFP+ transceivers with dumb fiber cabling. I’d go with OM3 or OM4 MMF.
I do actually live in the UK where the house size is quite a bit smaller than the US. I have 10m Fibre DAC cables connecting the upstairs study with the downstairs studio.
My computers are pretty powerful.. plenty capable to source/sink 10G of traffic I would have thought.. server is 2012 era Intel Xeon 8 CPU with 32GB DDR ram running Ubuntu - a Lenovo Thinkstation C 30, Studio PC is 2020 12 gen Intel i7 CPU with 16GB DDR4 RAM running Windows 11.
I was previously using a Mellonex network card in the server and was getting comparable speeds - I switched that out, so I’m pretty sure it’s not the NIC.
The only thing that I can think of that MIGHT be causing a problem and might need to be replaced is the 10m DAC from server to switch.. if you have any recommendations of high quality 10m DACs I’d be very interested.
I suppose it might also be the PCI express lanes on my 2012 motherboard.. I would have thought they were more than capable of handling 5Gb/sec though at least.
That looks like your problem to me. I wasn’t in the 10G market in 2012, but I don’t remember anything “workstation” grade coming with 10G NICs, not even the top-end Mac Pro. That didn’t start happening until 2016-2017.
PCI express lanes on my 2012 motherboard..more than capable of handling 5Gb/sec
The 10GTek product page doesn’t specify a PCIe version, so we have to assume PCIe 1.0, where the total transfer is around 16Gbit/sec for 8 lanes, but that includes your graphics card and whatever else is on that shared bus. You’re right on the ragged edge here, if it’s possible at all.
Stupid question, but whats your MTU? to get ~10g routed traffic on my 2004 i have to turn the MTU up to around 8k, as well as run multiple TCP connections of iperf in parallel.
Across the OP’s CRS309, standard-sized Ethernet packets suffice, if we can go by my CRS328 tests here. I have yet to be able to justify jumbo packets with any test I’ve been able to devise here. That’s not to say it isn’t useful somewhere, but that I think people too often reach for this solution as an easy thing to try and then leave it enabled without first finding out if it actually helped.
TCP brutal
OP specified Windows on both ends up-thread.
In any case, how does congestion control adjustment help on a point-to-point test, on an idle link? Regular TCP will asymptotically approach the link’s maximum data rate, then settle in. I don’t see a change here helping unless you’re wanting shorter iperf3 runs than the default of 10 seconds.
If you’re still worried about it, run the test up to 30 seconds. I’d expect that to swamp any congestion control algorithm differences in a test like the one discussed here.
well well, then forget my practical advise and continue with your professional support, pointing at pcie links xD
If you haven’t gotten it, the congestion control point was not very serious.
kthxby
