Community discussions

MikroTik App
 
Malvineous
just joined
Topic Author
Posts: 12
Joined: Wed Jul 14, 2021 6:21 pm

Can't get over 1G on CRS309 10G ports

Thu Jul 22, 2021 1:58 pm

Hi all,

I just purchased my first Mikrotik device - a CRS309-1G-8S+IN - and after a bit of a rocky start (had to RMA the device due to the PoE input dying on the second use) I finally have the replacement unit up and running.

Unfortunately, despite it being connected to my servers at 2x 10G links each, I can only get between 0.9G and 1.1G transfer rates and I can't work out why.

The device arrived with all 8 SFP+ ports plus the 1G port in a bridge configuration. I removed ports 1-6 and used them to create three bonded interfaces to get a 20G link to each of the three servers I wish to connect. Previously two of these servers were directly connected and I was able to get approx. 18G transfer between them. Now the CRS309 sits in between them with an additional set of 10G direct attach cables, but there's no other change. The bond interfaces are set to balanced-rr on both ends (the servers run Linux). ethtool reports all links at 10G on the server side, and the RouterOS web page reports all links at 10G on the Mikrotik side, so the physical side of things appears to be fine.

However now, with the three bonded ports (2x SFP+), two unconnected SFP+ ports, and the 1G Ethernet port, I can barely reach 1G between any two servers that should each have a 20G link. I expected to get at least 17G but not 0.9G.

I tried temporarily disabling the 1G port in case it was trying to send all traffic over it for some reason, but even with the 1G port disabled, the 20G bonded links could barely make 1G.

Obviously I've done something wrong as 1G isn't even close to the unbonded 10G single-link speed, let alone the 20G available from the dual link, but I've gone through all the menus and looked around online and can't see anything that could explain it.

I'm not using any routing features (one of the machines is a Linux server that's doing my routing) so I suppose I could give SwOS a try, but it doesn't seem right that RouterOS would perform so poorly. So I'd like to try to understand what I've done wrong before I give up and try the other OS.

Any ideas what I might have missed?
 
User avatar
xvo
Forum Guru
Forum Guru
Posts: 1237
Joined: Sat Mar 03, 2018 1:12 am
Location: Moscow, Russia

Re: Can't get over 1G on CRS309 10G ports

Thu Jul 22, 2021 2:03 pm

The bond interfaces are set to balanced-rr on both ends (the servers run Linux).
Balance-rr can't be hw-offloaded, so it is procesed by switch's CPU - hence the bottleneck.
Use LACP or balance-xor.
 
Malvineous
just joined
Topic Author
Posts: 12
Joined: Wed Jul 14, 2021 6:21 pm

Re: Can't get over 1G on CRS309 10G ports

Thu Jul 22, 2021 3:00 pm

Thanks for the reply!

It does seem to provide a small speed increase going from balance-rr to balance-xor, crawling up from 0.9G to 3.9G, but that's still far from the 10G speed of a single link, let alone the 20G available. It also ultimately won't solve my issue as my understanding is that balance-xor and LACP top out at one interface's maximum speed between two machines. As far as I'm aware, balance-rr is the only way you can exceed a single link's bandwidth between only two machines, which is what got my attention with this product as I haven't seen balance-rr offered by any other brands of switch before.

At any rate, I'm not sure that it should matter whether the packets go via the CPU or not. The CRS309 is advertised as a 10G router with a "total non-blocking throughput of 81 Gbps." That means it can route at 80G so it should barely break a sweat at the 20G I am asking for.

There must be another config option I am missing because quite frankly if you're right and it's actually a 1G router falsely advertised as 10G with 80G throughput then it's going straight back to the shop and I'm never touching anything Mikrotik again!
 
biomesh
Long time Member
Long time Member
Posts: 561
Joined: Fri Feb 10, 2012 8:25 pm

Re: Can't get over 1G on CRS309 10G ports

Thu Jul 22, 2021 3:25 pm

It can switch at wire speed, not route. You don't want anything hitting the CPU if you can avoid it.

How are you running your tests? Iperf with multiple threads? Have you changed the window size to match between all devices?
Last edited by biomesh on Thu Jul 22, 2021 3:26 pm, edited 2 times in total.
 
User avatar
xvo
Forum Guru
Forum Guru
Posts: 1237
Joined: Sat Mar 03, 2018 1:12 am
Location: Moscow, Russia

Re: Can't get over 1G on CRS309 10G ports

Thu Jul 22, 2021 3:26 pm

It's not a router, it's a switch, and it has non-blocking switching throughput of 81 Gbps.

On it's CPU with ROS6 it can route something a little less than 1Gbit:
https://mikrotik.com/product/crs309_1g_ ... estresults

When ROS7 will be out, this switch will support L3-hw-offoading, so it will be able to route much more in simple scenarios, utilising the switch chip.
But that is still not the same thing - only some of the routing features can be hw-offloaded.
https://help.mikrotik.com/docs/display/ ... Offloading

And anyway bonding has nothing to do with routing, and I doubt that balance-rr is supported by the switch-chip, and can be ever offloaded from the CPU.
You are right about the limitations of LACP and balance-xor - you can't utilise both links for a single connection, however with L3+L4 hash, you can spread multiple connections even between the two machines.
There is simply nothing more you can do about it.

On the other hand - the result of only around 4gbit on a 20gbit bond still has to be investigated.
 
User avatar
xvo
Forum Guru
Forum Guru
Posts: 1237
Joined: Sat Mar 03, 2018 1:12 am
Location: Moscow, Russia

Re: Can't get over 1G on CRS309 10G ports

Thu Jul 22, 2021 3:31 pm

if you're right and it's actually a 1G router falsely advertised as 10G with 80G throughput
And yes - it is L2 (soon to be L3) 10G switch with 80G throughput and an 1G router in one device, and it's not advertised as anything more than that.

Who is online

Users browsing this forum: No registered users and 30 guests