Hi, I have RB5009 as router on a stick, connected to crs312 via sfp+ with 10G link. My issue is, that when I want to transfer data from one VLAN to another it tops at 2Gbit/s. When I look on the block diagram, the connection with cpu should be duplex 10G and when transferring the data the cpu does not go over 40% utilization, so my thinking is, that it should go faster than that. Am I missing something?
Hw offload probably not active ?
Without config, hard to tell...
I'm only inferring this, but if I understand correctly, you're doing the inter-vlan routing on the rb5009.
This device doesn't have hw offload capabilities (at least for routing.) So this won't help.
This device is roughly only capable of the sort of speeds you are seeing. What explains the cpu usage not increasing further to approach 100% is simply because the load balancing that the integrated NIC does hands off packets belonging to the same connection always to the same core. So what you're likely seeing is that one core is at the max, while the others are mostly idle. You can check this with the profile feature.
Two things will help you, probably radically.
The first is that the auto frequency scaling on the rb5009 usually causes massive performance loss. You can, however, lock the cpu frequency to the max of 1400 MHz, which resolves this completely. To do this, you'll have to enable the device-mode setting that allows you to do so. Also, you'll see a warning about a non-default frequency being selected - there's no reason to take this seriously. Some people worry about power consumption increasing by selecting this (and this worry is not irrational) but it simply doesn't happen.
The other is that you should enable fasttrack. With the appropriate firewall rule you can easily enable it selectively, e.g. only for inter-vlan traffic.
These two modifications will get you close to 10 Gbps routing.
I'm getting there, the only issue is, that even when I have fasttrack enabled (the default rule, which matches all established and related) it doesn't seem to work. The fasttrack counters are 0 and all traffic still goes through firewall. There is nothing special in the config or firewall, only thing, that I have is VRRP with second rb5009, which has the same issues and I have some wireguard tunnels and one gre tunnel. But I don't see any reason why would fasttrack not start. In /ip/settings fasttrack is ticked and fast path is enabled. I don't have any mangle rules setup. I have the mikrotik recommended raw firewall rules and some input rules limiting access to router. This is the start of forward chain:
chain=forward action=accept log=no log-prefix="" ipsec-policy=in,ipsec
chain=forward action=accept log=no log-prefix="" ipsec-policy=out,ipsec
chain=forward action=fasttrack-connection connection-state=established,related log=no log-prefix=""
chain=forward action=accept connection-state=established,related log=no log-prefix=""
can you provide a config file ? maybe I can help.
if you're aiming intervlan routing using RB5009 with matching CRS3xxx will be good.
you can get 9-10gb/s if it's between devices connected to the CRS3xxx.
I figured it out, I had DHCP snooping turned on on RB5009 bridge and because of that fasttrack wouldn't start. I've disabled it and I get 10G speeds no problem now. Thanks everyone for steering my head around.
Did you also lock the cpu frequency?
This detail could be of interest to anyone searching for the same thing.
Yes, I did, but didn't see any difference, maybe there is on start until the cpu spools up.
It mainly affects tcp streams because the initial packet loss leads to the congestion control dialing back the speed. Maybe you used udp, where this is not really present...
Anyway, thanks for the report.