Good Evening,
Please forgive me if this scenario has already been discussed elsewhere. I have an RB4011 as my main router connected to a CRS326 with a 10Gbps DAC. This router manages all of my vlans, dhcp, filtering etc. Performance is great, with 1 exception, any inter-vlan transfers seem to cap around 650-700Mbps, it appears to be the result of hitting a CPU bottleneck on 1 of the 4 cores as 1 core reaches 100% during a transfer test with iperf between a client on vlan 10 and the server on vlan 20.
While i understand that because this is a routing scenario it runs traffic through the cpu rather than switching it, i am curious if there is a way to have the routerboard use multiple cores for the transfer in an attempt to reach near full 1Gbps speeds. seems goofy that 3 cores sit idle while a transfer caps from a processor bottleneck.
Is there something i am missing, or a way to optimize this transfer. I already tried a fastrack on established and related but it didnt improve anything, at least not noticeably.
Thanks
Re: RB4011 Inter-VLAN routing performance
Established/related etc are states between WAN - LAN etc. not from VLAN to VLAN.
Check detailed resources information where CPU is spending time/cycles.
Especially firewall, if that is the culprit put early up in the list an accept rule for the inter vlan traffic you want to allow.
Rearrange the rules to lower CPU load.
For testing only (to see if firewall is the limit), disable all rules and run test...
Check detailed resources information where CPU is spending time/cycles.
Especially firewall, if that is the culprit put early up in the list an accept rule for the inter vlan traffic you want to allow.
Rearrange the rules to lower CPU load.
For testing only (to see if firewall is the limit), disable all rules and run test...
Re: RB4011 Inter-VLAN routing performance
Why do you think VLAN to VLAN traffic if somehow special?Established/related etc are states between WAN - LAN etc. not from VLAN to VLAN.
For multiple connections the device should utilise more than one core.
But still, this is the kind of traffic you should apply fasttrack to, in order to increase performance. Probably you didn't do it right.
Re: RB4011 Inter-VLAN routing performance
You need to go back to Networking school, because you are wrong.Established/related etc are states between WAN - LAN etc. not from VLAN to VLAN.
Re: RB4011 Inter-VLAN routing performance
I added the following rules at the top of the firewall rules list
7 ;;; FastTrack connection rule - established/related only
chain=forward action=fasttrack-connection connection-state=established,related
8 ;;; accept forward rule- fallback for non FastTrack-able packets - established/related only
chain=forward action=accept connection-state=established,related
I am fairly certain it catching and processing the packets through those rules as when i run my iperf test now i see a massive spike in bytes and packets on those to rules in Winbox only during the duration of the test. Absolutely no change in performance though. I still get a CPU spike of 100% during the transfer test and no difference in throughput speed.
7 ;;; FastTrack connection rule - established/related only
chain=forward action=fasttrack-connection connection-state=established,related
8 ;;; accept forward rule- fallback for non FastTrack-able packets - established/related only
chain=forward action=accept connection-state=established,related
I am fairly certain it catching and processing the packets through those rules as when i run my iperf test now i see a massive spike in bytes and packets on those to rules in Winbox only during the duration of the test. Absolutely no change in performance though. I still get a CPU spike of 100% during the transfer test and no difference in throughput speed.
Re: RB4011 Inter-VLAN routing performance
If i temporarily disable all the firewall rules i do see a persistent increase in performance of about 60 Mbit/s, however i am still not reaching full 1Gbit/s throughput. 1 core still hits 100% cpu and performance boost of 60 Mbit/s during the duration of the iperf test
Re: RB4011 Inter-VLAN routing performance
Quick update to this. if i run the iperf command as follows:
iperf3 -c x.x.x.x -P 2
It forces the iperf test to use 2 threads and balances the load between 2 cores the first time and i approach a sum throughput of 980Mbit/s. If I run the same command a second time regardless of the -P 2 flag still being specified the CPU pins on 1 core and the throughput drops to around 680Mbit/s per second. I repeated this 4 times and received this result over the 4 tests:
1. Balanced load between two CPU cores, combined throughput of 1GBit/s
2. Pinned CPU on 1 core throughput cap of 680Mbit/s
3. Balanced load between two CPU cores, combined throughput of 1GBit/s
4. Pinned CPU on 1 core throughput cap of 680Mbit/s
I don't understand why this is testing inconsistently.
Ultimately I would like to see a scenario where I can get a full 1Gbps connection speed when doing an SMB transfer...it seems however that my bottleneck in this scenario however may be that SMB will only ever use 1 CPU core as its a single connection...i may have to enable Multi-channel SMB? pure conjecture...i just want to be able to run full 1Gbit/s file transfers.
Thanks
iperf3 -c x.x.x.x -P 2
It forces the iperf test to use 2 threads and balances the load between 2 cores the first time and i approach a sum throughput of 980Mbit/s. If I run the same command a second time regardless of the -P 2 flag still being specified the CPU pins on 1 core and the throughput drops to around 680Mbit/s per second. I repeated this 4 times and received this result over the 4 tests:
1. Balanced load between two CPU cores, combined throughput of 1GBit/s
2. Pinned CPU on 1 core throughput cap of 680Mbit/s
3. Balanced load between two CPU cores, combined throughput of 1GBit/s
4. Pinned CPU on 1 core throughput cap of 680Mbit/s
I don't understand why this is testing inconsistently.
Ultimately I would like to see a scenario where I can get a full 1Gbps connection speed when doing an SMB transfer...it seems however that my bottleneck in this scenario however may be that SMB will only ever use 1 CPU core as its a single connection...i may have to enable Multi-channel SMB? pure conjecture...i just want to be able to run full 1Gbit/s file transfers.
Thanks
Re: RB4011 Inter-VLAN routing performance
Make sure that your problem is not affected by SMB - that protocol is, to put it mildly, horrible. Multichannel support was introduced with SMBv3 so check if you're forcing SMBv3 (by default modern clients will usually default to v2 for some reason).
Also, I was just poking around in my lab with a similar config: full duplex transfer between a computer tagging the traffic and RB4011 getting it via VLAN and then CPU-switching to a non-tagged ethernet. With ~1Gb/s symmetrical traffic (so in practice ~2Gb) using iperf3 I was getting ~970Mb/s both ways with 3-5% of CPU usage on RB4011.
Also, I was just poking around in my lab with a similar config: full duplex transfer between a computer tagging the traffic and RB4011 getting it via VLAN and then CPU-switching to a non-tagged ethernet. With ~1Gb/s symmetrical traffic (so in practice ~2Gb) using iperf3 I was getting ~970Mb/s both ways with 3-5% of CPU usage on RB4011.
Re: RB4011 Inter-VLAN routing performance
3-5%...i must have something configured incorrectly or im horribly misunderstanding your post...any test i run through iperf pins the cpu unless i specify multiple channels...I run iperf from a client on vlan 10 to an unraid server on vlan 13....and unless i specify the -p flag the processor on the rb4011 is 100% usage. From what i am hearing you saying, regardless of anything to do with SMB i should be able to get a near 1Gbps iperf test between 2 VLANS with 3-5% cpu usage?Make sure that your problem is not affected by SMB - that protocol is, to put it mildly, horrible. Multichannel support was introduced with SMBv3 so check if you're forcing SMBv3 (by default modern clients will usually default to v2 for some reason).
Also, I was just poking around in my lab with a similar config: full duplex transfer between a computer tagging the traffic and RB4011 getting it via VLAN and then CPU-switching to a non-tagged ethernet. With ~1Gb/s symmetrical traffic (so in practice ~2Gb) using iperf3 I was getting ~970Mb/s both ways with 3-5% of CPU usage on RB4011.
Re: RB4011 Inter-VLAN routing performance
Something is definitely wrong, the fasttrack rule should be hitted only once per connection, after that it's traffic should be fasttracked.I am fairly certain it catching and processing the packets through those rules as when i run my iperf test now i see a massive spike in bytes and packets on those to rules in Winbox only during the duration of the test.
And something is preventing that.
The amount of actually fasttracked traffic can be seen in the special dummy rules located above.
Or you can open individual connections on the connections tab, and see statistics there.
Also check IP -> Settings if Allow Fast Path is ticked.
Re: RB4011 Inter-VLAN routing performance
Fastpath is not enabled, does this have to be working for fastrack to work correctly? I have VLAN filtering enabled on the router. From what i was reading if vlan filtering is enabled fastpath is disabled. Packet counters increase exponentially in the firewall during the iperf test, however the packet counters do not increase accordingly under IP settings.Something is definitely wrong, the fasttrack rule should be hitted only once per connection, after that it's traffic should be fasttracked.I am fairly certain it catching and processing the packets through those rules as when i run my iperf test now i see a massive spike in bytes and packets on those to rules in Winbox only during the duration of the test.
And something is preventing that.
The amount of actually fasttracked traffic can be seen in the special dummy rules located above.
Or you can open individual connections on the connections tab, and see statistics there.
Also check IP -> Settings if Allow Fast Path is ticked.
Thanks
You do not have the required permissions to view the files attached to this post.
Re: RB4011 Inter-VLAN routing performance
I think you need to limit your config and find the culprit because there's definitely something wrong. Initially I was running L2 through CPU, I setup a quick test with L3 for you where I'm transferring data inter-VLAN:3-5%...i must have something configured incorrectly or im horribly misunderstanding your post...any test i run through iperf pins the cpu unless i specify multiple channels...
- Trunk port (say ether5) with two VLANs on it: ether5-30 & ether5-31
- Access port (say ether5) where a proxmox machine is connected
- Computer connected to trunk port (ether5) with packets tagged with VID=31
- RB4011 has a bridge1-30 with two ports in it: ether5-30 and ether6
- Bridge has no collision/filtering/firewall/VLAN enabled
Running "iperf3 -s" on a VM and "iperf3 --bidir -c VM" on the client. In this scenario RB4011 is doing a L3 between VLAN 30 & 31 in full duplex at once. The result is fine for me (lower than 2Gb/s due to other factors):
Code: Select all
[ ID][Role] Interval Transfer Bitrate Retr
[ 7][TX-C] 0.00-10.00 sec 980 MBytes 822 Mbits/sec sender
[ 7][TX-C] 0.00-10.00 sec 979 MBytes 821 Mbits/sec receiver
[ 9][RX-C] 0.00-10.00 sec 1.06 GBytes 906 Mbits/sec 28 sender
[ 9][RX-C] 0.00-10.00 sec 1.05 GBytes 904 Mbits/sec receiver
During the test the CPU is not unreasonably loaded:
I initially misread your message (different timezone, it was middle of the night here ;)). With just VLANs 3-5% average is normal, but if you add L3 routing to the mix you can see the average CPU for the whole system being around 10%.iperf test between 2 VLANS with 3-5% cpu usage?
This ↑ is probably your issue. FT rule should not count all the data: Look at https://wiki.mikrotik.com/wiki/Manual:I ... figuration - especially the part where you match the traffic. Maybe you can export the config with hide-sensitive?Something is definitely wrong, the fasttrack rule should be hitted only once per connection, after that it's traffic should be fasttracked.
And something is preventing that.
You do not have the required permissions to view the files attached to this post.
Re: RB4011 Inter-VLAN routing performance
Yes, it absolutely does.Fastpath is not enabled, does this have to be working for fastrack to work correctly?
That should be the solution to your problem.
Fastpath is used by different handlers.I have VLAN filtering enabled on the router. From what i was reading if vlan filtering is enabled fastpath is disabled.
And all that would be true for for bridged traffic.
Bridge fastpath handler setting is located in Bridge -> Settings.
Your inter-Vlan traffic, however, is not bridged.
And for it to be handled by fasttrack handler fastpath has to be enabled in IP -> Settings.
Re: RB4011 Inter-VLAN routing performance
4011 is capable of doing inter-VLAN routing with firewall at line rate (10Gbit)
For that to happen:
1. You just need to remove the bridge and configure VLANs directly on SFP+ port.
2. You need to enable fasttrack on the firewall
For that to happen:
1. You just need to remove the bridge and configure VLANs directly on SFP+ port.
Code: Select all
/interface vlan
add interface=sfp-sfpplus1 name=vlan16 vlan-id=16
add interface=sfp-sfpplus1 name=vlan18 vlan-id=18
2. You need to enable fasttrack on the firewall
Code: Select all
/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
Re: RB4011 Inter-VLAN routing performance
4011 is capable of doing inter-VLAN routing with firewall at line rate (10Gbit)
Not if you believe that official L3HW documentation is current and accurate.
What RB4011 can do since ROS v7 is L2 HW offload - bridging within same VLAN with vlan-filtering enabled on bridge.
Re: RB4011 Inter-VLAN routing performance
as @xvo correctly stated, fastpath is a requirement for fasttrack... https://help.mikrotik.com/docs/display/ ... n+RouterOS
fasttrack = fast path + connection tracking ...
fasttrack = fast path + connection tracking ...
Re: RB4011 Inter-VLAN routing performance
Guessing this is my problem...however what if I want to have multiple trunk interfaces on the RB4011 in addition to the SFP+ interface carrying the same VLAN's isnt the bridge interface required?4011 is capable of doing inter-VLAN routing with firewall at line rate (10Gbit)
For that to happen:
1. You just need to remove the bridge and configure VLANs directly on SFP+ port.
Code: Select all/interface vlan add interface=sfp-sfpplus1 name=vlan16 vlan-id=16 add interface=sfp-sfpplus1 name=vlan18 vlan-id=18
2. You need to enable fasttrack on the firewall
Code: Select all/ip firewall filter add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
Thanks
Re: RB4011 Inter-VLAN routing performance
...however what if I want to have multiple trunk interfaces on the RB4011 in addition to the SFP+ interface carrying the same VLAN's isnt the bridge interface required?
Yes, if you switch or bridge same L2 subnet (VLAN) over more than one port (i.e. you're using your RB4011 partially as a switch, partially as a router), then you better use bridge.