Dear all,

To make it short, my setup is a normal setup with:
CCR2004-1G-12S+2XS router <=> CRS312-4C+8XG-RM switch with VLANs <=> 10gb devices (fiber or RJ-45)

Testing with iPerf3, here are the results for single threads (one TCP steam):
VLAN A to Internet : 2 Gbit/s or more (fiber line limited to 2.5Gbit/s). Could be more on a 10Gbit fiber line.
VLAN A to VLAN B : 1.2 Gbit/s
VLAN A to VLAN A : 6 Gbit/s
VLAN B to VLAN B : 6 Gbit/s
I have around 25 rules with fasttrack enabled.

I want to use the CCR2004 router for VLAN inter routing as it is safer to keep all rules in one place.

I don't understand why inter-vlan routing is so slow.
Is this because the CCR2004-1G-12S+2XS does not have a switch chip?

When I purchased the CCR2004, I looked at the block diagram and saw 2x25Gbit full-duplex speed to CPU.
So I did not expect to reach only 1Gb/s in inter vlan routing.

Is there something wrong?



Kind regards,
FFries

Answer 1

Is this because the CCR2004-1G-12S+2XS does not have a switch chip?

Short Answer.. NO!
Most, if not all Cloud Core Router (CCR) from Mikrotik don't have a Switch-Chip
They are usually optimize for Routing over CPU



Answer 2

Is there something wrong?

I never owned a CCR2004-1G-12S+2XS,
but I would expect a bit more performance,
then you described in you Post.

Maybe another Forum-User can confirm,
but i would look for > 4.5 Gbps Routing between VLAN A / VLAN B



Answer 3:
VLAN A to VLAN A : 6 Gbit/s
Are you sure you are not at the limit of your iPerf3-PCs ?

I assume you used two PC's using iPerf3 to test the Network.
What kind of Speed to you get when connecting them directly together ?

I'm seeing the same speeds.

Setup:
- CCR2004 in the data center running 6.47.10 with 10Gbps Internet on one port and a VLAN-filtered bridge facing downstream
- PTP link to the shop is a 6Gbps radio with <500us latency
- CCR2004 running 7.1.1 at the shop with VLAN-filtered bridge configuration as well
- Speedtest.net server in the DC, and my laptop in the shop with 10GbE adapter.

Speed tests between laptop and server max out around 1.2Gbps, CPU @ 70-80%.
Onboard speed tests between CCR2004's (over dedicated VLAN) top out at 6Gbps, both one-way TCP and UDP. CPU @ 80-90%.

Fasttrack counters don't increment at all, making me wonder how much the CPU is doing on inter-VLAN routing for CCR2004.

On a thread about the CRS300 series and HW offloading, they mention that fasttrack between VLANs on VLAN-filtering bridges isn't working on 7.x yet, but is in the works. I would expect it to work on RouterOS 6, however. One commenter on a Reddit thread pointed out that ROS 6 on CCR2004 runs virtualized (or emulated, 32-bit OS on 64-bit platform), hence poor hardware performance.

There are methodological problems in my testing:
* There could be a limit with my network card (fiber) showing 6Gbit/s on the same VLAN (so direct connection using a Mikrotik switch). Maybe the limit is the other host, maybe I need a larger MTU (jumbo frames), I don't know precisely. I will soon receive a new 10Gb device so I will be able to do additional testing with proper hardware.
* I connect to a switch which is connected to the CCR2004 using a single SFP+ 10Gbit/s fiber module. Traffic needs to go to the CCR2004 and then flow back to the other host. So this is bidirectional and two streams cannot go 10Gb/s rather 5Gb/s each maximum. Of course there are two fibers, but with TCP there is always upstream and downstream at the same time. Therefore to do proper testing I either need to connect directly to the CCR2004 or use a bonding with at least two links. I cannot use a DAC because the switch is too far away from the CCR2004.

To make it short: to test inter-vlan routing, upstream links between CCR2004 and switch should always be a bonding with at least 2 interfaces.

I purchased a RB5009UG+S+IN just to compare inter-vlan speed.
The RB5009UG+S+IN will never be able to reach 10Gbit/s inter-vlan speed because it has only one SFP+ interface.

Please correct me if I am wrong.

Looking at the diagram, there are two full-duplex 25gbit/s links linking the CPU to the port extender.
If traffic is flowing through the CPU, the maximum theoretical inter VLAN routing speed is limited to 25Gbit/s.

This also suggests that the CCR2004 is not the right hardware for managing inter vlan because it lacks a switch chip.
Inter VLAN routing should always occur on the switch itself, not the CCR2004.

Correct me if I am wrong.

Inter VLAN routing should always occur on the switch itself, not the CCR2004.

Inter-VLAN routing is still routing which on vast majority of devices (CCR2004 and for now RB5009 both included) involves CPU. What most switch chips do well is intra-VLAN packet forwarding (a.k.a. switching).

The RB5009UG+S+IN will never be able to reach 10Gbit/s inter-vlan speed because it has only one SFP+ interface.

SFPs are generally full-duplex interfaces which means they are theoretically capable of transferring full speed in each direction simultaneously. For inter-VLAN routing this means receiving traffic @10Gbps from source VLAN and transmitting traffic @10Gbps to destination VLAN at the same time. It is true what you write about return traffic for TCP streams, where ACK packets from destination VLAN compete with forward traffic from source VLAN over same physical link in same direction and delaying ACK packets means lowering overall end-to-end throughput by margin larger than raw throughput needed for ACK packets. This effect can be overcome by using UDP for testing of raw routing/link speed, using (large number of) parallel TCP streams helps to certain extent as well.

As to using of bonding interfaces: most bonding Tx strategies imply using same physical interface for all traffic between a pair of hosts. Indeed it could be different physical interface for different direction and/or different VLAN, but it's a game of luck. The only strategy to keep traffic of two VLANs appart is to use two links, but configured to use one per VLAN. With increasing number of VLANs this becomes a bad strategy though. Which means that bonding is the way to go, but bonding might not improve speed (for certain connections at least ... even with L3+L4 Tx hash parallel connections might not spread over all bond members exactly evenly). However, with some luck using bonded trunk might allow tester to hit other device's bottlenecks.

Thank you for clarification.

I set up a bonding uplink to the CCR2004 and now intra vlan routing is worse:

[ ID] Interval Transfer Bitrate
[ 5] 0.00-3.78 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-3.78 sec 516 MBytes 1.15 Gbits/sec receiver

1.15Gbit/s on one stream and 2.5 on two streams from vlanA to vlanB.
1.15 Gbit/s sound like the CCR2004 is limited to gigabyte vlan routing.
Very strange.

I am waiting to receive more hardware to do more testing.

The general problem with bonding on devices without switch chip supporting bonding is that using bonding adds yet another layer of abstraction handled by already overloaded CPU ... through which all packets have to pass. So sometimes using bonding actually decreases performance and in those cases only benefit is link redundancy.

But then, general routing performance of most MT devices does not shine when testing with single high-throughput connection. Specially for devices with multiple CPU cores (i.e. all of the modern ones) it is highly recommended to run multiple parallel test streams (e.g. use "-P 8" when running iperf3 in TCP mode). This way packets belonging to different connections will be processed by different CPU cores and throughput will thus increase. N.b. in real-life scenarios, where router serves multiple clients, each communicating with multiple servers, router will process larger number of concurrent TCP connections and will actually perform in the line of proposed test case (iperf3 with multiple parallel TCP streams).

The issue was fixed with latest beta v7.2rc3.

*) bridge - added fast-path and inter-VLAN routing FastTrack support when vlan-filtering is enabled;

After enabling fast-path, I can reach wire speed in inter-vlan filtering.
Routing is also way faster as a result ...

So setting this issue to FIXED.
Users with inter-vlan filtering SHOULD upgrade to beta.

Thank you very much Mikrotik Team and Community.

Hi all!
Did I understand correctly that in RouterOS v6 this problem (slow inter-VLANs speed) has not been fixed and i should to upgrade to RouterOS v7 ?