Community discussions

MikroTik App
 
derdeagle
just joined
Topic Author
Posts: 24
Joined: Sat Jun 30, 2018 6:58 pm

VLAN performance issues in routing on RB3011

Tue Sep 13, 2022 11:09 pm

Hi all,

I am almost through migrating my old network setup with the new one. The new setup includes redundant routers and switches.

cr01 and cr02 are both RB3011UiAS-RM.
csw01 and csw02 are both CRS326-24G-2S+RM.

The PC is in VLAN 1000 and the storage is in VLAN 1010.

My problem now is that if I e.g. copy a file from the PC to the storage or vice versa the bandwidth is only ~ 150 Mbit/s (15 MB/s) and the CPU load of cr01 is at ~ 50% while the CPU load of csw01 is at about 2%. All the used ports are 1 Gbit/s and I am only using CAT 7 cables.

As soon as I put the PC in the same VLAN as the storage (1010) I get like ~ 900 Mbit/s bandwidth.

To avoid confusions: At first I had a connection between cr01, ether6 to csw02, ether 2 but changed the port on cr01 from ether 6 to ether3 because the router has two switching chips. I wanted to sort that possibility out. I even tried disconnecting cr02 and csw02 completely so that there are only cr01 and csw01 but with the same result.

Please see the image below for a stripped version - the redundant cabling for the devices are still missing.
screenshot_2022-09-13_21-48-04.png
I also attached the stripped configurations (wihtout DNS, CAPsMAN, address lists etc.) of the routers and switches.
The basic procedure of settings up VLANs have been taken from the post viewtopic.php?t=143620&sid=6be47775823e ... e795e6df15.

I know that routing is more expensive than switching but I highly doubt this is normal. Do I have a configuration issue somewhere?
You do not have the required permissions to view the files attached to this post.
 
derdeagle
just joined
Topic Author
Posts: 24
Joined: Sat Jun 30, 2018 6:58 pm

Re: Performance issues in routing on RB3011

Fri Sep 16, 2022 1:53 pm

Bumping in the hope for assistance.
 
holvoetn
Forum Guru
Forum Guru
Posts: 5405
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: Performance issues in routing on RB3011

Fri Sep 16, 2022 2:04 pm

I'm not offering a solution since I see the same behavior with inter-VLAN traffic on my Hex/AC3-setup and I am also searching for a solution.
So I am offering sympathy :lol:

When you look at details of load on cr01, I suspect one core will be maxed out close to or at 100% ?
Probably the VLAN stuff is being handled by one core only.
Same VLAN = nothing to do = much speedier.

How I circumvented it (for now, just like you already found out): devices which need to communicate frequently to each other, have been placed in the same VLAN.
And then I can go from AC3 to Hex over trunk to another AC3 using 930/940Mbps speeds ... (PC to iperf server)
But when a VLAN boundary needs to be crossed, it drops to 200-ish with Hex becoming the bottleneck on 1 core.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Performance issues in routing on RB3011

Fri Sep 16, 2022 3:32 pm

/interface vlan
add interface=general-bridge name=vl-999-mgmt vlan-id=999 OK
add interface=general-bridge name=vl-1000-int vlan-id=1000 ???
add interface=general-bridge name=vl-1010-media vlan-id=1010 ???
add interface=general-bridge name=vl-1020-wifi vlan-id=1020 ???
add interface=general-bridge name=vl-1030-tools vlan-id=1030 ???
add interface=general-bridge name=vl-1060-work vlan-id=1060 ???
You remove the rest of the configuration,
why you copy the VLAN 1000 and 1010 (and the others non-mgmt) to the CPU?
 
derdeagle
just joined
Topic Author
Posts: 24
Joined: Sat Jun 30, 2018 6:58 pm

Re: Performance issues in routing on RB3011

Fri Sep 16, 2022 7:28 pm

@rextended
For your question regarding the rest of the config ("???") - there is nothing more. What would you expect there?
why you copy the VLAN 1000 and 1010 (and the others non-mgmt) to the CPU?
It sounds like you know there is a better way. These are my first steps with Mikrotik and VLAN. Do you have a hint what I could do better?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Performance issues in routing on RB3011

Fri Sep 16, 2022 7:32 pm

While waiting for @anav to intervene,
I was wondering why those VLANs existed on the CPU.
If they are not used by other functions that you have removed from the export (it is always wrong to remove "what I think unuseful")
remove those useless VLANs. The "pass-through" VLANs that must not interact on the CPU must be present only in the bridge configuration.
 
derdeagle
just joined
Topic Author
Posts: 24
Joined: Sat Jun 30, 2018 6:58 pm

Re: Performance issues in routing on RB3011

Fri Sep 16, 2022 8:10 pm

I was further checking and playing around a bit with your idea.
You must have been talking about csw01/csw02 - as far as I can tell I can remove the VLAN interfaces there but not on cr01/cr02 as I need them there for VRRP to work.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Performance issues in routing on RB3011

Fri Sep 16, 2022 8:11 pm

Yes, I talk about the switches, not the routers, do not remove cpu mgmt access to the bridge or you lost the switch control.
 
derdeagle
just joined
Topic Author
Posts: 24
Joined: Sat Jun 30, 2018 6:58 pm

Re: Performance issues in routing on RB3011

Fri Sep 16, 2022 8:13 pm

Just to clarify: I removed firewall rules, static DHCP leases, comments which might leak personal information, SNMP etc.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: VLAN performance issues in routing on RB3011

Fri Sep 16, 2022 8:15 pm

Ok for the rest, but firewall rules can impact performance.
 
derdeagle
just joined
Topic Author
Posts: 24
Joined: Sat Jun 30, 2018 6:58 pm

Re: VLAN performance issues in routing on RB3011

Fri Sep 16, 2022 8:25 pm

That's why I stated in my first post that one of the first rules - the third in the forward chain to be exact - is to allow established and related traffic in the hope to reduce the impact in performance.
I now also disabled all the unnecessary VLAN interface on the csw01/csw02 - it is all working as before, so thank you for that hint!

Do you have any idea for the initial performance issue?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: VLAN performance issues in routing on RB3011

Fri Sep 16, 2022 10:37 pm

Sincerely not, probably I miss something that I do not notice on the config, sorry.

Another pair of eyes can help if someother check that...
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 887
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: Performance issues in routing on RB3011  [SOLVED]

Fri Sep 16, 2022 10:41 pm

When you look at details of load on cr01, I suspect one core will be maxed out close to or at 100% ?
Probably the VLAN stuff is being handled by one core only.
Same VLAN = nothing to do = much speedier.

How I circumvented it (for now, just like you already found out): devices which need to communicate frequently to each other, have been placed in the same VLAN.
And then I can go from AC3 to Hex over trunk to another AC3 using 930/940Mbps speeds ... (PC to iperf server)
But when a VLAN boundary needs to be crossed, it drops to 200-ish with Hex becoming the bottleneck on 1 core.
I would suspect the same, e.g. one core saturated.

But the reason that inter-vlan traffic is slower is because it has to be routed and go through the firewall. Stuff within the same vlan can be switched in the hardware switch ASIC. For example in @derdeagle's main diagram, if both the PC and Media server were on the same vlan, traffic between them would never even go across the trunk like to the RB3011, it would all happen at L2 in the CRS326.

Even if a more powerful router replaced the RB3011, inter-vlan traffic would still be limited to 1Gbps aggregate. This is because the trunk link between the RB3011 and CRS326 has to carry the traffic both ways, so a uni-directional 1Gbps flow between the media server and the PC would saturate both directions in the trunk link. This is discussed in post #13 of the Slow handover between vlans thread.
 
derdeagle
just joined
Topic Author
Posts: 24
Joined: Sat Jun 30, 2018 6:58 pm

Re: VLAN performance issues in routing on RB3011

Fri Sep 16, 2022 11:51 pm

Thank you both for your time and effort.
It really seems like I have to put the server into the same VLAN as the storage to achieve a higher throughput.
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 887
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: VLAN performance issues in routing on RB3011

Sat Sep 17, 2022 12:19 am

Do you have any idea for the initial performance issue?
It may be that you are using vlan filtering on the RB3011 and ROS does not support the QCA8337 switch ASIC for vlan-filtering, even in v7. In this respect, even MT7621A in the RB750Gr3 has better ROS support for vlans than the RB3011. See Bridge Hardware Offloading for what I mean.
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 887
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: VLAN performance issues in routing on RB3011

Sat Sep 17, 2022 12:26 am

The lack of vlan-filtering support should only make intra-vlan communication on the RB3011 slower. If you have to route between vlans anyway, the CPU is already going to be in the path, so whether there is vlan-filtering support or not will not make much difference.

The all intra-vlan traffic between ports on the RB3011 will be limited by the CPU on the RB3011 (and probably by one core).
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 887
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: VLAN performance issues in routing on RB3011

Sat Sep 17, 2022 12:31 am

I would expect that even if you configured ether4 and ether5 on the RB3011 to be in the same vlan, you would get worse performance than if you had two ports of the CRS326 in the same vlan.
QCA8337 does not support ROS hardware vlan filtering.png
Here are the paths the traffic would take if you use different vlans
inter-vlan routing.png
Here is the path the traffic would take if the two devices are in the same vlan.
intra-vlan switching.png
You do not have the required permissions to view the files attached to this post.
 
derdeagle
just joined
Topic Author
Posts: 24
Joined: Sat Jun 30, 2018 6:58 pm

Re: VLAN performance issues in routing on RB3011

Sun Sep 18, 2022 7:37 pm

@Buckeye
Thank you very much for the nice and helpful explanation. It does actually make sense.
In the meantime I was further playing around a bit and as my aim was to make most of my devices network-redundant I created a bond interface (active-backup) on the server and an additional VLAN interface (via the bond) so that the server now is in both VLANs - the "original" one and in the one where the storage is so that it actually takes the path you were showing in the last picture to the storage.
Nevertheless it's kind of a let-down that the RB3011 is quite limited in terms of throughput when VLANs are used.
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 887
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: VLAN performance issues in routing on RB3011

Mon Sep 19, 2022 6:41 am

Nevertheless it's kind of a let-down that the RB3011 is quite limited in terms of throughput when VLANs are used.
You may be able to get it to work at hardware level by disabling bridge vlan-filtering and using the switch vlan method like in the following youtube videos (on only the RB3011)

Configure VLAN on built-in switch chip in MikroTik by Inquirinity

Configuring VLAN's on MikroTik RouterBoard using the Switch Chip by Maher Haddad (MAICT Consult)

But that will only help when switching between switch ports on the same switch in the RB3011 that are in the same vlan. All inter-vlan traffic still needs to be routed, and so the CPU is still going to be in the path.

Internally the RB3011 is a "router on a stick" (with a vlan-aware switch connected). And in the RB3011 this is duplicated (two CPUS, and two switch chips). But without an external jumper, the two switch chips are only connected via a path through a CPU, as you can see in the RB3011UiAS-RM block diagram.
 
derdeagle
just joined
Topic Author
Posts: 24
Joined: Sat Jun 30, 2018 6:58 pm

Re: VLAN performance issues in routing on RB3011

Mon Sep 19, 2022 11:26 am

But that will only help when switching between switch ports on the same switch in the RB3011 that are in the same vlan. All inter-vlan traffic still needs to be routed, and so the CPU is still going to be in the path.

Internally the RB3011 is a "router on a stick" (with a vlan-aware switch connected). And in the RB3011 this is duplicated (two CPUS, and two switch chips). But without an external jumper, the two switch chips are only connected via a path through a CPU, as you can see in the RB3011UiAS-RM block diagram.
This is getting more and more interesting.

I will take a look at the videos and try to configure it in GNS3. Thank you very much, though I don't have high hopes as currently the router only comes into play with the inter-vlan-traffic which would according to your explanation still be done by the CPU instead of the ASIC.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11439
Joined: Thu Mar 03, 2016 10:23 pm

Re: VLAN performance issues in routing on RB3011

Mon Sep 19, 2022 12:06 pm

Inter-VLAN routing is done by CPU on all Mikrotik devices ... except for a few models capable of L3HW offloading if correctly configured and certain conditions are met. For the rest of devices HW offload means L2 only (ethernet level which is intra-VLAN).

Who is online

Users browsing this forum: Ahrefs [Bot], GoogleOther [Bot], neki and 81 guests