Community discussions

MikroTik App
 
Deathsight
just joined
Topic Author
Posts: 4
Joined: Wed May 27, 2020 8:54 pm

hAP ac2 – slow transfer speed between vlans

Thu Jul 16, 2020 3:57 pm

Hello everyone,

I would like to ask this awesome community for help with my setup. I tried to search older threads, but I didn’t find anything that would help me resolve this.

I have had hAP ac2 with a dumb switch for a few months and it worked great. Recently I got inspired by this awesome guide (viewtopic.php?t=143620), to upgrade my flat network and segment devices into vlans.

I have picked up CSS326-24G-2S+RM, configured SwOS and connected it to my hAP ac2 and setup vlans on single bridge according to the mentioned guide. Vlans themselves work great, every client gets correct IP from DHCP etc. But I have noticed significant slow down on file transfers from network shares on my Unraid server.

I did several tests using iperf3 to test throughput between vlans and it seems that its limited to roughly 395 Mbits/sec. between computers on the same vlan I’m getting the usual 900+ Mbits/sec. I have also noticed that during these transfers or iperf3 tests, only one core of the router is running at 100% and the rest are almost idle.

Here is my current config file
confighapac2.rsc
Right now I have added few firewall rules to allow all vlans everywhere to troubleshoot this issue, later I will have only few allow rules for things that need to talk to each other.

and here is diagram of my network
My Home Network v2.pdf
I was choosing between CSS326-24G-2S+RM and CRS326-24G-2S+RM. Would the CRS326 be better for my setup?

Thank you for your replies in advance.
You do not have the required permissions to view the files attached to this post.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: hAP ac2 – slow transfer speed between vlans

Thu Jul 16, 2020 4:18 pm

When two hosts in different VLANs communicate, that communication flows via router and any services it may have configured. Including firewall. In your case that's hAP ac2. In theory, this device should be able to route with speeds around (or above) 1Gbps even with some firewalling. However, this really depends on how optimal are firewall rules. So you should post full firewall configuration to get some useful feedback (execute /ip firewall export and /ipv6 firewall export if you're using IPv6). Also beware that ROSv6 doesn't play any advanced performance tricks with IPv6 (it has fasttrack for IPv4 which helps with firewalling performance a lot).

And no, selection of switch in this case doesn't play any role whatsoever, the whole burden is on router.

Edit:

I see you posted complete config. It seems just fine, I don't think you could optimize it any further.
There's a gotcha with throughput: any single TCP (and possibly UDP as well) stream, flowing cross the router, will be handled by single CPU. In that case single core performance can severely limit the throughput. When there are a few parallel TCP streams, they will be handled by different CPUs and the overall throughput will be much better. File transfer (using e.g. windows file sharing / SMB) is single connection so it's affected. You could verify this by running several file transfers in parallel, cumulative speed should eventually cap at wire speed (or only slightly lower).
 
Deathsight
just joined
Topic Author
Posts: 4
Joined: Wed May 27, 2020 8:54 pm

Re: hAP ac2 – slow transfer speed between vlans

Thu Jul 16, 2020 4:39 pm

Thank you for your reply. So far I have tested mostly with iperf3 using 1 to 8 streams, but ended up with similar numbers. Today I will try transfering several things in parallel between two PCs and report back with some numbers.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: hAP ac2 – slow transfer speed between vlans

Thu Jul 16, 2020 4:44 pm

So far I have tested mostly with iperf3 using 1 to 8 streams, but ended up with similar numbers.

iperf run with number of parallel streams should be able to max the transfer rate. But try to run profiler on hAP ac2 during transfers to see if CPU really is the bottleneck.
BTW, due to the problems you're seeing it's advised to have server in same VLAN as (most) of client machines ... unless you have a good reason not to.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: hAP ac2 – slow transfer speed between vlans

Thu Jul 16, 2020 6:49 pm

is the speed test being done with one end on a wifi connection or is this being done by a PC wired through the hapac?
 
biomesh
Long time Member
Long time Member
Posts: 561
Joined: Fri Feb 10, 2012 8:25 pm

Re: hAP ac2 – slow transfer speed between vlans

Fri Jul 17, 2020 12:01 am

You have vlan filtering enabled on your bridge which disables hardware offloading on the hap ac2.

https://wiki.mikrotik.com/wiki/Manual:I ... Offloading

You should really be doing the vlan tagging/filtering on your css326. It is swos, so there is no bridge config, just vlan config.

On the router side you should have vlan interfaces and let the router route the traffic between vlan interfaces. The traffic hitting the router should all be tagged and your ip addresses/services should be assigned to the specific vlan interfaces. The following youtube video from a previous MUM goes over this.

https://www.youtube.com/watch?v=_Tjcoq0aRR4
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: hAP ac2 – slow transfer speed between vlans

Fri Jul 17, 2020 7:43 am

@biomesh: when it comes to inter-VLAN routing, it doesn't matter how device deals with VLANs because routing is CPU task[*]. Hence my note about having server in the same VLAN as (most of) clients.

[*] For nit-pickers: yeah, with ROS v7, CRS317 can HW offload routing as well ... but we're talking about 99.999% of cases here.
 
biomesh
Long time Member
Long time Member
Posts: 561
Joined: Fri Feb 10, 2012 8:25 pm

Re: hAP ac2 – slow transfer speed between vlans

Fri Jul 17, 2020 1:55 pm

@mkx I agree with you 100%. The question is if the extra load from the bridging is causing the extra load. If he were to handle vlan tagging on the switch and just do a 'router on a stick' config would that get the performance where it needs to be?
 
Deathsight
just joined
Topic Author
Posts: 4
Joined: Wed May 27, 2020 8:54 pm

Re: hAP ac2 – slow transfer speed between vlans

Fri Jul 17, 2020 2:05 pm

All clients are wired to CSS326 and there is a trunk from CSS326 to Hap AC2.

I have done some tests doing parallel file transfers between clients. In all tests I was seeing around 45-49MB/s transfer rates. If I added another file trasnfer this speed was just split, but the total was still the same. These speeds are consistent with my iperf3 tests that showed around 395 Mbits/sec which is right around that 49MB/s mark.

I have monitored routers CPU load during these tests and it was always one core at 100% load, another core at around 30% and the other two cores idle 0-1%.

Having server on the same Vlan as clients would solve this issue, because with that setup im seeing expected near 1gig speeds. I would like to have my server in different Vlan for several reasons. My plan is to pay for public static IP from my ISP and host some gameservers (Minecraft, Factorio,ARK etc.) for my gaming group and running some public facing services like Nextcloud with reverse proxy for my family.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: hAP ac2 – slow transfer speed between vlans

Fri Jul 17, 2020 2:43 pm

@Deathsight: which process(es) did consume those 100% of single core? And a (desperate) suggestion: you might want to upgrade ROS to 6.47.1, it might make some difference.

@biomesh: when router deals with multiple VLANs, it has to tag frames passing CPU (IP packets being routed) in software regardless the underlying L2 config because both bridge and switch chip need them tagged when passed from CPU. The only difference between using bridge and switch chip is what happens on other ports ... and on trunk port towards switch nothing happens because tags are preserved both on ingress and egress, only filtering takes place.
The big difference is when device is used as switch ... where frames between ports either pass switch chip only (HW offload) or CPU (and switch chip twice - without HW offload).
 
biomesh
Long time Member
Long time Member
Posts: 561
Joined: Fri Feb 10, 2012 8:25 pm

Re: hAP ac2 – slow transfer speed between vlans

Fri Jul 17, 2020 2:59 pm

@mkx - My point was just going to have the hap ac2 as just a router (1 wan, 1 Lan) and have it only do the routing between vlans (and wan). This was to see if the performance was better than with the switch/bridge config.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: hAP ac2 – slow transfer speed between vlans

Fri Jul 17, 2020 9:37 pm

And I was essentialy saying that I'd be much surprised if there was any noticeable difference ... Because in that case a) you don't even need any bridge and b) there's nothing to offload to HW
 
User avatar
mutluit
Forum Veteran
Forum Veteran
Posts: 821
Joined: Wed Mar 25, 2020 4:04 am

Re: hAP ac2 – slow transfer speed between vlans

Sat Jul 18, 2020 1:51 am

I too had experienced similar dropping performance problems with the same router.
The reason was non-optimal firewall rules. After fixing it the performance came back to about 950 Mbps from previous about 250 Mbps. WAN/LAN as well LAN/LAN as well sameLAN.
I would suggest to try the following rules as the very first rules in your firewalls on both/all of the involved routers / switch-routers (CRS etc.):

/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related
add action=accept chain=forward connection-state=established,related,untracked
add action=fasttrack-connection chain=input connection-state=established,related
add action=accept chain=input connection-state=established,related,untracked
add action=fasttrack-connection chain=output connection-state=established,related
add action=accept chain=output connection-state=established,related,untracked
...

You can in the CLI "print" the rules, and "move" each to another position. As said, the above ones should be moved to the top of the rules table.
Hope it helps.
Btw, this is how I do in RouterOS, I think in SwOS this should be possible too, but I've no experience with it.
 
Deathsight
just joined
Topic Author
Posts: 4
Joined: Wed May 27, 2020 8:54 pm

Re: hAP ac2 – slow transfer speed between vlans

Tue Aug 04, 2020 12:36 pm

I have updated to ROS 6.47.1 just to make sure the problem is not in the ROS version.
After that I rerun my iperf3 tests. The results are the same. Around 400 Mbits/sec with 1, 2 and 8 parallel streams. Here are screenshots from my tests showing core usage on Hap ac2
https://imgur.com/a/cKdGQ26

I will try to work on my firewall rules, but I'm not sure if I will be able to make noticeable difference. I have fasttrack rules for forward chain in place already.
 
User avatar
xvo
Forum Guru
Forum Guru
Posts: 1237
Joined: Sat Mar 03, 2018 1:12 am
Location: Moscow, Russia

Re: hAP ac2 – slow transfer speed between vlans

Tue Aug 04, 2020 1:06 pm

Is fasttrack actually working?
Check in /ip firewall connections that connections between vlans (or any at all) are being fasttracked.
 
heinanr
just joined
Posts: 1
Joined: Sat Mar 13, 2021 8:13 pm

Re: hAP ac2 – slow transfer speed between vlans

Sun May 09, 2021 2:08 pm

I know this is relatively old discussion but felt like sharing my experience.

I'm having similar issues and similarly upgraded my network as was inspired by the same article and other good reviews about MikroTik CRS devices.
My network looks very similar on diagram to Deathsight.

Anyway I did not see any comments while doing my research that routing between VLAN's would be so heavily limited by CPU.
I have CRS328-24P-4S+ with 6.48.1 version.

I have been running tests using the latest version of iPerf3 and I have not been able to get much over 440 Mbits/sec between VLAN's.
This is after dropping all the non-essential firewall rules and using fastrack rules to optimize flows and flows are fastracked according to /ip firewall connections print command.
Before I started to optimize I was getting around 220 Mbit/sec between VLAN's. It took me a while to figure out it was actually CPU bound issue as it never even occurred to me that it would be so heavily restricted (I was so sure it was either my server or network cables I have made ... as in progress of converting everything to 10Gbit at home).
In comparisons within the same VLAN I get constantly around 9.4 Gbits/sec and CPU doesn't get impacted at all (obviously).

So I guess I have to redesign my network and either get more powerful router to separate networks or just make the network flat and just use it as a switch.
I should have done a bit more research .. it all seemed a bit too good to be true for the price you pay, which it was.

Anyway in overall I'm quite happy with the hardware just need to adjust my expectations.

As a final note I was surprised what difference adding 3rd SFP+ 10GT copper module made in fan noise.
With 2 SFP+ modules fans are "silent" (around 4500 rpm) but after adding 3rd module temperature rose too much and fans started to run closer to 7700 rpm.
Probably need to add front fan(s) to keep it cooler and quieter.

Who is online

Users browsing this forum: No registered users and 31 guests