Community discussions

MikroTik App
 
cjones277
just joined
Topic Author
Posts: 6
Joined: Tue Feb 07, 2023 11:40 pm

heX-S High CPU usage with Bridge VLAN switching.

Wed Feb 08, 2023 12:25 am

Hiya all. First time posting here.

I have had a heX-S and a hapAC for a bit over a year now a they have been great. I do a lot of virtualization with Proxmox off of a PowerEdge with two NICs and recently decided to implement VLANs to segment the network (as well as making a Guest Wi-Fi). Using what I think is a common setup with VLANs on a bridge (the only bridge), all seems to work as I desire. The exception is that when any connection streams data that crosses from one VLAN to another, there is a noticeable spike in the heX-S' CPU usage. If it is an inter-linked set of servers/containers where the data crosses a VLAN more than once, the spike can reach well into 70%.

I just updated to RouterOS 7.7, hoping to get the Hardware Offloading added in for the heX-s' switch chip. And while the ports indicate they are Hardware-Offloaded (same as in RouterOS 6.49), the CPU spike is still there.


Basic Network Map:
heX-S (PowerEdge T440 Proxmox node directly-attached to Ether 3 and 4, HP Laser printer on SFP1, hapAC on Ether 5) > hapAC for Wi-Fi devices. I use the Access List to VLAN tag certain devices.
I have NO dedicated switch as the heX-s provides enough ports - everything else is Wi-Fi.

VLAN 2
Not used yet. I was setting this one up for main LAN devices that are currently Untagged.

VLAN 5
Wi-Fi LAN - mostly IOT and Home Automation - The hapAC does not tag unless connected to the Guest Wi-Fi virtual SSID - I use the Access List to tag certain devices for this VLAN

VLAN 10
External - Internet-facing servers and containers such as a web server, Nextcloud, Minecraft and things like that. All Virtual and all coming from Ether 4

VLAN 20
Wi-Fi Guests - Tagged by a Virtual SSID on the hapAC.

Untagged/PVID 1
Everything else. By default even non-guest Wi-Fi devices end up here until I tag them in the hapAC

I imagine I am leaving something out. I will try to provide any information I can.

To reiterate, the VLAN itself is WORKING. Devices assigned to the VLANs IP addresses associated with them and firewall rules work. Just when streams (usually large ones like movies and such) cross VLANs, the CPU spikes.

Is it because not all traffic is tagged? Perhaps because this configuration existed before upgrading to RouterOS 7.7?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11433
Joined: Thu Mar 03, 2016 10:23 pm

Re: heX-S High CPU usage with Bridge VLAN switching.

Wed Feb 08, 2023 4:25 pm

If hEX S is used as router, then seeing traffic travelling between different VLANs hitting CPU is normal. Routing is not HW offloaded on hEX S, only switching within same VLAN is.
 
cjones277
just joined
Topic Author
Posts: 6
Joined: Tue Feb 07, 2023 11:40 pm

Re: heX-S High CPU usage with Bridge VLAN switching.

Wed Feb 08, 2023 4:53 pm

Yeah, that makes sense. I think a 30-70% CPU hit for a network transfer is a bit high for an almost 900MHz 2 core/4 thread CPU; even when traversing VLANs. But...it is what it is.

I guess I can mitigate the issue by moving the webserver and nextcloud (the two that primarily account for large file transfers) out of the external VLAN until I can get a beefier network appliance.

Thanks for the reply.
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 886
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: heX-S High CPU usage with Bridge VLAN switching.

Wed Feb 08, 2023 5:07 pm

I have had a heX-S and a hapAC for a bit over a year now a they have been great. I do a lot of virtualization with Proxmox off of a PowerEdge with two NICs
---snip---
Basic Network Map:
heX-S (PowerEdge T440 Proxmox node directly-attached to Ether 3 and 4
Are ether3 and ether4 is different vlans?

Are you bonding? That won't improver performance on the hEX S (at least I can't see how it would, based on the MT7621 architecture).
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11433
Joined: Thu Mar 03, 2016 10:23 pm

Re: heX-S High CPU usage with Bridge VLAN switching.

Wed Feb 08, 2023 5:11 pm

Each product has page with official test results, so does hEX S. The rule of thumb is that number under "routing, 25 filter rules, 512 byte packet size" best represents average real life performance. Numbers for older devices were obtained with ROS v6 runnung, and that one was a bit better in routing than v7. In your particular case the mentioned number might be too low if you only have a simple firewall. And another gotcha: firewall rules for some connection are always run by same CPU core. Which can be bottleneck for single connection performance (such as file copy using SMB). Run CPU profiler to see if single core is fully utilized.
 
cjones277
just joined
Topic Author
Posts: 6
Joined: Tue Feb 07, 2023 11:40 pm

Re: heX-S High CPU usage with Bridge VLAN switching.

Wed Feb 08, 2023 7:21 pm

I have had a heX-S and a hapAC for a bit over a year now a they have been great. I do a lot of virtualization with Proxmox off of a PowerEdge with two NICs
---snip---
Basic Network Map:
heX-S (PowerEdge T440 Proxmox node directly-attached to Ether 3 and 4
Are ether3 and ether4 is different vlans?

Are you bonding? That won't improver performance on the hEX S (at least I can't see how it would, based on the MT7621 architecture).
Yes, they are on different VLANs. No, they are not bonded.

Ether 3 is for the untagged VMs - which at the moment is just my file/media server and TrueNas VM I am testing.
- An additional note, implementing this setup was recent - in an attempt to both improve LAN-LAN performance of the heavy-transfer VMs and troubleshoot what I believed to poor performance for inter-VLAN routing on the hEX-s. Before this, ALL VMs, tagged and untagged just used Ether 4.

Ether 4 is for VLAN 10 - Internet-facing VMs and containers, such as my webserver, nextcloud server, a couple of game servers and such.
- I just tag the VMs in the network config in Proxmox.
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 886
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: heX-S High CPU usage with Bridge VLAN switching.

Wed Feb 08, 2023 8:34 pm

If you are using only a single port of the hex for each vlan, and they are all "access" ports (and there are no trunks), then you don't need vlans.
But I don't think using vlans is going to make a big difference when things are already being routed.

When you had everything in on LAN, the hEX wasn't even involved in the host to host transfers, it was all switched by the MT7530 switch ASIC "included" in the MT7621 SOC.

And there on the hEX you could have two full duplex wire speed transfers between 4 hosts connected to the switch, the switch won't be the bottleneck. But with the hEX you may be seeing the limitation due to the 1 G/b cpu/switch port.

If you aren't utilizing the ports as switch ports, you would probably get better performance with v6 (although I have never used v6 on my hEX S, other than when upgrading to 7).
 
cjones277
just joined
Topic Author
Posts: 6
Joined: Tue Feb 07, 2023 11:40 pm

Re: heX-S High CPU usage with Bridge VLAN switching.

Thu Feb 09, 2023 12:47 am

Yeah. I primarily set them up to make it easier to control access at the router level. Since I have several internet-facing "servers" and also work from home, so there is also a desire to keep work machines from poking around in the network too much. And they DO work in that capacity

For now, I will just move the heavy-hitters on the external LAN back to the main (untagged) LAN and just tighten the firewall rules on those devices to control what they can hit. If that fails to mitigate it enough, I will just use a single LAN (maybe a /16 so I can still organize like-devices).

I may either grab a switch at some point, or maybe a beefier router that has a CPU that can better absorb the hit.

This little router has been so great, I just must have gotten complacent about how much it could do. And honestly, it has not had a noticeable impact as of yet, but if ONE PC can hammer the router for up to 70% (depending on how many times the steam crosses VLANs), it won't take much to push it the rest of the way to 100.
 
llag
Frequent Visitor
Frequent Visitor
Posts: 72
Joined: Sat Aug 04, 2018 12:12 am

Re: heX-S High CPU usage with Bridge VLAN switching.

Thu Feb 09, 2023 2:00 am

For now, I will just move the heavy-hitters on the external LAN back to the main (untagged) LAN and just tighten the firewall rules on those devices to control what they can hit. If that fails to mitigate it enough, I will just use a single LAN (maybe a /16 so I can still organize like-devices).
You may want to consider making the heavy hitters (web server and nextcloud) dual homed by putting them in both the external facing VLAN and the main VLAN through a VLAN trunk (or 2 interfaces). I guess that Proxmox can route these VLANs to these heavy hitters on both VLANs That will avoid routing on the HEX-S for internal traffic as these functions will appear on both VLANs. That is assuming that the clients can address these functions on the IP-addresses on the main vlan. The HEX will do the routing of the incoming traffic to the web server and Nextcloud, while internal clients do not need to be routed.
the only limitation is that the clients need to be able to deal with this.
 
cjones277
just joined
Topic Author
Posts: 6
Joined: Tue Feb 07, 2023 11:40 pm

Re: heX-S High CPU usage with Bridge VLAN switching.

Thu Feb 09, 2023 8:28 pm

Yeah, I can try that too. Adding an interface in Proxmox is rather easy. I can add an untagged network interface on the bridge (Linux Bridge, not Mikrotik Bridge) that is connected to Ether 3, and set internal DNS to point to THAT IP for the "heavy-hitters". I will try that first.
 
cjones277
just joined
Topic Author
Posts: 6
Joined: Tue Feb 07, 2023 11:40 pm

Re: heX-S High CPU usage with Bridge VLAN switching.

Tue Feb 14, 2023 5:27 pm

UPDATE:

Well, I ran into some other issues with performance that do not appear to be VLAN-related - possibly just issues with RouterOS 7.7 on the hEX-S. Such as LAN-throughput and VPN performance/latency.

I ended up downgrading back to RouterOS 6.49.7. And since the little bugger refused to restore my pre-7.7 backup afterward, I am rebuilding my network from scratch - without VLANs for now.

Now I am toying with buying a beefier Mikrotik unit (RB4011 or RB5009 both look nice), or just build my own. One (or more) 4-port NIC cards in that PowerEdge (or another machine laying around) with a RouterOS license and I can easily build a beefier 'Tik than I can buy... I'll deal with that down the road. For now, the little hEX-S is happy back the way it was before my networking ambitions out-paced its hardware...

Thanks for the replies all!

Who is online

Users browsing this forum: No registered users and 26 guests