Inter VLAN traffic is limited to 1Gbps (RB5009)

RouterOS General
1

Board - RB5009UG+S+
Version - 7.11.2 (stable)

The Issue:

When connected into vlan90, 10Gbps speeds are seen, understandable as all traffic is on the 10Gbps switch at this point.
When connected into vlan90, 2.5Gbps speeds are also seen to the interface on eth1 which is configured in vlan90 - but this at least shows traffic from the switch to the eth1 interface on the router is multi-gig.

When connected into another vlan, say 70 or 10 all speeds to vlan90 are limited to 1Gbps - the traffic is all across 10Gbps links so i was expecting 10Gbps speeds.

My understanding is the traffic is traversing this type of path:

10Gbps client <=> vlan70 <=> switch <=> 10Gbps SFP+ <=> RB5009 <=> 10Gbps SFP+ <=> switch <=> vlan90 <=> 10Gbps NAS\VM Host or 2.5Gbps NAS

The Config:

RB5009 Interfaces
<=> SFP-SFPPlus1 <=> USG 10Gbps Switch <=> Various 10Gbps and 1Gbps clients across various vlans
<=> eth1 2.5Gbps <=> NAS with 2.5Gbps NIC configured on vlan90
<=> eth2 1Gbps <=> ISP\WAN

Vlan90 - Used for storage devices and VM Hosts
Vlan70 - Used for test Client
Vlan10 - Used for various clients
There are several other vlans, but like the above vlans, they are all configured on the same sfp-sfpplus1 interface

Only three interfaces are used on the device:

[]Bridge Configured on RB5009 sfp-sfpplus1 interface with mutliple vlans on same interface (I’ve actually reduced the bridge to just the sfp-sfpplus interface as no other interfaces are used)
[]eth1 interface configured on 2.5Gbps interface with vlan90 (temporarily providing a 2.5Gbps connection but useful for reference in this instance).
[*]eth2 is 1Gbps and the WAN interface, but irrelevant for this issue.

The Questions:

Does anyone know if there is a 1Gbps limitation on the bridge or with vlans for this device?
CPU Usage remains low, so it doesn’t appear to be a offload\resource issue?
Considering how few interfaces are in use, should i get rid of the bridge, what are the alternatives?
Does anyone have a working approach for getting > 1Gbps speeds between vlans?
Has this been asked before and I’m just not finding the forum post?

Clues gratefully received :slight_smile:

2

Regarding RB5009’s L3 Hardware Offloading expectation see @raimondsp 's post in L3 HW Offload support on RB5009 [SOLVED].

On the topic of RB5009’S 88E6393X switch chip capabilities check Bridge Hardware Offloading.

And on the final part about what may cause your issues check out Layer2 misconfiguration section of the documentation, especially the Bridge VLAN filtering on non-CRS3xx part.

3

Many thanks for the reply,

I’ve been reading the suggested links but I’m still unclear whether I should be able to get greater than 1Gbps between vlans on this device :confused:

The first post does set my expectations, but the OP also mentions he can get between 1.5Gbps and 5Gbps - which is encouraging, I would certainly settle for multi-Gbps connectivity - it seems like full 10Gbps is asking a lot from the RB5009 !

With your second link i understand hardware offloading is not possible, i just find throughput being stuck at a firm 1Gbps a little strange - if it can’t keep up i would expect some fluctuation in speeds rather than this hard limit - hardware limitations noted however !

I can’t see one of those layer 2 misconfigurations which would be applicable (I’ve checked and I’m not using VLAN filtering on the bridge) - a useful document however, and, it’s given me some further insights.

Reading some of the other topics in the forum have also been useful and making me consider scheduling some downtime with the users (kids and wife) and starting again… I don’t think i should be limited to 1Gbps between vlans and a fresh slate might help to understand and overcome this limitation.

4

Hello,

I would like to “chip in” in the discussion, because I have the same issue. We have inherited a client where the previous IT person set up unnecessarily complex network.
There were 19 bridges, 19 vlans, 800 firewall rules, etc. Needles to say the network was crawling along on a RB3011.

I have since modified it so that it uses 1 bridge with said vlans. I have the ports stating that they are hw.offloaded but the traffic between vlans is still maxing out at 1gps with basically 100% CPU usage. With RB4011 I’m getting more like 400mbs…

Is this the upper limit of the RB5009 or is there still limit in the configuration? Which router should I recommend the customer to achieve the “wire speeds” (CRS3xx, CCR2004(which I read it also doesn’t support full L3HW offloading) ? Needless to say I’m pretty novice in this as well and I don’t want to completely reconfigure the whole network.

5

First of all sorry for the late reply.

The RB5009s have a single Marvell Marvell 88E6393X switch chip. According to the Bridge Hardware Offloading section of the documentation this switch chip supports the following features with hardware offloading: “Features in Switch menu, Bridge STP/RSTP, Bridge MSTP, Bridge IGMP Snooping, Bridge DHCP Snooping, Bridge VLAN Filtering(3), Bonding” with the caveat (3) “… The switch does not support other ether-type 0x88a8 or 0x9100 (only 0x8100 is supported) and no tag-stacking. Using these features will disable HW offload.”. However the 88E6393X is not capable of Layer 3 Hardware Offloading (L3HW, otherwise known as … HW routing). Therefore the Bridge VLAN Filtering section of the documentation is relevant for this router which also entails that all of your LAN interfaces should be part of the bridge.
So for example (with VLAN ID 99 as management VLAN):

/interface bridge
set 0 add-dhcp-option82=yes auto-mac=no comment="Eth1, Eth3 to Eth8 \
    and SFP+" dhcp-snooping=yes igmp-snooping=yes igmp-version=3 \
    mld-version=2 multicast-router=permanent name=\
    bridge1 priority=0x7000 protocol-mode=rstp \
    vlan-filtering=no
/interface bridge port
set 0 bridge=bridge1 comment=\
    "NAS on Eth1" interface=ether1 internal-path-cost=\
    2500 path-cost=10000 trusted=no hw=yes pvid=90 \
    frame-types=admit-only-untagged-and-priority-tagged
set 1 bridge=bridge1 comment="defconf" \
    interface=ether3 internal-path-cost=10000 path-cost=10000 trusted=no hw=yes \
    pvid=99 frame-types=admit-only-untagged-and-priority-tagged
set 2 bridge=bridge1 comment="defconf" \
    interface=ether4 internal-path-cost=10000 path-cost=10000 trusted=no hw=yes
set 3 bridge=bridge1 comment="defconf" \
    interface=ether5 internal-path-cost=10000 path-cost=10000 trusted=no hw=yes
set 4 bridge=bridge1 comment="defconf" \
    interface=ether6 internal-path-cost=10000 path-cost=10000 trusted=no hw=yes
set 5 bridge=bridge1 comment="defconf" \
    interface=ether7 internal-path-cost=10000 path-cost=10000 trusted=no hw=yes
set 6 bridge=bridge1 comment="defconf" \
    interface=ether8 internal-path-cost=10000 path-cost=10000 trusted=no hw=yes
set 7 bridge=bridge1 comment=\
    "Optical link to Ubiquiti on SFP+" interface=sfp-sfpplus1 \
    internal-path-cost=1000 path-cost=1000 trusted=yes hw=yes frame-types=admit-only-vlan-tagged
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,sfp-sfpplus1 vlan-ids=10
add bridge=bridge1 tagged=bridge1,sfp-sfpplus1 vlan-ids=70
add bridge=bridge1 tagged=bridge1,sfp-sfpplus1 untagged=ether1 vlan-ids=90
add bridge=bridge1 tagged=bridge1,sfp-sfpplus1 untagged=ether3 vlan-ids=99
/interface vlan
add interface=bridge1 name=Various-Clients vlan-id=10
add interface=bridge1 name=Test-Client vlan-id=70
add interface=bridge1 name=Storage-and-VMs vlan-id=90
add interface=bridge1 name=MGMT vlan-id=99
/ip address
add address=192.168.1.1/24 interface=Various-Clients network=192.168.1.0
add address=192.168.2.1/24 interface=Test-Client network=192.168.2.0
add address=192.168.3.1/24 interface=Storage-and-VMs network=192.168.3.0
add address=192.168.99.1/24 interface=MGMT network=192.168.99.0
/interface bridge
set 0 add-dhcp-option82=yes auto-mac=no comment="Eth1, Eth3 to Eth8 \
    and SFP+" dhcp-snooping=yes igmp-snooping=yes igmp-version=3 \
    mld-version=2 multicast-router=permanent name=\
    bridge1 priority=0x7000 protocol-mode=mstp \
    region-name=myhome_-_mstp vlan-filtering=yes \
    ether-type=0x8100 ingress-filtering=yes
# Enabling multicast routing
/routing pimsm instance
add afi=ipv4 disabled=no name=\
    pimsm-instance-ipv4 \
    rp-hash-mask-length=30 rp-static-override=no switch-to-spt=yes \
    switch-to-spt-bytes=0vrf=main
add afi=ipv6 disabled=no name=\
    pimsm-instance-ipv6 \
    rp-hash-mask-length=30 rp-static-override=no switch-to-spt=yes \
    switch-to-spt-bytes=0 vrf=main
/routing pimsm interface-template
add disabled=no instance=\
    pimsm-instance-ipv4 interfaces=\
    lo,bridge1,dynamic join-tracking-support=yes \
    priority=2
add disabled=no instance=\
    pimsm-instance-ipv6 interfaces=\
    lo,bridge1,dynamic join-tracking-support=yes \
    priority=1
# Example DHCP Server section
/ip pool
add name=dhcp-default ranges=192.168.0.2-192.168.0.254
add comment="Various-Clients network DHCP pool" name=\
    dhcp-ipv4-various-clients ranges=192.168.1.2-192.168.1.254
add comment="Test-Client network DHCP pool" name=\
    dhcp-ipv4-test-clients ranges=192.168.2.2-192.168.2.254
add comment="Storage-and-VMs network DHCP pool" name=\
    dhcp-ipv4-storage-and-vms ranges=192.168.3.2-192.168.3.254
add comment="Management network DHCP pool" name=\
    dhcp-ipv4-mgmt ranges=192.168.99.2-192.168.99.254
/ip dhcp-server
add add-arp=yes address-pool=dhcp-default \
    allow-dual-stack-queue=no always-broadcast=yes comment=\
    "Default IPv4 DHCP server" interface=\
    bridge1 lease-time=60m name=\
    myrouter-ipv4-dhcps-default
add add-arp=yes address-pool=dhcp-ipv4-various-clients \
    allow-dual-stack-queue=no always-broadcast=yes comment=\
    "Various-Clients network IPv4 DHCP server" interface=\
    Various-Clients lease-time=60m name=\
    myrouter-ipv4-dhcps-various-clients
add add-arp=yes address-pool=dhcp-ipv4-test-clients \
    allow-dual-stack-queue=no always-broadcast=yes comment=\
    "Test-Clients network IPv4 DHCP server" interface=\
    Test-Client lease-time=60m name=\
    myrouter-ipv4-dhcps-test-clients
add add-arp=yes address-pool=dhcp-ipv4-storage-and-vms \
    allow-dual-stack-queue=no always-broadcast=yes comment=\
    "Storage-and-VMs network IPv4 DHCP server" interface=\
    Storage-and-VMs lease-time=60m name=\
    myrouter-ipv4-storage-and-vms
add add-arp=yes address-pool=dhcp-ipv4-storage-and-vms \
    allow-dual-stack-queue=no always-broadcast=yes comment=\
    "Management network IPv4 DHCP server" interface=\
    MGMT lease-time=60m name=\
    myrouter-ipv4-mgmt
/ip dhcp-server network
add address=192.168.0.0/24 dns-server=192.168.0.1 domain=lan gateway=\
    192.168.0.1 ntp-server=192.168.0.1

On your Ubiquiti switch you should enable IGMP snooping, DHCP snooping, set the SFP+ port connecting to the RB5009 as trunk and trusted (so it can receive DHCP advertisement on this port after Option 82 is enabled), enable DHCP Option 82, enable VLAN filtering with MSTP, use long path for cost, set bridge priority to according to your topology (6000 (hex) if it should be more important than the RB5009 or 8000 (hex) if it is less important), should not enable Multicast Querier (as the RB5009 is multicast router).

By the way the USG name suggest it is a (security) gateway not a switch as those usually go by the USW-something.

6

Please note that both RB3011UiAS-RM and RB4011iGS+RM has two switch chips. The former one has two QCA8337 with 1 Gbps link to each one of the two cores of the IPQ-8064 CPU and the later one has two RTL8367 with 2.5 Gbps link each to the AL21400 CPU (to which the sole SFP+ is directly connected via a 10 Gbps link).
Upon consulting the Switch Chip Features part of the documentation one can see that these two switch chips have different feature set, for example the QCA8337 has rule table while the RTL8367 does not (read the notes for important caveats). Also the two chips have different Bridge Hardware Offloading capabilities: enabling either MSTP or VLAN filtering on the former disables the hardware offloading on the given bridge (the footnotes are also contain important information on further limitations).

Further thing to keep in mind regarding the RB3011UiAS-RM is that in Port Settings On QCA8337 … switch chips, a default

vlan-header=leave-as-is

property should be used. The switch chip will determine which ports are access ports by using the

default-vlan-id

property. The

default-vlan-id

should only be used on access/hybrid ports to specify which VLAN the untagged ingress traffic is assigned to." Also have different Port isolation capabilities.

Since these routers have two switch chips they have to be configured according to this as it is described in the VLAN filtering with multiple switch chips section of the documentation’s Layer2 misconfiguration part.



Check out the CRS3xx, CRS5xx, CCR2116, CCR2216 switch chip features part of the documentation. Consult the L3HW Feature Support section of the documentation to get an overview of various features’ support and whether the offloading of a given feature is supported or not.

To sum it up if you wan to achieve way higher throughput on those devices than you have your work cut out for you as plenty of reading and comprehending awaits you than quite some reconfiguration, which may include not just the software configuration side however may hardware part as well (connecting the patch cables to different port in order to optimise the performance on the given hardware topology). Also you may get some inspiration from Edge Router & BNG Optimisation Guide for ISPs and the related thread How to: Edge router and BNG optimization for ISPs here on the forum.

One last advice: the risk averse method of moving to RouterOS v7 is by netinstalling the 7.13 or newer version on the router and rebuilding it from step by step using the exported (and copied to a computer) configuration

export show-sensitive file=a_filename_i_give_to_my_config

for information regarding what was the previous (working) setup.