Poor 10G performance when routed through a RB5009

wonders · July 21, 2024, 11:07am

I noticed my 10G network is giving me only 7Gbps when my traffic goes through my router, see the diagram below for more details

My RB5009 router’s SFP port is connected to my CRS305 switch, forming my 10G network. This network includes my PC and NAS server, as I only need 10G connectivity between these two devices.
To test the issue, I disconnected the router from the CRS305 and assigned static IPs to both the NAS and PC. When I tested the bandwidth between them, I achieved a solid 10Gbps connection.
Connected to the router again, monitored the system health on the RB5009 during the transfer, The CPU usage hardly reached 1%, and the memory was mostly unused. Despite this, I believe there might be a misconfiguration in my RB5009. I would appreciate it if someone could review my setup to identify any potential issues.
RB5009 config:

# 2024-07-21 11:48:50 by RouterOS 7.14.2
# model = RB5009UPr+S+
/interface bridge
add admin-mac=48:A9:8A:38:D1:D7 auto-mac=no comment=defconf name=bridge port-cost-mode=short protocol-mode=none
/interface ethernet
set [ find default-name=ether1 ] comment=ONT-WAN
set [ find default-name=ether2 ] comment=Main-Network
set [ find default-name=sfp-sfpplus1 ] comment=Fibre-Network l2mtu=9000 mtu=9000
/interface wireguard
add comment=Wireguard listen-port=13231 mtu=1420 name=WG
/interface vlan
add comment=IoT interface=ether2 name=VLAN_IOT vlan-id=500
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp ranges=10.0.0.10-10.0.0.200
add name=dhcp_pool1 ranges=10.0.0.11-10.0.0.254
add name=dhcp_pool2 ranges=10.0.10.2-10.0.10.20
add name=dhcp_pool3 ranges=10.0.10.2-10.0.10.254
add name=dhcp_pool4 ranges=192.168.0.2-192.168.0.254
add name=vpn ranges=192.168.89.2-192.168.89.255
add name=vpn-pool ranges=192.168.10.10-192.168.10.99
/ip dhcp-server
add address-pool=dhcp_pool1 interface=bridge lease-time=10m name=dhcp2
/ppp profile
add idle-timeout=1h local-address=192.168.10.250 name=vpn-profile remote-address=vpn-pool use-encryption=yes
add local-address=192.168.10.250 name=ovpn-local remote-address=vpn-pool use-encryption=yes
set *FFFFFFFE local-address=192.168.89.1 remote-address=vpn
/interface bridge port
add bridge=bridge comment=defconf interface=ether2 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether3 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether4 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether5 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether6 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether7 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether8 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=sfp-sfpplus1 internal-path-cost=10 path-cost=10
/ip firewall connection tracking
set udp-timeout=10s
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface detect-internet
set detect-interface-list=WAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=LAN
add interface=sfp-sfpplus1 list=LAN
add interface=WG list=LAN
/interface ovpn-server server
set auth=sha1 certificate=server-certificate cipher=aes128-cbc,aes192-cbc,aes256-cbc default-profile=vpn-profile enabled=yes require-client-certificate=yes
/interface sstp-server server
set default-profile=default-encryption enabled=yes
/interface wireguard peers
add allowed-address=192.168.50.2/32 interface=WG
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=192.168.88.0
add address=10.0.0.10/24 interface=bridge network=10.0.0.0
add address=192.168.50.1/24 interface=WG network=192.168.50.0
/ip cloud
set ddns-enabled=yes
/ip dhcp-client
add comment=defconf interface=ether1
/ip dhcp-server lease
add address=10.0.0.240 client-id=1:18:31:bf:27:54:83 mac-address=18:31:BF:27:54:83 server=dhcp2
add address=10.0.0.235 client-id=1:f0:6e:b:45:58:5e mac-address=F0:6E:0B:45:58:5E server=dhcp2
add address=10.0.0.228 client-id=1:e4:5f:1:5d:6b:89 mac-address=E4:5F:01:5D:6B:89 server=dhcp2
add address=10.0.0.32 client-id=1:0:2:c9:52:b6:b8 mac-address=00:02:C9:52:B6:B8 server=dhcp2
add address=10.0.0.37 client-id=1:0:2:c9:53:9d:5a mac-address=00:02:C9:53:9D:5A server=dhcp2
add address=10.0.0.40 client-id=1:c8:cb:b8:c7:da:96 mac-address=C8:CB:B8:C7:DA:96 server=dhcp2
add address=10.0.0.67 mac-address=8C:DC:D4:AB:FA:7C server=dhcp2
add address=10.0.0.192 client-id=1:f4:39:9:7:6c:31 mac-address=F4:39:09:07:6C:31 server=dhcp2
add address=10.0.0.63 mac-address=40:CA:63:29:89:DD server=dhcp2
add address=10.0.0.105 mac-address=8C:DC:D4:AB:FA:7D server=dhcp2
add address=10.0.0.125 client-id=1:2c:d:a7:c7:a5:1f mac-address=2C:0D:A7:C7:A5:1F server=dhcp2
/ip dhcp-server network
add address=10.0.0.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=10.0.0.10
add address=10.0.10.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=10.0.10.1
add address=192.168.0.0/24 gateway=192.168.0.1
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall address-list
add address=10.0.0.0/24 list=Allowed-to-the-router
add address=192.168.10.0/24 list=vpn-allowed-to-router
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add action=accept chain=forward dst-address=10.0.0.23 dst-port=44158 in-interface=ether1 protocol=tcp
add action=accept chain=input dst-port=1194 protocol=tcp
add action=accept chain=forward in-interface=all-ppp
add action=accept chain=input comment="default configuration" connection-state=established,related
add action=accept chain=input src-address-list=Allowed-to-the-router
add action=accept chain=input src-address-list=vpn-allowed-to-router
add action=accept chain=input protocol=icmp
add action=accept chain=input comment=Wireguard dst-port=13231 in-interface=ether1 protocol=tcp
add action=drop chain=input comment="Block everything else"
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat out-interface=ether1
add action=dst-nat chain=dstnat comment="Home Assistant" disabled=yes dst-port=8123 in-interface=ether1 protocol=tcp to-addresses=10.0.0.228 to-ports=8123
add action=dst-nat chain=dstnat dst-port=34444 in-interface=ether1 protocol=tcp to-addresses=10.0.0.10 to-ports=80
add action=masquerade chain=srcnat comment="masq. vpn traffic" src-address=192.168.89.0/24
add action=masquerade chain=srcnat src-address=192.168.8.0/24
add action=masquerade chain=srcnat src-address=192.168.10.0/24
add action=masquerade chain=srcnat src-address=192.168.50.0/24
/ip route
add disabled=no dst-address=10.0.0.40/32 gateway=lo
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes port=5500
set api disabled=yes
set api-ssl disabled=yes
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN

sindy · July 21, 2024, 12:02pm

I can’t spot anything unusual in the RB5009 configiration. Did the connection between the RB5009 and the CRS310 establish on 10 Gbit/s? The CRS310 probably doesn’t move on to the next frame until the current one has been sent in all directions it’s intended to be sent, so all the multicast and broadcast traffic, which has to egress through all active ports, may slow down the total throughput if one of the egress ports is slower than 10 Gbit/s.

biomesh · July 21, 2024, 12:07pm

Just to have everyone verify I would also post the crs config.

jaclaz · July 21, 2024, 12:10pm

Nice explanation, I was wondering what could be a possible reason, thanks.

But can this slow down the whole stuff by 30%?

I mean, multicast and broadcast should be a teeny-tiny fraction of the traffic, a direct download or upload to the NAS should be affected only marginally (or not)?

wonders · July 21, 2024, 12:14pm

The CRS305 is factory default, I didn’t change anything there, just plugged it in and it worked, how can I dump the config from that? Doesn’t seem to have any terminal options like RouterOS?

Also on CRS305, the other ports are empty, what is shown int he diagram is what is used there!

wonders · July 21, 2024, 12:17pm

Sorry forgot to ask, how can I check speed between RB5009 and CRS305, I tried using bandwidth test in tools, but it doesn’t connect! Keep saying connecting!

biomesh · July 21, 2024, 12:21pm

While I don’t think that these may be the issue they are areas of concern:

No stp/rstp protocol on the bridge
the sfp+ port has an mtu of 9000, does that match your switch and other devices?

sindy · July 21, 2024, 12:27pm

If there’s Windows involved, there may be much more multicast traffic than you would expect.

For confirmation, you can use switch chip rules (that are excuted in hardware, hence at wire speed) to prevent multicast and broadcast traffic to be forwarded from the PC and NAS interfaces to the RB5009 one. A single rule should be sufficient:
/interface/ethernet/switch/rule/add ports=sfp-sfpplus2,sfp-sfpplus3 dst-mac-address=01:00:00:00:00:00/01:00:00:00:00:00 new-dst-ports=sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,ether1

This way, moving the RB5009 to sfp-sfpplus4 will make it work as before, and connecting it to sfp-sfpplus1 (the only one missing in the list in new-dst-ports) will make it not get the broadcast and multicast traffic, which will also break routing from the PC through there as the PC will be unable to determine the MAC address of its default gateway as the ARP request is a broadcast packet. So this is a first step, and if it removes the brake, we can add additional rules to selectively allow the ARP which is indeed not so frequent.

sindy · July 21, 2024, 12:29pm

xSTP does not subtly reduce speed, it just cuts traffic completely if misconfigured.
non-matching settings of MTU/L2MTU on interconnected devices do affect traffic, but only one that passes through the link with configuration mismatch - not the case here.

wonders · July 21, 2024, 12:36pm

It was set to rstp, setting that to none increased the speed to 8Gbps
Yes all MTUs are set to 9000

anav · July 21, 2024, 12:37pm

Why would you think the RB5009 can route 10G?
Straight routing and no rules etc. should yield 9.5 gigs
At 25 filter rules the best you can hope for is around 3-4 Gigs.
Assuming you have somewhat less than 25 rules if getting 7gigs…

You should get a a 2116, or 2216 for 10gig type numbers.

I understand that funny looking bird now! Its pinocchio reincarnated!!

jaclaz · July 21, 2024, 12:39pm

If I get it right the CRS305 is currently running SWos, not Ros?

@anav
The issue is not in the routing(or in the RB5009), it is in the switching (on the CRS305) between the Pc and the Nas.
Everything happens (or should happen) within the borders of the green rectangle.

anav · July 21, 2024, 12:41pm

Title of the thread reads…
Poor 10G performance when routed through a RB5009

Sadly another first post fail, where the configs of the CRS305 and RB5009 were not provided.

jaclaz · July 21, 2024, 12:49pm

Sure , you shouldn’t judge a book by its cover or title, the book titled “The eye of the needle” by Ken Follett is not about tailors exchanging ideas on how to better thread needles…

wonders · July 21, 2024, 12:50pm

Yes, SWOS on the CRS305

Sorry for the confusion, I purchased RB5009 recently mainly because CRS305 was getting too hot and dropping connection and speeds.

I used to use CRS305 as my switch and router with RouterOS running on it, with that alone, I was getting 9.5Gbps which was what I expected,
Adding RB5009 I don’t expect to get 10G but at least 9.5Gbps but nothing more than 7 or 8Gbps

sindy · July 21, 2024, 12:58pm

In the defense of @anav, this is a technical forum, not a bookstore or a newspaper. So a spot-on topic title (or a book title for that matter) helps avoid confusion. As an example, if I were in my grumpy mood or just busy with something more complicated, I might have simply skipped that topic, assuming that someone else will respond the obvious (routing speed does not match the link speed).

sindy · July 21, 2024, 1:03pm

To use the test rule I have suggested, you’ll have to switch back to RouterOS, in SwOS you cannot configure switch chip rules. But you can still use the default RouterOS configuration where ithe CRS305 will acts as a mere switch on wirespeed.

The CPUs in CRS products are intended for management and light routing related to the management, not for routing the payload traffic. And even the 5009 will not pass 10 Gb/s through even if you connected it using a 10 Gb/s SFP.

wonders · July 21, 2024, 1:28pm

Thanks very much, will try that and update the post!

anav · July 21, 2024, 3:43pm

To be fair to jaclaz, he was just being a jackazz, in a humorous way. Still I did read through the thread and it was still not clear to me that the issue was something else than the 5009.

Paternot · July 21, 2024, 5:01pm

If I understood it correctly, the issue is slower speed on the 10Gbps network ONLY when the RB5009 is connected - even without traffic crossing the 5009.