RB5009 drops packets and speed on eth1 (vlan + pppoe)

trmns · April 23, 2024, 4:48pm

      ┌─────────┐ ┌─────────────────────────────────────┐          
      │   ISP   │ │  RB5009                             │          
      │   ONT   │ │                                     │          
      │         │ ├────────┬────────────────────────────┤          
      │         │ │  eth1  │           bridge           │          
      │         │ │  vlan  ├────────┬──────────┬────────┤          
      │         │ │  PPPoE │  eth2  │  eth3-7  │  eth8  │          
      └───┬─────┘ └──▲─────┴──┬─────┴──────────┴────────┘          
          │          │        │                                    
          └──────────┘     ┌──┴─────┐                              
                           │  Comp  │                              
                           └────────┘

I have the setup shown in the diagram, this is how my interfaces look:

/interface bridge
add admin-mac=78:9A:18:BD:4D:8C auto-mac=no comment=defconf dhcp-snooping=yes \
    name=bridge port-cost-mode=short
/interface ethernet
set [ find default-name=ether1 ] comment=ONT l2mtu=1526 mtu=1512
set [ find default-name=ether2 ] comment=Comp rx-flow-control=auto \
    tx-flow-control=auto
set [ find default-name=ether4 ] rx-flow-control=auto tx-flow-control=auto
set [ find default-name=ether7 ] comment="AP"
set [ find default-name=ether8 ] comment="AP"
set [ find default-name=sfp-sfpplus1 ] comment="sfp fiber" disabled=yes
/interface wireguard
add comment=Wireguard listen-port=13231 mtu=1420 name=wg0
/interface vlan
add comment="vlan6 for ISP" interface=ether1 loop-protect=off mtu=1508 \
    name=vlan6 vlan-id=6
/interface pppoe-client
add add-default-route=yes allow=pap comment="ISP PPPoE" disabled=no \
    interface=vlan6 max-mru=1500 max-mtu=1500 name=ISP-pppoe profile=\
    ISP use-peer-dns=yes user=user@ISP
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add comment="bridge only" name=BRIDGE
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge comment=defconf interface=ether2 internal-path-cost=10 \
    path-cost=10
add bridge=bridge comment=defconf interface=ether3 internal-path-cost=10 \
    path-cost=10
add bridge=bridge comment=defconf interface=ether4 internal-path-cost=10 \
    path-cost=10
add bridge=bridge comment=defconf interface=ether5 internal-path-cost=10 \
    path-cost=10
add bridge=bridge comment=defconf interface=ether6 internal-path-cost=10 \
    path-cost=10
add bridge=bridge comment=defconf interface=ether7 internal-path-cost=10 \
    path-cost=10 trusted=yes
add bridge=bridge comment=defconf interface=ether8 internal-path-cost=10 \
    path-cost=10 trusted=yes
add bridge=bridge comment=defconf interface=sfp-sfpplus1 internal-path-cost=10 \
    path-cost=10
/interface detect-internet
set detect-interface-list=WAN internet-interface-list=WAN lan-interface-list=\
    LAN wan-interface-list=WAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ISP-pppoe list=WAN
add comment="Add Wireguard to LAN, for IPv6 Firewall rules" interface=wg0 list=\
    LAN
add comment="for Mikrotik neighbor discovery" interface=bridge list=BRIDGE
/interface wireguard peers
<wg peers>

/ip firewall connection tracking
set udp-timeout=10s
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input comment="accept WireGuard connections" dst-port=13231 protocol=udp
add action=accept chain=input comment="allow access to RouterOS's DNS server via WireGuard" dst-port=53 in-interface=wg0 protocol=udp
add action=accept chain=input comment="allow access to RouterOS mobile app via WireGuard" dst-port=8291 in-interface=wg0 protocol=tcp
add action=accept chain=input comment="allow access to RouterOS webinterface via WireGuard" dst-port=80 in-interface=wg0 protocol=tcp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" disabled=yes ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" disabled=yes ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat comment=https://help.mikrotik.com/docs/display/ROS/NAT#NAT-HairpinNAT dst-address=192.168.88.0/24 dst-port=22 out-interface-list=LAN protocol=tcp src-address=\
    192.168.88.0/24
add action=dst-nat chain=dstnat comment="This rule changes the IP of all packets which are from the external network and directed to tcp:<external_ip>:22 to tcp:192.168.88.124:22" dst-port=22 \
    in-interface-list=WAN protocol=tcp to-addresses=192.168.88.254 to-ports=22
add action=dst-nat chain=dstnat comment="This rule changes the IP of all packets which are from the internal network and directed to tcp:<external_ip>:22 to tcp:192.168.88.124:22" dst-address=\
    <external_ip> dst-port=22 in-interface-list=LAN protocol=tcp to-addresses=192.168.88.254 to-ports=22
add action=dst-nat chain=dstnat comment="This rule changes the IP of all packets which are from the external network and directed to tcp:<external_ip>:2222 to tcp:192.168.88.253:22" dst-port=2222 \
    in-interface-list=WAN protocol=tcp to-addresses=192.168.88.253 to-ports=22
add action=dst-nat chain=dstnat comment="This rule changes the IP of all packets which are from the internal network and directed to tcp:<external_ip>:2222 to tcp:192.168.88.253:22" dst-address=\
    <external_ip> dst-port=2222 in-interface-list=LAN protocol=tcp to-addresses=192.168.88.253 to-ports=22
add action=dst-nat chain=dstnat comment="allow iperf3 to comp" dst-port=5201 in-interface-list=WAN protocol=tcp to-addresses=192.168.88.254 to-ports=5201
/ip firewall service-port
set ftp disabled=yes
set h323 disabled=yes
set pptp disabled=yes

/queue type
add fq-codel-interval=30ms fq-codel-limit=100 kind=fq-codel name=fq_codel-tuning
add kind=fq-codel name=fq_codel-default
/queue interface
set ether1 queue=fq_codel-tuning
set ether2 queue=fq_codel-tuning
set ether3 queue=fq_codel-tuning
set ether4 queue=fq_codel-tuning
set ether5 queue=fq_codel-tuning
set ether6 queue=fq_codel-tuning
set ether7 queue=fq_codel-tuning
set ether8 queue=fq_codel-tuning

With this configuration I can get 900 Mbit down:

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   105 MBytes   881 Mbits/sec
[  5]   1.00-2.00   sec   107 MBytes   901 Mbits/sec
[  5]   2.00-3.00   sec   108 MBytes   904 Mbits/sec
[  5]   3.00-4.00   sec   111 MBytes   932 Mbits/sec
[  5]   4.00-5.00   sec   106 MBytes   893 Mbits/sec
[  5]   5.00-6.00   sec   110 MBytes   921 Mbits/sec
[  5]   6.00-7.00   sec   104 MBytes   877 Mbits/sec
[  5]   7.00-8.00   sec   106 MBytes   890 Mbits/sec
[  5]   8.00-9.00   sec   108 MBytes   904 Mbits/sec
[  5]   9.00-10.00  sec   104 MBytes   872 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.05 GBytes   900 Mbits/sec  64607             sender
[  5]   0.00-10.00  sec  1.04 GBytes   897 Mbits/sec                  receiver

and also around 900 Mbit of upload

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   105 MBytes   878 Mbits/sec  156    697 KBytes
[  5]   1.00-2.00   sec  92.6 MBytes   777 Mbits/sec    0    793 KBytes
[  5]   2.00-3.00   sec   101 MBytes   848 Mbits/sec    0    887 KBytes
[  5]   3.00-4.00   sec   110 MBytes   919 Mbits/sec    0    977 KBytes
[  5]   4.00-5.00   sec  94.2 MBytes   791 Mbits/sec   53    775 KBytes
[  5]   5.00-6.00   sec   104 MBytes   873 Mbits/sec    0    867 KBytes
[  5]   6.00-7.00   sec   105 MBytes   883 Mbits/sec   22    670 KBytes
[  5]   7.00-8.00   sec  87.5 MBytes   734 Mbits/sec    0    764 KBytes
[  5]   8.00-9.00   sec  97.0 MBytes   814 Mbits/sec    0    856 KBytes
[  5]   9.00-10.00  sec   106 MBytes   887 Mbits/sec    0    947 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1002 MBytes   840 Mbits/sec  231             sender
[  5]   0.00-10.00  sec   999 MBytes   838 Mbits/sec                  receiver

As you can see though the speed is not consistent, but it fluctuates between 700 and 900 Mbit.

If I change my setup like this:

┌─────────┐  ┌────────┐
│   ISP   │  │  Comp  │
│   ONT   │  │        │
└───┬─────┘  └─┬──────┘
    │          │       
    └──────────┘

I get consistent 930 Mbit upload without any retries. I can’t show a speedtest right now, because re-cabling is annoying, but I can prove it should the proof really be needed. Also this does not depend on the speedtest server, because this behavior is reproducible.

So my question is, surely there must be a config in which I can get the same performance with a 200 dollar device?

CGGXANNX · April 24, 2024, 4:48am

Did you try to reset the interface queue setting back to only-hardware-queue? That might interfere with the consistency of the transfer rate.

From your configuration: there is no needs to set the MTU of ether1 to 1512. MTU 1508 for ether1 is enough for vlan6 to have an MTU of 1508. You also don’t need to increase l2mtu on ether1 to 1526. Doing so actually use double the internal buffer compared to the default value of 1514. L2MTU 1514 is fully sufficient to have MTU 1508 on the vlan interface. I know that because this is my setup with my RB5009 (I have all interfaces under the same bridge, including the interface sfp-plus1 used by PPPoE, and bridge vlan filtering enable on that single bridge, the WAN port is treated like a normal VLAN under the bridge):

And because of this video from MikroTik https://www.youtube.com/watch?v=7a_z1jAdIME. L2MTU only needs to greater than or equal to MTU. In this case L2MTU will be lowest on the vlan6 interface (1510 because 4 bytes were spent on the 802.1Q header) and the vlan6 interface requires MTU 1508 so that ISP-pppoe can have an MTU of 1500. But 1508 is still smaller than 1510, so you don’t need to change that value.

You can also switch the port-cost-mode to long and clear the path-cost setting. Refer to this document from @tangent https://tangentsoft.com/mikrotik/wiki?name=Configuration%20Flotsam under “Bridge Interface Path Costs”.

Unrelated but in your firewall configuration, the rules with the comments “allow access to RouterOS…” are redundant, becase wg0 is already in the interface list LAN and thus traffic coming from it is not blocked by the

add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN

rule.

trmns · April 24, 2024, 11:12am

Thanks for your reply. I was actually running MTU 1508 for the majority of the time, I only changed it recently thinking I was smart. I’ve changed it back to 1508 and also the L2MTU, but nothing really changed.

regarding only-hardware-queue. That’s the default but I had noticed in the interface stats, that when running this I’d get a lot of TX queue drops. Using something else like mq-ethernet-default helped, or like in this case fq_codel. But changing this setting back also doesn’t really change anything with the reliability of the upload. It is still choppy compared to connecting to the ONT directly.

And yeah, I only recently added wg to the LAN list, so I forgot about that side-effect!

pe1chl · April 24, 2024, 11:58am

Actually it is not necessary (and not a good idea) to increase the MTU on an ethernet device “to accomodate VLAN”.
VLAN header overhead is incorporated in the ethernet device settings automatically.

You ONLY need to increase the MTU for the PPPoE overhead, so the device where pppoe-out with MTU 1500 is created needs an additional MTU of 8 bytes. That seems to be vlan1000 in this case, so that needs MTU 1508.
Set all others back to 1500. Even ether1 MTU does not have to be increased: the vlan1000 knows by itself it is >1500 and the MTU of ether1 is only for direct transmissions on ether1 (which there aren’t).

trmns · April 24, 2024, 3:14pm

Thanks for your feedback by now my config has changed a bit, but sadly literally nothing has changed:

/interface bridge
add admin-mac=78:9A:18:BD:4D:8C auto-mac=no comment=defconf dhcp-snooping=yes name=bridge
/interface ethernet
set [ find default-name=ether1 ] comment=ONT
set [ find default-name=ether2 ] comment=Comp rx-flow-control=auto tx-flow-control=auto
set [ find default-name=ether4 ] rx-flow-control=auto tx-flow-control=auto
set [ find default-name=ether7 ] comment="AP"
set [ find default-name=ether8 ] comment="AP"
set [ find default-name=sfp-sfpplus1 ] comment="sfp fiber" disabled=yes
/interface wireguard
add comment=Wireguard listen-port=13231 mtu=1420 name=wg0
/interface vlan
add arp=disabled comment="vlan6 for ISP" interface=ether1 loop-protect=off mtu=1508 name=vlan6 vlan-id=6
/interface pppoe-client
add add-default-route=yes allow=pap comment="ISP PPPoE" disabled=no interface=vlan6 keepalive-timeout=20 max-mru=1500 max-mtu=1500 name=ISP-pppoe profile=ISP use-peer-dns=yes user=user@ISP.nl
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add comment="bridge only" name=BRIDGE
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7 trusted=yes
add bridge=bridge comment=defconf interface=ether8 trusted=yes
add bridge=bridge comment=defconf interface=sfp-sfpplus1
/interface detect-internet
set detect-interface-list=WAN internet-interface-list=WAN lan-interface-list=LAN wan-interface-list=WAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ISP-pppoe list=WAN
add comment="Add Wireguard to LAN, for IPv6 Firewall rules" interface=wg0 list=LAN
add comment="for Mikrotik neighbor discovery" interface=bridge list=BRIDGE

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  85.6 MBytes   718 Mbits/sec   48   1.11 MBytes
[  5]   1.00-2.00   sec   104 MBytes   868 Mbits/sec    0   1.19 MBytes
[  5]   2.00-3.00   sec   106 MBytes   887 Mbits/sec  132    935 KBytes
[  5]   3.00-4.00   sec  84.9 MBytes   712 Mbits/sec   22    735 KBytes
[  5]   4.00-5.00   sec  94.2 MBytes   791 Mbits/sec    0    834 KBytes
[  5]   5.00-6.00   sec   104 MBytes   871 Mbits/sec    0    923 KBytes
[  5]   6.00-7.00   sec   106 MBytes   893 Mbits/sec    0   1010 KBytes
[  5]   7.00-8.00   sec   107 MBytes   894 Mbits/sec    0   1.06 MBytes
[  5]   8.00-9.00   sec   101 MBytes   848 Mbits/sec  132    850 KBytes
[  5]   9.00-10.00  sec  99.8 MBytes   837 Mbits/sec    0    940 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   992 MBytes   832 Mbits/sec  334             sender
[  5]   0.00-10.00  sec   990 MBytes   830 Mbits/sec                  receiver

It also bums me out that the ISP supplied FRITZ!Box performs better, because I don’t have this issue with that device either

pe1chl · April 24, 2024, 4:09pm

I have read in other places that there are performance problems with the switch on the RB5009.
Not sure what is really true, some people claim it helps to remove the 2.5G from the advertisement list on ether1 but i can’t believe that.
I have a couple of RB5009 in the company but none in a configuration where I can push them to the limit.

Note that PPPoE is quite heavy on MikroTik devices. It is not hardware-assisted, even though in some cases the hardware is able to do that and on other manufacturer’s devices it is being done (so a lot less CPU load).

You can try to run Tools->Profile or look at System->Resources while doing the test to see if you are CPU limited.
If not, it may be a switch problem.

trmns · April 24, 2024, 4:41pm

Yeah the profiler says that one core will use around 72% of its ressources on upload tests. That was a good hint, because on downloads it is only 30%. So I fixed the CPU frequency from ‘auto’ to 1400 MHz and I go from this:

[  5]   0.00-1.00   sec  93.0 MBytes   779 Mbits/sec  110    827 KBytes
[  5]   1.00-2.00   sec  99.5 MBytes   835 Mbits/sec   41    645 KBytes
[  5]   2.00-3.00   sec  83.5 MBytes   700 Mbits/sec    0    734 KBytes
[  5]   3.00-4.00   sec  79.8 MBytes   669 Mbits/sec   11    585 KBytes
[  5]   4.00-5.00   sec  77.9 MBytes   653 Mbits/sec    0    680 KBytes
[  5]   5.00-6.00   sec  87.5 MBytes   734 Mbits/sec    0    773 KBytes
[  5]   6.00-7.00   sec  95.8 MBytes   803 Mbits/sec    0    864 KBytes
[  5]   7.00-8.00   sec   108 MBytes   906 Mbits/sec    0    956 KBytes
[  5]   8.00-9.00   sec  91.9 MBytes   771 Mbits/sec   69    755 KBytes
[  5]   9.00-10.00  sec  97.0 MBytes   813 Mbits/sec    0    840 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   914 MBytes   766 Mbits/sec  231             sender
[  5]   0.00-10.01  sec   911 MBytes   763 Mbits/sec                  receiver

to

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   108 MBytes   901 Mbits/sec  108   1.10 MBytes
[  5]   1.00-2.00   sec   109 MBytes   918 Mbits/sec    0   1.22 MBytes
[  5]   2.00-3.00   sec   108 MBytes   906 Mbits/sec    0   1.31 MBytes
[  5]   3.00-4.00   sec   110 MBytes   920 Mbits/sec   47   1004 KBytes
[  5]   4.00-5.00   sec   109 MBytes   917 Mbits/sec    0   1.06 MBytes
[  5]   5.00-6.00   sec   109 MBytes   917 Mbits/sec    0   1.13 MBytes
[  5]   6.00-7.00   sec   109 MBytes   918 Mbits/sec    0   1.20 MBytes
[  5]   7.00-8.00   sec   109 MBytes   916 Mbits/sec    0   1.27 MBytes
[  5]   8.00-9.00   sec   109 MBytes   915 Mbits/sec   48    994 KBytes
[  5]   9.00-10.00  sec   108 MBytes   905 Mbits/sec    0   1.07 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.06 GBytes   913 Mbits/sec  203             sender
[  5]   0.00-10.01  sec  1.06 GBytes   910 Mbits/sec                  receiver

Sounds like a firmware bug? It should boost to 1400 MHz automatically I suppose. Interestingly enough it still performs fine on 700 MHz capped. So the automatic frequency changing is an issue here.

I also removed the 2.5G advertisement, but that doesn’t change anything either. Changing the CPU frequency is what really made an impact.

pe1chl · April 24, 2024, 8:10pm

Yeah, I think the CPU governor should keep it at 1400 MHz for some time when it has deemed that to be required, not fall back so quickly. It was probably coded (or tuned) more for a workstation than for a router.
One of the problems probably is that the PPPoE is single-threaded (to avoid packet re-ordering) so even when you load one core to nearly 100% due to the PPPoE processing, the total CPU load still is 25% max so the governor sees no real need to increase the clock frequency to the max, I guess.
You could see if the temperature of the router increases at 1400 MHz. For my 4011 at home I was never able to detect much influence on the temperature. For my PC that is completely different, though.

trmns · April 25, 2024, 5:02pm

The device runs at around 47C with its day-to-day load on a fixed 1400 MHz. I guess I can keep running it like that! Still, it doesn’t feel too good knowing that the CPU governor is not being efficient. I guess I will open a support ticket with my findings.

pe1chl · April 25, 2024, 7:35pm

Yes, that would be a good idea. Probably mentioning that it does not seem to perform well when the task at hand loads only a single core.
47C is not bad. I now have 3 of those RB5009 running and two are on shelves in 19" cabinets, run at 48/49C and one is on my desk and it is 43C.
(all of them with CPU freq auto)
When you want it much cooler you likely need a fan… E.g. here at home I have a 4011 with an 8-port switch on top and a VDSL modem on top of that, and to get reasonable temperatures in the summer I have put a small fan alongside which is powered by 7V instead of 12V. It provides a tiny breeze of air and can not be heard, but that takes temps down by 15 degrees.

trmns · April 26, 2024, 12:48am

I actually have a hAP ax3 sitting on top of it for AP access, that’s not beneficial for the temp I suppose

pe1chl · April 26, 2024, 7:51am

Yes that is like here, a little breeze makes a lot of difference.
But this temperature is nothing to worry about, it would be a good idea to do something when it gets above 60 or 70.