Suddenly reduced bandwidth over PPPoE/FTTP internet connection from RB5009 only

RouterOS General
1

I have a RB5009UG+S+ (RouterOS 7.15.1) connected to a standard OpenReach ONT and establishing a PPPoE session for my 550/75 FTTP package. This has all worked perfectly for 3+ years and I have seen solid 500Mbit+ download speeds. However, sometime last week (I think) max download speeds dropped to roughly 98-99 Mbits throughput on the WAN ethernet interface when running a test on speedtest.net from various wired ethernet clients on the LAN.

Given that suspicious Fast Ethernet-like speed, my first thought was cabling. However, Mikrotik reports a 1Gbps negotiated link on the WAN port. If I remove the WAN cable from the router and plug it into my Windows laptop directly, I can bring up a PPPoE connection and download at 500+MBits. Also, If I connect my laptop to the ONT end of the cable I can link at 1Gbps to the router.

Any suggestions welcome, as I’m not sure where to take this one next. The PPPoE debug logs (without packet-level info) are below, unfortunately I don’t know enough about PPPoE to know if the PADO drops and LCP ConfReq state errors are significant. My sanitised config is attached.

16:45:35 pppoe,ppp,info pppoe-out1: initializing…
16:45:35 pppoe,ppp,info pppoe-out1: connecting…
16:45:35 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:35 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:35 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:35 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:35 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:35 pppoe,ppp,debug pppoe-out1: LCP lowerup
16:45:35 pppoe,ppp,debug pppoe-out1: LCP open
16:45:35 pppoe,ppp,debug pppoe-out1: LCP opened
16:45:35 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:35 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:35 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:35 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:35 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:35 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:36 pppoe,ppp,debug pppoe-out1: LCP received ConfReq in opened state: restarting
16:45:36 pppoe,ppp,debug pppoe-out1: LCP closed
16:45:36 pppoe,ppp,debug pppoe-out1: LCP opened
16:45:36 pppoe,ppp,debug pppoe-out1: LCP received ConfReq in opened state: restarting
16:45:36 pppoe,ppp,debug pppoe-out1: LCP closed
16:45:36 pppoe,ppp,debug pppoe-out1: LCP opened
16:45:36 pppoe,ppp,info pppoe-out1: authenticated
16:45:36 pppoe,ppp,debug pppoe-out1: IPCP lowerup
16:45:36 pppoe,ppp,debug pppoe-out1: IPCP open
16:45:36 pppoe,ppp,debug pppoe-out1: IPV6CP lowerup
16:45:36 pppoe,ppp,debug pppoe-out1: IPV6CP open
16:45:36 pppoe,ppp,debug pppoe-out1: MPLSCP lowerup
16:45:36 pppoe,ppp,debug pppoe-out1: MPLSCP open
16:45:36 pppoe,ppp,debug pppoe-out1: BCP open
16:45:36 pppoe,ppp,debug pppoe-out1: CCP lowerup
16:45:36 pppoe,ppp,debug pppoe-out1: CCP open
16:45:36 pppoe,ppp,debug pppoe-out1: IPCP opened
16:45:36 pppoe,ppp,info pppoe-out1: connected
16:45:36 interface,info pppoe-out1 detect UNKNOWN
16:45:37 pppoe,ppp,debug pppoe-out1: IPV6CP timer
16:45:37 pppoe,ppp,debug pppoe-out1: IPV6CP opened
16:45:38 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:38 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:38 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:38 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:38 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:38 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:38 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:38 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:38 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:38 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:38 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:38 pppoe,debug ether1 (WAN): received PADO with unknown host-uniq, dropping
16:45:42 interface,info pppoe-out1 detect INTERNET
16:45:47 pppoe,ppp,debug pppoe-out1: MPLSCP timer
config.txt (13 KB)

2

I have the same issue on a Hex S. Suddenly speeds have slowed to a crawl and bypassing the mikrotik yields results in line with the ISP line speed. Up dated firmware with marginal improvement but still under 50% of the 100Mbps

3

There is something very wrong with Your PPPoE connection. I’m running 7.14.3, and this is what I see from the router, on my remote log server:

2024-06-19T11:17:12-03:00 RB5009 pppoe:  PredialnetPPPoE: rcvd LCP EchoReq id=0xc3
2024-06-19T11:17:12-03:00 RB5009 pppoe:     <magic 0x84dc0def>
2024-06-19T11:17:12-03:00 RB5009 pppoe:     <data len=4>
2024-06-19T11:17:12-03:00 RB5009 pppoe:  PredialnetPPPoE: sent LCP EchoRep id=0xc3
2024-06-19T11:17:12-03:00 RB5009 pppoe:     <magic 0x1d2fc8ef>
2024-06-19T11:17:12-03:00 RB5009 pppoe:     <data len=4>
2024-06-19T11:18:42-03:00 RB5009 pppoe:  PredialnetPPPoE: rcvd LCP EchoReq id=0xc4
2024-06-19T11:18:42-03:00 RB5009 pppoe:     <magic 0x84dc0def>
2024-06-19T11:18:42-03:00 RB5009 pppoe:     <data len=4>
2024-06-19T11:18:42-03:00 RB5009 pppoe:  PredialnetPPPoE: sent LCP EchoRep id=0xc4
2024-06-19T11:18:42-03:00 RB5009 pppoe:     <magic 0x1d2fc8ef>
2024-06-19T11:18:42-03:00 RB5009 pppoe:     <data len=4>
2024-06-19T11:20:12-03:00 RB5009 pppoe:  PredialnetPPPoE: rcvd LCP EchoReq id=0xc5
2024-06-19T11:20:12-03:00 RB5009 pppoe:     <magic 0x84dc0def>
2024-06-19T11:20:12-03:00 RB5009 pppoe:     <data len=4>
2024-06-19T11:20:12-03:00 RB5009 pppoe:  PredialnetPPPoE: sent LCP EchoRep id=0xc5
2024-06-19T11:20:12-03:00 RB5009 pppoe:     <magic 0x1d2fc8ef>
2024-06-19T11:20:12-03:00 RB5009 pppoe:     <data len=4>
2024-06-19T11:21:42-03:00 RB5009 pppoe:  PredialnetPPPoE: rcvd LCP EchoReq id=0xc6
2024-06-19T11:21:42-03:00 RB5009 pppoe:     <magic 0x84dc0def>
2024-06-19T11:21:42-03:00 RB5009 pppoe:     <data len=4>
2024-06-19T11:21:42-03:00 RB5009 pppoe:  PredialnetPPPoE: sent LCP EchoRep id=0xc6
2024-06-19T11:21:42-03:00 RB5009 pppoe:     <magic 0x1d2fc8ef>
2024-06-19T11:21:42-03:00 RB5009 pppoe:     <data len=4>
2024-06-19T11:23:12-03:00 RB5009 pppoe:  PredialnetPPPoE: rcvd LCP EchoReq id=0xc7
2024-06-19T11:23:12-03:00 RB5009 pppoe:     <magic 0x84dc0def>
2024-06-19T11:23:12-03:00 RB5009 pppoe:     <data len=4>
2024-06-19T11:23:12-03:00 RB5009 pppoe:  PredialnetPPPoE: sent LCP EchoRep id=0xc7
2024-06-19T11:23:12-03:00 RB5009 pppoe:     <magic 0x1d2fc8ef>
2024-06-19T11:23:12-03:00 RB5009 pppoe:     <data len=4>

It just goes like this, with nothing else. Sometimes (once every 20 - 40 days?) the connection drops and is reestablished. The it shows on the logs - but nothing like what you are getting.

4

I’ve (mostly) fixed it. MTU issue.

My ISP, IDNet, was claiming that nothing had changed at their end and that it must be failing hardware, so I remembered and dug out my old Hex S from the attic. This was my main router before the RB5009 and it turned out contained a 3 year old config on RouterOS 6.9 that still connected - a good opportunity to probe for hardware problems, OS regressions and configuration regressions.

It showed the same issues, with speed tests coming in just under 100Mbits, ruling out a hardware issue. However, when I reset the configuration and set up the PPoE WAN interface using WebFig’s quick config, I got the expected 500MBps+.

The RB5009 ppoe-client configuration contained additional properties not in the fresh Hex S config: keepalive-timeout=60, max-mru=1480, max-mtu=1480, mrru=1600, service-name=internet. On removing these, actual MTU in the PPPoE status went down from 1520 to 1492, and download speeds were restored.

TBH I can’t remember when/why I introduced these manually set properties - but I’ve been with IDNet for about 8 years and it was likely fairly early on in that period. Clearly some recent change in the PPPoE negotiation and/or routing path suddenly resulted in packet fragmentation and an >5x decrease in speed. I hadn’t realised MTU problems could be that dramatic.

I am now seeing a log message about every 3 mins saying “invalid mtu 1500 on pppoe-out1 from fe80::262:ecff:fedb:c00”. Advice welcome on whether I should tune the MTU up to 1500 or if it’s safe to ignore it as per http://forum.mikrotik.com/t/radvd-invalid-mtu/167666/1

5

These messages are safe to ignore (it seems that IPv6 RA daemon on ISP side is not aware of PPPoE and it’s lower MTU … it’s quite common or so it seems). You can simply stop logging them by setting

/system/logging
set 2 topics=warning,!radvd

Make sure you run print command beforehand and verify that setting with index 2 is indeed the one with topics set to warning …

You most probably should not increase MTU to 1500, not many ISPs support that over PPPoE connection.

6

Thanks. It looked similar to the forum answer I linked, but the MTU was order of magnitude bigger there so wanted to check it was still applicable.

IDNet have indeed separately recommended 1492. They’re still claiming no change from their end, which is a bit hard to square with the results of the experiments above, but hey ho, it’s fixed now - even if I had to pull out most of my hair over the last few days to get there. I’ll pass on the possible RA misconfiguration to them.

7

Hi @martinm

I just wanted to give you a big thank you for mentioning this. This morning I completed a switch from AAISP to IDNET here in the UK. While on AAISP, my RB5009 always gave me 900/900 line which is what I pay for, however when this was switched to IDNET (just to save a few pounds), I then suffered 95/900 connection with same hardware/config etc (just with user/pass details updated).

Original:

/interface pppoe-client
add allow=chap comment=AAISP disabled=no interface=vlan911 max-mru=1500 max-mtu=1500 mrru=1500 name=pppoe0 user=*****

❯ iperf3 -c speedtest.idnet.net

Connecting to host speedtest.idnet.net, port 5201

[  7] local 10.29.30.20 port 59526 connected to 212.69.36.111 port 5201

[ ID] Interval           Transfer     Bitrate

[  7]   0.00-1.00   sec   101 MBytes   850 Mbits/sec
[  7]   1.00-2.01   sec   111 MBytes   926 Mbits/sec
[  7]   2.01-3.00   sec   108 MBytes   911 Mbits/sec
[  7]   3.00-4.00   sec   108 MBytes   908 Mbits/sec
[  7]   4.00-5.00   sec   111 MBytes   927 Mbits/sec
[  7]   5.00-6.01   sec   111 MBytes   928 Mbits/sec
[  7]   6.01-7.00   sec   104 MBytes   878 Mbits/sec
[  7]   7.00-8.01   sec   108 MBytes   904 Mbits/sec
[  7]   8.01-9.01   sec   110 MBytes   927 Mbits/sec
[  7]   9.01-10.01  sec   111 MBytes   928 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate
[  7]   0.00-10.01  sec  1.06 GBytes   909 Mbits/sec                  sender
[  7]   0.00-10.01  sec  1.06 GBytes   907 Mbits/sec                  receiver
 
iperf Done.
 
~   10s

❯ iperf3 -c speedtest.idnet.net -R

Connecting to host speedtest.idnet.net, port 5201

Reverse mode, remote host speedtest.idnet.net is sending

[  7] local 10.29.30.20 port 59528 connected to 212.69.36.111 port 5201
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-1.00   sec  11.1 MBytes  93.3 Mbits/sec
[  7]   1.00-2.01   sec  11.4 MBytes  95.0 Mbits/sec
[  7]   2.01-3.01   sec  11.2 MBytes  94.4 Mbits/sec
[  7]   3.01-4.01   sec  11.2 MBytes  94.4 Mbits/sec
[  7]   4.01-5.00   sec  11.2 MBytes  94.8 Mbits/sec
[  7]   5.00-6.01   sec  11.4 MBytes  95.0 Mbits/sec
[  7]   6.01-7.00   sec  11.2 MBytes  94.6 Mbits/sec
[  7]   7.00-8.00   sec  11.2 MBytes  94.5 Mbits/sec
[  7]   8.00-9.00   sec  11.2 MBytes  94.1 Mbits/sec
[  7]   9.00-10.00  sec  11.2 MBytes  94.5 Mbits/sec

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Retr
[  7]   0.00-10.01  sec   114 MBytes  95.3 Mbits/sec    0            sender
[  7]   0.00-10.00  sec   113 MBytes  94.5 Mbits/sec                  receiver

This didn't work with IDNET, while connection was brought up fine and uploads also fine, the download was heavily impacted.

Now changed to:

/interface pppoe-client
add allow=chap comment=IDNET disabled=no interface=vlan911 name=pppoe0 user=******

❯ iperf3 -c speedtest.idnet.net
Connecting to host speedtest.idnet.net, port 5201
[  7] local 10.29.30.20 port 61010 connected to 212.69.36.111 port 5201
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-1.01   sec  99.0 MBytes   826 Mbits/sec
[  7]   1.01-2.01   sec   111 MBytes   929 Mbits/sec
[  7]   2.01-3.00   sec   108 MBytes   910 Mbits/sec
[  7]   3.00-4.00   sec   108 MBytes   907 Mbits/sec
[  7]   4.00-5.00   sec   108 MBytes   903 Mbits/sec
[  7]   5.00-6.00   sec   111 MBytes   928 Mbits/sec
[  7]   6.00-7.00   sec   105 MBytes   880 Mbits/sec
[  7]   7.00-8.00   sec   105 MBytes   881 Mbits/sec
[  7]   8.00-9.00   sec   111 MBytes   929 Mbits/sec
[  7]   9.00-10.01  sec   111 MBytes   928 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-10.01  sec  1.05 GBytes   902 Mbits/sec                  sender
[  7]   0.00-10.01  sec  1.05 GBytes   901 Mbits/sec                  receiver

iperf Done.

~   10s
❯ iperf3 -c speedtest.idnet.net -R
Connecting to host speedtest.idnet.net, port 5201
Reverse mode, remote host speedtest.idnet.net is sending
[  7] local 10.29.30.20 port 61012 connected to 212.69.36.111 port 5201
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-1.00   sec  93.9 MBytes   785 Mbits/sec
[  7]   1.00-2.00   sec  93.0 MBytes   783 Mbits/sec
[  7]   2.00-3.00   sec  93.1 MBytes   781 Mbits/sec
[  7]   3.00-4.00   sec  92.6 MBytes   777 Mbits/sec
[  7]   4.00-5.00   sec  91.8 MBytes   767 Mbits/sec
[  7]   5.00-6.00   sec  96.2 MBytes   807 Mbits/sec
[  7]   6.00-7.01   sec  95.0 MBytes   796 Mbits/sec
[  7]   7.01-8.01   sec  90.9 MBytes   762 Mbits/sec
[  7]   8.01-9.01   sec  96.2 MBytes   807 Mbits/sec
[  7]   9.01-10.00  sec  95.6 MBytes   805 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  7]   0.00-10.01  sec   948 MBytes   795 Mbits/sec  63673            sender
[  7]   0.00-10.00  sec   938 MBytes   787 Mbits/sec                  receiver

and now the line is 900/900.

Thank you again for posting the solution, this saved me even more debugging then I already have done.