Mark "LCP EchoRep" with PCP 7 to mitigate PPPoE over VRRP VXLAN vBNG

[Anime4000]

Lately, PPPoE become very unstable when I start uploading terabytes of data, so I investigate what cause PPPoE get terminated when I saturated the upload

> /tool bandwidth-test address=104.16.133.229 protocol=udp direction=transmit user=mikrotik password=l23LjWHP2kf0vrECuyJz01JhhA80s2kB
                status: running  
              duration: 4m26s    
            tx-current: 1162.0Mbps
  tx-10-second-average: 1055.2Mbps
      tx-total-average: 1161.7Mbps
           random-data: no       
             direction: transmit 
               tx-size: 1472     
      connection-count: 20       
        local-cpu-load: 30%      
       remote-cpu-load: 24%

long day logs

 2026-02-13 12:42:15 pppoe,ppp,info pppoe-out1: terminating... (terminated by peer)
 2026-02-13 12:42:18 pppoe,ppp,info pppoe-out1: connected
 ...
 2026-02-13 15:24:39 pppoe,ppp,info pppoe-out1: terminating... (terminated by peer)
 2026-02-13 15:24:42 pppoe,ppp,info pppoe-out1: connected
 ...
 2026-02-13 17:14:23 pppoe,ppp,info pppoe-out1: terminating... (terminated by peer)
 2026-02-13 17:14:26 pppoe,ppp,info pppoe-out1: connected
 ...
 2026-02-14 00:38:48 pppoe,ppp,info pppoe-out1: terminating... (terminated by peer)
 2026-02-14 00:38:51 pppoe,ppp,info pppoe-out1: connected
 ...
 2026-02-14 11:52:53 pppoe,ppp,info pppoe-out1: terminating... (terminated by peer)
 2026-02-14 11:52:57 pppoe,ppp,info pppoe-out1: connected
 ...
 2026-02-18 11:51:01 pppoe,ppp,info pppoe-out1: terminating... (terminated by peer)
 2026-02-18 11:51:04 pppoe,ppp,info pppoe-out1: connected
 ...
 2026-02-18 16:07:05 pppoe,ppp,info pppoe-out1: terminating... - hungup
 2026-02-18 16:07:08 pppoe,ppp,info pppoe-out1: connected
 ...
 2026-02-20 03:48:15 pppoe,ppp,info pppoe-out1: terminating... - disconnected
 2026-02-20 03:48:18 pppoe,ppp,info pppoe-out1: connected
 ...
 2026-02-19 20:19:57 pppoe,ppp,info pppoe-out1: terminating... - disconnected
 2026-02-19 20:20:12 pppoe,ppp,info pppoe-out1: connected
 ...
 2026-02-20 07:50:25 pppoe,ppp,info pppoe-out1: terminating... (terminated by peer)
 2026-02-20 07:50:38 pppoe,ppp,info pppoe-out1: connected
 ...
 2026-02-21 09:45:27 pppoe,ppp,info pppoe-out1: terminating... (terminated by peer)
 2026-02-21 09:45:31 pppoe,ppp,info pppoe-out1: connected

Recent disconnect

Feb/21/2026 15:36:52 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP EchoReq id=0x1e
Feb/21/2026 15:36:52 pppoe,ppp,debug,packet     <magic 0x9cd40bec>
Feb/21/2026 15:36:52 pppoe,ppp,debug,packet  pppoe-out1: sent LCP EchoRep id=0x1e
Feb/21/2026 15:36:52 pppoe,ppp,debug,packet     <magic 0x15ec4918>
Feb/21/2026 15:37:12 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP EchoReq id=0x1f
Feb/21/2026 15:37:12 pppoe,ppp,debug,packet     <magic 0x9cd40bec>
Feb/21/2026 15:37:12 pppoe,ppp,debug,packet  pppoe-out1: sent LCP EchoRep id=0x1f
Feb/21/2026 15:37:12 pppoe,ppp,debug,packet     <magic 0x15ec4918>
Feb/21/2026 15:37:32 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP TermReq id=0x3
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: LCP closed
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: CCP lowerdown
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: BCP lowerdown
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: BCP down event in starting state
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: IPCP lowerdown
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: IPCP closed
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: IPV6CP lowerdown
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: IPV6CP closed
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: MPLSCP lowerdown
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: MPLSCP down event in starting state
Feb/21/2026 15:37:32 pppoe,ppp,debug,packet  pppoe-out1: sent LCP TermAck id=0x3
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: LCP lowerdown
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: CCP close
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: BCP close
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: IPCP close
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: IPV6CP close
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: MPLSCP close
Feb/21/2026 15:37:32 pppoe,ppp,info pppoe-out1: terminating...
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: LCP lowerdown
Feb/21/2026 15:37:32 pppoe,ppp,debug pppoe-out1: LCP down event in starting state
Feb/21/2026 15:37:32 pppoe,ppp,info pppoe-out1: disconnected
Feb/21/2026 15:37:32 pppoe,ppp,info pppoe-out1: initializing...
Feb/21/2026 15:37:32 pppoe,ppp,info pppoe-out1: connecting...
Feb/21/2026 15:37:33 pppoe,ppp,debug pppoe-out1: LCP lowerup
Feb/21/2026 15:37:33 pppoe,ppp,debug pppoe-out1: LCP open
Feb/21/2026 15:37:33 pppoe,ppp,debug pppoe-out1: LCP timer
Feb/21/2026 15:37:33 pppoe,ppp,debug,packet  pppoe-out1: sent LCP ConfReq id=0x7
Feb/21/2026 15:37:33 pppoe,ppp,debug,packet    <mru 1500>
Feb/21/2026 15:37:33 pppoe,ppp,debug,packet    <magic 0x3580eca1>
Feb/21/2026 15:37:33 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP ConfAck id=0x7
Feb/21/2026 15:37:33 pppoe,ppp,debug,packet    <mru 1500>
Feb/21/2026 15:37:33 pppoe,ppp,debug,packet    <magic 0x3580eca1>
Feb/21/2026 15:37:34 pppoe,ppp,debug pppoe-out1: LCP timer
Feb/21/2026 15:37:34 pppoe,ppp,debug,packet  pppoe-out1: sent LCP ConfReq id=0x8
Feb/21/2026 15:37:34 pppoe,ppp,debug,packet    <mru 1500>
Feb/21/2026 15:37:34 pppoe,ppp,debug,packet    <magic 0x3580eca1>
Feb/21/2026 15:37:34 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP ConfAck id=0x8
Feb/21/2026 15:37:34 pppoe,ppp,debug,packet    <mru 1500>
Feb/21/2026 15:37:34 pppoe,ppp,debug,packet    <magic 0x3580eca1>
Feb/21/2026 15:37:35 pppoe,ppp,debug pppoe-out1: LCP timer
Feb/21/2026 15:37:35 pppoe,ppp,debug,packet  pppoe-out1: sent LCP ConfReq id=0x9
Feb/21/2026 15:37:35 pppoe,ppp,debug,packet    <mru 1500>
Feb/21/2026 15:37:35 pppoe,ppp,debug,packet    <magic 0x3580eca1>
Feb/21/2026 15:37:35 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP ConfAck id=0x9
Feb/21/2026 15:37:35 pppoe,ppp,debug,packet    <mru 1500>
Feb/21/2026 15:37:35 pppoe,ppp,debug,packet    <magic 0x3580eca1>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP ConfReq id=0x2
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet    <mru 1492>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet    <magic 0xcc4802ee>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet    <auth pap>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: sent LCP ConfAck id=0x2
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet    <mru 1492>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet    <magic 0xcc4802ee>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet    <auth pap>
Feb/21/2026 15:37:36 pppoe,ppp,debug pppoe-out1: LCP opened
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: sent PAP AuthReq id=0x3
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     <user username@unifi>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     <password 12345678>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: rcvd PAP AuthAck id=0x3
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     Authentication success,Welcome!
Feb/21/2026 15:37:36 pppoe,ppp,info pppoe-out1: authenticated
Feb/21/2026 15:37:36 pppoe,ppp,debug pppoe-out1: IPCP lowerup
Feb/21/2026 15:37:36 pppoe,ppp,debug pppoe-out1: IPCP open
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: sent IPCP ConfReq id=0x5
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     <addr 0.0.0.0>
Feb/21/2026 15:37:36 pppoe,ppp,debug pppoe-out1: IPV6CP lowerup
Feb/21/2026 15:37:36 pppoe,ppp,debug pppoe-out1: IPV6CP open
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: sent IPV6CP ConfReq id=0x3
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     <interface-identifier b06c:5cca:0:25>
Feb/21/2026 15:37:36 pppoe,ppp,debug pppoe-out1: MPLSCP open
Feb/21/2026 15:37:36 pppoe,ppp,debug pppoe-out1: BCP open
Feb/21/2026 15:37:36 pppoe,ppp,debug pppoe-out1: CCP lowerup
Feb/21/2026 15:37:36 pppoe,ppp,debug pppoe-out1: CCP open
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: rcvd IPCP ConfReq id=0x1
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     <addr 115.133.39.254>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: sent IPCP ConfAck id=0x1
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     <addr 115.133.39.254>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: rcvd IPV6CP ConfReq id=0x1
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     <interface-identifier 200:5eff:fe00:1d4>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: sent IPV6CP ConfAck id=0x1
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     <interface-identifier 200:5eff:fe00:1d4>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: rcvd IPCP ConfNak id=0x5
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     <addr 115.133.38.247>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: sent IPCP ConfReq id=0x6
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     <addr 115.133.38.247>
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: rcvd IPV6CP ConfAck id=0x3
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     <interface-identifier b06c:5cca:0:25>
Feb/21/2026 15:37:36 pppoe,ppp,debug pppoe-out1: IPV6CP opened
Feb/21/2026 15:37:36 pppoe,ppp,info pppoe-out1: connected
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet  pppoe-out1: rcvd IPCP ConfAck id=0x6
Feb/21/2026 15:37:36 pppoe,ppp,debug,packet     <addr 115.133.38.247>
Feb/21/2026 15:37:36 pppoe,ppp,debug pppoe-out1: IPCP opened
Feb/21/2026 15:37:56 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP EchoReq id=0x1
Feb/21/2026 15:37:56 pppoe,ppp,debug,packet     <magic 0xcc4802ee>
Feb/21/2026 15:37:56 pppoe,ppp,debug,packet  pppoe-out1: sent LCP EchoRep id=0x1
Feb/21/2026 15:37:56 pppoe,ppp,debug,packet     <magic 0x3580eca1>
Feb/21/2026 15:38:16 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP EchoReq id=0x2
Feb/21/2026 15:38:16 pppoe,ppp,debug,packet     <magic 0xcc4802ee>
Feb/21/2026 15:38:16 pppoe,ppp,debug,packet  pppoe-out1: sent LCP EchoRep id=0x2
Feb/21/2026 15:38:16 pppoe,ppp,debug,packet     <magic 0x3580eca1>

I have contacted ISP Support, NOC Engineer come to investigate... the ISP have move many their infra to Cloud: vBNG, VRRP, VXLAN, etc...

ISP decommission Juniper Networks equipment in favor Huawei NE9000 vBNG

I told them that was a mistake to virtualized everything, because PPPoE are sensitive with VRRP/VXLAN jitter:

Feb/21/2026 15:36:52 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP EchoReq id=0x1e
Feb/21/2026 15:36:52 pppoe,ppp,debug,packet     <magic 0x9cd40bec>
Feb/21/2026 15:36:52 pppoe,ppp,debug,packet  pppoe-out1: sent LCP EchoRep id=0x1e
Feb/21/2026 15:36:52 pppoe,ppp,debug,packet     <magic 0x15ec4918>
Feb/21/2026 15:37:12 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP EchoReq id=0x1f
Feb/21/2026 15:37:12 pppoe,ppp,debug,packet     <magic 0x9cd40bec>
Feb/21/2026 15:37:12 pppoe,ppp,debug,packet  pppoe-out1: sent LCP EchoRep id=0x1f
Feb/21/2026 15:37:12 pppoe,ppp,debug,packet     <magic 0x15ec4918>
Feb/21/2026 15:37:32 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP TermReq id=0x3
                                                         ----------------
														         |
																 |-- last "sent LCP EchoRep" lost in VXLAN

ISP Engineer show their ONR (Optical Network Router) have proper PCP to accommodate PPPoE over jitter VXLAN, by marking PPP LCP Echo inside 0x8864 with PCP 7

PPP LCP Echo on EtherType 0x8864 plus VLAN took only around 112 bytes, which Mikrotik didn't mark it PCP 7 and ISP side QoS drop it.

ISP stock ONR (Optical Network Router) have modified pppd by MediaTek EcoNet to inject proper PCP 7, but when in Bridge Mode, it just pass traffic and Mikrotik responsible to insert PCP 7 for PPP LCP Echo

I think, vBNG/VRRP/VXLAN is bad technology and doesn't care ancient PPPoE

[itimo01]

Isnt it basically a known thing that Huawei vBNGs are bad?

[Anime4000]

before was Juniper

now VRRP,

ISP told me whole BNG is now Huawei, latency from my home to vBNG is 3ms

PPPoE are sensetive to jitter, their stock ONR have modified pppd to pace PPP LCP over VXLAN

and Yes, vBNG is bad, I don't get it why ISP want to cloud everything, even OLT is Virtualized now, Mikrotik need prepare ISP infra going Cloud now,

[Znevna]

What if you set PCP 7 for all the pppoe packets?

or.. just don’t saturate the link?

and … make a feature request

[Anime4000]

I think Mikrotik don't have Layer 2 Mangle, not possible to PCP 7 whole 0x8864

I paid 2Gbps download, 1Gbps upload... sure I will use whole link

Recent disconnect, I limit bandwidth to 500Mbps, still have disconnect

Feb/22/2026 10:53:03 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP EchoReq id=0x57
Feb/22/2026 10:53:03 pppoe,ppp,debug,packet     <magic 0x4eecc2bb>
Feb/22/2026 10:53:03 pppoe,ppp,debug,packet  pppoe-out1: sent LCP EchoRep id=0x57
Feb/22/2026 10:53:03 pppoe,ppp,debug,packet     <magic 0x1e7053be>
Feb/22/2026 10:53:23 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP EchoReq id=0x58
Feb/22/2026 10:53:23 pppoe,ppp,debug,packet     <magic 0x4eecc2bb>
Feb/22/2026 10:53:23 pppoe,ppp,debug,packet  pppoe-out1: sent LCP EchoRep id=0x58
Feb/22/2026 10:53:23 pppoe,ppp,debug,packet     <magic 0x1e7053be>
Feb/22/2026 10:53:43 pppoe,ppp,debug,packet  pppoe-out1: rcvd LCP TermReq id=0x3
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: LCP closed
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: CCP lowerdown
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: BCP lowerdown
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: BCP down event in starting state
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: IPCP lowerdown
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: IPCP closed
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: IPV6CP lowerdown
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: IPV6CP closed
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: MPLSCP lowerdown
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: MPLSCP down event in starting state
Feb/22/2026 10:53:43 pppoe,ppp,debug,packet  pppoe-out1: sent LCP TermAck id=0x3
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: LCP lowerdown
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: CCP close
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: BCP close
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: IPCP close
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: IPV6CP close
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: MPLSCP close
Feb/22/2026 10:53:43 pppoe,ppp,info pppoe-out1: terminating...
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: LCP lowerdown
Feb/22/2026 10:53:43 pppoe,ppp,debug pppoe-out1: LCP down event in starting state
Feb/22/2026 10:53:43 pppoe,ppp,info pppoe-out1: disconnected

[Znevna]

Ye I’ve tried on my RB5009 using switch rules but apparently it’s not supported by the switch in this unit, spits an unsupported error, some switches can do this though.

Or you can try setting your wan through a bridge, and set it using bridge filters, just for a test.

At what point those TX drops increase? just before a disconnect? Or something else is causing those packets to never even leave your router?

Device? port used?

[Anime4000]

I checked

it appear do not have in depth Layer 2 pattern to match

when I put sfp1 on new bridge, and pppoe on that new bridge, my speed become worst and all CCR2004 core 100% when iperf3 over PPPoE

[Znevna]

[Anime4000]

that's make whole PPPoE Session with PCP 7, still cause jitter as PPP LCP fighting with others

[Znevna]

Only a feature request then might be the only hope.

[Anime4000]

I have contacted Mikrotik support since last year, Mikrotik seem unaware ISP migrating to Cloud and no way to maintain pace PPP link over fragile VRRP and VXLAN

Only way to stable is double NAT, use ISP Router and NAT again to Mikrotik, Unless ISP ONR support PPPoE Half-Bridge

[Anime4000]

I wondering what cause these spike?

[Anime4000]

Telekom Malaysia Unifi Elite has identified the issue, they admit that Huawei vBNG terminate connection when I use all 1Gbps upload

I run bandwidth test (since windows version never work)

flooding upload pppoe-out1 -> vlan500 -> sfp-sfpplus1

in mean time, I do this

/interface ethernet switch port set sfp-sfpplus1 egress-rate=1240M
/queue type add name=cake-l2-egress-gpon kind=cake cake-bandwidth=1100M
/queue interface set sfp-sfpplus1 queue=cake-l2-egress-gpon

then queue with FastTrack support

/queue type add name=fq-codel-default kind=fq-codel
/queue tree add name=qos-upload parent=pppoe-out1 packet-mark=no-mark queue=fq-codel-default bucket-size=0.01 max-limit=1024M

I been googling, asking ChatGPT, Google Gemini, Deepseek about Cake Overhead correct value for GPON, from value 22, 48, , 58 all didn't work

so I thought, cake diffserv did not understand PPP LCP, since it's for layer 3 classification

then I try zero cake overhead, since I going to attach cake on the physical interface that see vlan500 and pppoe-out1 egress traffic, this seem to work,

or ISP already fixed their stupid Huawei vBNG