WAN throughput degradation after terminating PPPoE with RB2011UAS

gerhard · Fri Oct 29, 2021 1:25 pm

I used to have the following network setup:

GPON ONT (syncs at 1Gbps, symmetric)
|
|- Unifi DreamMachine Pro (1Gbps) - acting as PPPoE client

The problem with this was sporadic packet loss, ISP connection drops, etc.

I have since introduced a MikroTik RB2011UAS into the network chain, which is acting as the PPPoE client. The connection is now stable with almost no packet loss:

The problem now is that the WAN throughput is highly variable and can be as much as 20x lower.

BEFORE, GPON ONT -> UDM Pro (PPPoE) -> Linux host

root@linux-host:~# iperf3 -c w.gerhard.io
Connecting to host w.gerhard.io, port 5201
[  5] local 192.168.1.11 port 48876 connected to 176.58.103.232 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  93.8 MBytes   787 Mbits/sec    0   3.10 MBytes
[  5]   1.00-2.00   sec   110 MBytes   923 Mbits/sec   17   1.54 MBytes
[  5]   2.00-3.00   sec   106 MBytes   891 Mbits/sec    1   1.63 MBytes
[  5]   3.00-4.00   sec   109 MBytes   912 Mbits/sec    0   1.68 MBytes
[  5]   4.00-5.00   sec   109 MBytes   912 Mbits/sec    0   1.73 MBytes
[  5]   5.00-6.00   sec   110 MBytes   923 Mbits/sec    0   1.77 MBytes
[  5]   6.00-7.00   sec   110 MBytes   923 Mbits/sec    0   1.82 MBytes
[  5]   7.00-8.00   sec   109 MBytes   912 Mbits/sec    0   1.86 MBytes
[  5]   8.00-9.00   sec   111 MBytes   933 Mbits/sec    0   1.90 MBytes
[  5]   9.00-10.00  sec   110 MBytes   923 Mbits/sec    0   1.94 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.05 GBytes   904 Mbits/sec   18             sender
[  5]   0.00-10.04  sec  1.05 GBytes   899 Mbits/sec                  receiver

root@linux-host:~# iperf3 -c w.gerhard.io -R
Connecting to host w.gerhard.io, port 5201
Reverse mode, remote host w.gerhard.io is sending
[  5] local 192.168.1.11 port 48880 connected to 176.58.103.232 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   104 MBytes   876 Mbits/sec
[  5]   1.00-2.00   sec   111 MBytes   928 Mbits/sec
[  5]   2.00-3.00   sec   111 MBytes   933 Mbits/sec
[  5]   3.00-4.00   sec   107 MBytes   894 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   936 Mbits/sec
[  5]   5.00-6.00   sec   111 MBytes   934 Mbits/sec
[  5]   6.00-7.00   sec   111 MBytes   934 Mbits/sec
[  5]   7.00-8.00   sec   111 MBytes   934 Mbits/sec
[  5]   8.00-9.00   sec   110 MBytes   921 Mbits/sec
[  5]   9.00-10.00  sec   111 MBytes   928 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  1.08 GBytes   921 Mbits/sec    3             sender
[  5]   0.00-10.00  sec  1.07 GBytes   922 Mbits/sec                  receiver

AFTER, GPON ONT -> RB2011UAS (PPPoE) -> UDM Pro -> Linux host

root@linux-host:~# iperf3 -c w.gerhard.io
Connecting to host w.gerhard.io, port 5201
[  5] local 192.168.1.11 port 49414 connected to 176.58.103.232 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  7.97 MBytes  66.8 Mbits/sec    1    113 KBytes
[  5]   1.00-2.00   sec  9.68 MBytes  81.2 Mbits/sec    1    114 KBytes
[  5]   2.00-3.00   sec  9.56 MBytes  80.2 Mbits/sec    0    165 KBytes
[  5]   3.00-4.00   sec  13.1 MBytes   109 Mbits/sec    1    151 KBytes
[  5]   4.00-5.00   sec  10.1 MBytes  84.3 Mbits/sec    1    145 KBytes
[  5]   5.00-6.00   sec  11.8 MBytes  98.7 Mbits/sec    1    192 KBytes
[  5]   6.00-7.00   sec  15.0 MBytes   125 Mbits/sec    1    170 KBytes
[  5]   7.00-8.00   sec  11.0 MBytes  92.5 Mbits/sec    2    110 KBytes
[  5]   8.00-9.00   sec  6.99 MBytes  58.6 Mbits/sec    1    127 KBytes
[  5]   9.00-10.00  sec  9.99 MBytes  83.8 Mbits/sec    1    128 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   105 MBytes  88.1 Mbits/sec   10             sender
[  5]   0.00-10.04  sec   104 MBytes  86.8 Mbits/sec                  receiver

root@linux-host:~# iperf3 -c w.gerhard.io -R
Connecting to host w.gerhard.io, port 5201
Reverse mode, remote host w.gerhard.io is sending
[  5] local 192.168.1.11 port 49430 connected to 176.58.103.232 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  60.0 MBytes   503 Mbits/sec
[  5]   1.00-2.00   sec  28.6 MBytes   240 Mbits/sec
[  5]   2.00-3.00   sec  23.2 MBytes   195 Mbits/sec
[  5]   3.00-4.00   sec  21.0 MBytes   176 Mbits/sec
[  5]   4.00-5.00   sec  22.5 MBytes   189 Mbits/sec
[  5]   5.00-6.00   sec  20.4 MBytes   171 Mbits/sec
[  5]   6.00-7.00   sec  22.8 MBytes   192 Mbits/sec
[  5]   7.00-8.00   sec  19.9 MBytes   167 Mbits/sec
[  5]   8.00-9.00   sec  23.4 MBytes   196 Mbits/sec
[  5]   9.00-10.00  sec  26.5 MBytes   222 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   272 MBytes   227 Mbits/sec  619             sender
[  5]   0.00-10.00  sec   268 MBytes   225 Mbits/sec                  receiver

This is the full network layout used for iperf3 testing:

|- w.gerhard.io (2.5Gbps) - public iperf3 server
|
GPON ONT (syncs at 1Gbps, symmetric)
|
|- MikroTik RB2011UAS (1Gbps) - acting as PPPoE client
  |
  |- Mac host A (1Gbps)
  |- Unifi DreamMachine Pro (1Gbps)
    |
    |- Linux host (1Gbps)
    |- FreeBSD host (1Gbps)
    |- Mac host B (10Gbps)

I have no issues inside the Unifi network, as all 3 hosts consistently max out 1Gbps.
As soon as I go through the RB2011UAS, I get the up to 20x slower & highly variable throughput. Starts at 500Mbps and drops to as low as 50Mbps

I thought it may be the double NAT, so I ran iperf3 from Mac host A which is wired directly into the RB2011UAS:

gerhard@mac-host-a ~ % iperf3 -c w.gerhard.io
Connecting to host w.gerhard.io, port 5201
[  7] local 192.168.88.251 port 60581 connected to 176.58.103.232 port 5201
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-1.00   sec  77.1 MBytes   647 Mbits/sec
[  7]   1.00-2.00   sec  77.3 MBytes   648 Mbits/sec
[  7]   2.00-3.00   sec  62.2 MBytes   522 Mbits/sec
[  7]   3.00-4.00   sec  35.0 MBytes   293 Mbits/sec
[  7]   4.00-5.00   sec  18.6 MBytes   156 Mbits/sec
[  7]   5.00-6.00   sec  26.2 MBytes   220 Mbits/sec
[  7]   6.00-7.00   sec  23.8 MBytes   200 Mbits/sec
[  7]   7.00-8.00   sec  26.8 MBytes   225 Mbits/sec
[  7]   8.00-9.00   sec  22.8 MBytes   191 Mbits/sec
[  7]   9.00-10.00  sec  24.6 MBytes   206 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-10.00  sec   394 MBytes   331 Mbits/sec                  sender
[  7]   0.00-10.01  sec   394 MBytes   330 Mbits/sec                  receiver

gerhard@mac-host-a ~ % iperf3 -c w.gerhard.io -R
Connecting to host w.gerhard.io, port 5201
Reverse mode, remote host w.gerhard.io is sending
[  7] local 192.168.88.251 port 60595 connected to 176.58.103.232 port 5201
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-1.00   sec  56.0 MBytes   470 Mbits/sec
[  7]   1.00-2.00   sec  36.7 MBytes   308 Mbits/sec
[  7]   2.00-3.00   sec  35.1 MBytes   295 Mbits/sec
[  7]   3.00-4.00   sec  31.3 MBytes   262 Mbits/sec
[  7]   4.00-5.00   sec  16.9 MBytes   142 Mbits/sec
[  7]   5.00-6.00   sec  13.9 MBytes   117 Mbits/sec
[  7]   6.00-7.00   sec  17.5 MBytes   146 Mbits/sec
[  7]   7.00-8.00   sec  21.4 MBytes   179 Mbits/sec
[  7]   8.00-9.00   sec  22.5 MBytes   189 Mbits/sec
[  7]   9.00-10.00  sec  20.4 MBytes   171 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  7]   0.00-10.01  sec   275 MBytes   230 Mbits/sec  1023             sender
[  7]   0.00-10.00  sec   272 MBytes   228 Mbits/sec                  receiver

Should the RB2011UAS be able to sustain 1Gbps TCP traffic over PPPoE? Do I have a misconfiguration?

This is my current RB2011UAS config (removed the less relevant sections like ipv6, lcd, etc.)

# oct/29/2021 11:20:20 by RouterOS 6.49
# software id = 0YXJ-NGJP
#
# model = 2011UAS
# serial number = 3747022BB7F7
/interface bridge
add admin-mac=D4:CA:6D:79:23:12 auto-mac=no comment=defconf name=bridge
/interface vlan
add interface=ether1 name=Vodafone vlan-id=911
/interface pppoe-client
add add-default-route=yes disabled=no interface=Vodafone keepalive-timeout=disabled name="Vodafone PPPoE" password=XXX user=XXX
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge name=defconf
/tool user-manager customer
set admin access=own-routers,own-users,own-profiles,own-limits,config-payment-gw
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=ether8
add bridge=bridge comment=defconf interface=ether9
add bridge=bridge comment=defconf interface=ether10
add bridge=bridge comment=defconf interface=sfp1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface="Vodafone PPPoE" list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=192.168.88.0
/ip dhcp-client
add comment=defconf disabled=no interface=ether1
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
/system identity
set name=MikroTik-RB2011UAS

tdw · Fri Oct 29, 2021 1:51 pm

As soon as I go through the RB2011UAS, I get the same slow & highly variable throughput. Starts at 500Mbps and drops to as low as 50Mbps
Should the RB2011UAS be able to sustain 920Mbps over PPPoE?

Unlikely given the CPU it has, however this should only cap the throughput not introduce huge random changes in throughput.

The most common GPON installations use 2.5Gbps down / 1.25Gbps up shared across all the clients on each segment from the headend, and there may also be other contention from the headend to the ISPs datacentre. The headline figure for your service may be 920Mbps but that is unlikely to be the continuous guaranteed bandwidth.

You should configure a VLAN and PPPoE client on a PC/Mac, connect it direcly to the ONT and see what results you get from iperf3 then to eliminate the Ubquiti and Mikrotik elements.

gerhard · Fri Oct 29, 2021 3:30 pm

You should configure a VLAN and PPPoE client on a PC/Mac

I don't have a PC, and I when I configure PPPoE on a Mac, I keep getting No route to host errors. Setting the VLAN ID is not straightforward on a Mac, and when I configure that via a virtual interface, I can't configure the PPPoE service on that interface. Networking on Macs is a mess, I know, and I also suspect that it's a tangent (see below).

As mentioned at the beginning of this thread, if I terminate PPPoE with Unifi, I get the full 920Mbps speed consistently. The problem with that setup is that PPPoE drops frequently, and even when it doesn't drop out, I get sporadic 100% packet loss which makes the connection unreliable. Fast, but unreliable.

When I terminate PPPoE with Mikrotik, I get highly variable throughput in iperf3, but otherwise a really stable PPPoE connection, which is essential to me.

Interestingly, when I use speedtest instead of iperf3, I get much better throughput: https://www.speedtest.net/result/12254379428

This makes me wonder if iperf3 default TCP mode behaves in a way that makes RB2011UAS throughput inconsistent. Running in UDP mode confirms this theory:

root@linux-host:~# iperf3 -c w.gerhard.io -u -b 0
Connecting to host w.gerhard.io, port 5201
[  5] local 192.168.1.11 port 57866 connected to 176.58.103.232 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   114 MBytes   955 Mbits/sec  83570
[  5]   1.00-2.00   sec   114 MBytes   953 Mbits/sec  83450
[  5]   2.00-3.00   sec   114 MBytes   953 Mbits/sec  83450
[  5]   3.00-4.00   sec   114 MBytes   953 Mbits/sec  83440
[  5]   4.00-5.00   sec   114 MBytes   953 Mbits/sec  83450
[  5]   5.00-6.00   sec   114 MBytes   953 Mbits/sec  83450
[  5]   6.00-7.00   sec   114 MBytes   953 Mbits/sec  83450
[  5]   7.00-8.00   sec   114 MBytes   953 Mbits/sec  83450
[  5]   8.00-9.00   sec   114 MBytes   953 Mbits/sec  83450
[  5]   9.00-10.00  sec   114 MBytes   953 Mbits/sec  83440
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.11 GBytes   953 Mbits/sec  0.000 ms  0/834600 (0%)  sender
[  5]   0.00-10.06  sec   895 MBytes   746 Mbits/sec  0.023 ms  172511/829707 (21%)  receiver

So I am able to achieve 953Mbit/s with UDP in the following network setup:

|- w.gerhard.io (2.5Gbps) - public iperf3 server
|
GPON ONT (syncs at 1Gbps, symmetric)
|
|- MikroTik RB2011UAS (1Gbps) - acting as PPPoE client
  |
  |- Unifi DreamMachine Pro (1Gbps)
    |
    |- Linux host (1Gbps)

What could make the RB2011UAS under-perform when using TCP instead of UDP?

I am suspecting that it's the +9 years old CPU which clocks at 600Mhz, but considering that this maxes out at 23% when running the benchmark, I'm suspecting that it's something else, but I'm not sure what exactly. I am wondering if this would solve my problem: viewtopic.php?t=179868

mkx · Fri Oct 29, 2021 8:22 pm

Code: Select all
[  5]   0.00-10.06  sec   895 MBytes   746 Mbits/sec  0.023 ms  172511/829707 (21%)  receiver
So I am able to achieve 953Mbit/s with UDP in the following network setup:

No, you're not able to achieve 953Mbps, see the line above (I've left it from your test results).

The big thing when testing with iperf is to understand that UDP has no feedback from receiver ... not realtime.

So what you see on sending side is throughput between sender and first L3 entity on the path ... which is your router. Any L3 entity on path can drop packets if needed. When testing using UDP that means one must check receiver's report to assess the test results (or ideally run iperf server interactively and observe stats real-time). When testing with TCP, L4 protocol provides feedback and sender knows real end-to-end throughput (perhaps slightly skewed compared to receiver but not much).

Which likely explains low TCP throughput: you're experiencing substantial packet loss (test result says 21%) even with RB as PPPoE server, but TCP survives it by using retransmissions and bandwidth throttling, thus lower speed.

Your experience with Unifi PPPoE client indicates that packet loss quite likely occurs between customer port of ONT and PPPoE server of your ISP. It's impossible to tell where exactly, could as well be contaminated (dirty) FO connector.

gerhard · Fri Oct 29, 2021 8:33 pm

OK, so that 21% must be load-shedding. I am using my 10Gbit host to test this theory:

iperf3 -c w.gerhard.io -u -b 0
Connecting to host w.gerhard.io, port 5201
[  7] local 192.168.1.121 port 64743 connected to 176.58.103.232 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  7]   0.00-1.00   sec   468 MBytes  3.92 Gbits/sec  343290
[  7]   1.00-2.00   sec   457 MBytes  3.83 Gbits/sec  335300
[  7]   2.00-3.00   sec   470 MBytes  3.94 Gbits/sec  345190
[  7]   3.00-4.00   sec   471 MBytes  3.95 Gbits/sec  345700
[  7]   4.00-5.00   sec   468 MBytes  3.93 Gbits/sec  343710
[  7]   5.00-6.00   sec   476 MBytes  4.00 Gbits/sec  349880
[  7]   6.00-7.00   sec   444 MBytes  3.72 Gbits/sec  325990
[  7]   7.00-8.00   sec   455 MBytes  3.81 Gbits/sec  333820
[  7]   8.00-9.00   sec   471 MBytes  3.95 Gbits/sec  346120
[  7]   9.00-10.00  sec   462 MBytes  3.87 Gbits/sec  339080
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  7]   0.00-10.00  sec  4.53 GBytes  3.89 Gbits/sec  0.000 ms  0/3408080 (0%)  sender
[  7]   0.00-10.01  sec   932 MBytes   780 Mbits/sec  0.016 ms  2702716/3386843 (80%)  receiver

Does the above mean that this host maxes UDP out at 780Mbps, and the surplus traffic gets dropped?

This is the view from the server side - I assume this is what you meant by interactive mode:

Accepted connection from 90.241.31.80, port 64929
[  5] local 176.58.103.232 port 5201 connected to 90.241.31.80 port 64743
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  90.1 MBytes   756 Mbits/sec  0.012 ms  252197/318365 (79%)
[  5]   1.00-2.00   sec  96.7 MBytes   812 Mbits/sec  0.019 ms  264105/335145 (79%)
[  5]   2.00-3.00   sec  95.1 MBytes   798 Mbits/sec  0.024 ms  275702/345537 (80%)
[  5]   3.00-4.00   sec  95.3 MBytes   800 Mbits/sec  0.019 ms  275552/345553 (80%)
[  5]   4.00-5.00   sec  95.9 MBytes   804 Mbits/sec  0.010 ms  273240/343638 (80%)
[  5]   5.00-6.00   sec  94.8 MBytes   795 Mbits/sec  0.013 ms  281744/351360 (80%)
[  5]   6.00-7.00   sec  95.3 MBytes   800 Mbits/sec  0.021 ms  254720/324710 (78%)
[  5]   7.00-8.00   sec  82.3 MBytes   691 Mbits/sec  0.017 ms  274022/334480 (82%)
[  5]   8.00-9.00   sec  90.5 MBytes   759 Mbits/sec  0.013 ms  277973/344427 (81%)
[  5]   9.00-10.00  sec  94.5 MBytes   793 Mbits/sec  0.023 ms  270382/339770 (80%)
[  5]  10.00-10.01  sec  1.06 MBytes   619 Mbits/sec  0.016 ms  3079/3858 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.01  sec   932 MBytes   780 Mbits/sec  0.016 ms  2702716/3386843 (80%)  receiver

As for TCP, this is what I'm getting currently in reverse mode (client -> server) - notice the 325 retransmits:

Accepted connection from 90.241.31.80, port 64946
[  5] local 176.58.103.232 port 5201 connected to 90.241.31.80 port 64947
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  44.4 MBytes   373 Mbits/sec  276    513 KBytes
[  5]   1.00-2.00   sec  36.2 MBytes   304 Mbits/sec    0    562 KBytes
[  5]   2.00-3.00   sec  40.0 MBytes   336 Mbits/sec    0    612 KBytes
[  5]   3.00-4.00   sec  43.8 MBytes   367 Mbits/sec    0    664 KBytes
[  5]   4.00-5.00   sec  45.0 MBytes   377 Mbits/sec    0    708 KBytes
[  5]   5.00-6.00   sec  48.8 MBytes   409 Mbits/sec    0    752 KBytes
[  5]   6.00-7.00   sec  51.2 MBytes   430 Mbits/sec    0    800 KBytes
[  5]   7.00-8.00   sec  53.8 MBytes   451 Mbits/sec    0    849 KBytes
[  5]   8.00-9.00   sec  48.8 MBytes   409 Mbits/sec   48    452 KBytes
[  5]   9.00-10.00  sec  32.5 MBytes   273 Mbits/sec    0    501 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   444 MBytes   372 Mbits/sec  324             sender

This is the topology that I used for the above tests:

|- w.gerhard.io (2.5Gbps) - public iperf3 server
|
GPON ONT (syncs at 1Gbps, symmetric)
|
|- MikroTik RB2011UAS (1Gbps) - acting as PPPoE client
  |
  |- Unifi DreamMachine Pro (1Gbps)
    |
    |- Unifi XG6POE (10Gbps)
      |
      |- Mac host B (10Gbps)

I am going to re-run it with the UDMPro as the PPPoE client, and remove the RB2011UAS from the chain.

tdw · Fri Oct 29, 2021 8:41 pm

I don't have a PC, and I when I configure PPPoE on a Mac, I keep getting No route to host errors. Setting the VLAN ID is not straightforward on a Mac, and when I configure that via a virtual interface, I can't configure the PPPoE service on that interface. Networking on Macs is a mess, I know, and I also suspect that it's a tangent (see below).

A quick search suggests you have to set the IPv4 configuration for the PPPoE connection to 'Using PPP', it may not default to that even though there is no other option in PPPoE. You could use a managed switch, or even the gigabit switch chip in the 2011, to handle the VLAN tagging so the Mac doesn't have to. Doing a direct test would prove or disprove any issues with the connection.

mkx · Fri Oct 29, 2021 8:51 pm

Does the above mean that this host maxes UDP out at 780Mbps, and the surplus traffic gets dropped?

I'm not sure what exactly you mean by "host". I understand either your linux machine or server. And no, sender (linux host) doesn't throttle down to 700Mbps, in case if 1Gbps test it's pushing out 953Mbos and in 10Gbps case it's pushing out short of 4Gbps. And receiver is likely to successfully ingest everything delivered at its end.

It's other equipment on the way dropping packets ... in 10Gbps case it's clear (from topology) that even L2 device does it (in my previous post I wrongly wrote that only L3 entities are doing it) because of Rx overruns ... either Unifi XG6POE or Unifi DreamMachine Pro, depending on the rate of connection between the two. And then there are other devices between your router and server which are dropping packets (down to short of 800Mbps).

As I already indicated: every packet drop hits TCP performance big time and it builds back slowly ... it's the way TCP works. Delay affects TCP as well, but to lesser extent (all nicely-behaving traffic shapers do it by delaying packets, not by dropping them).

gerhard · Tue Jul 12, 2022 10:24 am

My network setup & experience has improved considerably since this post (one more than the other!), thank you all for helping out.

Before I start the next home office network improvement phase, I wanted to share two important checkpoints:

1.

How I found my lost network packets - check the screenshots & pictures under Notes & Links
2. Latest speedtest
3. Current network topology:

GPON ONT (syncs at 1Gbps, symmetric)
|
|- MikroTik RB5009UG (1Gbps) - acting as PPPoE client
  |
  |- MikroTik CRS312 (10Gbps) - hw offloaded switching only
    |
    |- Mac host (10Gbps)
    |- Linux host (10Gbps)

Both 10Gbps hosts are connected via the CRS312. This is what an iperf3 between the two looks like:

-----------------------------------------------------------
Server listening on 5201 (test #6)
-----------------------------------------------------------
Accepted connection from 192.168.88.17, port 56838
[  5] local 192.168.88.22 port 5201 connected to 192.168.88.17 port 56839
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.09 GBytes  9.35 Gbits/sec
[  5]   1.00-2.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5]   2.00-3.00   sec  1.09 GBytes  9.41 Gbits/sec
[  5]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5]   4.00-5.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5]   5.00-6.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5]   6.00-7.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5]   8.00-9.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5]   9.00-10.00  sec  1.09 GBytes  9.40 Gbits/sec
[  5]  10.00-10.00  sec  5.56 MBytes  9.41 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #7)
-----------------------------------------------------------
Accepted connection from 192.168.88.17, port 56840
[  5] local 192.168.88.22 port 5201 connected to 192.168.88.17 port 56841
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   932 MBytes  7.81 Gbits/sec    0   2.00 MBytes
[  5]   1.00-2.00   sec   926 MBytes  7.77 Gbits/sec    0   2.00 MBytes
[  5]   2.00-3.00   sec   930 MBytes  7.80 Gbits/sec    0   2.00 MBytes
[  5]   3.00-4.00   sec   931 MBytes  7.81 Gbits/sec    0   2.00 MBytes
[  5]   4.00-5.00   sec   932 MBytes  7.82 Gbits/sec    0   2.00 MBytes
[  5]   5.00-6.00   sec   934 MBytes  7.83 Gbits/sec    0   2.00 MBytes
[  5]   6.00-7.00   sec   930 MBytes  7.80 Gbits/sec    0   2.00 MBytes
[  5]   7.00-8.00   sec   934 MBytes  7.83 Gbits/sec    0   2.00 MBytes
[  5]   8.00-9.00   sec   925 MBytes  7.76 Gbits/sec    0   2.00 MBytes
[  5]   9.00-10.00  sec   931 MBytes  7.81 Gbits/sec    0   2.00 MBytes
[  5]  10.00-10.00  sec  2.50 MBytes  6.72 Gbits/sec    0   2.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.09 GBytes  7.81 Gbits/sec    0             sender
-----------------------------------------------------------
Server listening on 5201 (test #8)
-----------------------------------------------------------

Thanks again for helping, will mark this one as solved.

WAN throughput degradation after terminating PPPoE with RB2011UAS [SOLVED]

WAN throughput degradation after terminating PPPoE with RB2011UAS

Re: WAN throughput degradation after terminating PPPoE with RB2011UAS

Re: WAN throughput degradation after terminating PPPoE with RB2011UAS

Re: WAN throughput degradation after terminating PPPoE with RB2011UAS

Re: WAN throughput degradation after terminating PPPoE with RB2011UAS

Re: WAN throughput degradation after terminating PPPoE with RB2011UAS

Re: WAN throughput degradation after terminating PPPoE with RB2011UAS [SOLVED]

Re: WAN throughput degradation after terminating PPPoE with RB2011UAS

Who is online