ICMP Packet loss when WAN is saturated

RouterOS General
1

Something I’ve not experienced on previous routers, presumably because it was something baked in and on Mikrotik I need to configure it, however;

I experience packet loss both in and out while saturating the connection - particularly running speedtests.

Connection is 900/100 over PPPoE over FTTP.

I’ve tried prioritising ICMP, as well as creating a simple queue with dest=pppoe and experimented with max-limit, but it seems I have to significantly reduce my max download speed before the packet loss goes away.

I assume the speed includes overheads. The currently configured 95/940 gives me around 90/800 real world.

This ICMP packet loss is observed by pinging out while running speedtests, but also with a ping monitor;





[admin@MikroTik] > export compact hide-sensitive
# apr/29/2021 09:50:09 by RouterOS 6.48.2
# software id = Y7QR-K6J3
#
# model = RB4011iGS+
# serial number = D4460DEE27EA
/interface bridge
add admin-mac=00:BB:01:32:00:00 auto-mac=no comment=defconf name=bridge
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 name=pppoe-out1 use-peer-dns=yes user=*
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip kid-control
add name=Lily
/ip pool
add name=dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge lease-script="# When \"1\" all DNS entries with IP address of D\
    HCP lease are removed\r\
    \n:local dnsRemoveAllByIp \"1\"\r\
    \n# When \"1\" all DNS entries with hostname of DHCP lease are removed\r\
    \n:local dnsRemoveAllByName \"1\"\r\
    \n# When \"1\" addition and removal of DNS entries is always done also for non-FQDN hostname\r\
    \n:local dnsAlwaysNonfqdn \"1\"\r\
    \n# DNS domain to add after DHCP client hostname\r\
    \n:local dnsDomain \"lan\"\r\
    \n# DNS TTL to set for DNS entries\r\
    \n:local dnsTtl \"00:15:00\"\r\
    \n# Source of DHCP client hostname, can be \"lease-hostname\" or any other lease attribute, like \"host-name\"\
    \_or \"comment\"\r\
    \n:local leaseClientHostnameSource \"lease-hostname\"\r\
    \n\r\
    \n:local leaseComment \"dhcp-lease-script_\$leaseServerName_\$leaseClientHostnameSource\"\r\
    \n:local leaseClientHostname\r\
    \n:if (\$leaseClientHostnameSource = \"lease-hostname\") do={\r\
    \n  :set leaseClientHostname \$\"lease-hostname\"\r\
    \n} else={\r\
    \n  :set leaseClientHostname ([:pick \\\r\
    \n    [/ip dhcp-server lease print as-value where server=\"\$leaseServerName\" address=\"\$leaseActIP\" mac-ad\
    dress=\"\$leaseActMAC\"] \\\r\
    \n    0]->\"\$leaseClientHostnameSource\")\r\
    \n}\r\
    \n:local leaseClientHostnames \"\$leaseClientHostname\"\r\
    \n:if ([:len [\$dnsDomain]] > 0) do={\r\
    \n  :if (\$dnsAlwaysNonfqdn = \"1\") do={\r\
    \n    :set leaseClientHostnames \"\$leaseClientHostname.\$dnsDomain,\$leaseClientHostname\"\r\
    \n  } else={\r\
    \n    :set leaseClientHostnames \"\$leaseClientHostname.\$dnsDomain\"\r\
    \n  }\r\
    \n}\r\
    \n:if (\$dnsRemoveAllByIp = \"1\") do={\r\
    \n  /ip dns static remove [/ip dns static find comment=\"\$leaseComment\" and address=\"\$leaseActIP\"]\r\
    \n}\r\
    \n:foreach h in=[:toarray value=\"\$leaseClientHostnames\"] do={\r\
    \n  :if (\$dnsRemoveAllByName = \"1\") do={\r\
    \n    /ip dns static remove [/ip dns static find comment=\"\$leaseComment\" and name=\"\$h\"]\r\
    \n  }\r\
    \n  /ip dns static remove [/ip dns static find comment=\"\$leaseComment\" and address=\"\$leaseActIP\" and nam\
    e=\"\$h\"]\r\
    \n  :if (\$leaseBound = \"1\") do={\r\
    \n    :delay 1\r\
    \n    /ip dns static add comment=\"\$leaseComment\" address=\"\$leaseActIP\" name=\"\$h\" ttl=\"\$dnsTtl\"\r\
    \n  }\r\
    \n}" name=defconf
/queue simple
add limit-at=512k/512k max-limit=512k/512k name=ICMP packet-marks=icmp-pkt target=""
add dst=pppoe-out1 max-limit=95M/940M name=queue1 target=""
/user group
set full policy=\
    local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,password,web,sniff,sensitive,api,romon,dude,tikapp
/interface bridge port
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=ether8
add bridge=bridge comment=defconf interface=ether9
add bridge=bridge comment=defconf interface=ether10
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge interface=ether2
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=pppoe-out1 list=WAN
add interface=ether3 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=ether2 network=192.168.88.0
/ip dhcp-client
add comment=defconf interface=ether1
add add-default-route=no !dhcp-options disabled=no interface=ether3
/ip dhcp-server lease
add address=192.168.88.245 client-id=1:0:11:32:b4:c0:4d mac-address=00:11:32:B4:C0:4D server=defconf
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 domain=lan gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes servers=1.1.1.1,1.0.0.1
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related \
    disabled=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-connection chain=prerouting new-connection-mark=icmp-con passthrough=yes protocol=icmp
add action=mark-connection chain=postrouting new-connection-mark=icmp-con passthrough=yes protocol=icmp
add action=mark-packet chain=prerouting connection-mark=icmp-con new-packet-mark=icmp-pkt passthrough=yes \
    protocol=icmp
add action=mark-packet chain=postrouting connection-mark=icmp-con new-packet-mark=icmp-pkt passthrough=yes \
    protocol=icmp
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
/ip route
add check-gateway=ping disabled=yes distance=1 gateway=8.8.8.8
add check-gateway=ping disabled=yes distance=2 gateway=8.8.4.4
add disabled=yes distance=1 dst-address=8.8.4.4/32 gateway=192.168.8.1 scope=10
add disabled=yes distance=1 dst-address=8.8.8.8/32 gateway=51.148.72.22 scope=10
/system clock
set time-zone-name=Europe/London
/tool e-mail
set address=smtp-relay.gmail.com from="Home Router <router@*net>" start-tls=yes user=\
    me@*.net
/tool graphing interface
add interface=pppoe-out1
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
2

Does the packet loss go away if you go back to your previous router? Are you sure the packet loss is at your end and not happening upstream at your ISP?

3

Yes, no packet loss on previous router;

~12pm is when I switched to the mikrotik from OpenWRT on a RPI. 2PM is me playing with WAN failover.

Outbound packet loss at least seems to start at the first hop outside of my network.

I notice every other PPPoE device I’ve tried asks for connection speed, I wonder if this is trait of PPPoE and I need to do a better job of controlling traffic my end?

4

Needing to know the speed is not some inherent trait of PPPoE, but the router can’t really prioritise anything if it doesn’t know how much bandwidth is available.

Try disabling discovery on the PPPoE interface:
http://forum.mikrotik.com/t/routeros-6-44-x-pppoe-client-issue/129390/10

5

OpenWRT handles saturation much better due to fq_codel / cake schedulers, not yet available on Mikrotik. You have to cap your bandwidth significantly below link saturation point to avoid buffers being flooded.

6

Thanks, that makes a lot of sense.

Is this normal to run a Mikrotik this way? It seems like quite a significant issue.