Some Websites Not Reachable (MTU size?)

tikker · April 10, 2022, 8:55am

Hey, Good Morning, Everybody

Did some testing this morning and like to share the results. I also try to address some questions that arose:

First: No, there is no VPN involved.

The hping bisect was done with the mangle rule disabled: No issues with a data size up to 1451. At 1452 starting to “feel sluggish”. At 1453 and over no success.

On the direct link, I had no issues at all with random data sizes between 1000 and 5000.

Adding the input accept rule for ICMP does not change the situation! Some sites like duckduckgo still not responding.

Instant response when activating the mangle workaround. To make it clear at this point: I know this is a workaround and I’d rather prefer not to live with it, but find the root cause instead!

So I’d really appreciate if you like to investigate further.

tangent · April 10, 2022, 9:09am

How high up in the firewall list did you put it? Without seeing your rules, I can’t give you a position, but “high.” I believe it’s about #3 in the stock firewall.

tikker · April 10, 2022, 9:35am

First rule, right on top. Packet count steadily rising.

tikker · April 10, 2022, 12:42pm

Of course it was.

What failure message? Firefox comes back with a network timeout.

Like what? What do you need?

Amm0 · April 10, 2022, 1:04pm

Also, there is mention of a RB5009, but also mention of a WLAN here. So begs the question, are these tests from a device connected to a port on the RB5009, or via the Wi-Fi.

Is there some reason you can’t just share the config? Sure the MTU could be involved, but boy a config file tell a lot. If you’re not doing the tests via the RB5009 but though an access point, try same from via ethernet. Result same?

tikker · April 10, 2022, 1:44pm

Connectivity issues are the same over wifi and wire. I have two dedicated ports for the tests on the router. One port uses an untagged VLAN port the other is a single port with DHCP which is NATted directly.

tikker · April 11, 2022, 9:12am

To frequent request, my config:

# apr/10/2022 15:45:21 by RouterOS 7.2
# software id = ZU1X-J0KM
#
# model = RB5009UG+S+
/interface bridge
add name=bridge1 vlan-filtering=yes
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether8 keepalive-timeout=\
    disabled name=pppoe-wan service-name=XXX user=XXX
/interface eoip
add allow-fast-path=no !keepalive mac-address=00:00:00:00:00:00 name=\
    eoip-001 remote-address=192.168.255.101 tunnel-id=42
add !keepalive mac-address=00:00:00:00:00:00 name=eoip-map01 remote-address=\
    10.31.0.8 tunnel-id=4711
/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1
/interface vlan
add interface=bridge1 name=vlan_31 vlan-id=31
add interface=bridge1 name=vlan_32 vlan-id=32
add interface=bridge1 name=vlan_33 vlan-id=33
add interface=bridge1 name=vlan_34 vlan-id=34
add interface=bridge1 name=vlan_35 vlan-id=35
/caps-man datapath
add bridge=bridge1 client-to-client-forwarding=yes local-forwarding=yes name=\
    datapath-31 vlan-id=31 vlan-mode=use-tag
add bridge=bridge1 client-to-client-forwarding=no local-forwarding=yes name=\
    datapath-32 vlan-id=32 vlan-mode=use-tag
add bridge=bridge1 client-to-client-forwarding=no local-forwarding=yes name=\
    datapath-34 vlan-id=34 vlan-mode=use-tag
add bridge=bridge1 client-to-client-forwarding=yes local-forwarding=yes name=\
    datapath-35 vlan-id=35 vlan-mode=use-tag
/caps-man security
add authentication-types=wpa2-psk encryption=aes-ccm name=wpa2-psk-aes
/caps-man configuration
add country=germany datapath=datapath-31 multicast-helper=full name=\
    wlan_trusted security=wpa2-psk-aes ssid=XXX
add country=germany datapath=datapath-32 multicast-helper=full name=\
    wlan_untrusted security=wpa2-psk-aes ssid=XXX
add country=germany datapath=datapath-34 multicast-helper=full name=\
    wlan_guest security=wpa2-psk-aes ssid=XXX
add country=germany datapath=datapath-35 multicast-helper=full name=\
    wlan_smarthome security=wpa2-psk-aes ssid=XXX
/caps-man interface
add configuration=wlan_trusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=none name=ac-cap01-1 radio-mac=\
    00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_untrusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=ac-cap01-1 name=ac-cap01-1-1 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_trusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=none name=ac-hap01-1 radio-mac=\
    00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_untrusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=ac-hap01-1 name=ac-hap01-1-1 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_trusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=none name=ac-hap02-1 radio-mac=\
    00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_untrusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=ac-hap02-1 name=ac-hap02-1-1 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_trusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=none name=ac-hap03-1 radio-mac=\
    00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_untrusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=ac-hap03-1 name=ac-hap03-1-1 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_trusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=none name=gn-cap01-1 radio-mac=\
    00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_untrusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=gn-cap01-1 name=gn-cap01-1-1 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_guest disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=gn-cap01-1 name=gn-cap01-1-2 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_smarthome disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=gn-cap01-1 name=gn-cap01-1-3 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_trusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=none name=gn-hap01-1 radio-mac=\
    00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_untrusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=gn-hap01-1 name=gn-hap01-1-1 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_guest disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=gn-hap01-1 name=gn-hap01-1-2 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_smarthome disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=gn-hap01-1 name=gn-hap01-1-3 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_trusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=none name=gn-hap02-1 radio-mac=\
    00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_untrusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=gn-hap02-1 name=gn-hap02-1-1 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_guest disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=gn-hap02-1 name=gn-hap02-1-2 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_smarthome disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=gn-hap02-1 name=gn-hap02-1-3 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_trusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=none name=gn-hap03-1 radio-mac=\
    00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_untrusted disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=gn-hap03-1 name=gn-hap03-1-1 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_guest disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=gn-hap03-1 name=gn-hap03-1-2 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
add configuration=wlan_smarthome disabled=no l2mtu=1600 mac-address=\
    00:00:00:00:00:00 master-interface=gn-hap03-1 name=gn-hap03-1-3 \
    radio-mac=00:00:00:00:00:00 radio-name=000000000000
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=pool_31 ranges=10.31.0.100-10.31.0.254
add name=pool_32 ranges=10.32.0.100-10.32.0.254
add name=pool_33 ranges=10.33.0.100-10.33.0.254
add name=pool_34 ranges=10.34.0.100-10.34.0.254
add name=pool_35 ranges=10.35.0.100-10.35.0.254
add name=pool_test ranges=192.168.10.100-192.168.10.200
/ip dhcp-server
add address-pool=pool_31 interface=vlan_31 lease-time=1d name=dhcp_31
add address-pool=pool_32 interface=vlan_32 lease-time=1d name=dhcp_32
add address-pool=pool_33 interface=vlan_33 lease-time=1d name=dhcp_33
add address-pool=pool_34 interface=vlan_34 lease-time=6h name=dhcp_34
add address-pool=pool_35 interface=vlan_35 lease-time=1d name=dhcp_35
add address-pool=pool_test bootp-support=none interface=ether5 lease-time=5m \
    name=testserver
/caps-man manager
set ca-certificate=auto certificate=auto enabled=yes
/caps-man provisioning
add action=create-enabled hw-supported-modes=gn master-configuration=\
    wlan_trusted name-format=prefix-identity name-prefix=gn \
    slave-configurations=wlan_untrusted,wlan_guest,wlan_smarthome
add action=create-enabled hw-supported-modes=ac master-configuration=\
    wlan_trusted name-format=prefix-identity name-prefix=ac \
    slave-configurations=wlan_untrusted
/interface bridge port
add bridge=bridge1 comment="trunk port: main switch" interface=ether2 pvid=31
add bridge=bridge1 comment="trunk port" interface=ether3 pvid=31
add bridge=bridge1 comment="VLAN 32: Pi-Hole" interface=ether4 pvid=32
add bridge=bridge1 comment="trunk port: hap03" interface=ether7
add bridge=bridge1 interface=eoip-001 pvid=31
add bridge=bridge1 comment="VLAN 31 untagged" interface=ether6 pvid=31
add bridge=bridge1 comment="VLAN 32 for EoIP to map01" interface=eoip-map01 \
    pvid=32
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,ether3,ether2,ether7,eoip-001 \
    vlan-ids=31
add bridge=bridge1 tagged=bridge1,ether3,ether2,ether7,eoip-001 \
    vlan-ids=32
add bridge=bridge1 tagged=bridge1,ether3,ether2,ether7,eoip-001 \
    vlan-ids=33
add bridge=bridge1 tagged=bridge1,ether3,ether2,ether7,eoip-001 \
    vlan-ids=34
add bridge=bridge1 tagged=bridge1,ether3,ether2,ether7,eoip-001 \
    vlan-ids=35
/interface ovpn-server server
set auth=sha1,md5
/interface wireguard peers
add allowed-address=10.39.0.2/32 comment="iPhone" interface=\
    wireguard1 public-key="XXX"
/ip address
add address=10.31.0.1/24 interface=vlan_31 network=10.31.0.0
add address=10.32.0.1/24 interface=vlan_32 network=10.32.0.0
add address=10.33.0.1/24 interface=vlan_33 network=10.33.0.0
add address=192.168.30.1/24 interface=ether8 network=192.168.30.0
add address=10.34.0.1/24 interface=vlan_34 network=10.34.0.0
add address=10.35.0.1/24 interface=vlan_35 network=10.35.0.0
add address=192.168.255.100/24 interface=ether1 network=192.168.255.0
add address=10.39.0.1/24 interface=wireguard1 network=10.39.0.0
add address=192.168.10.1/24 interface=ether5 network=192.168.10.0
/ip dhcp-server lease
   ...
/ip dhcp-server network
add address=10.31.0.0/24 dns-server=10.32.0.2 domain=trusted.local gateway=\
    10.31.0.1 netmask=24 ntp-server=10.31.0.1
add address=10.32.0.0/24 dns-server=10.32.0.2 domain=untrusted.local gateway=\
    10.32.0.1 ntp-server=10.32.0.1
add address=10.33.0.0/24 dns-server=10.32.0.2 domain=phone.local gateway=\
    10.33.0.1 ntp-server=10.33.0.1
add address=10.34.0.0/24 dns-server=10.32.0.2 domain=guest.local gateway=\
    10.34.0.1 ntp-server=10.34.0.1
add address=10.35.0.0/24 dns-server=10.32.0.2 domain=smarthome.local gateway=\
    10.35.0.1 ntp-server=10.35.0.1
add address=192.168.10.0/24 dns-server=8.8.8.8 domain=test.local gateway=\
    192.168.10.1 ntp-server=192.168.10.1
/ip dns
set servers=10.32.0.2
/ip firewall filter
add action=accept chain=input protocol=icmp
add action=drop chain=input comment="drop invalid input packets" \
    connection-state=invalid
add action=drop chain=input comment="drop input packets with TCP urgent flag" \
    protocol=tcp tcp-flags=urg
add action=accept chain=input comment="allow all input from trusted VLAN 31" \
    in-interface=vlan_31
add action=accept chain=input comment=\
    "allow input from ether 5 (testing interface)" in-interface=ether5
add action=accept chain=input comment="allow all input from WireGuard" \
    in-interface=wireguard1
add action=accept chain=input comment="DHCP, NTP for all but external" \
    dst-port=67,68,123 in-interface=!pppoe-wan protocol=udp
add action=accept chain=input comment=\
    "allow input for established and related" connection-state=\
    established,related
add action=accept chain=input comment="allow UDP WireGuard" dst-port=13231 \
    protocol=udp
add action=drop chain=input comment="final DROP rule on input" log-prefix=\
    DROP_IN
add action=drop chain=forward comment="drop invalid forward packets" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "drop forward packets with TCP urgent flag" protocol=tcp tcp-flags=urg
add action=accept chain=forward comment="forward established and related" \
    connection-state=established,related
add action=accept chain=forward comment=\
    "forward to everywhere from trusted VLAN 31" in-interface=vlan_31
add action=accept chain=forward comment=\
    "allow forwarding on ether 5 to everywhere (for testing)" in-interface=\
    ether5
add action=accept chain=forward comment=\
    "forward to everywhere from WireGuard" in-interface=wireguard1
add action=accept chain=forward comment=\
    "forward TCP DNS requests to Pi-Hole as long as not coming via WAN" \
    dst-address=10.32.0.2 dst-port=53 in-interface=!pppoe-wan protocol=tcp
add action=accept chain=forward comment=\
    "forward UDP DNS requests to Pi-Hole as long as not coming via WAN" \
    dst-address=10.32.0.2 dst-port=53 in-interface=!pppoe-wan protocol=udp
add action=accept chain=forward comment="allow internet from VLAN 33 (phone)" \
    out-interface=pppoe-wan src-address=10.33.0.0/24
add action=accept chain=forward comment="allow internet for VLAN 34 (guests)" \
    out-interface=pppoe-wan src-address=10.34.0.0/24
add action=accept chain=forward comment="allow internet for Home Assistant" \
    out-interface=pppoe-wan src-address=10.35.0.2
add action=accept chain=forward comment="allow SNMP for Home Assistant" \
    dst-port=161,162 protocol=udp src-address=10.35.0.2
add action=accept chain=forward comment="allow syslog to praios" dst-address=\
    10.31.0.10 dst-port=514 in-interface=!pppoe-wan protocol=udp
add action=accept chain=forward comment=\
    "allow homeassistant to modem (for SNMP queries)" out-interface=ether8 \
    src-address=10.35.0.2
add action=accept chain=forward comment=\
    "forward Home Assistant to untrusted LAN" dst-address=10.32.0.0/24 \
    dst-port="" src-address=10.35.0.2
add action=jump chain=forward comment=\
    "apply specific rules for outbound WAN from untrusted VLAN 32" \
    in-interface=vlan_32 jump-target=forward_untrusted out-interface=\
    pppoe-wan
add action=drop chain=forward comment="final DROP rule on forward" \
    log-prefix=DROP_FWD
add action=accept chain=forward_untrusted comment=\
    "allow internet for Pi-Hole" src-address=10.32.0.2
add action=accept chain=forward_untrusted comment=\
    "allow internet for internet radio" src-mac-address=\
    00:00:00:00:00:00
add action=accept chain=forward_untrusted comment=\
    "Amazon Fire TV Stick" src-mac-address=00:00:00:00:00:00
add action=drop chain=forward_untrusted comment=\
    "final DROP rule on forward_untrusted" log-prefix=DROP_32_WAN
/ip firewall mangle
add action=change-mss chain=forward comment="workaround for MTU issues (https:\
    //forum.mikrotik.com/viewtopic.php\?t=127108#p625671)" new-mss=\
    clamp-to-pmtu passthrough=yes protocol=tcp tcp-flags=syn
/ip firewall nat
add action=masquerade chain=srcnat comment="masquerade VLAN 31 (trusted)" \
    out-interface=pppoe-wan src-address=10.31.0.0/24
add action=masquerade chain=srcnat comment="masquerade VLAN 32 (untrusted)" \
    out-interface=pppoe-wan src-address=10.32.0.0/24
add action=masquerade chain=srcnat comment="masquerade VLAN 33 (phone)" \
    out-interface=pppoe-wan src-address=10.33.0.0/24
add action=masquerade chain=srcnat comment="masquerade VLAN 34 (guest)" \
    out-interface=pppoe-wan src-address=10.34.0.0/24
add action=masquerade chain=srcnat comment="masquerade VLAN 35 (smarthome)" \
    out-interface=pppoe-wan src-address=10.35.0.0/24
add action=masquerade chain=srcnat comment="masquerade WireGuard clients" \
    out-interface=pppoe-wan src-address=10.39.0.0/24
add action=masquerade chain=srcnat comment=\
    "masquerade testing interface ether5" out-interface=pppoe-wan \
    src-address=192.168.10.0/24
add action=dst-nat chain=dstnat comment="redirect TCP DNS over Pi-Hole" \
    dst-address=!10.32.0.2 dst-port=53 in-interface=!pppoe-wan protocol=tcp \
    src-address=!10.32.0.2 to-addresses=10.32.0.2
add action=dst-nat chain=dstnat comment="redirect UDP DNS over Pi-Hole" \
    dst-address=!10.32.0.2 dst-port=53 in-interface=!pppoe-wan protocol=udp \
    src-address=!10.32.0.2 to-addresses=10.32.0.2
/ip ssh
set always-allow-password-login=yes
/snmp
set enabled=yes
/system clock
set time-zone-name=Europe/Berlin
/system identity
set name=router
/system ntp client
set enabled=yes
/system ntp server
set broadcast=yes broadcast-addresses=\
    10.31.0.255,10.32.0.255,10.33.0.255,10.34.0.255,10.35.0.255 enabled=yes \
    manycast=yes
/system ntp client servers
add address=192.53.103.104
add address=192.53.103.108

On the RB5009, wired VLAN tests conducted over ether6, wired “direct” tests over ether5.

Sure, there’s some wifi behind it, but as the issue is reproducible on the RB5009 alone, I’m curious what you find out.

Buckeye · April 11, 2022, 10:10am

You have pppoe which is reducing your effective MTU from 1500 to 1492.

Watch the video I suggested in my first post #3.

It’s at the end of the post.

tikker · April 11, 2022, 12:33pm

So from the video I infer:

PMTUD does not work properly anyhow…
MSS clamping is not a workaround but the way to go.

Question remains: why did this arise on my side after updating to ROS 7.2?

tangent · April 11, 2022, 1:09pm

What I got from that video is that PMTUD breakage in the PPPoE case is usually a few steps down the line. I don’t see how that squares with your report that “hping3 -T” fails at the first step.

I do believe you’re right that there might be some RouterOS involvement with this problem.

You’ve assembled enough hard data now to make a formal support request.

Cloudyfa · April 12, 2022, 12:09pm

Hi tikker,
I myself faced the same issue as yours, and so strange that it seemed only me having this problem at first (noone talking about it, and no known bug - since I keep looking for 7.2 reports) so I must roll back to rOS 7.1.5 (not having this issue). I disabled all my firewall rules to check if there were something wrong with them (as others mentioned, easily caused by bad firewall rules).
As you did, I tried to search about the MTU and found that workaround method (firewall mangle rule), yes it worked, but not as good as i think. After I fixed wesite loading with that mangle rule, website loaded ok but I hardly can check the new firmware again (error loading changelog when checking new firmware in System\Packages).

I fixed this issue by adding Max MTU value to PPPoE interface (in my case, I used 1500) and everything worked perfectly (yes, I removed the workaround mangle rule also), what I never had to do with older rOS versions.

Again, this only happen with 7.2 and 7.2.1, not any other versions I known (in my case), maybe something related to what they change in rOS 7.2:

*) pppoe - use default MTU of 1492

Not sure if it gonna work with you or not, but hope that they gonna fix it next version (if this is really a bug )

Best regards.

Buckeye · April 13, 2022, 1:01am

If you are referring to post #10, he never said that it was the first step. He only said it was an . He avoided giving much useful information though.

So my guess is that “internal router ip” was the ISP assigned ip on the pppoe interface.

But as you said it doesn’t explain the difference between 7.2rc2 and 7.2 given the claims in post #13

In my opinion which is evidently different than @tangent’s, when using a pppoe interface, you should be clamping mss to a maximum of 1452 in both directions. (During the 3 way TCP handshake, each side can tell the other the maximum segment it can receive. Your PC client will be happy to receive 1500, and the remote host will be happy to receive 1500, but you want to make sure the other side is told your PC can only receive 1492, and if the other side says it can receive 1500, you want that to be reduced to 1492 as well)

I have not configured an MikroTik router with a pppoe interface, only EdgeRouters. On the EdgeRouters, the setup wizard does this for you and uses a conservative value of 1412 by default, by you can change to 1452. Some users claim that 1412 is more optimal than 1452 when using DSL (due to ATM) but I don’t think there is probably much effective difference between the two.

But I am also not convinced by any evidence that has been provided, that it worked differently with v7.2rc2

@tikker, can you reload v7.2rc2 and verify that it works without the mss clamping, I think that is what your original claim was.

tangent · April 13, 2022, 6:42am

My replies are based on sheer incredulity that we’ve still got broken PMTUD on the Internet in 2022. If you’re right, that sucks. The last time I had to deal with manual MTU/MSS hackery was when we had a Hughes satellite internet setup where the uplink was via analog telephone modem, and only the downlink was via satellite, creating a round-trip time of roughly a second.

That said, your linked video’s point about PMTUD creating an extra round-trip per TCP connection is compelling. Even on links where PMTUD works, clamping the MSS makes connection establishment faster, so you’d want to do that even if PMTUD was working properly.

Buckeye · April 13, 2022, 7:47am

I still need to do some study on how QUIC deals with PMTU, since it is built on top of UDP.
Improve Performance with DPLPMTUD give some hints, but I haven’t really digested it yet.
From the article:

By default, the maximum size of QUIC packets is limited to 1252 bytes (or 1232 bytes on IPv6). This is done to increase the chances of a successful QUIC connection, for research has shown that there are paths on the Internet that will drop packets larger than a certain threshold.

But Chris Greer’s How QUIC Works - The Handshake show wireshark captures with the intial packets being 1392 (still under 1400, but it seems there could be issues running QUIC over a VPN link, if there isn’t some way for it to negotiate down).

Znevna · April 13, 2022, 8:12am

On RB5009, ethernet L2MTU defaults to 1514 for some reason, did you try increasing that?
Or a better thought, I haven’t seen in any of your posts a test of your L2 network (testing if your VLAN can handle 1500) debug that first by pinging the lan ip of your router from inside a vlan.

Buckeye · April 13, 2022, 6:07pm

I wonder why that would have been set? It seems that is almost guaranteed to cause issues with tagged ethernet frames.

I don’t have an RB5009, but that seems like it would have been fixed pretty quick. What versions of RouterOS is l2mtu set to 1514 on the RB5009? Is that still true for v7.2 stable?

jmszuch1 · April 14, 2022, 1:41am

Just wanted to chime regarding that point specifically. It does seem to match with how my RB5009 is set. It’s presently running the 7.3 beta although it was a fresh 7.2 install before that upgrade
Screenshot 2022-04-13 212628.png

Buckeye · April 14, 2022, 3:43am

That’s funny, that the vlan interface is showing with a L2 MTU of 1510. My guess is there is an “off by 4” error in the reporting code.

But this is what an RB760iGS hEX S shows, and it makes more sense to me. Why would the RB5009 have the default L2 set so low? That’s not large enough to carry IEEE 802.1Q tagged ethernet frames for L3 packets with MTU 1500. But I am wondering if it is just being reported incorrectly, because it’s hard to imagine that there would not have been an uproar if something as fundamental as tagged vlans didn’t work on the RB5009.

Can someone verify that tagged vlans do work correctly with the RB5009?

and here with vlan interfaces.

Amm0 · April 14, 2022, 5:43pm

I have a couple RB5009s, so I check this out. Never paid much attention to the L2MTU for a VLAN – figured it be right . But I see the L2MTU as 1510 under 7.1.3, 7.2.1, 7.3rc33, in both winbox & the CLI too – so not just winbox.

/interface/vlan/print proplist=vlan-id,name,mtu,l2mtu
Flags: R - RUNNING
Columns: VLAN-ID, NAME, MTU, L2MTU
#   VLAN-ID  NAME                 MTU  L2MTU
0 R      22  vlan22-lan-general  1500   1510

:put [/system/resource/get version]
7.3beta33 (testing)
:put [/system/resource/get board-name]
RB5009UG+S+

A wAPacR shows a L2MTU for a VLAN as 1596. So doesn’t seem some general ROS issue. But 1510 seem wrong BUT hard to no one run into issues. Maybe PMTUD just hides any problem, or it’s just some UI display bug… I haven’t looked at packet traces yet but certainly curious.

Buckeye · April 15, 2022, 2:34am

I think I found the answer, MikroTik’s L2MTU definition is different than what my working definition was (mine is what MikroTik refers to as Full frame MTU). My definition of L2MTU is more along the lines of this blog post.

But this is from https://help.mikrotik.com/docs/display/ROS/MTU+in+RouterOS

MTU in RouterOS
MikroTik MTU.png

Full frame MTU
Full frame MTU indicates the actual size of the frame that is sent by a particular interface. Frame Checksum is not included as it is removed by an ethernet driver as soon as it reaches its destination.

MAC/Layer-2/L2 MTU
L2MTU indicates the maximum size of the frame without the MAC header that can be sent by this interface.

In RouterOS L2MTU values can be seen in the “/interface” menu. L2MTU support is added for all Routerboard related Ethernet interfaces, VLANs, Bridge, VPLS and wireless interfaces. Some of them support the configuration of the L2MTU value. All other Ethernet interfaces might indicate L2MTU only if the chipset is the same as Routerboard Ethernets.

This will allow users to check if the desired setup is possible. Users will be able to utilize additional bytes for VLAN and MPLS tags, or simply increase of interface MTU to get rid of some unnecessary fragmentation.

So even with the vlan interface’s L2MTU being 1510, there are still 10 unused bytes, so there is room for another stacked vlan tag and an MPLS header with 2 spare bytes. But I think their terminology would be confusing to most people that hadn’t started by reading MikroTik’s documentation.

I dislike their definition, because it displays the vlan “header” in the incorrect position (after the 14 byte ethernet header), instead of after the DstMAC and SrcMAC addresses. And it doesn’t show how stacked frames are handled. But it is what it is. It is probably just showing what contributes to the size of the frame, not necessarily an indication of how it is laid out. I usually consider the ethernet frame header to be variable size. Almost everything now use DIX ethernet frame format. At least that’s what I see; I don’t don’t remember seeing ethernet frames with an IEEE 802.3 length or 802.2 LLC and SNAP headers in recent memory when I use wireshark.

But it seems that some of the MikroTik developers were also confused by the term, otherwise, why chose 1514, which seems like an odd value (not a multiple of 4) greater than the L3 MTU (what MiktoTik refers to as IP MTU in the graphic).