My goal is to have ether8 (and VLAN 2 tagged packets from my WAP) not communicate with the rest of the main network or be able to use my main IP, but rather independently be routed through the VPN via WireGuard.
I have this all set up, and it works quite well, for the most part.
But I discovered quickly that many sites load just fine, while others don't, and that some apps just won't work at all.
So I looked around and found that people recommend using
Code: Select all
/ip firewall mangle add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes protocol=tcp tcp-flags=synI poked around some more and found that some people recommended lowering the Wireguard interface MTU from 1370 to something even lower, like 1330. I tried this, and when I did, I found that even more stuff was broken than before. I then tried upping the MTU to 1500 on the WireGuard interface, and lo and behold, everything that was broken started working.
Now, I assume this is a problem because, at least I'm guessing, the packets I am sending out, after overhead, are growing larger than 1500 bytes and at risk for fragmentation and possible loss on the Internet between my network and the VPN server.
My question is this: why isn't it working with a lower MTU? What's going on here, and what can I do about it?
I don't know much about how MTU and fragmentation is handled inside the router. Here's what I think is going on (probably wrong), but I mention it only to let you know where I am right now in my thought process:
Because my computer is hooked up to, as I test, the WAP and not attached directly to the SurfShark interface, it probably is using a 1500 MTU. That means the packets are ingressing at 1500 bytes, but when they hit the smaller MTU wireguard interface one of three things must be happening:
a) the packets are being dropped because they are too large
b) the packets are getting passed as is and screwing up something later
c) the packets are being fragmented and not getting assembled properly downstream
So, maybe the thing to do is to change the MTU of all of the computers that attach to the computer via DHCP so that it matches the MTU of the smallest MTU used by any one interface in order to prevent packet framentation.
But, then, another part of me thinks, why can't the router rewrite these packets in a way that results in a TCP stream that isn't fragmented to begin with? But of course, even if you did this, it wouldn't help any with oversized UDP packets.
So, what gives here? What's the accepted normal practice for dealing with this?
Update: After doing some reading, here is what my current thought process is:
Clamp-to-pmtu is supposed to do path mtu discover according to https://help.mikrotik.com/docs/display/ROS/Mangle
However, my guess is that mangling all packets to do this is a hacky workaround that is really an abuse of the intended purpose of this function. I suspect the discovery process is slow and possibly prone to failure if ICMP packets are not returned properly, and if my understanding is correct, this only works for TCP connections and not UDP connections and is intended as a failsafe if a piece of equipment outside of your control "out there on the internet" has a smaller MTU requirement than what you are using.
I'm guessing this hack can lead to very slow performance if the MTU used by a client is larger than what the router knows is the maximum MTU supported by its internet connection. Say, for example, the DHCP server tells the client to use MTU 1500, but then forwards packets across wireguard at MTU 1450, the clamping function will have to kick in and do MTU discovery which could take a while. Additionally, UDP packets that are too large will just get dropped.
So, AFAIK, the correct way to deal with this is to discover the smallest mtu that you know your own configuration can handle and set the MTU in your client server to match that value --- that way, the packets will never need to be fragmented on your own network, regardless of which path they take, and ideally will not need to be fragmented once they leave your network and are out of your control.
That means if you route some packets over wireguard and others over the ethernet, you should not set your client to use 1500 MTU, even though the internet at large can handle it --- but instead, should set the client MTU to use whatever the wireguard link between you and the other end can handle.
If I'm right, then the correct course of action is to realize that the MTU for wireguard, if you want to use IPV6, is 1420, and the correct thing to do to fix the fact that some sites won't load isn't to clamp the MSS, but to set the MTU of the CLIENTS to 1420. Ideally you would do this via HDCP, except that most clients won't respect that option, so you have to configure them manually and... I guess rely on the MSS clamp and setting the wireguard MTU to 1420?
But this is scary, because if the MSS clamp worked as expected, why did lowering the MTU of the Wiregard interface screw things up so badly? Is it because the largest possible UDP packet size is too small and it's UDP that's getting lost and not TCP? And will 1420 eventually cause me problems? Maybe it's better just to set the wireguard MTU to 1500 since my packets appear to be making it to the ISP ok that way?
Thanks.
Here's my config for reference:
Code: Select all
# 2023-08-31 10:59:14 by RouterOS 7.11
# software id = Z8XQ-IISM
# model = RB5009UPr+S+
/interface bridge
add admin-mac=... auto-mac=no name=mainBridge vlan-filtering=yes
/interface wireguard
add listen-port=51820 name=SurfShark_wireguard private-key=...
/interface vlan
add interface=mainBridge name=rentalVLAN vlan-id=2
/disk
set usb1 type=hardware
set usb2 type=hardware
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec policy group
add name=group-ike2
/ip ipsec profile
add dh-group=modp1024 enc-algorithm=aes-256 hash-algorithm=sha256 name="phase 1"
/ip ipsec peer
add exchange-mode=ike2 name=peer-ike2 passive=yes profile="phase 1"
/ip ipsec proposal
add enc-algorithms=aes-128-cbc name="phase 2" pfs-group=none
/ip pool
add name=dhcp ranges=192.168.1.100-192.168.1.254
add name=pool-ike2 ranges=10.0.10.3-10.0.10.254
add name=rentalHDCPIPPool ranges=192.168.102.100-192.168.102.254
/ip dhcp-server
add address-pool=dhcp interface=mainBridge lease-time=10m name=mainDHCP
add address-pool=rentalHDCPIPPool interface=rentalVLAN name=rentalHDCP
/ip ipsec mode-config
add address-pool=pool-ike2 address-prefix-length=32 name=conf-ike2 split-dns="" split-include=192.168.1.2/32
/routing table
add fib name=rentalRouting
/user-manager user
add attributes=Framed-IP-Address:10.0.10.2 name=...
/interface bridge port
add bridge=mainBridge interface=ether2
add bridge=mainBridge interface=ether3
add bridge=mainBridge interface=ether4
add bridge=mainBridge interface=ether5
add bridge=mainBridge interface=ether6
add bridge=mainBridge interface=ether7
add bridge=mainBridge interface=ether8
add bridge=mainBridge interface=sfp-sfpplus1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface bridge vlan
#ether6 is the WAP, needs tagged VLAN 2 for correct SSID, ether2/3 are test ports not configured correctly ATM.
add bridge=mainBridge tagged=mainBridge,ether6 untagged=ether3,ether2 vlan-ids=2
add bridge=mainBridge untagged=ether4,ether5,ether7,ether8,ether6 vlan-ids=1
/interface list member
add interface=mainBridge list=LAN
add interface=ether1 list=WAN
add interface=rentalVLAN list=LAN
/interface wireguard peers
add allowed-address=0.0.0.0/0 endpoint-address=45.39.230.77 endpoint-port=51820 interface=SurfShark_wireguard persistent-keepalive=25s public-key=...
/ip address
add address=192.168.1.1/24 interface=mainBridge network=192.168.1.0
add address=192.168.102.1/24 interface=rentalVLAN network=192.168.102.0
add address=10.14.0.2/31 interface=SurfShark_wireguard network=10.14.0.2
/ip dhcp-client
add interface=ether1
/ip dhcp-server network
add address=192.168.1.0/24 dns-server=192.168.1.1 gateway=192.168.1.1 netmask=24
add address=192.168.102.0/24 dns-server=192.168.102.1 gateway=192.168.102.1 netmask=24
/ip dns
set allow-remote-requests=yes cache-max-ttl=1h use-doh-server=https://dns.dnswarden.com/uncensored
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input dst-port=80 protocol=tcp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input dst-port=4500,500 protocol=udp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes protocol=tcp tcp-flags=syn
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat out-interface=SurfShark_wireguard
/ip ipsec identity
add auth-method=eap-radius certificate=letsencrypt-autogen_2023-08-04T23:24:58Z generate-policy=port-override mode-config=conf-ike2 peer=peer-ike2 policy-template-group=group-ike2
/ip ipsec policy
add dst-address=10.0.10.0/24 group=group-ike2 proposal="phase 2" src-address=0.0.0.0/0 template=yes
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=SurfShark_wireguard pref-src="" routing-table=rentalRouting scope=30 suppress-hw-offload=no target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www-ssl certificate=letsencrypt-autogen_2023-08-04T23:24:58Z
set api disabled=yes
set api-ssl disabled=yes
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
/radius
add address=127.0.0.1 service=ipsec
/routing rule
add action=lookup-only-in-table disabled=no src-address=192.168.102.0/24 table=rentalRouting
/system clock
set time-zone-name=Pacific/Honolulu
/system leds settings
set all-leds-off=after-1min
/system logging
add topics=ipsec,!debug
add topics=dhcp
add topics=wireguard
/system note
set show-at-login=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/user-manager
set certificate=letsencrypt-autogen_2023-08-04T23:24:58Z enabled=yes
/user-manager router
add address=127.0.0.1 name=router
