Config broke after upgrade

Jonas47372 · November 1, 2022, 12:15pm

I upgraded from 7.2.3 to 7.6 on hap ac^2, and as result my wireguard tunnel config stopped working. Rollback to 7.2.3 solved the problem without any config change. I tried also some versions between those, all resulting in the same problem.
I don’t know if this is a bug or a problem with my configuration of the tunnel or the routing with routing-marks.
Is there anything I can do, that a upgrade is possible without losing my tunnel-conection?

I observed after the upgrade, that the wireguard tunnel is running, but the Rx packet count stays at 0.

Shot description of my setup:
ether1 is connected to the internet, almost all traffic to the internet should go directly through ether1.
Only connections that came though the tunnel or from TCP 192.168.99.2 port 25 (at ether4 / Bridge DMZ) are marked and routed trough the tunnel.

output of the /export command with some unrelated sections removed. On version 7.6 the output is the same (except version number):

# nov/01/2022 12:26:16 by RouterOS 7.2.3
# software id = Y4CQ-DF7G
#
# model = RBD52G-5HacD2HnD

/interface bridge
add name=DMZ
add name=Gast-Wlan
add name=Wlan
add name=solar protocol-mode=none


/interface ethernet
set [ find default-name=ether1 ] comment=WAN
set [ find default-name=ether2 ] comment=InternNet
set [ find default-name=ether3 ] comment=Solar
set [ find default-name=ether4 ] comment=DMZ
set [ find default-name=ether5 ] comment=DMZ


/interface wireguard peers
add allowed-address=0.0.0.0/0 endpoint-address=***.***.***.*** endpoint-port=\
    51820 interface=Server-Tunnel persistent-keepalive=10s public-key=\
    "*******************************"
    
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-g/n country=germany frequency=auto \
    mode=ap-bridge security-profile=******* ssid=****** wps-mode=\
    disabled
set [ find default-name=wlan2 ] country=germany mode=ap-bridge \
    security-profile=******* ssid=****** wps-mode=disabled
add keepalive-frames=disabled mac-address=BA:69:F4:4D:3B:42 master-interface=\
    wlan1 mode=station multicast-buffering=disabled name=LTE-Bridge \
    security-profile=****** ssid=****** wds-cost-range=0 \
    wds-default-cost=0 wps-mode=disabled
add keepalive-frames=disabled mac-address=BA:69:F4:4D:3B:43 master-interface=\
    wlan1 multicast-buffering=disabled name=GastWiFi ssid=******* \
    wds-cost-range=0 wds-default-cost=0 wps-mode=disabled
add keepalive-frames=disabled mac-address=BA:69:F4:4D:3B:44 master-interface=\
    wlan1 multicast-buffering=disabled name=solar_wlan ssid=solar \
    wds-cost-range=0 wds-default-cost=0 wps-mode=disabled

/interface bridge port
add bridge=Wlan ingress-filtering=no interface=wlan1 multicast-router=\
    disabled
add bridge=Wlan ingress-filtering=no interface=wlan2 multicast-router=\
    disabled
add bridge=DMZ ingress-filtering=no interface=ether4 multicast-router=\
    disabled
add bridge=DMZ ingress-filtering=no interface=ether5 multicast-router=\
    disabled
add bridge=Gast-Wlan ingress-filtering=no interface=GastWiFi
add bridge=solar interface=ether3
add bridge=solar interface=solar_wlan
add bridge=solar disabled=yes interface=*16

/ip address
add address=192.168.1.1/24 interface=ether2 network=192.168.1.0
add address=192.168.2.1/24 interface=Wlan network=192.168.2.0
add address=192.168.3.1/24 interface=Gast-Wlan network=192.168.3.0
add address=192.168.99.1/24 interface=DMZ network=192.168.99.0
add address=192.168.255.2/24 interface=ether1 network=192.168.255.0
add address=192.168.5.1/24 interface=solar network=192.168.5.0
add address=10.10.10.2/24 interface=Server-Tunnel network=10.10.10.0

/ip firewall filter
add action=drop chain=forward disabled=yes dst-port=25 protocol=tcp
add action=drop chain=input comment="Drop invalid Input" connection-state=\
    invalid
add action=accept chain=input comment="Accept Established" connection-state=\
    established,related,untracked
add action=drop chain=input comment="Drop inbound DNS" dst-port=53 \
    in-interface-list=WAN protocol=udp
add action=accept chain=input comment="Accept Input from LANs" \
    in-interface-list=LAN
add action=accept chain=input comment="Accept Winbox" dst-port=8291 \
    in-interface-list=LAN protocol=tcp
add action=drop chain=input comment="Drop all other Input"
add action=drop chain=forward comment="Drop Invalid Forward" \
    connection-state=invalid
add action=fasttrack-connection chain=forward comment="Accept Established" \
    connection-state=established,related,untracked disabled=yes hw-offload=\
    yes
add action=accept chain=forward comment="Accept Established" \
    connection-state=established,related,untracked
add action=accept chain=forward comment="Accept LANs->WAN" in-interface-list=\
    LAN out-interface-list=WAN
add action=accept chain=forward comment="Allow DNS for LAN" dst-address=\
    192.168.99.4 dst-port=53 in-interface-list=LAN protocol=udp
add action=accept chain=forward comment="Accept from InternNet" \
    in-interface-list=InternNet
add action=accept chain=forward comment=Mail dst-address=192.168.99.2 \
    dst-port=25,143,587 protocol=tcp
add action=accept chain=forward comment="Webserver RPi4" dst-address=\
    192.168.99.4 dst-port=80,443 protocol=tcp
add action=accept chain=forward comment="Webserver RPi2" dst-address=\
    192.168.99.2 dst-port=80,443 protocol=tcp
add action=accept chain=forward comment="Accept DST-NAT" \
    connection-nat-state=dstnat
add action=accept chain=forward comment="allow Solar" disabled=yes \
    in-interface=solar out-interface-list=WAN
add action=drop chain=forward comment="Drop all other forward"

/ip firewall mangle
add action=mark-connection chain=prerouting comment=\
    "\DCber Server kommende Verbindung" in-interface=Server-Tunnel \
    new-connection-mark=server-tunnel-connection passthrough=yes
add action=mark-connection chain=prerouting comment="Mailserver Outbound" \
    connection-state=new dst-port=25 new-connection-mark=\
    server-tunnel-connection passthrough=yes protocol=tcp src-address=\
    192.168.99.2
add action=change-mss chain=forward comment="MSS Clamping for DSL" new-mss=\
    1452 out-interface=ether1 passthrough=yes protocol=tcp tcp-flags=syn \
    tcp-mss=1453-65535
add action=change-mss chain=forward comment="MSS Clamping for DSL" \
    in-interface=ether1 new-mss=1452 passthrough=yes protocol=tcp tcp-flags=\
    syn tcp-mss=1453-65535
add action=change-mss chain=forward comment="MSS Clamping tunnel" \
    connection-mark=server-tunnel-connection new-mss=1380 passthrough=yes \
    protocol=tcp tcp-flags=syn tcp-mss=1381-65535
add action=mark-routing chain=prerouting connection-mark=\
    server-tunnel-connection dst-address=!192.168.0.0/16 new-routing-mark=\
    server passthrough=no
add action=mark-routing chain=prerouting new-routing-mark=main passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
# LTE-Bridge not ready
add action=masquerade chain=srcnat out-interface=LTE-Bridge
add action=dst-nat chain=dstnat comment=Mail dst-port=25,143,587 \
    in-interface=ether1 protocol=tcp to-addresses=192.168.99.2
add action=dst-nat chain=dstnat comment=Webserver dst-port=80,443,8448 \
    in-interface=ether1 protocol=tcp to-addresses=192.168.99.4
add action=dst-nat chain=dstnat comment="SSH RPi4" dst-port=22 in-interface=\
    ether1 protocol=tcp to-addresses=192.168.99.4
add action=dst-nat chain=dstnat comment="SSH RPi4" dst-port=2222 \
    in-interface=ether1 protocol=tcp to-addresses=192.168.99.2 to-ports=22

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.10.10.1 pref-src=\
    "" routing-table=server scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.255.1 \
    pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10

anav · November 1, 2022, 12:28pm

Going from betas to a a released version in the life of 7, has caused issues.
Best to netinstall 7.6 and manually put in the config and it should work.

(1) one thing I would change is
FROM:
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.10.10.1 pref-src=
“” routing-table=server scope=30 suppress-hw-offload=no target-scope=10
TO:
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway**=Server-Tunnel** pref-src=
“” routing-table=server scope=30 suppress-hw-offload=no target-scope=10

(2) Where is your table? I see no config rule creating the table “server”

(3) Since you didnt make it clear, the device you are showing is a client device. What is the other end? A mikrotik router, 3rd party VPN??

(4) Why are you attempting to mangle traffic from 192.168.22.X that is going through the tunnel.

(5) I see no routing rule that would actually direct traffic go out the tunnel?

Jonas47372 · November 1, 2022, 1:40pm

thanks for your reply.
I discovered that no IPv4 connection from or to the router works on Version 7.6. I can’t ping the router and the router can’t ping any other device, not even itself. But forwarding IPv4 works. This could be a reason for the not working tunnel.

So I try the netinstall and reply later, when i configured the router again.

The answers to the questions:

changed it, no effect.
I assumed the command in 1) implicitly creates the routing table ‘server’.
the other end is ad Debian server. It forwards the traffic to the internet with NAT
My ISP assignes to me only a residential IP. Many Mail servers reject traffic from those IPs, so I tunnel the outgoing mail traffic trough my external server, which has a accepted IP.
I mark the Connections in /ip firewall mangle with

add action=mark-connection chain=prerouting comment=\
    "\DCber Server kommende Verbindung" in-interface=Server-Tunnel \
    new-connection-mark=server-tunnel-connection passthrough=yes
    add action=mark-connection chain=prerouting comment="Mailserver Outbound" \
    connection-state=new dst-port=25 new-connection-mark=\
    server-tunnel-connection passthrough=yes protocol=tcp src-address=\
    192.168.99.2

and then add a routing-mark to the packets of the maked connection

    add action=mark-routing chain=prerouting connection-mark=\
    server-tunnel-connection dst-address=!192.168.0.0/16 new-routing-mark=\
    server passthrough=no

Jonas47372 · November 1, 2022, 3:40pm

I reinstalled 7.6 via netinstall. While adding the configuration step by step I found the entry that broke it:
in /ip/firewall/mangle I assigned in the last rule the routing-mark ‘main’ to all packets that reached that rule.

chain=prerouting action=mark-routing new-routing-mark=main passthrough=no log=no log-prefix=""

disabling this rule solved the problem, as ‘main’ should be the default.