Hairpin NAT fail when PCC is active

mercenaruss · August 31, 2023, 1:14pm

Hi there,
I really in need of help with my config,i hope somebody will direct me to fix this issue.
I have next config:
2 ISP, 1 static IP (ether1) , 1 dinamic IP from Starlink Wifi Router( wlan 1)
I setup PCC Load balancing and HairPin NAT.
Currently port forwarding is working only when Starlink is down(interface disabled), I know i did some mistake in Mangle Rules but cant identify and get desperate.
Here is my config,where:
x.x.x.x is my static IP from Fiber Provider.
x.x.y.y is my gateway from Fiber Provide

/interface bridge
add name=Bridge

/interface list
add name=WAN
add name=LAN

/routing table
add disabled=no fib name=to-isp1
add disabled=no fib name=to-isp2

/interface list member
add interface=ether1 list=WAN
add interface=Bridge list=LAN
add interface=wlan1 list=WAN

/ip address
add address=x.x.x.x/29 comment=ISP-1 interface=ether1 network=\
    x.x.y.y
add address=192.168.200.1/23 interface=Bridge network=192.168.200.0
add address=192.168.1.10/24 comment=ISP-2 interface=wlan1 network=192.168.1.0

/ip firewall address-list
add address=192.168.200.0/23 list=LOCAL-IP

/ip firewall filter
add action=accept chain=input comment="accept establish & related" \
    connection-state=established,related
add action=drop chain=input comment="drop invalid" connection-state=invalid
add action=accept chain=input comment="accept ICMP" protocol=icmp
add action=drop chain=input comment="drop all not from lan" in-interface=\
    !Bridge
add action=accept chain=forward comment="accept established,related" \
    connection-state=established,related
add action=drop chain=forward comment="drop invalid" connection-state=invalid
add action=drop chain=forward comment="drop all from WAN to LAN" \
    connection-nat-state=!dstnat connection-state=new in-interface=ether1
add action=drop chain=forward connection-nat-state=!dstnat connection-state=\
    new in-interface=wlan1

/ip firewall mangle
add action=accept chain=prerouting comment="LB PCC" dst-address-list=LOCAL-IP \
    src-address-list=LOCAL-IP
add action=accept chain=postrouting comment="LB PCC" dst-address-list=\
    LOCAL-IP src-address-list=LOCAL-IP
add action=accept chain=forward comment="LB PCC" dst-address-list=LOCAL-IP \
    src-address-list=LOCAL-IP
add action=accept chain=input comment="LB PCC" dst-address-list=LOCAL-IP \
    src-address-list=LOCAL-IP
add action=accept chain=output comment="LB PCC" dst-address-list=LOCAL-IP \
    src-address-list=LOCAL-IP
add action=mark-connection chain=input comment="LB PCC" in-interface=ether1 \
    new-connection-mark=cm-ether1 passthrough=yes
add action=mark-connection chain=input comment="LB PCC" in-interface=wlan1 \
    new-connection-mark=cm-wlan1 passthrough=yes
add action=mark-routing chain=output comment="LB PCC" connection-mark=\
    cm-ether1 new-routing-mark=to-isp1 passthrough=yes
add action=mark-routing chain=output comment="LB PCC" connection-mark=\
    cm-wlan1 new-routing-mark=to-isp2 passthrough=yes
add action=mark-connection chain=prerouting comment="LB PCC" \
    dst-address-type=!local in-interface=Bridge new-connection-mark=cm-ether1 \
    passthrough=yes per-connection-classifier=src-address-and-port:2/0
add action=mark-connection chain=prerouting comment="LB PCC" \
    dst-address-type=!local in-interface=Bridge new-connection-mark=cm-wlan1 \
    passthrough=yes per-connection-classifier=src-address-and-port:2/1
add action=mark-routing chain=prerouting comment="LB PCC" connection-mark=\
    cm-ether1 in-interface=Bridge new-routing-mark=to-isp1 passthrough=yes
add action=mark-routing chain=prerouting comment="LB PCC" connection-mark=\
    cm-wlan1 in-interface=Bridge new-routing-mark=to-isp2 passthrough=yes
	
/ip firewall nat
add action=src-nat chain=srcnat comment="Masquarade NET Access - Coolink" \
    out-interface=ether1 src-address=192.168.200.0/23 to-addresses=\
    x.x.x.x
add action=src-nat chain=srcnat comment="Masquarade NET Access - Starlink" \
    out-interface=wlan1 src-address=192.168.200.0/23 to-addresses=\
    192.168.1.10
add action=dst-nat chain=dstnat comment="HairPin NAT - NextCloud" \
    dst-address=x.x.x.x dst-port=443 protocol=tcp to-addresses=\
    192.168.200.40 to-ports=443
add action=masquerade chain=srcnat dst-address=192.168.200.40 out-interface=\
    Bridge protocol=tcp src-address=192.168.200.0/23
add action=dst-nat chain=dstnat comment="HairPin NAT - WireGuard Forwarding" \
    dst-address=x.x.x.x dst-port=51820 protocol=udp to-addresses=\
    192.168.200.40 to-ports=51820
	
/ip route
add check-gateway=ping comment="LB PCC" distance=1 gateway=x.x.y.y \
    routing-table=to-isp1
add check-gateway=ping comment="LB PCC" distance=1 gateway=192.168.1.1 \
    routing-table=to-isp2
add check-gateway=ping comment="LB PCC" distance=1 gateway=x.x.y.y
add check-gateway=ping comment="LB PCC" distance=2 gateway=192.168.1.1

Really hope somebody will direct me on this issue.

anav · August 31, 2023, 3:09pm

Before I even start to look at a config…

Okay to get this straight,
ISP1 is WIRED and a fibre connection
ISP2 is wifi from a starlink router a CGNAT useless WANIP for most things other than just plain internet access.

WHY ON earth would you want to include starlinkin the PCC equation and not just use if for backup?? Is it that fast/throughput and that reliable???

+++++++++++++++++++++++++++++++++++++++++

Just to confirm, you have servers on your subnet that you force local LAN users to access by WANIP address instead of local LANIP ???

If so which WANIP?

Do you have external users coming in via WAN1 for servers?
Do you have external users coming in via WAN2 for servers?
Do you have any VPNs setup and if so how?

mercenaruss · September 4, 2023, 8:44am

Hi,
ISP1 is wired FiberLink with static white IP
ISP2 is Starlink,i get ip from their router over wireless in format 192.168.1.10 255.255.25.0 192.168.1.1.

I included Stalink in PCC equation just because our fiber link is just 50/50 mbps(cost 800USD/month) and is not enough.
Starlink is pretty reliable in our location.

Yes,i have users in the office what are travelling to projects and come back to office,so this why i decide to use HairPin Nat.
All connections to server are passing only thru WAN1(ISP1) Fiber, Starlink is used only for outgoing connections from the office(browsing,calls,meetings,etc)

anav · September 4, 2023, 3:18pm

(1) Lets adjust forward chain rules.
From:
add action=drop chain=forward comment=“drop all from WAN to LAN”
connection-nat-state=!dstnat connection-state=new in-interface=ether1
add action=drop chain=forward connection-nat-state=!dstnat connection-state=
new in-interface=wlan1
TO:
add action=accept chain=forward comment=“internet traffic” in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment=“port forwarding” connection-nat-state=dstnat
add action=drop chain=forward comment=“drop all else”

(2) Sourcnat rule required… ( only one hairpin nat rule is required ), fixed the others for you!!

/ip firewall nat
add action=src-nat chain=srcnat comment=“NET Access - Coolink”
out-interface=ether1 to-addresses=X.X.X.X
add action=src-nat chain=srcnat comment=“NET Access - Starlink”
out-interface=wlan1 to-addresses=192.168.1.10
add action=masquerade chain=srcnat comment=“hairpin nat” src-address=192.168.200.0/23 dst=address=192.168.200.0/23

(3) Port forwarding rules ( only one rule is required ( if to ports the same not required ), the second rule is covered above, and wireguard is not for port forwarding )

add action=dst-nat chain=dstnat comment=“NextCloud”
dst-address=x.x.x.x dst-port=443 protocol=tcp to-addresses=
192.168.200.40

NOTE: you should avoid port forwarding if possible. Suggest your users wireguard into the router and then access servers.
However if this is not possible then have your users provide you with
a. their fixed static WANIP
OR
b. their dyndns name, which the MT router will resolve in a firewall address list. There are plenty of free offering for such names and thus there is no reason not to have this additional limitation on who can access the server. Adding this also makes the port invisible on scans vice visible but closed.

add add action=dst-nat chain=dstnat comment=“NextCloud”
dst-address=x.x.x.x dst-port=443 protocol=tcp to-addresses=
192.168.200.40 src-address-list=Authorized

(3) Mangles…

Step 1: Ensuring users coming in on WAN1 go out WAN1, in this case we dont need to ensure traffic coming in WAN2 goes out WAN2 because you will never forward through WAN2.
The second WAN is strictly for OUTGOING TRAFFIC PCC. This rule ensures two things. Any external incoming traffic to a router service coming on WAN1 will go back out WAN1, and second any external users coming in to servers via WAN1 will go back out via WAN1.

/ip firewall mangle
add action=mark-connection chain=prerouting in-interface=ether1
connection-mark=no-mark new-connection-mark=incomingWAN1 passthrough=yes
add action=mark-routing chain=output connection-mark=incomingWAN1
new-routing-mark=to_ISP1 passthrough=no

Step2: Ensuring bridge users use PCC for outbound traffic only.
add action=mark-connection chain=prerouting comment=“LB PCC” connection-mark=no-mark
dst-address-type=!local in-interface=Bridge new-connection-mark=cm-ether1
passthrough=yes per-connection-classifier=src-address-and-port:2/0
add action=mark-connection chain=prerouting comment=“LB PCC” connection-mark=no-mark
dst-address-type=!local in-interface=Bridge new-connection-mark=cm-wlan1
passthrough=yes per-connection-classifier=src-address-and-port:2/1
add action=mark-routing chain=prerouting comment=“LB PCC” connection-mark=
cm-ether1 in-interface=Bridge new-routing-mark=to-isp1 passthrough=yes
add action=mark-routing chain=prerouting comment=“LB PCC” connection-mark=
cm-wlan1 in-interface=Bridge new-routing-mark=to-isp2 passthrough=yes

Thats it, no other rules required!
Any traffic coming from bridge will be split among the two WANs.
It will only PCC traffic with no marks on it coming from the bridge, so it will not touch external originated traffic being returned by the servers.

Now you can put back in the fastrack rule… with a slight change.
add action=fasttrack-connection chain=forward comment=“defconf: fasttrack” connection-state=established,related connection-mark=no-mark

(4) Routes
Assuming you have two viable WANs for PCC, that if one WAN goes down you want the other WAN to handle all the traffic. Right now,every other session will go no where.

Step1: First we need the two routes in the main table as per normal. Nothing special for distance for main table and special routes, because we are using PCC.
_/ip route
add dst-address=0.0.0.0/0 comment=“LB PCC” gateway=x.x.y.y routing-table=main
add dst-address=0.0.0.0/0 comment=“LB PCC” gateway=192.168.1.1 routing-table=main
add dst-address=0.0.0.0/0 comment=“LB PCC” gateway=x.x.y.y routing-table=use-ISP1
add dst-address=0.0.0.0/0 comment=“LB PCC” gateway=192.168.1.1 routing-table=use-ISP2
_

Step2: We need to provide the ability for the router to use either WAN for special routing and here we use distance to distinguish primary and secondary for each special table. By the way I never uses distance 1 when distance is required, as I always leave room to put a route distance behind, and after…

/ip route
distance=2 dst-address=0.0.0.0/0 gateway=x.x.y.y routing-table=to-ISP1
distance=4 dst-address=0.0.0.0/0 gateway=192.168.1.1 routing-table=to-ISP1

distance=2 dst-address=0.0.0.0/0 gateway=192.168.1.1 routing-table=to-ISP2
distance=4 dst-address=0.0.0.0/0 gateway=x.x.y.y routing-table=to-ISP2

mercenaruss · September 5, 2023, 7:37pm

Hello,
this was amazing detailed instruction, no words can describe how thankful I am.
I followed your instruction and i succed to solve the main issue.
But still have problem with firewall itself, I get locked out(good i was in safe mode) when i applied your recommendation from:
(1) Lets adjust forward chain rules:

From:
add action=drop chain=forward comment="drop all from WAN to LAN" \
connection-nat-state=!dstnat connection-state=new in-interface=ether1
add action=drop chain=forward connection-nat-state=!dstnat connection-state=\
new in-interface=wlan1
TO:
add action=accept chain=forward comment="internet traffic" in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment="port forwarding" connection-nat-state=dstnat
add action=drop chain=forward comment="drop all else"

(3) : Will try to force them to use VPN and move current VPN from internal server to Mikrotik, this why i needed port forwarding for it.
(4): Fasttrack rule should be first in the list? Here i get small confused.
(5): IP Routes also get me confused, because i have only 2 routing marks: to-isp1 & to-isp2.
You added there another routing mark, what was use-ISP1 & use-ISP1

Current config looks like this and is working perfectly with both ISP and port forwarding:

## Interfaces and Interface List

/interface bridge
add name=Bridge

/interface list
add name=WAN
add name=LAN

/routing table
add disabled=no fib name=to-isp1
add disabled=no fib name=to-isp2

/interface list member
add interface=ether1 list=WAN
add interface=Bridge list=LAN
add interface=wlan1 list=WAN

## Firewall Settings

/ip firewall address-list
add address=192.168.200.0/23 list=LOCAL-IP

/ip firewall filter
add action=accept chain=input comment="accept establish & related" connection-state=established,related
add action=drop chain=input comment="drop invalid" connection-state=invalid
add action=accept chain=input comment="accept ICMP" protocol=icmp
add action=drop chain=input comment="drop all not from lan" in-interface=!Bridge
add action=accept chain=forward comment="accept established,related" connection-state=established,related
add action=drop chain=forward comment="drop invalid" connection-state=invalid
add action=drop chain=forward comment="drop all from WAN to LAN" connection-nat-state=!dstnat connection-state=new in-interface=ether1 new in-interface=wlan1

/ip firewall mangle
add action=fasttrack-connection chain=forward comment=Fasttrack connection-mark=no-mark connection-state=established,related
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether1 new-connection-mark=incomingWAN1 passthrough=yes
add action=mark-connection chain=prerouting comment="LB PCC" connection-mark=no-mark dst-address-type=!local in-interface=Bridge new-connection-mark=cm-ether1 passthrough=yes per-connection-classifier=src-address-and-port:2/0
add action=mark-connection chain=prerouting comment="LB PCC" connection-mark=no-mark dst-address-type=!local in-interface=Bridge new-connection-mark=cm-wlan1 passthrough=yes per-connection-classifier=src-address-and-port:2/1
add action=mark-routing chain=prerouting comment="LB PCC" connection-mark=cm-ether1 in-interface=Bridge new-routing-mark=to-isp1 passthrough=yes
add action=mark-routing chain=prerouting comment="LB PCC" connection-mark=cm-wlan1 in-interface=Bridge new-routing-mark=to-isp2 passthrough=yes
	
/ip firewall nat
add action=src-nat chain=srcnat comment="NET Access - Coolink" out-interface=ether1 to-addresses=x.x.x.x
add action=src-nat chain=srcnat comment="NET Access - Starlink" out-interface=wlan1 to-addresses=192.168.1.10
add action=masquerade chain=srcnat comment="Hairpin NAT" dst-address=192.168.200.0/23 src-address=192.168.200.0/23
add action=dst-nat chain=dstnat comment=NextCloud dst-address=x.x.x.x dst-port=443 protocol=tcp to-addresses=192.168.200.40
add action=dst-nat chain=dstnat comment=Wireguard dst-address=x.x.x.x dst-port=51820 protocol=udp to-addresses=192.168.200.40
	
## IP Route

/ip route
add comment="Cooling" dst-address=0.0.0.0/0 gateway=x.x.y.y routing-table=main
add comment="Starlink" dst-address=0.0.0.0/0 gateway=192.168.1.1 routing-table=main
add distance=2 dst-address=0.0.0.0/0 gateway=x.x.y.y routing-table=to-isp1
add distance=4 dst-address=0.0.0.0/0 gateway=192.168.1.1 routing-table=to-isp1
add distance=2 dst-address=0.0.0.0/0 gateway=192.168.1.1 routing-table=to-isp2
add distance=4 dst-address=0.0.0.0/0 gateway=x.x.y.y routing-table=to-isp2