Help to get Hairpin NAT working please?

a752412341 · February 14, 2015, 6:15pm

Hi all,

My LAN is on 192.168.27.0/24 on Ether1 There is a server at 192.168.27.4 that I want to access from LAN, and outside on a random port 21392.

I have 2 ADSL lines which are bonded togeter using PPoE on Ether2 and Ether3

I have put int the src-nat entry as per the wiki but can’t get it to work. Any pointers greatly appreciated?

Thanks!

Here are the NAT Rules

[admin@MikroTik] /ip firewall nat> print
Flags: X - disabled, I - invalid, D - dynamic
 0    ;;; CAM incoming
      chain=dstnat action=dst-nat to-addresses=192.168.27.4 to-ports=21392
      protocol=tcp in-interface=all-ppp dst-port=21392 log=no log-prefix=""

 1    ;;; HS3 incoming
      chain=dstnat action=dst-nat to-addresses=192.168.27.8 to-ports=44443
      protocol=tcp in-interface=all-ppp dst-port=44443 log=yes log-prefix=""

 2    ;;; HS3 HSTOUCH incoming
      chain=dstnat action=dst-nat to-addresses=192.168.27.8 to-ports=44444
      protocol=tcp in-interface=all-ppp dst-port=44444 log=yes log-prefix=""

 3    ;;; Hairpin NAT - CAM
      chain=srcnat action=masquerade src-address=192.168.27.0/24
      dst-address=192.168.27.4 out-interface=ether1-LAN log=no log-prefix=""

 4    ;;; Outbound Internet Access
      chain=srcnat action=masquerade out-interface=all-ppp log=no
      log-prefix=""
[admin@MikroTik] /ip firewall nat>

Here are Firewall filter rules

[admin@MikroTik] /ip firewall filter> print
Flags: X - disabled, I - invalid, D - dynamic
 0    ;;; LAN traffic can go anywhere
      chain=forward action=accept in-interface=ether1-LAN log=no
      log-prefix=""

 1    ;;; Established traffic
      chain=forward action=accept connection-state=established log=no
      log-prefix=""

 2    ;;; Related traffic
      chain=forward action=accept connection-state=related log=no
      log-prefix=""

 3    ;;; ICMP
      chain=forward action=accept protocol=icmp log=no log-prefix=""

 4    ;;; cam allow incoming traffic
      chain=forward action=accept protocol=tcp dst-address=192.168.27.4
      dst-port=21392 log=yes log-prefix=""

 5    ;;; HS3 allow incoming traffic
      chain=forward action=accept protocol=tcp dst-address=192.168.27.8
      dst-port=44443 log=yes log-prefix=""

 6    ;;; HS3 HSTOUCH allow incoming traffic
      chain=forward action=accept protocol=tcp dst-address=192.168.27.8
      dst-port=44444 log=yes log-prefix=""

 7    ;;; Drop the rest
      chain=forward action=drop log=no log-prefix=""

 8    chain=output action=accept log=no log-prefix=""

 9    ;;; LAN traffic can go anywhere
      chain=input action=accept in-interface=ether1-LAN log=no log-prefix=""

10    ;;; Established traffic
      chain=input action=accept connection-state=established log=no
      log-prefix=""

11    ;;; Related traffic
      chain=input action=accept connection-state=related log=no log-prefix=""

12    ;;; ICMP
      chain=input action=accept protocol=icmp limit=5,5 log=no log-prefix=""

13    ;;; Drop the rest
      chain=input action=drop log=no log-prefix=""

[admin@MikroTik] /ip firewall filter>

a752412341 · February 17, 2015, 12:22am

Anyone got any pointers? Cheers

DLNoah · February 17, 2015, 8:42pm

The src-nat rule isn’t matching anything because the original dst-nat rules aren’t matching packets that came from your LAN (only matching in from the ppp interfaces).

The normal setup for using Hairpin NAT would be similar to the following:

Public IP of 1.2.3.4, clients requesting on port 21392
Private IP of 192.168.27.4, server listening on port 21392

/ip firewall nat
add chain=dstnat action=dst-nat to-addresses=192.168.27.4 protocol=tcp dst-address=1.2.3.4 dst-port=21392 log=no
add chain=srcnat action=masquerade src-address=192.168.27.0/24 dst-address=192.168.27.4 log=no

For your dst-nat rule, you don’t want to specify the in-interface – you want to dst-nat all traffic that requests the NAT’d public IP, whether it came from inside your network or from outside. You don’t need to specify the to-ports, because you’re not changing the original port (if no to-port is specified, the port number will remain unchanged).

On the hairpin rule, you don’t really need to specify the in-interface (the src-address is sufficient to create a unique match for traffic from the LAN to the after-NAT internal IP of the server)

a752412341 · February 17, 2015, 10:24pm

Hi DLNoah, thank you for the reply.

I have made the changes you suggest, and now I sometimes get a “connected” from the browser but it then hangs, and doesn’t return the web page. What am I doing wrong?

[admin@MikroTik] /ip firewall nat> print
Flags: X - disabled, I - invalid, D - dynamic
 0    ;;; CAM incoming
      chain=dstnat action=dst-nat to-addresses=192.168.27.4 protocol=tcp
      dst-address=[PUBLIC IP] dst-port=21392 log=no log-prefix=""

 1    ;;; CAM hairpin NAT
      chain=srcnat action=masquerade src-address=192.168.27.0/24
      dst-address=192.168.27.4 log=no log-prefix=""

 2    ;;; HS3 incoming
      chain=dstnat action=dst-nat to-addresses=192.168.27.8 protocol=tcp
      dst-address=[PUBLIC IP] dst-port=44443 log=no log-prefix=""

 3    ;;; HS3 hairpin NAT
      chain=srcnat action=masquerade src-address=192.168.27.0/24
      dst-address=192.168.27.8 log=no log-prefix=""

 4    ;;; HS3 HSTOUCH incoming
      chain=dstnat action=dst-nat to-addresses=192.168.27.8 protocol=tcp
      dst-address=[PUBLIC IP] dst-port=44444 log=yes log-prefix=""

 5    ;;; Outbound Internet Access
      chain=srcnat action=masquerade out-interface=all-ppp log=no
      log-prefix=""

[admin@MikroTik] /ip firewall nat>

here are the filter rules

[admin@MikroTik] /ip firewall filter> print
Flags: X - disabled, I - invalid, D - dynamic
 0    ;;; LAN traffic can go anywhere
      chain=forward action=accept in-interface=ether1-LAN log=no log-prefix=""

 1    ;;; Established traffic
      chain=forward action=accept connection-state=established log=no log-prefix=""

 2    ;;; Related traffic
      chain=forward action=accept connection-state=related log=no log-prefix=""

 3    ;;; ICMP
      chain=forward action=accept protocol=icmp log=no log-prefix=""

 4    ;;; cam allow incoming traffic
      chain=forward action=accept protocol=tcp dst-address=192.168.27.4 dst-port=21392 log=yes log-prefix=""

 5    ;;; HS3 allow incoming traffic
      chain=forward action=accept protocol=tcp dst-address=192.168.27.8 dst-port=44443 log=yes log-prefix=""

 6    ;;; HS3 HSTOUCH allow incoming traffic
      chain=forward action=accept protocol=tcp dst-address=192.168.27.8 dst-port=44444 log=yes log-prefix=""

 7    ;;; Drop the rest
      chain=forward action=drop log=no log-prefix=""

 8    chain=output action=accept log=no log-prefix=""

 9    ;;; LAN traffic can go anywhere
      chain=input action=accept in-interface=ether1-LAN log=no log-prefix=""

10    ;;; Established traffic
      chain=input action=accept connection-state=established log=no log-prefix=""

11    ;;; Related traffic
      chain=input action=accept connection-state=related log=no log-prefix=""

12    ;;; ICMP
      chain=input action=accept protocol=icmp limit=5,5 log=no log-prefix=""

13    ;;; Drop the rest
      chain=input action=drop log=no log-prefix=""
[admin@MikroTik] /ip firewall filter>

DLNoah · February 17, 2015, 11:46pm

Uhm, the most obvious thing I can think of would be that there’s more than one stream as part of the connection. But if that were the case, I would expect external users to have problems to, and I’m reading that you’re not having that problem?

What does Torch show when you’re trying to make the connection from an internal user?

a752412341 · February 18, 2015, 12:02am

Hi, it works fine externally.

Here is the torch screenshot after putting http://domain name.com:21392 into a browser which resolves to my external ip. I’m not sure what this is telling me.
ScreenClip.png

DLNoah · February 18, 2015, 1:33am

It looks like the NAT connection isn’t getting created correctly in the firewall. Can you try rebooting the firewall or going to IP > Firewall > Connections and deleting all entries? If you had connections open in conntrack from before you changed the rules, they don’t get automatically flushed on rule change.

a752412341 · February 18, 2015, 10:06am

I tried rebooting, but still the same issue.

For completeness here is the routes screenshot.
ScreenClip.png

strods · February 18, 2015, 10:15am

Seems to me that situation must be fixed if you change this rule:
1 ;;; CAM hairpin NAT
chain=srcnat action=masquerade src-address=192.168.27.0/24
dst-address=192.168.27.4 log=no log-prefix=“”

Try to run:
“/ip firewall nat set 1 out-interface=ether1-LAN”

a752412341 · February 18, 2015, 10:24am

Hi Strods, thank you for the comment.

I just tried that, but no change.

There are some packets getting through somehow, because firefox says connected, and eventually says transferring data…

BartoszP · February 18, 2015, 11:24am

Try this - it works for my local Lotus server which could be accessed from LAN via external public IP.

add action=masquerade chain=srcnat comment="HarpIn Lotus" dst-address=192.168.1.0/24 src-address=192.168.1.0/24
add action=dst-nat chain=dstnat comment="Lotus SATDAT step 1" dst-address-list=ETH_WAN_GROUP dst-port=1352 protocol=tcp to-addresses=LOTUS_LOCAL_IP
add action=src-nat chain=srcnat comment="Lotus SATDAT step 2" dst-address=LOTUS_LOCAL_IP dst-port=1352 protocol=tcp to-addresses=THIS_ROUTER_IP

When the packet from your computer goes to the server via it’s external address then router changes dst address to local server’s ip and sends the packet to server’s local ip.
Then server, which sees the packet coming from local address responds to this packet with own local ip and sends directly to your computer as it knows that src-address is from LAN so why it should be send back to router -it is shortening the path and it is IMHO the problem you are facing.
Your computer sees the packet coming from server with local ip but the computer does not know nothing about this incoming connection from server as it expects to match packet coming from external ip. Packet is dropped.

The crucial part of these three lines above is line #1. It is making masquarade for packets coming from LAN to LAN when the packet is and substitutes src address of this packet (your computer) with router LAN IP. With this masquarade your local server responds to router as it sees the masquaraded original packet with router src-address substituted, router unmasquarades this packet and sends it back to your computer. There is no “short path” which fools your computer.

I could be wrong but it works for me

DLNoah · February 18, 2015, 1:39pm

I see from your route table you’re using routing marks. Can you post your mangle rules that are applying the routing marks? I think that may be causing the problem; if the initial connection is hitting that routing mark, it isn’t going to run the directly connected route to 192.168.27.0/24 in the main routing table, and that would then cause the “two connections” that are showing in torch.

a752412341 · February 18, 2015, 2:26pm

Hi DLNoah, here you go.

thanks, Al

[admin@MikroTik] /ip firewall mangle> print
Flags: X - disabled, I - invalid, D - dynamic
 0    chain=prerouting action=mark-packet new-packet-mark=one passthrough=yes
      dst-address=0.0.0.0/0 in-interface=ether1-LAN nth=2,1 log=no
      log-prefix=""

 1    chain=prerouting action=mark-packet new-packet-mark=two passthrough=yes
      dst-address=0.0.0.0/0 in-interface=ether1-LAN nth=2,2 log=no
      log-prefix=""

 2    chain=prerouting action=mark-routing new-routing-mark=one passthrough=no
      packet-mark=one log=no log-prefix=""

 3    chain=prerouting action=mark-routing new-routing-mark=two passthrough=no
      packet-mark=two log=no log-prefix=""
[admin@MikroTik] /ip firewall mangle>

DLNoah · February 18, 2015, 2:37pm

Ok, those are definitely effecting this traffic.

Can you try replacing dst-address=0.0.0.0/0 with dst-address=![PUBLIC IP]

Again, after you change the firewall rule, you will need to delete all connections (or restart the router) before the changes will take full effect.

a752412341 · February 18, 2015, 4:02pm

I tried that but didn’t make any difference.

[admin@MikroTik] /ip firewall mangle> print
Flags: X - disabled, I - invalid, D - dynamic
 0    chain=prerouting action=mark-packet new-packet-mark=one passthrough=yes
      dst-address=![public ip] in-interface=ether1-LAN nth=2,1 log=no
      log-prefix=""

 1    chain=prerouting action=mark-packet new-packet-mark=two passthrough=yes
      dst-address=![public ip] in-interface=ether1-LAN nth=2,2 log=no
      log-prefix=""

 2    chain=prerouting action=mark-routing new-routing-mark=one passthrough=no
      packet-mark=one log=no log-prefix=""

 3    chain=prerouting action=mark-routing new-routing-mark=two passthrough=no
      packet-mark=two log=no log-prefix=""
[admin@MikroTik] /ip firewall mangle>

DLNoah · February 18, 2015, 5:09pm

Can you post another Torch, and a filtered “IP > Firewall > Connections” list, please?

a752412341 · February 18, 2015, 6:07pm

Here’s another screenshot of Torch and Connections while opening the url in the browser. There’s also an external connection happening which I was testing.

ScreenClip3.png
ScreenClip2.png

DLNoah · February 18, 2015, 6:22pm

Hm, I’m not sure what else to look at. The rules you have in place work for me on RouterOS v6.19, for similar functionality I have up and going.

When you double-click one of the IP > Firewall > Connections from your internal connection attempt, what does it show for “Reply Src Address” and “Reply Dst Address”?

a752412341 · February 18, 2015, 8:16pm

I’m on Firmware version 3.17 - it is a Routerboard 951G-2HnD

The odd thing is that it is almost working - I get half a web page, very slowly.

The packet marking rules are to split the upload between the 2 links. My ISP splits the download packets between the 2 links on their end.

It seems like perhaps half the packets are getting through one way or another?

Here is the screenshot of one of the connections.
ScreenClip4.png

DLNoah · February 18, 2015, 8:41pm

I assume by “firmware version” you mean the output of System > RouterBOARD. What version shows in the title bar when you log in: “user@ip (identity) - Winbox vX.XX …”?

The connection you’re showing is what I would expect to see with the NAT rules in place.

My expectation when you posted the mangle rules was that half the time (at random) it would work, since half of your packets were getting marked with routing mark “two”, which had a special routing rule active, and the other half were getting marked with routing mark “one”, which didn’t have any active special routing rules.

I can think of two possible troubleshooting steps from here:

Use Tools > Packet Sniffer in order to record the traffic while you try to make a connection from an internal computer. After you record the data with Packet Sniffer, you can download the resulting file to your computer and open it with Wireshark/tcpdump or a similar program. This will let you verify the actual responses coming back from the server – maybe you’re getting an ICMP reject because of a firewall or other setting on the server?
Make one more set of changes to the mangle rules, to make it so IP > Firewall > Connections should potentially have useful information to show.
Change these:

 0    chain=prerouting action=mark-packet new-packet-mark=one passthrough=yes
      dst-address=![public ip] in-interface=ether1-LAN nth=2,1 log=no
      log-prefix=""

 1    chain=prerouting action=mark-packet new-packet-mark=two passthrough=yes
      dst-address=![public ip] in-interface=ether1-LAN nth=2,2 log=no
      log-prefix=""

Change the action from “mark-packet” to “mark-connection”, remove “new-packet-mark” and insert “new-connection-mark=one” (or two) – no other changes, just switch from a packet mark to a connection mark. In general, for PCC load balancing and bonding, it’s recommended to mark connections instead of packets, so that you can apply special handling to both directions of the traffic. In your case, the connection-mark isn’t strictly necessary, because you only care about the round-robin handling on the outbound leg – responses don’t have to travel out the exact same interface they came in.

Along with that change, you would also need to change:

 2    chain=prerouting action=mark-routing new-routing-mark=one passthrough=no
      packet-mark=one log=no log-prefix=""

 3    chain=prerouting action=mark-routing new-routing-mark=two passthrough=no
      packet-mark=two log=no log-prefix=""

Remove the packet-mark match, and add a connection-mark match with the same match (so match connection-mark=one or two, depending on which rule).

The advantage we get in this particular case is that IP > Firewall > Connections will show us which connection mark is applied (if any) to a connection, but cannot show us packet or routing marks.

Personally, I would probably try the Packet Sniffer step first, because it should be a lot shorter and less disruptive to do.