Brain fade: routing to internally hosted servers doesn't work

Complete brain fade here. What’s the term/phrase used when routing to internal servers doesn’t work? I’ve got an instance of SimpleHelp hosted on my own server. It’s accessed via help.maltsystems.co.uk. By default, this resolves to the public IP address. Works flawlessly externally but any computers on the LAN can’t route correctly because the router gets confused over routing out and the back. This “thing” has a name. Just can’t recall what it’s called!

I’ve implemented the workaround on the Windows DNS servers on my domain by adding a new zone for maltsystems.co.uk and manually adding the internal IP addresses so that help.maltsystems.co.uk resolves to the internal IP address.

Hairpin nat.

It’s not the router that’s confused in these instances, but the client btw.

Hairpin NAT - that’s it! I recall there is an in-depth article on this subject. What perplexed me is that before I installed the Mikrotik router, I used the Virgin Media Superhub and I didn’t have the same issue with that. I need to do some more reading.

A lot of consumer routers include a blanket masquerade rule of the form

/ip firewall nat add chain=srcnat action=masquerade

This is of course bad: e.g. even external dst-nat connections appear to come from the router instead of the proper external IP. It makes things a bit easier to configure, and they don’t care.

A more proper rule looks like this:

/ip firewall nat 
add chain=srcnat action=masquerade in-interface=srv-bridge dst-address=se.rv.er.ip protocol=tcp dst-port=80,443

Dst-addr/proto/port should match your dst-nat to-address/to-port. In ye olden times the in-interface matcher was unavailable in the srcnat chain, on these versions src-address=su.bn.et.0/24 should be used instead.

I usually setup my local network to use the router ip for DNS. Then I can manage things like this using static entries or forwarders so the internal machines use the internal address instead of the public.

These days the only time I use hairpin is for special cases where someone internally might be using a VPN and needs to go to another DNS, but still needs internal access to an internal server, usually email or web.

Those use cases have to be considered individually to determine if it’s a security concern or not to do that.

You bring up some interesting points that merit consideration.

Redirecting DNS is an okay solution. The first problem with it is that it’s increasingly difficult to do: due to constant ill-intentioned(1) meddling with DNS, many systems and programs default to not using the resolver specified by DHCP or even use DNS-over-HTTPS by default. In many scenarios this can be mitigated, but in many it cannot. The other problem is that the TTL of the public DNS entry must be sufficiently short to not interfere with this scheme via caching.

As to security concerns. Basically I only suggest hairpin for scenarios where the entire world was already given access to a given resource. This resource either contains information that is sufficiently public as to enable this, or has some sort of adequate authentication as to enable making the gateway open. The only associated problem is that the requester’s IP remains cloaked - but only in the sense that we don1t know which host on the subnet made the request, which would also be unavailable for anyone behind NAT, so everyone else as well.

Anyway, I am of the opinion that hosted services should generally reside in a non-user (non-client, whatever) subnet, where this becomes moot.

(1) By ill-intentioned meddling I mean things like returning the addresses of ad-laden landing pages instead of NXDOMAIN (which breaks a lot of things), or hijacking the DNS of ad networks, again to insert the provider’s ads into other people’s pages. Both were done primarily in the US by cable-turned-ISP monopolies. The default usage of external DNS was mostly a reaction to such breakage.

I like that option as well when available.

I’ll have a look at that option as well. Shouldn’t be too difficult to implement. The additional zone on the DNS servers works fine most of the time but I’m currently building a PC image for a client and it works 90% of the time but sometimes doesn’t get the internal IP but the external. That’s probably a problem with my DNS set-up, it’s mainly a lab environment.

I will read up on hairpin as I do generally like to understand that’s going on.

Bull pucky Lurker, , your sourcenat rule is OVERCOOKED.
It should be kept a simple as possible, its NOT meant to take over responsibilities normally handled by other tools such as IP route or Firewall Rules, there are exceptions of course.

In addition, clearly servers being accessed from external are best handled in a separate vlan from local users, for serveral reasons.
Removes hairpin nat issues as this only occurs if user are in the same subnet
Is generally a more secure approach.

You’re too kind. :slight_smile:

I know of many more convoluted ways, but none that are simpler. Maybe you can give an example?

Keep in mind that a host in a given subnet may attempt to send a packet to another client in the same subnet via the gateway. In this case the gateway router should do two things:

  • forward that packet with unchanged addressing
  • issue an ICMP redirect, notifying the sender of a more direct path

As to the best practice being to put servers in another subnet: I fully agree (as I stated).

The subject is hairpin nat, not how uneccessarily convoluted one can make the sourcenat settings.

The normal sourcenat ( masquerade or not ) is NOT the issue.
To address hairpin shenanigans there are several viable methods.
One is a simple hairpin sourcenat rule
add chain=srcnat action=masquerade src-address=serversubnet dst-address=serversubnet

DONE!

The DNS solution as indicated is probably the better choice, if one cannot move the server to its own subnet…

Concur, port forwarding should be avoided if possible, use wireguard instead to reach a device internally.

Just saying “done” doesn’t make it so. Let me illustrate the problem. With your rule installed:

Assume the following: 192.168.80.1 is the router, 192.168.80.3 and 4 are both hosts on this /24 subnet. Watch what happens is 192.168.80.3 pings 192.168.80.4 via 192.168.80.1:

Packets on .3:

#  NUM  TIME   DIRECTION  SRC-MAC            DST-MAC            SRC-ADDRESS   DST-ADDRESS   IP-PROTOCOL
0    1  2.491  tx         4A:8F:5A:84:1F:98  DE:2C:6E:48:95:17  192.168.80.3  192.168.80.4  icmp       
1    2  2.492  rx         DE:2C:6E:48:95:17  4A:8F:5A:84:1F:98  192.168.80.4  192.168.80.3  icmp       
2    3  3.495  tx         4A:8F:5A:84:1F:98  DE:2C:6E:48:95:17  192.168.80.3  192.168.80.4  icmp       
3    4  3.496  rx         DE:2C:6E:48:95:17  4A:8F:5A:84:1F:98  192.168.80.4  192.168.80.3  icmp       
4    5  4.49   tx         4A:8F:5A:84:1F:98  DE:2C:6E:48:95:17  192.168.80.3  192.168.80.4  icmp       
5    6  4.49   rx         DE:2C:6E:48:95:17  4A:8F:5A:84:1F:98  192.168.80.4  192.168.80.3  icmp 

Packets on .4:

#  NUM  TIME   DIRECTION  SRC-MAC            DST-MAC            SRC-ADDRESS   DST-ADDRESS   IP-PROTOCOL
0    1  7.43   rx         DE:2C:6E:48:95:17  4A:8F:5A:84:20:64  192.168.80.1  192.168.80.4  icmp       
1    2  7.43   tx         4A:8F:5A:84:20:64  DE:2C:6E:48:95:17  192.168.80.4  192.168.80.1  icmp       
2    3  8.434  rx         DE:2C:6E:48:95:17  4A:8F:5A:84:20:64  192.168.80.1  192.168.80.4  icmp       
3    4  8.434  tx         4A:8F:5A:84:20:64  DE:2C:6E:48:95:17  192.168.80.4  192.168.80.1  icmp       
4    5  9.428  rx         DE:2C:6E:48:95:17  4A:8F:5A:84:20:64  192.168.80.1  192.168.80.4  icmp       
5    6  9.428  tx         4A:8F:5A:84:20:64  DE:2C:6E:48:95:17  192.168.80.4  192.168.80.1  icmp

As you can see, the exact opposite of what I said should happen is observed:

  • addresses are changed
  • an ICMP redirect is not emitted

With my overly convoluted rule, this behavior does not occur.

You can play make up new requirements or use a corner case that is outlandish.......doesnt change the facts.

I actually half agree with you. If it works, why bother with edge cases. My point is simply that one should be aware of them. I have two specific reasons.

One is that I’ve dealt with a few industrial networks, where the routers masqueraded basically all routed traffic. This allows working setups with incorrect configurations. I admit that I have a bit of PTSD from this. I think that a router should “just route” if it’s a possibility, and only go around changing addresses when it’s a requirement.

The other case in which this is bad is a configuration not commonly discussed on the forum, but quite common and popular in some circles. This is PVLAN (which is a misnomer, and actually doesn’t have much, if anything, to do with vlans…) where all traffic, even within the same subnet, is routed. In these cases this sort if hairpin rule actually breaks things completely.

Again, for the standard home setups it’s fine. I just wanted to highlight the cases where things an go wrong.

Got it, yes overused or reliance on masquerade, means one has to be very precise with firewall rules....