Multiple WAN IP's and NAT

RouterOS General
1

Firstly I am new to MikroTik, but I figure if this is possible then RouterOS is the way to do it! Also, I don’t normally get this involved with the specifics of IP traffic routing and management so please excuse some newbie remarks/skills..

Secondly, apologies for the topic, I’m not sure what to even call this and after much searching and experimentation in a virtual environment I’m a little lost.. Not even sure it can be done with a single router device.. Hopefully this illustrates the basic concept..

Essentially I have multiple WAN IP addresses, and I would like some of these to be passed through to their respective servers which are statically assigned with these WAN IP’s, but have some simple firewall rules, outbound allow all, inbound only allow very select protocols to those IP’s/servers.

But then also have NAT on another WAN IP behind which there are various servers/services which have LAN IP’s and respond on different ports, again with simple firewall rules, outbound allow all, inbound only allow select protocols to the respective server/service, with routing passing respective port traffic to the required LAN IP.

E.G. Incoming HTTPS to A.B.C.70 is allowed and hits the statically configured PSK server with that IP, Incoming HTTPS to A.B.C.66 is allowed and goes through NAT to hit the OME server (10.0.0.3).

Passthrough of WAN IP’s with evaluation of firewall access rules to permit/deny traffic, as well as NAT to a select LAN group.. NAT and not NATting at the same time, with the one router..

It makes sense to have a WAN side router configured without NAT, this connects all the PSK servers and a second NAT enabled router, behind the second NAT router are the VPS, OME, etc servers.

But can it be done with a single router? And if so, pointers in the right direction.. Any input would be greatly appreciated.

BGP and Public IP configuration
2

You can dstnat the publics to the DMZ - since the LAN is in a different subnet, you won’t have to worry about hairpin NAT.

If you want the publics directly on the hosts, the best way to do it is make a WAN bridge, and connect ether1 and ether2 to it, and configure WAN bridge to use IP firewall so you can filter the traffic like you want.

  • personally, I’d go with DSTNAT. You lose fast path/fasttrack if you enable bridge firewalls, and furthermore, if you add IPv6 then your topology is already completely compatible.
3

After much experimentation it seems the answer to my question is NO, you can’t do both at the same time with the one router.

I have been able to configure 1-TO-1 NAT using dst-nat to forward the required ports to the required internal LAN IP’s and src-nat to bind the respective LAN IP’s to their WAN IP, and then add in some firewall rules to allow/deny as required.

With reference to my original illustration - I can change the configuration to bridge the PSK server (ETH1) to ETH0 and with ‘use IP firewall’ selected for the bridge create some firewall rules to allow/deny traffic, but once you do this the router changes to bridge mode and the previous dts-nat/src-nat rules under the firewall start doing some really strange things with traffic to the VPS and OME servers.

If I look at the Packet Flow diagram in the manual I guess it makes sense, the very first evaluation is “is the IN interface bridged”? It can only be Yes or No, not both.

So I think there is only two ways to achieve what I originally illustrated:

  • Use two routers - the first is a WAN bridge handing public IP’s direct to PSK servers and to a second router which does 1-TO-1 NAT as required for VPS and OME servers

  • Get another transit from ISP - one could connect to ETH0 which is bridged to ETH1 and has available the public IP’s which will be on the PSK servers, the other is connected to ETH3 (or other) with remaining public IP’s available and which has dts-nat/src-nat rules with the in-interface as ETH3 directing traffic to LAN IP’s connected to ETH2 (this is untested but I can’t see why it wouldn’t work)

4

I believe it is doable. Bridging all devices with public ips is the right approach. Than make inner bridge as network of the natted devices and use the outer bridge as wan (masquerade to it). You will be routing between the two bridges and you can set the port forwarding. In firewall rules you will be using only reference to the bridges and not to physical ethernets.

5

That actually makes sense Jarda, I think I may have even gotten close to a configuration like that but didn’t change my default masquerade rule to use the bridge, it was still aiming for ETH1 and a lot of things didn’t work.. Such a simple step..

Some other rules were using the bridge/s but some would still have been using interfaces. I was also thinking along the line of bridges ‘side-by-side’..

They way you describe it opens my mind up, ‘bridges within bridges’, an _outer _WAN bridge to connect ISP transit and server/s using public IP’s, and an _inner _ LAN bridge which uses the WAN bridge for internet and allows NATing to the private IP’s.

As you said all the NAT and firewall rules would reference the bridges, I’m not sure if my public IP’s (currently specified on the ETH1 interface) should be on the WAN bridge instead, likewise the LAN IP subnet should be on the LAN bridge under ‘Address List’, DHCP obviously should be to the LAN bridge..

When I get time I will reconfigure the test environment and see if I can get it to work.

6

Might I offer a second solution that should not put as much overhead on the performance of the router?

Instead of bridging to the public IP hosts and enabling IP firewall on the bridge (which makes fastpath impossible), you can achieve the same result with proxy ARP and /32 interfaces on the Mikrotik for each host that should get the public IP directly connected to it, and continue to use NAT for the hosts where you just want to use that.

on public-facing router:
configure WAN interface as a single ethernet interface (e.g. ether1) with standard IP address:
ip address = a.b.c.65/28
Configure the WAN interface for arp=proxy-arp
black-hole route any addresses you intend to use src/dstnat - e.g. A.B.C.71/32 → blackhole

For any host where you intend to configure the public IP address directly on the host, configure a stand-alone (unbridged, non-switched) ethernet interface for each host:
So if you want to connect a host with address A.B.C.72/28 to interface ether3:
/ip address add address=a.b.c.65/32 network=a.b.c.72 interface=ether3
/interface ethernet set ether3 arp=proxy-arp

Then configure the host as usual, IP=a.b.c.72 / Mask=255.255.255.240 / GW = a.b.c.65 / DNS=8.8.8.8 (or whatever server you use)

Be sure that the default behavior in NAT tables is to ignore a.b.c.64/28

Any rules for hosts with NAT should be added with /32 of the host being natted… and be consistent when using srcnat/dstnat or netmap targets.
Don’t netmap in one direction and srcnat in the other - because src/dst nat are stateful targets, where netmap is stateless and must evaluate for each packet, but causes minimum interference by NAT (i.e. it’s closer to having a real IP on the end-host than when using src/dst nat)

7

Thanks for the additional input ZeroByte, I have given this a try and it seems to be working as planned, so far there is only one IP directly assigned and one NATed, it might get interesting when I try adding more.. I am yet to try a bridged solution but will in due course.

A couple of things with the ‘proxy ARP’ approach..

What is the purpose of routing IP’s intended for src/dstnat to blackhole?

I put this in place as suggested but during some troubleshooting to get the configuration to work I disabled the route, consequently once I got things working I tried it disabled and enabled and there seems to be no obvious difference in the behaviours of incoming or outgoing traffic to the direct IP’s or the NATed ones.

The other thing with this configuration is that firewall filter rules when enabled are instantly effective, when disabled however it takes a long time to come into effect - e.g. rules ALLOWing port 80 to direct and NATed IP’s are disabled, the server on that IP keeps responding to requests from the WAN for anywhere up to a minute+ after the rule is disabled, only after that time does it start blocking traffic, when you enable the rules again the response is immediate.

I’m not sure why the delay? It’s not caching, I’m using a simple web server for testing and making small random changes to the index page, when you disable the ALLOW rule, make a change, and reload the page from the WAN, it reloads the most recent change. I didn’t time it exactly, but it’s not just a few seconds catching up, it is at least a minute, perhaps more.

8

Delay can be caused by connection tracking. Once the router holds the connection, it keeps it flow. This is some kind of feature that speeds up the firewall as the router checks only the first packet and decide what to do with it. When accepted, all other packets considered to be replies or related traffic are passed through. See your firewall filter rules and their order.
Surely you can check every packet, if you set it so.

9

The thing with this delay that stood out was I’m reasonably certain it has only shown up since I’ve been testing out the “proxy ARP” approach to a solution.

This is my first taste of RouterOS, and to be honest it’s also the first time I’ve gotten this deep into the bowels of network routing and traffic management, prior to this it was generally dealt with by someone else or a fool-proof GUI configuration.. So I’m on a learning curve!

Initial configuration was simply get Firewall NAT dstnat + srcnat rules right to achieve 1-TO-1 NAT, then associated Filter Rules to allow required WAN traffic for this NAT, after that default allow LAN stuff and finally lock everything else down so it can’t happen..

During this I disabled and enabled a lot of NAT and Filter Rules while getting my head around how RouterOS does things, and I would disable, test, enable, test, etc, never noticing any delays beyond the time RouterOS took to check changes to the rule once applied.

So when I say “reasonably certain” I am confident that prior to this approach it was almost an immediate effect when enabling and disabling Firewall rules.

10

This is so that the Mikrotik will have some destination for the address that is pointed to an interface other than the WAN interface. This is necessary for proxy ARP because if the router thinks that a certain IP lives “outside” the router on the WAN interface, then it is not going to answer ARP requests for that IP address (the other guy is going to answer for himself). When the IP lives somewhere further “inside” the network, then the Mikrotik knows that it must answer ARP for this address on behalf of the inner host.
If your setup works with or without the blackhole route, then either you have already added the additional IP addresses to your WAN interface (which causes the Mikrotik to just answer ARPs directly - not a proxy arp), or else there is already some other route in the routing table which points away from the WAN interface for the particular IP you’re dealing with.

Proxy arp has nothing to do with this behavior. Probably you have made other changes in your firewall design which lead to the delay you’re seeing, and didn’t notice the difference until you started enabling/disabling while testing the proxy arp functionality. All proxy arp does is tell the router to answer ARPs for things that it knows how to reach, and that live “behind” the router.
As for the delayed behavior - Jarda’s explanation is spot-on (as usual) but to elaborate: if your firewall has a rule “allow established,related connections” and this rule is very early in the chain of rules, then this rule matches packets sooner than other rules… This is the best configuration for maximizing the performance of your router, but it does have the drawback that once a connection is accepted, it stays “acceptable” for the lifetime of the connection. Any rules AFTER the “allow established” rule are now basically criteria that only get checked for new connections. This is why the packet counters / traffic counters on a rule that accepts http traffic will have much smaller packet counts / traffic totals than you would otherwise expect.

When you disable the allow-http rule, you’re basically locking the front door, but you’re not kicking out any customers already inside the store, so to speak.

If you were to place a rule “block port 80” sooner in the chain than the rule that accepts established connections, then it would take immediate effect whenever you enable it.

Also note that I’m talking about the filter chain, not the NAT chain - the NAT chain ONLY checks the first packet of new connections when using dstnat/srcnat/masquerade actions. If using netmap action, then this disables state tracking for that connection, so every packet must be compared to the NAT table. (but it can actually be faster than stateful NAT, amazingly) Stateless NAT has another advantage: Suppose the host/server communication follows some behavior that gets lost in the state tracking - for example an ssh connection that’s left open with zero activity for 48 hours (you go home for the weekend without logging out) - a stateful NAT will time this connection out and delete it from the table, so when you come back to your desk later, the connection will be broken, forcing you to log in again. A stateless NAT won’t care - it will just map the addresses by the rule’s configuration regardless of connection state, so even after 48 hours of inactivity, the NAT session will not be broken. (note that the state tracking in the FILTER table might still block this, depending on the configuration)

11

That is all good info thanks, very much appreciated, it has taken me a while to get back to this (I needed to get some live stuff working)!

ZeroByte wrote:
If your setup works with or without the blackhole route … you have already added the additional IP addresses to your WAN interface

Yes, it was still there from previous configuration - I actually ended up wiping the lot and started from scratch to make sure it was a clean config in the test environment. The direct IP’s work as expected, but the only way I can get traffic to the NATed IP’s is when I put the address back on the WAN interface.

However this seems to work OK, both the direct IP and NAT servers have access to the WAN, services on both are responsive from the WAN, and firewall rules allowing traffic to the direct IP or the LAN IP for NATed servers take effect and control traffic..

I played around with my src/dst-nat rules to see if something needed changing but there was no configuration that worked (except adding the NATed WAN IP back to the WAN interface as /32 address), NAT rules are as follows:

/ip firewall nat
add action=dst-nat chain=dstnat comment=“NAT IIS 1” dst-address=192.168.1.66 dst-address-type=local dst-port=80 protocol=tcp to-addresses=10.0.0.2 to-ports=80
add action=src-nat chain=srcnat src-address=10.0.0.2 to-addresses=192.168.1.66

Filter rules are basically accept dst-address=10.0.0.2 port=80 in-interface=ether1 for the NATed traffic, and accept dts-address=192.168.1.70 port=80 in-interface=ether1 for the direct IP traffic, with fairly standard/template accept LAN/ping/related/established and deny everything else rules below that.

The rest of the config is per previous advice - ether1 and ether5 are set to proxy-arp, ether1 connects to WAN with router IP=192.168.1.65, ether5 connects direct IP server 192.168.1.70, ether2 is LAN and connects NATed servers 10.0.0.2, ether5 has address=192.168.1.65 network=192.168.1.70.. But then there is also ether1 address=192.168.1.66 network=192.168.1.66 to get the NATed traffic working.

Is there any apparent/obvious issue doing it this way? Again I must stress I’m in new territory here so I am just basing it on “it all seems to work OK so what could possibly be wrong” which is fine in a test environment but not so confident to make such a thing live without further understanding. It’s like it will be doing ARP for the NATed IP’s specified on the WAN interface, and proxy-ARP for the direct IP’s specified on their respective connected interface.


The delay makes sense, and since I started from a fresh empty config it now seems to be consistently “taking a while to come into effect” during test requests from the same browser/session, the configuration where it didn’t seem to do this had a LOT of compounded additions/deletions/enabling/disabling/etc dating several weeks back to my first exposure to RouterOS so it could have been literally anything anywhere.

12

The reason the direct IP hosts work w/o needing you to add secondary IPs to the WAN interface is that when you create the /32 address facing the direct-IP host, this creates a static /32 route in your IP routing table. Since this route leads to an interface OTHER than the WAN interface, the router will now respond to ARP requests on the WAN interface which are looking for the /32 in question.

When you specify a NAT rule, however, this is not the same thing. All a NAT rule does is instruct the router to modify the contents of the IP header of matching packets.

Imagine a postal worker whose job it was to place a sticker over the return addresses on envelopes. This worker could only cover the addresses on envelopes which reach her desk. Adding a NAT rule does nothing to modify the system’s routing table. Therefore, if the router considers x.x.x.32/29 as existing on the WAN interface, then adding a NAT rule for x.x.x.37 will not change this fact. Without also changing the routing table, the router will not respond to ARPs for x.x.x.37 received on the WAN interface - because the intended recipient is “outside” the router, and must obviously answer the ARP on its own behalf. Since in the case of dstnat, the x.x.x.37 address is NOT attached to the WAN network, the router must respond to these ARP requests on behalf of the intended destination host.

Again - The DSTNAT rule only instructs the router to change the contents of the IP headers. In the example, it will change the destination address of x.x.x.37 into (e.g.) 192.168.100.37 on any packet it sees. It will also remember this translation so that whenever it sees a reply from 192.168.100.37, the router will replace the 192.168.100.37 source address with x.x.x.37 on the packet’s “from” field. The remote host will be none the wiser that anything unusual has taken place. Since the NAT rule doesn’t affect any other IP behavior, we must take another action to cause these packets destined for x.x.x.37 to end up in the router’s hands so that it can work its magic.

How do we do that?

One way is to have the upstream network explicitly route this IP to your router. Your connection is not configured this way, so it’s not an option for you. Instead, your router must respond to any ARP requests for x.x.x.37 so that the packet will be delivered to the router for processing.

There are two ways of doing this:

  1. apply the address directly to the WAN interface. (“actual” arp)
  2. proxy arp - which works when the router knows that the IP destination of x.x.x.37/32 lives somewhere other than on the WAN interface.

Method 1 is obvious - tell the router that it’s actually responsible for this IP address directly, and it will naturally reply to ARP requests for x.x.x.37 - it really IS the router’s IP address.

Method 2 is how your direct-IP hosts are being serviced as I stated above - the router knows that x.x.x.36/32 lives on some other interface. In the case of direct-IP hosts, the router doesn’t have any instructions to mess with the IP headers, so it just forwards the packets intact, and forwards the replies intact.

In order to get the proxy-arp behavior for a dstnat IP, you must also tell the router that x.x.x.37/32 lives somewhere other than on the WAN interface. One simple choice is a black hole - because the dstnat IP doesn’t really live anywhere, right?

/ip route add dst=x.x.x.37/32 type=blackhole

Adding this route will tell your router that .37 lives somewhere other than the WAN interface, so the WAN interface will begin responding to ARP requests for .37 in proxy for the blackhole…

But before the black hole gets to gobble up the packets, the DSTNAT rule will modify x.x.x.37 to be the true internal IP of the host. After making this change, the router will then see that the packet’s new destination IP is not the black hole (or the router itself if using method 1), but some other host. Thus the IP forwarding engine in the router will now process the packet with its new destination address, and will forward the packet wherever is most appropriate.