2 IP Addresses 1 interface - DHCP on the first, firewall on the second

IwanGreyling · Tue Aug 30, 2022 5:44 pm

Hey guys.

I'm not sure where to start with this one.
What I should search, how I should handle this.

I have 5 static IP Addresses from my ISP, we currently only use 2, one for general internet traffic, it gets firewalled and natted in our RB2011.
The other is going to our server, and used to just be plugged directly into the ISPs router with no firewall. (Yes. this is a terrible idea, but the boss was more concerned with convenience and overruled my please, untill we got a nice little ransomware attack a few days ago)
I want it to go through our router, and use it as a basic firewall, and just let some ports through this interface, our ERP Server and some API Interfaces we use for development.

So basically, 1 WAN interface, 2 IP addresses, one going through a masquerade, the other just some firewall rules.

Is this possible?
Can someone point me in the right direction?

anav · Tue Aug 30, 2022 6:47 pm

Post your current config
/export hide sensitive file=anynameyouwish and ensure no public WANIP info is shown.

sindy · Tue Aug 30, 2022 7:47 pm

Is this possible?
Can someone point me in the right direction?

Yes and yes, but first please confirm that the five public addresses are in the same /29 subnet, as that affects the correct approach.

gotsprings · Tue Aug 30, 2022 7:52 pm

So he wants his server to get one of the Public IPs via DHCP?

IwanGreyling · Wed Aug 31, 2022 9:18 am

Post your current config
/export hide-sensitive file=anynameyouwish and ensure no public WANIP info is shown.

I attatched the file
Thanks for taking the time!

Is this possible?
Can someone point me in the right direction?
Yes and yes, but first please confirm that the five public addresses are in the same /29 subnet, as that affects the correct approach.

Yep they are all on the /29 subnet

So he wants his server to get one of the Public IPs via DHCP?

Id rather it not get it's IP by DHCP. But if it comes to that I can...

Also right now, my setup looks as such:

Router.png

sindy · Wed Aug 31, 2022 9:48 am

OK. So

the public address of the server must respond to ARP requests on the WAN so that the ISP router knew where to send IP packets for it,
the server itself must either have that address as a /32 one on itself and it must be connected to a dedicated interface (physical Ethernet or VLAN) on the 2011, or it must have a private address and a dst-nat rule on the 2011 must be used to forward requests coming to the public one to the private one. Depending on the application running at the server, it may be necessary to put the public address also up on the server and use a dst-nat rule on the server itself to revert the NAT. What OS is running on the server?

IwanGreyling · Wed Aug 31, 2022 11:10 am

OK. So
the public address of the server must respond to ARP requests on the WAN so that the ISP router knew where to send IP packets for it,

the server itself must either have that address as a /32 one on itself and it must be connected to a dedicated interface (physical Ethernet or VLAN) on the 2011, or it must have a private address and a dst-nat rule on the 2011 must be used to forward requests coming to the public one to the private one. Depending on the application running at the server, it may be necessary to put the public address also up on the server and use a dst-nat rule on the server itself to revert the NAT. What OS is running on the server?

Alright. That sounds close to how I have it right now.

I got the server plugged directly into ETH 4, it has a static IP assigned to it in Ubuntu Network Manager; x.x.x.85/32 (The public IP Address)
(It's running Ubuntu 18.04 LTS)
Is there any config I need to do to get the server to respond to ARP Requests?

Would it be better to have a private IP set up for the server, have it DHCP wit all the other traffic, and have a separate NAT for the x.x.x.85 address?

anav · Wed Aug 31, 2022 5:23 pm

I am just following this thread but I am curious as to the firewall rules for this what seems to be a basic ONE to ONE NAT???

sindy · Wed Aug 31, 2022 6:31 pm

Good. First, maybe you want to edit the configuration file to hide the public prefix?

Before starting, set up the forward chain of the firewall filter so that it wouldn't forward anything coming from WAN (ether1) except packets belonging to established, related, or untracked connections, and except icmp - that's the very goal of the exercise, isn't it. Later on, you'll permit forwarding (without dst-nat) of the chosen ports to the.pub.lic.85.

/ip firewall filter
add chain=forward connection-state=established,related,untracked action=accept
add chain=forward protocol=icmp action=accept
add chain=forward in-interface=ether1 action=drop

Next, to have the public IP up directly on eth0 of the server while not wasting another three public IPs for broadcast, network address, and gateway, you have to use the point-to-point Ethernet setup, where the local address is a /32 one and the remote address is, well, another /32 one. I don't know how to cofigure that using the NetworkManager, but using the plain ip command, you set it the following way:
ip address add the.pub.lic.85/32 peer 10.20.30.40 dev eth0
ip route add default via 10.20.30.40

The corresponding setting at the Mikrotik end of the cable is
/ip address add interface=ether4 address=10.20.30.40/32 network=the.pub.lic.85
(of course you first remove ether4 from the bridge connecting it with ether3).

This way, Mikrotik will add a route to the.pub.lic.85 via ether4:
[me@MyTik] > ip route print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
...
17 ADC the.pub.lic.85/32 10.20.30.40 ether4 0
...

A smaller subnet (longer prefix) always wins so if RouterOS receives a packet for the.pub.lic.85 from anywhere, it will send it via ether4, although the.pub.lic.85 also fits into the the.pub.lic.80/29 subnet attached to ether1.

The dst-nat to a private address and back would be a just workaround if you wouldn't be able to find how to set this point-to-point configuration up in the startup configuration of the Ubuntu.

Next, you have to tell the Mikrotik to respond to ARP requests arriving to the main WAN and asking about the.pub.lic.85 with its own MAC address; to do that, you use
/ip arp add interface=ether1 address=the.pub.lic.85 published=yes

That's it.

IwanGreyling · Thu Sep 01, 2022 9:24 am

Good. First, maybe you want to edit the configuration file to hide the public prefix?

Ooof... I should have double checked. I guess I never saved the file after removing it all?

/ip firewall filter
add chain=forward connection-state=established,related,untracked action=accept
add chain=forward protocol=icmp action=accept
add chain=forward in-interface=ether1 action=drop

I like to understand what I'm doing so I'm going to just ask some questions:

Firewall is executed in order, so where in my firewall should these rules be?
The last one I guess is supposed to be just before my "Drop everything else" rule?
What is the function of that last rule?

Next, to have the public IP up directly on eth0 of the server while not wasting another three public IPs for broadcast, network address, and gateway, you have to use the point-to-point Ethernet setup, where the local address is a /32 one and the remote address is, well, another /32 one. I don't know how to cofigure that using the NetworkManager, but using the plain ip command, you set it the following way:
ip address add the.pub.lic.85/32 peer 10.20.30.40 dev eth0
ip route add default via 10.20.30.40

Our gateway is x.x.x.81 so do I substitute the 10.20.30.40 with that?
If that's the case all of this is done.

Just wanna get a little clarity before proceeding.
Will probably do all this the afternoon when most people went home.

Thank you very much for taking the time to assist me!

sindy · Thu Sep 01, 2022 9:47 am

I like to understand what I'm doing so I'm going to just ask some questions:
Firewall is executed in order, so where in my firewall should these rules be?

Firewall rules within a given chain are executed in order. Since your chain forward in filter was completely empty in the export (i.e. you had no other protection of the LAN except the partial one provided by NAT), just enter them in this order. It doesn't matter whether you put them before or after those in chain input; you could even interleave them but it is bad for human reading

The last one I guess is supposed to be just before my "Drop everything else" rule?

What is the function of that last rule?

The last one is a shortcut, an equivalent of
chain=forward in-interface=!ether1 action=accept (allow initiation of conections from LAN, or more exactly from any other in-interface than the WAN one)
chain=forward action=drop (aka "drop everything else")

Our gateway is x.x.x.81 so do I substitute the 10.20.30.40 with that?

Of course you don't, but you may use any other non-conflicting IP address from the private ranges. The whole trick is that you exclude the server's x.x.x.85 from the /29 from the point of view of the 2011 and the server itself, but keep it there from the point of view of the ISP router. So the 2011 impersonates the server to the ISP router and responds to its ARP requests for x.x.x.85 with its own MAC address, so the ISP router then sends pakets for x.x.x.85 to the Mikrotik, and Mikrotik routes them further to the server. It would probably be possible to use some tricks with bridge nat rules to keep the settings at the server as they are and let the Mikrotik answer also ARP requests for x.x.x.81 coming from the server with its own MAC address, but let's spare this ugly workaround for the moment you give up convincing NetworkManager to set up this /32 to /32 network

sindy · Thu Sep 01, 2022 10:01 am

One more thing, I'd really recommend to set use-ip-firewall under /interface bridge settings to no. The purpose of the yes here is to push even bridged frames through the IP firewall layers to facilitate queueing them for QoS, and it causes havoc to NAT. Since your mangle rules and queues do not suggest you actually use it for QoS handling of the bridged traffic between ether3 and ether4, only the havoc on NAT remains.

IwanGreyling · Thu Sep 01, 2022 5:58 pm

Firewall rules within a given chain are executed in order. Since your chain forward in filter was completely empty in the export (i.e. you had no other protection of the LAN except the partial one provided by NAT), just enter them in this order. It doesn't matter whether you put them before or after those in chain input; you could even interleave them but it is bad for human reading

Ah! Right!

Our gateway is x.x.x.81 so do I substitute the 10.20.30.40 with that?
Of course you don't, but you may use any other non-conflicting IP address from the private ranges. The whole trick is that you exclude the server's x.x.x.85 from the /29 from the point of view of the 2011 and the server itself, but keep it there from the point of view of the ISP router. So the 2011 impersonates the server to the ISP router and responds to its ARP requests for x.x.x.85 with its own MAC address, so the ISP router then sends pakets for x.x.x.85 to the Mikrotik, and Mikrotik routes them further to the server. It would probably be possible to use some tricks with bridge nat rules to keep the settings at the server as they are and let the Mikrotik answer also ARP requests for x.x.x.81 coming from the server with its own MAC address, but let's spare this ugly workaround for the moment you give up convincing NetworkManager to set up this /32 to /32 network

So... 10.20.30.40 is some special address?
Or are we having a miscommunication?

x.x.x.81 is the gateway address that the ISP Provided. Should I use that rather than 10.20.30.40(IE is 10.20.30.40 a placeholder)

One more thing, I'd really recommend to set use-ip-firewall under /interface bridge settings to no. The purpose of the yes here is to push even bridged frames through the IP firewall layers to facilitate queueing them for QoS, and it causes havoc to NAT. Since your mangle rules and queues do not suggest you actually use it for QoS handling of the bridged traffic between ether3 and ether4, only the havoc on NAT remains.

Yeah. I plan on disabling that. It was something I tried that really didn't work.

sindy · Thu Sep 01, 2022 7:11 pm

Or are we having a miscommunication?

Seems to be the case. A call might be a faster way ahead if you want to share your contact.

Let me try again. You keep the WAN (ether1) IP settings unchanged on the Mikrotik. But you need to insert the 2011 as a router, not as a bridge, into the path between x.x.x.81 and x.x.x.85. To achieve that, the x.x.x.85 must be in a smaller subnet than the x.x.x.80/29, so that the 2011 itself knew that packets for the .85 must not be sent via ether1 but via the interface to which that smaller subnet is attached, i.e. the ether4.

Now as the only smaller subnet than a /29 is a /30 one on Mikrotik (RouterOS does not support /31 subnets), so you would waste half of the /29 for this 2011-to-server subnet, you have to use the point-to-point approach where the two hosts connected by an Ethernet link do not share any common subnet - instead, each of them has its own /32 and an information what the other one's /32 is.

10.20.30.40 is just a "nice" address, but you may as well use 192.168.17.236 or any other private IP if you please. It just must not be part of any subnet you use in your network.

Sob · Fri Sep 02, 2022 5:26 pm

It would probably be possible to use some tricks with bridge nat rules to keep the settings at the server as they are and let the Mikrotik answer also ARP requests for x.x.x.81 coming from the server with its own MAC address, but let's spare this ugly workaround for the moment you give up convincing NetworkManager to set up this /32 to /32 network

No need for ugly:

/ip arp
add interface=ether4 address=the.pub.lic.81 published=yes

IwanGreyling · Fri Sep 02, 2022 5:34 pm

Alright. So I made the changes, however the server now has no internet connection.

I added the new export file (Making double sure to hide my IP address this time. XD)

Im not sure if netplan(Not NetworkManager as I said earlier d'oh) and the IP settings are clashing...
And Im not sure how to check if the settings applied...

sindy · Fri Sep 02, 2022 5:39 pm

No need for ugly:

Right you are, at least it is the same trick at both ends. Unless @IwanGreyling needs the server to be able to talk to the other IPs in the /29.

IwanGreyling · Fri Sep 02, 2022 5:44 pm

I just had to flush the IP Address that was assigned previously.
I can ping from the server now!

Now I'm guessing since theres no NAT in this setup, I need to allow ports such as our ERP Server using /ip firewall filter-rules?

sindy · Fri Sep 02, 2022 5:50 pm

Alright. So I made the changes, however the server now has no internet connection.

NetPlan/NetworkManager/whatever Ubuntu calls that thing, doesn't matter, just a blackbox to me.

I'd say @Sob's suggestion is great as it allows to keep the configuration of the server unchanged. So revert whatever changes you've done on the server, and add that line to your config,
/ip arp add interface=ether4 address=x.x.x.81 published=yes
without changing anything else (i.e. at the Mikrotik end, the address=10.20.30.40 network=x.x.x.85 will stay in place).

This will save you the need to find out how to make the current setting (with "peer 10.20.30.40") reboot-proof.

Now I'm guessing since theres no NAT in this setup, I need to allow ports such as our ERP Server using /ip firewall filter-rules?

Exactly. In the forward chain, before the "drop what came in via ether1", add rules like
in-interface=ether1 dst-address=x.x.x.85 protocol=tcp dst-port=yyyyzzz,ttt action=accept

Sob · Fri Sep 02, 2022 5:51 pm

Unless @IwanGreyling needs the server to be able to talk to the other IPs in the /29.

Adding those the same way should work. That said, I see point to point /32 config on server as clean and better way. Only problem may be convincing UI to accept it, because it's slightly unusual config. At least I had some trouble with that in the past.

IwanGreyling · Fri Sep 02, 2022 6:07 pm

It looks to be working with the netstat setup!
Thanks guys!

It looks like the firewall isn't letting things through though.
But I'll mission on for a bit, I'm sure this part I can figure out.

IwanGreyling · Sat Sep 03, 2022 9:39 am

Firewall is working! You guys are the best!
Sindy and Sob really are gurus!

Thanks for having patience with me XD.
I was employed to route circuit boards but sort of became the unofficial sys admin where I work

IwanGreyling · Mon Oct 02, 2023 3:27 pm

So, I found a better way of doing this, if anyone will find this usefull one day.

I want the servers IP to be 333.333.333.33
The server is already on the local lan as 192.168.1.100 (static DHCP)

First thing I did was add the address I wanted to the WAN interface:
/ip address
add address=222.222.222.222/29 interface=ether1 network=000.000.000.000 ///Routers public IP address
add address=333.333.333.333/29 interface=ether1 network=000.000.000.000 ///Servers public IP address

then I added some NAT rules to allow the ports to the relevant IP addresses:
/ip firewall nat
add action=dst-nat chain=dstnat dst-address=333.333.333.333 dst-port=80 \
protocol=tcp to-addresses=192.168.1.100 to-ports=80
add action=dst-nat chain=dstnat dst-address=333.333.333.333 dst-port=443 \
protocol=tcp to-addresses=192.168.1.100 to-ports=443

Quick, Clean, Easy, Painless.

I didn't know you can specify the dst-address like that.
I always thought it had to be an interface.

Hope this helps someone one day.

sindy · Mon Oct 02, 2023 8:38 pm

So, I found a better way of doing this, if anyone will find this usefull one day.
...
The server is already on the local lan as 192.168.1.100 (static DHCP)
...
Quick, Clean, Easy, Painless.

This was the essence of my alternative suggestion back then:

... or it must have a private address and a dst-nat rule on the 2011 must be used to forward requests coming to the public one to the private one. Depending on the application running at the server, it may be necessary to put the public address also up on the server and use a dst-nat rule on the server itself to revert the NAT. What OS is running on the server?

But it somehow quickly left the focus as we were busy dealing with the caveats of keeping the public IP directly on the server

Good that you found your way to it on your own.

2 IP Addresses 1 interface - DHCP on the first, firewall on the second

2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Re: 2 IP Addresses 1 interface - DHCP on the first, firewall on the second

Who is online