Wrong output interface after dstnat

[shunkica]

I have a pppoe connection with dynamic IP from ISP.
Needed a static IP, so I got one from a company that offers them through a pptp connection.
This pptp connection is established over the pppoe connection which is the only internet access I have.

Setup a dstnat rule:

/ip firewall nat add action=jump chain=dstnat in-interface=pptp-static jump-target=static-chain
/ip firewall nat add action=dst-nat chain=static-chain dst-port=21 protocol=tcp to-addresses=10.20.10.1 to-ports=2100

When I try to establish a connection the packets come to the destination server:

forward: in:pptp-static out:br_local, proto TCP (SYN), <client ip>:31128->10.20.10.1:2100, NAT <client ip>:31128->(<static ip>:21->10.20.10.1:2100), len 60

But the response is not sent back via the pptp-static interface, it is sent back via the pppoe interface:

forward: in:br_local out:pppoe-internet, src-mac <hidden>, proto TCP (SYN,ACK), 10.20.10.1:2100-><client ip>:31128, NAT (10.20.10.1:2100-><static ip>:21)-><client ip>:31128, len 60

Of course this doesn’t work and the client does not pick up the response.

I realize I could do something like:

/ip route add distance=1 gateway=pptp-static routing-mark=Route-Over-Static
/ip firewall mangle add action=mark-routing chain=prerouting new-routing-mark=Route-Over-Static passthrough=no protocol=tcp src-address=10.20.10.1 src-port=2100

But this would mean that the service would now only be available through the static connection, and makes it harder to add new services (ports) on this static ip connection.
The other problem is that this is a FTP server and when the mode switches to passive, the port will change and the response will go over pppoe again.

Is there some way to make the response always use the same interface from which the request came?

EDIT: I should also mention that I don’t want all my traffic going through the static IP, it should only be used for specific things like this. The rest of the traffic needs to go through the pppoe interface.

[jkroon]

Hi,

At a routing level the only stuff the kernel looks at is the destination address.

Every packet however first passes through a rule based engine, which is a sequence of rules dictating which routing table to use. Since this is a sequential check the number of these rules should be kept low (for obvious reasons). These rules can be used to set up alternative routing tables and direct packets to not use the main routing table (there is a little known “local” table which the kernel actually uses to route local destined frames before looking at other rules). As an example of what these rules look like:

jkroon@plastiekpoot ~ $ ip rule show
0:	from all lookup local
1000:	from 192.168.42.204 lookup srcrt_ppp0
1001:	from all fwmark 0x1 lookup srcrt_ppp0
32766:	from all lookup main
32767:	from all lookup default

This is on my desktop, and to the best of my knowledge routeros only exposes the fwmark option here. default is generally empty, and main is primarily used. Rule priorities 1000 and 1001 here is non-default, and RouterOS uses this mechanism in order to implement “marked routing”.

Since the routing table only looks at the destination, and Mikrotik only exposes the fwmark rule from above, you have NO WAY other than using the marks you describe to achieve “the right thing”. I use a variation of what you describe here, but I believe for your use-case your rule would be sufficient. You will also note that I use fwmark above AS WELL AS from, that’s because the initial routing decision is made prior to applying source nat (which is used to “undo” DNAT before placing it back on the wire).

https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg - this remains one of the best packet flow diagrams I’ve ever seen for the Linux kernel. This relates because it give you a better understanding of where you can interact with the whole process, and in many cases, where certain processing in the flow of a packet happens, there are explicit statements here that “nat tables are not consulted except for state new” - which is accurate, but what is not mentioned is that this point in the flow is also where previously NATed frames are adjusted again, depending on whether it’s the source or destination being adjusted (both happens for all frames, and could simply be a no-op). Consider your DNAT, the destination gets altered, but for return traffic the source needs to get “reverted”, so DNAT/SNAT simply indicates the change on the ORIGINAL NEW frame, the same gets applied to further frames for the same flow in the same direction, and the reverse on future frames in the opposite direction.

So for not-NEW frames, the NAT table locations also applies the following:

PREROUTING - DNAT
OUTPUT - DNAT
INPUT - SNAT
POSTROUTING - SNAT

There are certain (xfrm - or x transformation, mostly ipsec) cases where it’s possible for the “same” frame to pass through the same chains multiple times, but this is not he norm in my experience.

So at this point, when the routing decision is made, the frame goed from 10.20.10.1 to external_ip. So the routing rules can’t match on the static IP because it’s not IN THE FRAME. As such, the ONLY workable solution is to use packet marks. The reason we use both is because for those applications that allows to specify a bind address (eg, ping -I, ssh -b) they will then also follow the “natural” path.

You mark the specific return traffic which is fine, but a more generic approach is to mark the connections on NEW (not the packet) and to then restore the connection mark to the packet mark (fwmark) upon ingress from the appropriate internal interface. This solves both your concerns of:

1. Not having to add additional mark routes for additional DNATs.
2. Service only being accessible through the static connection.

[shunkica]

Thank you for this exhaustive answer.
Some of the things you said however go straight over my head.

I will try to summarize what my conclusions after reading this are:

I should mark the new connections to :21 with some connection mark
Mark the packets originating from which have this connection mark with route mark

So something like this:

/ip firewall nat
add action=jump chain=dstnat in-interface=pptp-static jump-target=static-chain
add action=dst-nat chain=static-chain dst-port=21 protocol=tcp to-addresses=10.20.10.1 to-ports=2100

/ip firewall mangle
add action=mark-connection chain=prerouting connection-state=new dst-address=<static ip> dst-port=21 new-connection-mark=static-ip-input passthrough=no protocol=tcp
add action=mark-routing chain=prerouting connection-mark=static-ip-input new-routing-mark=Route-Over-Static passthrough=no src-address=10.20.10.1

/ip route
add distance=1 gateway=pptp-static routing-mark=Route-Over-Static

Is this what you meant?

Will this work with FTP passive mode? Eg. when server sends a different port to the client to connect to. This connection will remain marked with “static-ip-input” mark?

[Sob]

Use this, it will cover passive mode ports:

/ip firewall mangle
add chain=prerouting in-interface=pptp-static dst-address=<static ip> connection-state=new action=mark-connection new-connection-mark=static-ip-input passthrough=no

[jkroon]

Thank you for this exhaustive answer.
Some of the things you said however go straight over my head.

Sorry .

I should mark the new connections to :21 with some connection mark
Mark the packets originating from which have this connection mark with route mark

Halfway there, if you look at this:

@Sob

Use this, it will cover passive mode ports:

/ip firewall mangle
add chain=prerouting in-interface=pptp-static dst-address=<static ip> connection-state=new action=mark-connection new-connection-mark=static-ip-input passthrough=no

It over-covers, I’d drop the dst-address from this entirely. In other words:

/ip firewall mangle
add chain=prerouting in-interface=pptp-static connection-state=new action=mark-connection new-connection-mark=static-ip-input passthrough=no

This marks all NEW connections that was FIRST SEEN on INGRESS from the interface. So we should route it back out that way.

/ip firewall nat
add action=jump chain=dstnat in-interface=pptp-static jump-target=static-chain
add action=dst-nat chain=static-chain dst-port=21 protocol=tcp to-addresses=10.20.10.1 to-ports=2100

This seems reasonable, but I wouldn’t bother creating a second chain just for that, but if you have many such external interfaces it makes sense from a manageability perspective.

The last portion then that we need which is still missing is to mark packets before the routing decision. The problem we have now is RouterOS doesn’t seem to support -j CONNMARK --restore-mark in any way, so we end up having to restore the possible mappings one by one (you have passthrough=no above, so I believe we can safely assume that further processing will not happen for these frames, otherwise, just change above to jump to a separate chain which just adds the mark and then accepts):

/ip firewall mangle
... prev rule
add action=mark-routing chain=prerouting connection-mark=static-ip-input new-routing-mark=Route-Over-Static passthrough=no
add action=mark-routing chain=output connection-mark=static-ip-input new-routing-mark=Route-Over-Static passthrough=no

You’ll note I dropped the src-address, so it’ll apply to ALL connections that originally came into the RB on the static IP pptp. Specifically, if the firewall permits say winbox into the local host on this incoming link, then that will work. Firewall with care.

/ip route
add distance=1 gateway=pptp-static routing-mark=Route-Over-Static

And this is spot on.

Will this work with FTP passive mode? Eg. when server sends a different port to the client to connect to. This connection will remain marked with “static-ip-input” mark?

If you’re using the ftp connection tracking + nat helper module, yes it should.

[jkroon]

Actually, no, the above is NOT good enough, drop the connection-state=new on the connection mark, this should then re-mark the connection on any ingress frame, however, this should not matter.

The problem without that is that follow-up ingress frames will get the routing mark set, resulting in the frame being routed back out. This is bad.

[Sob]

Right, dst-address= shouldn’t be needed, mainly because packets to any other destination shouldn’t come from there.

The last problem can be solved by OP’s src-address=10.20.10.1 in route marking rule. Or it can be in-interface(-list)=, in-interface=!pptp-static, or anything that will cause that only responses will get routing marked.

Regarding passive mode, there’s no problem, it will work fine even without any NAT helper, because it’s again just new incoming conections. Only you’d have to manually forward range of passive ports to server. But that’s needed anyway if you want encryption and not stick with old FTP using even plaintext passwords. The problematic one would be active mode where server is connecting to client. But it’s pretty much dead nowadays, because all sane clients use passive mode by default.

[jkroon]

Right, dst-address= shouldn’t be needed, mainly because packets to any other destination shouldn’t come from there.

The last problem can be solved by OP’s src-address=10.20.10.1 in route marking rule. Or it can be in-interface(-list)=, in-interface=!pptp-static, or anything that will cause that only responses will get routing marked.

Regarding passive mode, there’s no problem, it will work fine even without any NAT helper, because it’s again just new incoming conections. Only you’d have to manually forward range of passive ports to server. But that’s needed anyway if you want encryption and not stick with old FTP using even plaintext passwords. The problematic one would be active mode where server is connecting to client. But it’s pretty much dead nowadays, because all sane clients use passive mode by default.

Passive … yes, you’re right, but you still need related,established unless you forward the entire passive range (which we generally don’t want to do). Also, if the FTP service runs SSL there is no way for the conntrack module to function and then you need the ports forwarded (even if you’re running a firewall on the same host as ftps you still need these ports open since the kernel won’t “see” the port numbers in the ftp conversation so won’t be able to accept based on related-established). What I dislike about that is the fact that the kernel would normally restrict the source of the connection if I’m not mistaken, which is useful.

And no, if you’re using marked routing in combination with connection marking you don’t need to be aware of internal/external IP addresses at all in mangle, ingress from pptp interface marks the connection, in pre-routing, if the in-interface isn’t the one for the connection mark, mark the packet for routing. Is the short version. How you implement that there are different options, the above discussion can probably be used as the basis for quite a number of those.

[Sob]

Forwarding all possible passive ports is not ideal, but as we both wrote, if you need encryption/SSL, it’s the only way. Not counting switching to active mode and doing similar thing on client, which would be possible if you control the client and you know what you’re doing, but it’s unusable with random clients that you don’t control.

As for marking, I don’t think we disagree. System assigns connection mark to all packets that belong to the connection, and then you need to assign routing marks, but you want those only for one direction. Using internal source address is just one of ways how to match right packets. Alternatively, you could mark routing for all packets, and use routing rule to basically ignore them when destination is local address or subnet, e.g.:

/ip route rule
add dst-address=10.20.10.1 action=lookup-only-in-table table=main

[shunkica]

I do not understand why this would be needed:

add action=mark-routing chain=output connection-mark=static-ip-input new-routing-mark=Route-Over-Static passthrough=no

I set my rules without this on the output chain and everything seems to be working. Here is my final config.

/ip firewall nat
add action=jump chain=dstnat in-interface=pptp-static jump-target=static-chain
add action=dst-nat chain=static-chain dst-port=21 protocol=tcp to-addresses=10.20.10.1 to-ports=2100

/ip firewall mangle
add action=mark-routing chain=prerouting connection-mark=static-ip-input new-routing-mark=Route-Over-Static passthrough=no src-address=10.20.10.1
add action=mark-connection chain=prerouting connection-state=new in-interface=pptp-static new-connection-mark=static-ip-input passthrough=no

/ip firewall filter
add action=accept chain=forward connection-nat-state=dstnat connection-state=new

/ip firewall service-port
set ftp ports=21

[shunkica]

Use this, it will cover passive mode ports:

/ip firewall mangle
add chain=prerouting in-interface=pptp-static dst-address=<static ip> connection-state=new action=mark-connection new-connection-mark=static-ip-input passthrough=no

Actually it worked with the config I posted in my second post.
It looks like ROS marks related connections with the same mark as the “parent” connection.
Either that or something else is going on, but it worked.

[jkroon]

Use this, it will cover passive mode ports:

/ip firewall mangle
add chain=prerouting in-interface=pptp-static dst-address=<static ip> connection-state=new action=mark-connection new-connection-mark=static-ip-input passthrough=no

Actually it worked with the config I posted in my second post.
It looks like ROS marks related connections with the same mark as the “parent” connection.
Either that or something else is going on, but it worked.

Well, there I learn something: https://home.regit.org/netfilter-en/netfilter-connmark/ relates:

CONNMARK is a cool feature of Netfilter. It provides a way to have a mark which is linked to the a connection tracking entry. Once a connmark is set, it also apply for RELATED connection entry. So, if you add a connmark to an FTP connection, the same connmark will be put of connections from ftp-data.

Now that is good to know. Yes, ROS builds on top of Netfilter.

[pir]

Actually it worked with the config I posted in my second post.
It looks like ROS marks related connections with the same mark as the “parent” connection.
Either that or something else is going on, but it worked.

This post got me to the point where dstnat connections to either of my WAN connections work but the automatic connection tracking doesn’t see to be working for me. My mangle rules are:

/ip firewall mangle
add action=mark-connection chain=prerouting in-interface-list=WAN2 log-prefix=wan2 new-connection-mark=wan2-connect passthrough=no
add action=mark-connection chain=prerouting in-interface-list=WAN1 log-prefix=wan1 new-connection-mark=wan1-connect passthrough=no
add action=mark-routing chain=output connection-mark=wan2-connect new-routing-mark=wan2 passthrough=no
add action=mark-routing chain=output connection-mark=wan1-connect new-routing-mark=wan1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=wan2-connect new-routing-mark=wan2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=wan1-connect new-routing-mark=wan1 passthrough=no

and then to do the routing:

/routing table
add fib name=wan1
add fib name=wan2
/ip route
add gateway=PPPoE routing-table=wan2
add gateway=ether1 routing-table=wan1

It functions if I do not have

connection-state=new

and the firewall connections tab in winbox then shows the connection getting established. if I add the new state requirement the connections tabs only sees the SYN and any return packets go out the default (wrong) interface when I check with sniffer.