Hi there!

While debugging some unrelated issues I re-discovered a problem in my setup that I was and still am stuck at. Here is my setup (relevant config export in attachment):

- I have two ISPs, ISP1 and ISP2
- ISP1 gives dst-nats me a public IP to 10.11.33.1 (let's call iIP1 the public IP address dst-natted here).
- ISP2 gives me public IP 1.2.3.82 (let's IP2 the public IP address) anonymized, don't want even more bots knocking on my door)
- ISP1 is the primary ISP, ISP2 is my backup. The failover is realized by marking connections in the mangle table and setting up routes appropriately. This works perfectly.

I have DNS names to IP1 and IP2 to access my NAS machine. From the outside, everything works well. From the inside, accessing IP1 works well too. However, accessing IP2 does not. After analyzing the rules, I reached the conclusion to what's happening, that is pretty close:

1) Machine from LAN tries to initiate a connection to the NAS, sends out TCP SYN to IP2.
2) Router sees the packet, sends it over ISP1 (primary ISP).
3) Router sees the packet again returning from ISP2 (backup ISP), but it does not mark it at this time (at least I don't see the mark in the table, and I believe the marking is done after SYN-ACK is seen).
4) Target machine doing the NAS responds SYN-ACK, but for some reason it is probably sent out using the default rule to ISP1.

Now, if I hardcoded the IP1 and IP2 into to router, I'm sure I can come up with a few firewall rules to handle this specific issue. If those IPs would be directly on the interfaces, I could also build a shortcut using srcnat (I think). But I wonder if there is a standard solution for this kind of configuration? I think it's pretty common. Ideas and suggestions appreciated.

Cheers,
Krakonos

PS: I'm not so desperate to fix it, so I won't be cooking crazy solutions - we mostly don't need the backup interface to be accessible, but I'm curious about it.

I think you want this instead of your current two dstnat rules:

Confused, why do you need mangle etc when you only have one working WAN at a time.
I can see the need for mangling when you have two concurrent WANIPs running??

Probably because if there are both connected, they both work. Where do you see only one?

Not the point!
I have two WANS up ALL the time. A primary and a secondary.
The secondary is used for all email pulls.
Guess what, no hands and no mangles PFM !!!

TO QUOTE THE OP
- ISP1 is the primary ISP, ISP2 is my backup. The failover

This is a simple recursive setup scenario WITHOUT mangling.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

What the OP should do is more clearly state why he has two DYNDNS rules for his NAS.
I am assuming that in case the Primary goes down he wants folks to be able to reach the NAS via the secondary WAN.
In which case, I would say the onus is on him to use a single DYNDNS client that switches to the operating available WANIP - could even use IP cloud for that.......

In this way, the NAS is always reachable via one URL at all times.
Sure there will be a break in connection at the time of switch over but thats all..........

If on the other hand his initial statement was not completely true (not primary for NAS but both at the same time) then he/she needs to come clean and make it clearer.
If so then mangling is the only option.

And do you also have some internal server that should be accessible from internet using both WANs at the same time? Try that without mangle rules.

And do you also have some internal server that should be accessible from internet using both WANs at the same time? Try that without mangle rules.

Why would I want them accessible from both at the same time. As I said, one is primary and one is backup.
If your going to use them at the same time dont give me primary failover crap give me PCC load balancing truth and then we can talk mangling

Maybe because you don't care that much about accessing internet from internal network, but you want internal server always immediatelly accessible from internet. And maybe it's for something where clients need to be connected for a long time, like OP's ssh. Obviously, if client uses WAN1 and it goes down, it will have to reconnect and use WAN2. But the when WAN1 goes up again, you don't want to disconnect that client again, because WAN2 still works.

Btw, it can be done without mangle rules, but you have to "cheat" a bit.

Hi!
Quite a few replies, let's reply to them all:
1. Sob, makes sense, I'll morph into that. But I don't think it will fix the issue I am having, or should it?
2. anav, both WANs are active all the time and can be used for incoming connections. There is no DYNDNS, as I didn't want to fuss with that. In the rare event of failure, the user will just connect to the backup DNS name.
3. Why mangling? Why not PCC load balancing? Actually, I did hide the disabled PCC rules, as I no longer use them and the mangle is a leftover from that. The problem is, that the backup link is very asymmetrical (DSL), so if the connection wants to upload and is unlucky to load balance onto the backup link, it's around 10 times slower than it should be.
4. anav, I could do that. However, in case the primary comes back online, I want the connections made over backup to continue working, as there is no reason they should not work.
5. Would be interested in a way without mangle rules, but I actually think my way is reasonably OK and clear to understand, with the only problem being the one mention in the question.

I was also experimenting with "manually" load balancing, that is putting the download heavy clients on ISP2 and symmetrical/upload heavy on ISP1, but that's too much work for too little gain...

A few minutes later update:

Sob's solution in the first reply appears to fix the issue. The connection never receives a mark and I guess it never leaves the internet. It appears having all the IPs in the dstnat rule make the srcnat rule work as expected, I didn't think it would in this scenario. Thanks a bunch, I think the problem is solved!

I think you made mistake in original description (mixed IP1 and IP2), it's ISP2's 1.2.3.82 which should always work, because it's on your router. ISP1's x.x.x.x (let's use e.g. 3.4.5.6) isn't and router doesn't have any idea that this address is yours.

If you're connecting from LAN:

a) to 1.2.3.82, then it always ends up on your router and dstnat catches it.
b) to 3.4.5.6 when ISP1 is preferred, then router forwards it to ISP1 and it can either work (if ISP does hairpin NAT) or fail (if not).
c) to 3.4.5.6 when ISP2 is preferred, then router forwards it to ISP2 (after masquerading it to 1.2.3.82), it goes through internet to ISP1, gets dstnatted to 10.11.33.1, dstnat catches it and forwards it to server. Server sends response to 1.2.3.82, and it fails, because it's not from 3.4.5.6 as expected.

-

And the solution without mangle rules and "cheating" bit, server would need two addresses, e.g. 10.79.80.10 and 10.79.80.11. Dstnat on WAN1 would forward ports to .10 and dstnat on WAN2 would use .11. And then you'd need routing rules to make traffic from .10 use WAN1 and from .11 WAN2.

That clears it up. I'll test variant b) next week, not sure what the ISP does exactly.

The cheating is actually a good idea, though does only work for the machine set up in this way, and more work is still needed for connections initiated from LAN, so in my case I'll keep the mangling rules.

Thanks again and keep up the good work, this was solved way faster than I anticipated!

Problem with b) is that it depends on ISP. And even if it works, it's no good to send packets to their router and back.

For the record, even c) could be made to work (key part would be mangle rule with action=route), but it doesn't make any sense, other than exercise for fun.

And the solution with two addresses, I wouldn't use that either. Maybe in some specific case it could be good idea (I don't have any example), but it was mainly a puzzle inspired by @anav's fear of mangle rules.