Dual WAN mangle rules

RouterOS Beginner Basics
1

I’ve got two WAN’s (main-fast = WAN-1, backup-slow=WAN-2). I want that all incoming traffic from WAN-2 (eth11-WAN-2) also goes out through the same gateway (WAN-2).

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=eth11-WAN-2 \
    new-connection-mark=MARK-WAN-2 passthrough=yes
add action=return chain=routing dst-address-type=local,broadcast,multicast
add action=mark-routing chain=prerouting connection-mark=MARK-WAN-2 new-routing-mark=WAN-2 \
    passthrough=no

And the corresponding routes…

/ip route
add comment=WAN-1 distance=1 gateway=[gateway-ip-WAN-1]
add comment=WAN-2 distance=2 gateway=[gateway-ip-WAN-2]
add comment=WAN-2 distance=2 gateway=[gateway-ip-WAN-2] routing-mark=WAN-2

Unfortunately when I active the mangle rules, I can’t access the service at all from the WAN-2 gateway (since no traffic is coming through at all). The mangle rules show however that they are processed (both the mark-connection, as well as mark-routing). The mangle rule “return” however is never hit. I have this code from a post by “Sob” on this forum. Any help is highly appreciated!

2

I’ll deny everything! :slight_smile:

But ok, it can be mine, but you clearly took only part of it. There’s no chain=routing used in any way by your other rules. Move that rule to regular chain=prerouting and it should work. Or you can get rid of it completely and instead do either:

a) Add in-interface= to route marking rule.
b) Add routing rule:

/ip route rule
add action=lookup-only-in-table dst-address=<your LAN subnet> table=main
3

I dont understand the request or the reply.

Firstly, all traffic going out WAN1 will go back through WAN1
All traffic going out WAN2 will go back through WAN2 and this is based on our SRCNAT RUles.
YOu should have a masquerade rule for both (or action srcnat if a static public IP)

Assuming WAN1 is your primary and WAN2 the secondary.
When WAN1 goes down, WAN2 will pick up the slack and when WAN1 comes back on line, the traffic will switch back to WAN1
What happens during the switches exactly is unknown to me, but SOB probably has some nifty clear all connections command that would help.

Okay as far as traffic coming into WAN1 or WAN2
I have to assume you mean unsolicited traffic where you are expecting outside users to come and access servers on your LAN???
If so you will need to create DSTNAT rules for this to happen.

As the ADMIN you decide which interface people are going to use to access your servers!!
Typical way to do this is use a DYNDNS type name which you associate with WAN1…
You can have a backup that points to WAN2 I suppose.

So far I dont see any need for mangling anything. What am I missing in my tired state?

4

The request was for incoming traffic, i.e. forwarded port(s) to internal server.

If it comes from WAN1, it will work by default, because WAN1 is default outgoing route.

If it comes from WAN2, it will not work by default, because WAN1 is still default outgoing route and router will send reply there.

In short, no, it’s not what srcnat does. It only changes addresses/ports, it does not influence routing.

I guess it’s mainly that this is only small part of config, and all the rest what’s required and you might be thinking about is there and doesn’t need to be shown.

5

I like simple solutions, because that is all my little brain can handle :wink:
So you say that this rule will make sure that incoming traffic from WAN1 will go out at WAN1, and incoming traffic from WAN2 will go out at WAN2?

/ip route rule
add action=lookup-only-in-table dst-address=10.42.0.0/16 table=main

How does that work? That means any packets going INTO the LAN should “remember where they came from”, and “lookup their routing only for their corresponding routing table”? Is that how to read that rule?

6

Im in the Oriion camp of small brains LOL.
I also dont get your devilishly clever programming. What is it saying/doing?

(PS the mangle rules looked okay to me except the middle one which is not needed and not sure why it was there?)

7

Well, I am totally not an expert, but I understood that the “return” rule being run BEFORE the “mark-routing” rule acts as a sort of “if-then clause”, with the effect that any traffic that is “local” and won’t go out through the WAN-interface(s) will not even reach the “mark-routing” rule. So it’s sort of an “optimization” in order to minimize CPU load on the “router / mangling” code.

8

Sob’s original post (where I got this solution from) also used “jump” rules, which I understood even less, and therefor ignored entirely. I understood they were further (speed) optimizations…?

9

I have never seen return chain used but I dont think it is needed or does what you think it does.
I would have also kept the in-interface=wan2 for the second mark rule. Other than that I think its fine with the two mangle rules (connection mark and route mark)

It will interesting to see what SOB says for his non standard method of avoiding mangle rules LOL

10

I’m pretty sure that was one of my earlier configs, possibly not the best one. Plus it had some other things, and jumps and returns made sense there, I guess. I remember it worked, so it wasn’t entirely bad.

The routing rule was meant only instead of this one:

/ip firewall mangle
add action=return chain=routing dst-address-type=local,broadcast,multicast

The other two you had (to mark connection and then routing) you want to keep.

And to explain it for curious users, it’s very simple. You first mark connection for anything coming from WAN2. And then you mark routing for packets that belong to previously marked connections. When you do it for outgoing packets, they will use the right routing table, where WAN2 is default gateway. Trouble is, when you do it also for incoming packets, the same happens. Unfortunately for you, routing table WAN-2 doesn’t have route to 10.42.0.0/16, only default one via ISP’s gateway, so it will bounce incoming packets for marked connections right back.

You can either limit route marking only for outgoing traffic (in-interface=) or just tell router that 10.42.0.0/16 should always be looked up in main routing table, where correct route exists.

And I was wrong before, the return rule would not work like this, it would have to be e.g.:

/ip firewall mangle
action=accept chain=prerouting in-interface=!<LAN>

So anything coming from other source than LAN would be accepted and not allowed to reach following rule. But don’t use this, there’s no need to have another rule, plus it could break something, if there were other rules.

11

Still zinging over my head…
I understand marking connections initiated from web coming in on WAN2
I understand marking route for those connections.

Thus all packets coming on WAN2 to a destination on the LAN will have these markings.
I guess it boils down to what happens when a server on the LAN responds (incoming information request), with information now heading back out the router to the web originator.
Since its the same session, I would think that the return packets get marked as well???
Otherwise what would have been the point of marking the inbound connection???
If the return outbound packets are not also marked, then what was the point of marking the inbound packets to begin with.

So we have… with the only change I would make is distance=1 for the routing-mark rule…
/ip route
add comment=WAN-1 distance=1 gateway=[gateway-ip-WAN-1]
add comment=WAN-2 distance=2 gateway=[gateway-ip-WAN-2]
add comment=WAN-2 distance=1 gateway=[gateway-ip-WAN-2] routing-mark=WAN-2

WHY is that extra line required?
/ip route rule
add action=lookup-only-in-table dst-address=10.42.0.0/16 table=main

12

Thank you Sob for your explanations and help!
So I will keep these two mangle rules

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=eth11-WAN-2 \
    new-connection-mark=MARK-WAN-2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=MARK-WAN-2 new-routing-mark=WAN-2 \
    passthrough=no

And in addition (to make the whole thing work) I’ll add this:

/ip route rule
add action=lookup-only-in-table dst-address=10.42.0.0/16 table=main

However, I still don’t quite understand what this rule does?
I am especially baffled about the “table=main” part…
Could you explain please?

Btw, I have seen other examples where they use chain input, output or forward… looks a lot more complex. I’d like to keep things simple, as long as it works.

13

Btw, I do have another rule in my setup:

/ip route rule
add action=lookup src-address=[IP-WAN2]/32 table=WAN-2

This rule is supposed to make sure that all communication with the router (ping, winbox and sstp) will send their packets back out on WAN-2, if being accessed from WAN-2. If it works, I am not entirely sure…

14

Ok you both, imagine packet from internet coming to your public address. It’s going to port 80 and you have dstnat rule for that, to send it to your internal webserver. Dstnat happens in prerouting. It’s dstnat chain in RouterOS, but like prerouting, it happens before a routing decision is made. So it changes destination of packet to e.g. 10.42.0.20:80. And because it’s incoming connection, it already got marked with connection mark. Now the route marking rule will kick in and mark routing for this packet, i.e. set the mark to WAN-2, which means that routing table WAN-2 should be used to find out where the packet should go to. And this routing table looks like this:

/ip route
add comment=WAN-2 distance=2 gateway=[gateway-ip-WAN-2] routing-mark=WAN-2

Now think as router. Where will you send packet to 10.42.0.20? To gateway-ip-WAN-2, of course, because that’s the only possible route in this table. And that’s why you need the routing rule (or exception in mangle rules, to not mark routing for incoming packets).

Why “main”? Because that’s the default routing table. If you do:

/ip route print detail

then all entries without routing-mark= mean the same as routing-mark=main. And dynamic connected route to 10.42.0.0/16 is one of them.

The other routing rule you posted can only work for traffic from router itself. Not for forwarded ports, because reply packets for those go from 10.42.x.x. Srcnat later changes source to address on WAN port, but it’s after the routing was already decided.

15

Sob, thank you so much for your help and explanation! It works! This rule (in conjunction with the two mangle rules) fixes the problem:

/ip route rule
add action=lookup-only-in-table dst-address=10.42.0.0/16 table=main

However, I still don’t understand what this rule actually does?
I can totally follow your explanation until this line:

And that’s why you need the routing rule (or exception in mangle rules, to not mark routing for incoming packets).
Why “main”? Because that’s the default routing table.

and so forth…

I guess with “routing rule” you refer to the “/ip route rule” listed above? So this routing rule defines a sort of exception to the “mangling” process? It reads as “do not mangle packets heading for 10.42.0.0/16, instead lookup routing in the main table”? I am pretty sure that interpretation is wrong, because it makes no sense in my brain.
I know now what the main table is - in fact I figured that before you gave your explanation. But what exactly are we defining here, to tell the router to specifically use the “main routing table”? Under what circumstances we force the router to use the main routing table only?

Sob, thank you so much for your help, I highly appreciate. Ideally I’d just like to understand what this rule actually really does…

16

I mentioned before that I already used a routing rule before (that I found somewhere in the forums, many moons ago)

/ip route rule
add action=lookup src-address=[IP-WAN2]/32 table=WAN-2

This rule is supposed to make sure that all communication with the router itself (ping, winbox and sstp) will send their packets back out on WAN-2, if being accessed from WAN-2.

I do not know that with the new rules (2 x manngle + 1 x routing) I even still need this? It is still active at the moment.
I can create a SSTP tunnel both to WAN-1 and WAN-2. And according to speedtest I can figure that WAN-2 routes correctly back out on WAN-2 (btw, is there a more sophisticated approach to check if the routing - in and out - is routing the way it’s supposed to be?)
However, I am curious that in both cases (SSTP via WAN-1 or WAN-2) if I google for “my ip”, it always shows the IP of WAN-1. I would have expected that if I SSTP via WAN-2, then “my ip” should show as WAN-2 IP?

Is that a sign that something is wrong, or am I missing something in my logic?

17

Nope, stone cold lack of understanding.
Your speaking a foreign language. I asked about whether or not the packets retain their routing marks after hitting the server they were intended to reach. Not even close to being discussed or answered.
I assume that they are marked inbound and reach the server. What happens next to the packets in this session is what was needed for an explanation for me anyway because what you said resulted in zero comprehension. :frowning: In other words, what was the purpose of marking these packets in the first place if they dont retain them on the way back out. IF they dont retain them on the way back out, correct they will go out the default and wrong route.

I dont see how your addition addresses this scenario.
What one needs is some way to say… This response packets in this response session to marked packets inbound to the server must be treated as if they had marks themselves.
How is this done…???

I hope sindy reads this thread because he/she asked why I made noise of his/her help being obtuse and this thread has helped crystalize my thoughts…
You suffer from SINDYS’s disease :wink: - unable to walk a mile in a non-trained networking chaps shoes…

18

Mangle rules still work. But they are in prerouting, literally before the routing decision happens. Result of your mangle rules is (for some packets) a new routing mark. If there isn’t anything else, it means that routing decision will use given routing table. But then there’s optional next step, and it’s routing rules (actually, there are some implicit invisible rules, but don’t worry about them, just think about it as simple “if a routing mark is set, use that routing table, otherwise use main one”). Routing rules are processed after prerouting, it’s the actual routing decision, and they allow you to change default behaviour.

So the rule I posted tells the router “hey, if destination is 10.42.0.0/16, forget about everything else you’d otherwise do, and just look it up in main routing table”. That’s it.

Your routing rule is what I referred to in last paragraph in my previous post. It does exactly what you describe. The same thing is often done with mangle rules (e.g. in PCC example, rules in output chain), and it’s probably because the WAN address is often dynamic, so routing rule like yours would have to be created by some script (and scripts are generally not user friendly).

And if I correctly understand what you mean with SSTP (router as VPN server, you connect to it as client, and access internet through tunnel), then what you see is correct. Even if you connect to address on WAN2, it’s only the tunnel. Traffic inside the tunnel going to internet uses default route, which is WAN1.

19

@anav: Routing marks are for packets. Aside from being internal thing that doesn’t leave the router, they couldn’t stick to reply packets from server anyway, because those are not the same packets. Packets are not like envelopes you can reuse, server is sending new ones. :slight_smile: What does stick is connection mark, thanks to our magic friend called conntrack.

20

Thank you Sob :slight_smile:

Mangle rules still work. But they are in prerouting, literally before the routing decision happens. Result of your mangle rules is (for some packets) a new routing mark. If there isn’t anything else, it means that routing decision will use given routing table. But then there’s optional next step, and it’s routing rules (actually, there are some implicit invisible rules, but don’t worry about them, just think about it as simple “if a routing mark is set, use that routing table, otherwise use main one”). Routing rules are processed after prerouting, it’s the actual routing decision, and they allow you to change default behaviour.

Ok, understood!


So the rule I posted tells the router “hey, if destination is 10.42.0.0/16, forget about everything else you’d otherwise do, and just look it up in main routing table”. That’s it.

You are right, I would read the rule like that. But why? Both the traffic from WAN1 and WAN2 will always go to 10.42.0.0/16 - everything is in 10.42.0.0/16. So if that rule has the power to change the default behavior (override what the mangle rules do), then wouldn’t packets with the WAN2 tag still be forced to use the main-routing table?
OR, if this routing rule leaves packets with specific routing-marks untouched, and only touches those without a routing-mark… that would make sense (in my understanding) - but wouldn’t that be the default behavior anyway? Like, any packets (without a routing-mark) will always use the main-route? So why is this rule even necessary? Obviously it IS, because things don’t work without it…