Dual WAN mangle rules

OriiOn · Fri Jun 22, 2018 3:56 am

I've got two WAN's (main-fast = WAN-1, backup-slow=WAN-2). I want that all incoming traffic from WAN-2 (eth11-WAN-2) also goes out through the same gateway (WAN-2).

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=eth11-WAN-2 \
    new-connection-mark=MARK-WAN-2 passthrough=yes
add action=return chain=routing dst-address-type=local,broadcast,multicast
add action=mark-routing chain=prerouting connection-mark=MARK-WAN-2 new-routing-mark=WAN-2 \
    passthrough=no

And the corresponding routes...

/ip route
add comment=WAN-1 distance=1 gateway=[gateway-ip-WAN-1]
add comment=WAN-2 distance=2 gateway=[gateway-ip-WAN-2]
add comment=WAN-2 distance=2 gateway=[gateway-ip-WAN-2] routing-mark=WAN-2

Unfortunately when I active the mangle rules, I can't access the service at all from the WAN-2 gateway (since no traffic is coming through at all). The mangle rules show however that they are processed (both the mark-connection, as well as mark-routing). The mangle rule "return" however is never hit. I have this code from a post by "Sob" on this forum. Any help is highly appreciated!

Sob · Fri Jun 22, 2018 4:22 am

I'll deny everything!

But ok, it can be mine, but you clearly took only part of it. There's no chain=routing used in any way by your other rules. Move that rule to regular chain=prerouting and it should work. Or you can get rid of it completely and instead do either:

a) Add in-interface=<LAN> to route marking rule.
b) Add routing rule:

/ip route rule
add action=lookup-only-in-table dst-address=<your LAN subnet> table=main

anav · Fri Jun 22, 2018 5:05 am

I dont understand the request or the reply.

Firstly, all traffic going out WAN1 will go back through WAN1
All traffic going out WAN2 will go back through WAN2 and this is based on our SRCNAT RUles.
YOu should have a masquerade rule for both (or action srcnat if a static public IP)

Assuming WAN1 is your primary and WAN2 the secondary.
When WAN1 goes down, WAN2 will pick up the slack and when WAN1 comes back on line, the traffic will switch back to WAN1
What happens during the switches exactly is unknown to me, but SOB probably has some nifty clear all connections command that would help.

Okay as far as traffic coming into WAN1 or WAN2
I have to assume you mean unsolicited traffic where you are expecting outside users to come and access servers on your LAN????
If so you will need to create DSTNAT rules for this to happen.

As the ADMIN you decide which interface people are going to use to access your servers!!
Typical way to do this is use a DYNDNS type name which you associate with WAN1........
You can have a backup that points to WAN2 I suppose.

So far I dont see any need for mangling anything. What am I missing in my tired state?

Sob · Fri Jun 22, 2018 10:03 am

The request was for incoming traffic, i.e. forwarded port(s) to internal server.

If it comes from WAN1, it will work by default, because WAN1 is default outgoing route.

If it comes from WAN2, it will not work by default, because WAN1 is still default outgoing route and router will send reply there.

All traffic going out WAN2 will go back through WAN2 and this is based on our SRCNAT RUles.

In short, no, it's not what srcnat does. It only changes addresses/ports, it does not influence routing.

What am I missing in my tired state?

I guess it's mainly that this is only small part of config, and all the rest what's required and you might be thinking about is there and doesn't need to be shown.

OriiOn · Fri Jun 22, 2018 1:33 pm

I like simple solutions, because that is all my little brain can handle

So you say that this rule will make sure that incoming traffic from WAN1 will go out at WAN1, and incoming traffic from WAN2 will go out at WAN2?

/ip route rule
add action=lookup-only-in-table dst-address=10.42.0.0/16 table=main

How does that work? That means any packets going INTO the LAN should "remember where they came from", and "lookup their routing only for their corresponding routing table"? Is that how to read that rule?

anav · Fri Jun 22, 2018 1:48 pm

Im in the Oriion camp of small brains LOL.
I also dont get your devilishly clever programming. What is it saying/doing?

(PS the mangle rules looked okay to me except the middle one which is not needed and not sure why it was there?)

OriiOn · Fri Jun 22, 2018 1:54 pm

Well, I am totally not an expert, but I understood that the "return" rule being run BEFORE the "mark-routing" rule acts as a sort of "if-then clause", with the effect that any traffic that is "local" and won't go out through the WAN-interface(s) will not even reach the "mark-routing" rule. So it's sort of an "optimization" in order to minimize CPU load on the "router / mangling" code.

OriiOn · Fri Jun 22, 2018 1:58 pm

Sob's original post (where I got this solution from) also used "jump" rules, which I understood even less, and therefor ignored entirely. I understood they were further (speed) optimizations...?

anav · Fri Jun 22, 2018 3:59 pm

I have never seen return chain used but I dont think it is needed or does what you think it does.
I would have also kept the in-interface=wan2 for the second mark rule. Other than that I think its fine with the two mangle rules (connection mark and route mark)

It will interesting to see what SOB says for his non standard method of avoiding mangle rules LOL

Sob · Fri Jun 22, 2018 7:40 pm

I'm pretty sure that was one of my earlier configs, possibly not the best one. Plus it had some other things, and jumps and returns made sense there, I guess. I remember it worked, so it wasn't entirely bad.

The routing rule was meant only instead of this one:

/ip firewall mangle
add action=return chain=routing dst-address-type=local,broadcast,multicast

The other two you had (to mark connection and then routing) you want to keep.

And to explain it for curious users, it's very simple. You first mark connection for anything coming from WAN2. And then you mark routing for packets that belong to previously marked connections. When you do it for outgoing packets, they will use the right routing table, where WAN2 is default gateway. Trouble is, when you do it also for incoming packets, the same happens. Unfortunately for you, routing table WAN-2 doesn't have route to 10.42.0.0/16, only default one via ISP's gateway, so it will bounce incoming packets for marked connections right back.

You can either limit route marking only for outgoing traffic (in-interface=<LAN>) or just tell router that 10.42.0.0/16 should always be looked up in main routing table, where correct route exists.

And I was wrong before, the return rule would not work like this, it would have to be e.g.:

/ip firewall mangle
action=accept chain=prerouting in-interface=!<LAN>

So anything coming from other source than LAN would be accepted and not allowed to reach following rule. But don't use this, there's no need to have another rule, plus it could break something, if there were other rules.

anav · Fri Jun 22, 2018 8:42 pm

Still zinging over my head..........
I understand marking connections initiated from web coming in on WAN2
I understand marking route for those connections.

Thus all packets coming on WAN2 to a destination on the LAN will have these markings.
I guess it boils down to what happens when a server on the LAN responds (incoming information request), with information now heading back out the router to the web originator.
Since its the same session, I would think that the return packets get marked as well????
Otherwise what would have been the point of marking the inbound connection?????
If the return outbound packets are not also marked, then what was the point of marking the inbound packets to begin with.

So we have......... with the only change I would make is distance=1 for the routing-mark rule.....
/ip route
add comment=WAN-1 distance=1 gateway=[gateway-ip-WAN-1]
add comment=WAN-2 distance=2 gateway=[gateway-ip-WAN-2]
add comment=WAN-2 distance=1 gateway=[gateway-ip-WAN-2] routing-mark=WAN-2

WHY is that extra line required?
/ip route rule
add action=lookup-only-in-table dst-address=10.42.0.0/16 table=main

OriiOn · Fri Jun 22, 2018 9:22 pm

Thank you Sob for your explanations and help!
So I will keep these two mangle rules

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=eth11-WAN-2 \
    new-connection-mark=MARK-WAN-2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=MARK-WAN-2 new-routing-mark=WAN-2 \
    passthrough=no

And in addition (to make the whole thing work) I'll add this:

/ip route rule
add action=lookup-only-in-table dst-address=10.42.0.0/16 table=main

However, I still don't quite understand what this rule does?
I am especially baffled about the "table=main" part...
Could you explain please?

Btw, I have seen other examples where they use chain input, output or forward... looks a lot more complex. I'd like to keep things simple, as long as it works.

OriiOn · Fri Jun 22, 2018 9:27 pm

Btw, I do have another rule in my setup:

/ip route rule
add action=lookup src-address=[IP-WAN2]/32 table=WAN-2

This rule is supposed to make sure that all communication with the router (ping, winbox and sstp) will send their packets back out on WAN-2, if being accessed from WAN-2. If it works, I am not entirely sure...

Sob · Fri Jun 22, 2018 9:40 pm

Ok you both, imagine packet from internet coming to your public address. It's going to port 80 and you have dstnat rule for that, to send it to your internal webserver. Dstnat happens in prerouting. It's dstnat chain in RouterOS, but like prerouting, it happens before a routing decision is made. So it changes destination of packet to e.g. 10.42.0.20:80. And because it's incoming connection, it already got marked with connection mark. Now the route marking rule will kick in and mark routing for this packet, i.e. set the mark to WAN-2, which means that routing table WAN-2 should be used to find out where the packet should go to. And this routing table looks like this:

/ip route
add comment=WAN-2 distance=2 gateway=[gateway-ip-WAN-2] routing-mark=WAN-2

Now think as router. Where will you send packet to 10.42.0.20? To gateway-ip-WAN-2, of course, because that's the only possible route in this table. And that's why you need the routing rule (or exception in mangle rules, to not mark routing for incoming packets).

Why "main"? Because that's the default routing table. If you do:

/ip route print detail

then all entries without routing-mark=<something> mean the same as routing-mark=main. And dynamic connected route to 10.42.0.0/16 is one of them.

The other routing rule you posted can only work for traffic from router itself. Not for forwarded ports, because reply packets for those go from 10.42.x.x. Srcnat later changes source to address on WAN port, but it's after the routing was already decided.

OriiOn · Sat Jun 23, 2018 1:33 am

Sob, thank you so much for your help and explanation! It works! This rule (in conjunction with the two mangle rules) fixes the problem:

/ip route rule
add action=lookup-only-in-table dst-address=10.42.0.0/16 table=main

However, I still don't understand what this rule actually does?
I can totally follow your explanation until this line:

And that's why you need the routing rule (or exception in mangle rules, to not mark routing for incoming packets).
Why "main"? Because that's the default routing table.

and so forth...

I guess with "routing rule" you refer to the "/ip route rule" listed above? So this routing rule defines a sort of exception to the "mangling" process? It reads as "do not mangle packets heading for 10.42.0.0/16, instead lookup routing in the main table"? I am pretty sure that interpretation is wrong, because it makes no sense in my brain.
I know now what the main table is - in fact I figured that before you gave your explanation. But what exactly are we defining here, to tell the router to specifically use the "main routing table"? Under what circumstances we force the router to use the main routing table only?

Sob, thank you so much for your help, I highly appreciate. Ideally I'd just like to understand what this rule actually really does...

OriiOn · Sat Jun 23, 2018 1:42 am

I mentioned before that I already used a routing rule before (that I found somewhere in the forums, many moons ago)

/ip route rule
add action=lookup src-address=[IP-WAN2]/32 table=WAN-2

This rule is supposed to make sure that all communication with the router itself (ping, winbox and sstp) will send their packets back out on WAN-2, if being accessed from WAN-2.

I do not know that with the new rules (2 x manngle + 1 x routing) I even still need this? It is still active at the moment.
I can create a SSTP tunnel both to WAN-1 and WAN-2. And according to speedtest I can figure that WAN-2 routes correctly back out on WAN-2 (btw, is there a more sophisticated approach to check if the routing - in and out - is routing the way it's supposed to be?)
However, I am curious that in both cases (SSTP via WAN-1 or WAN-2) if I google for "my ip", it always shows the IP of WAN-1. I would have expected that if I SSTP via WAN-2, then "my ip" should show as WAN-2 IP?

Is that a sign that something is wrong, or am I missing something in my logic?

anav · Sat Jun 23, 2018 4:52 am

Nope, stone cold lack of understanding.
Your speaking a foreign language. I asked about whether or not the packets retain their routing marks after hitting the server they were intended to reach. Not even close to being discussed or answered.
I assume that they are marked inbound and reach the server. What happens next to the packets in this session is what was needed for an explanation for me anyway because what you said resulted in zero comprehension.

In other words, what was the purpose of marking these packets in the first place if they dont retain them on the way back out. IF they dont retain them on the way back out, correct they will go out the default and wrong route.

I dont see how your addition addresses this scenario.
What one needs is some way to say................... This response packets in this response session to marked packets inbound to the server must be treated as if they had marks themselves.
How is this done.......????

I hope sindy reads this thread because he/she asked why I made noise of his/her help being obtuse and this thread has helped crystalize my thoughts..........
You suffer from SINDYS's disease

- unable to walk a mile in a non-trained networking chaps shoes............

Sob · Sat Jun 23, 2018 5:12 am

Mangle rules still work. But they are in prerouting, literally before the routing decision happens. Result of your mangle rules is (for some packets) a new routing mark. If there isn't anything else, it means that routing decision will use given routing table. But then there's optional next step, and it's routing rules (actually, there are some implicit invisible rules, but don't worry about them, just think about it as simple "if a routing mark is set, use that routing table, otherwise use main one"). Routing rules are processed after prerouting, it's the actual routing decision, and they allow you to change default behaviour.

So the rule I posted tells the router "hey, if destination is 10.42.0.0/16, forget about everything else you'd otherwise do, and just look it up in main routing table". That's it.

Your routing rule is what I referred to in last paragraph in my previous post. It does exactly what you describe. The same thing is often done with mangle rules (e.g. in PCC example, rules in output chain), and it's probably because the WAN address is often dynamic, so routing rule like yours would have to be created by some script (and scripts are generally not user friendly).

And if I correctly understand what you mean with SSTP (router as VPN server, you connect to it as client, and access internet through tunnel), then what you see is correct. Even if you connect to address on WAN2, it's only the tunnel. Traffic inside the tunnel going to internet uses default route, which is WAN1.

Sob · Sat Jun 23, 2018 5:18 am

@anav: Routing marks are for packets. Aside from being internal thing that doesn't leave the router, they couldn't stick to reply packets from server anyway, because those are not the same packets. Packets are not like envelopes you can reuse, server is sending new ones.

What does stick is connection mark, thanks to our magic friend called conntrack.

OriiOn · Sat Jun 23, 2018 1:38 pm

Thank you Sob

Mangle rules still work. But they are in prerouting, literally before the routing decision happens. Result of your mangle rules is (for some packets) a new routing mark. If there isn't anything else, it means that routing decision will use given routing table. But then there's optional next step, and it's routing rules (actually, there are some implicit invisible rules, but don't worry about them, just think about it as simple "if a routing mark is set, use that routing table, otherwise use main one"). Routing rules are processed after prerouting, it's the actual routing decision, and they allow you to change default behaviour.

Ok, understood!

So the rule I posted tells the router "hey, if destination is 10.42.0.0/16, forget about everything else you'd otherwise do, and just look it up in main routing table". That's it.

You are right, I would read the rule like that. But why? Both the traffic from WAN1 and WAN2 will always go to 10.42.0.0/16 - everything is in 10.42.0.0/16. So if that rule has the power to change the default behavior (override what the mangle rules do), then wouldn't packets with the WAN2 tag still be forced to use the main-routing table?
OR, if this routing rule leaves packets with specific routing-marks untouched, and only touches those without a routing-mark... that would make sense (in my understanding) - but wouldn't that be the default behavior anyway? Like, any packets (without a routing-mark) will always use the main-route? So why is this rule even necessary? Obviously it IS, because things don't work without it...

OriiOn · Sat Jun 23, 2018 1:44 pm

And if I correctly understand what you mean with SSTP (router as VPN server, you connect to it as client, and access internet through tunnel), then what you see is correct. Even if you connect to address on WAN2, it's only the tunnel. Traffic inside the tunnel going to internet uses default route, which is WAN1.

Ah yeah! You are right, that makes total sense!

So in effect that means I can get rid of that rule:

/ip route rule
add action=lookup src-address=[IP-WAN2]/32 table=WAN-2

Because traffic to the router itself (the router is part of 10.42.0.0/16) is covered as well. So communication with the router like ping, winbox, SSTP-VPN, will be covered by the same mangle and routing rules we defined in this thread? My (speed) tests seem to indicate just that.

So that is exactly the setup I was looking for! Thanks a lot again

Sob · Sat Jun 23, 2018 3:11 pm

Both the traffic from WAN1 and WAN2 will always go to 10.42.0.0/16 - everything is in 10.42.0.0/16.

No, it won't. Once more, there's packet coming from WAN2. Dstnat changes its destination to to 10.42.x.x. It's marked as WAN2 connection and it gets routing mark WAN-2. So router will use this routing table and only route it contains is:

dst-address = 0.0.0.0/0, gateway = [gateway-ip-WAN-2]

Do you see any route to 10.42.x.x? The answer is no. So the packet goes to gateway-ip-WAN-2.

Routing rules use same principle as firewall rules. There are many options and only those you enter will be used. So if the only option for this specific rule is dst-address, then it's the only thing the rule looks for. Doesn't matter what routing mark the packet has, if you didn't specify any option for that, so it will match all.

Like, any packets (without a routing-mark) will always use the main-route?

True. But your packets from WAN2 do have routing mark WAN-2, so they wouldn't use main routing table by default.

So communication with the router like ping, winbox, SSTP-VPN, will be covered by the same mangle and routing rules we defined in this thread?

No. Mangle rules in this thread mark connections to both router and forwarded ports. But routing is marked only in prerouting, which covers only forwarded ports, not router's own output. If you'd get rid of your routing rule, you'd have to add another mangle rule, to mark routing for output (as in PCC example).

anav · Sat Jun 23, 2018 6:55 pm

Okay lets see if I have a better grasp.
We mark the inbound connections in pre-routing with connection marks.
We do not bother route marking yet because these packets will not retain the route markings once they hit the server and return traffic is generated.

However the return traffic will retain the connecting marks??

So how do we identify return traffic from the LAn that has connection marks associated with them identifying them as coming through WAN2.

What you are saying is that we have to tell this traffic to not follow the standard routing (out WAN1), but to follow the mangle route.
BUT there is no such thing as /ip route connection-mark = ??

So I am stuck in many ways here. I am beginning to think marking inbound traffic is a complete waste of time.

Sob · Sat Jun 23, 2018 8:17 pm

Let me clarify my previous reply. All marks (connection/packet/routing) are local to router and none of them actually stick to packet, in a way that they would be sent with it to other devices.

I did write that connection mark sticks, but it works in a different way. Remember conntrack, our magic friend? Conntrack watches every packet coming to, from or through router. And it's like a nosy neighbour, it keeps track of all of them.

So packet comes from internet (1.2.3.4:56789) to router's public address (2.3.4.5:80), conntrack takes a note and files it as connection #176524. Mangle rule marks the connection with connection mark "WAN2" and conntrack adds this mark to record. Dstnat changes destination to 192.168.88.20:80, but it doesn't fool conntrack, it still knows that it's connection #176524. Server receives packet and sends reply to 1.2.3.4:5678. Conntrack sees packet from 192.168.88.20:80 to 1.2.3.4:56789 and thinks "hey, this little fellow looks suspicious!", goes through its records and finds that it matches connection #176524. It inspect the record, finds that this connection should have mark "WAN2", so it adds mark to packet, to make it available for firewall rules. And that's it.

Next step is firewall rules. In prerouting, you check for connection marks and if packets has "WAN2", it gets assigned new routing mark.

Final step with connection tracking (we don't care about this now, just to make it complete) is when packet is about to be sent out to ISP, conntrack steps in and tells NAT to change source to 2.3.4.5:80, because that's where the original request from internet came to, and where the reply must come from. Conntrack is the big boss, NAT is only servant who couldn't do anything without conntrack.

That was connections marks.

Now for route marks, think about them as hints for routing. Imagine packets as people going on the road. When the road splits, they by default take the left one, because it's better or whatever. But you want them to use both, so a little before the split, you make a stand and give out cards with "left" and "right" written on them to those who you think should take specific way. Then at the split you put policeman who checks the cards and whoever has one, will be sent as the card says. That's what happens by default. But the policeman doesn't blindly follow what's on cards, he has other rules, e.g. that fat people can't take right way, because it's uphill and it would be too much for them. So when a fat guys comes with "right" card, he's sent to left anyway. That's routing rules.

Clear enough?

mozerd · Sat Jun 23, 2018 8:44 pm

When the road splits, they by default take the left one, because it's better or whatever. But you want them to use both, so a little before the split, you make a stand and give out cards with "left" and "right" written on them to those who you think should take specific way. Then at the split you put policeman who checks the cards and whoever has one, will be sent as the card says. That's what happens by default. But the policeman doesn't blindly follow what's on cards, he has other rules, e.g. that fat people can't take right way, because it's uphill and it would be too much for them. So when a fat guys comes with "right" card, he's sent to left anyway.

@Sob, FIRST I love your sense of humour. SECOND "When the road splits, they by default take the left one, because it's better or whatever" -- a very deep philosophical thought !
Let me declare that I always take the 'right' path because the 'left' path always leads to disaster.

THIRD -- I now understand mangle --- thank YOU!

Sob · Sat Jun 23, 2018 9:46 pm

What can I say, analogies are hard. They rarely match 1:1, some details have to be sacrificed. Why would people take only left way? They wouldn't. Maybe with "no trespassing" sign on right, but then it would be wrong to forcefully send them there. So it wasn't really deep thought, just me giving up. They are simply special kind of people and they prefer left.

anav · Sat Jun 23, 2018 10:16 pm

Good explanations result in fewer questions.
No more questions....
Just kidding LOL

So how was I supposed to assume that the second PREROUTING MANGLE RULE for mark route magically happened or was applied ONLY on and when the return packets from the server were sent? Where is this described? I think you skipped as step!! I am thinking its not magic and so what is the logic?
Logic would dictate that both actions are carried out on the way IN and since we are not routing on the way in, the mark route is useless and has no effect (is this your missing piece?).
So then you state magically the return packets still have the connection mark (glad we have a connection tracker) and because they do, the second mangle rule is applied again and this time routing will not dropped as the packet is on its way out the door.

So if the packet has the associated connection mark and is tagged with the Route mark.
Why on earth do you need a fourth IP route rule. The router will see that the router mark matches for an existing route and should send the packet out WAN2 regardless what the default is.
Otherwise why the heck include the route-mark in the standard third IP route rule?? The problem I thought was he had a higher distance than the default. It should be the same just to be on the safe side.

I dont see how return traffic coming in from WAN2 will go out anything but WAN2???

Sob · Sat Jun 23, 2018 10:55 pm

... since we are not routing on the way in, the mark route is useless and has no effect ...

No no no no no. There's always some routing (well, unless there's bridging instead, but that's not our case). And nothing unexpected happens. What you set is what you get. We have two rules:

1) RULE1:

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=WAN2 new-connection-mark=WAN2conn passthrough=yes

2) RULE2:

/ip firewall mangle
add action=mark-routing chain=prerouting connection-mark=WAN2conn new-routing-mark=WAN2route passthrough=no

Initial packet from internet comes to interface WAN2. It's start of new connection, so it doesn't have any connection mark. Does RULE1 match? Yes, it does, so the connection gets WAN2conn mark. Next is RULE2, does it match? Yes, it does, because the connection was already marked a moment before by RULE1. So the packet gets routing mark WAN2route. And that's also the moment when it gets screwed, because our routing table WAN2route has only default route via ISP2's gateway and no route to LAN subnet.

Edit: In case it's not clear, the problem is with packets going in, not out.

Sun Jun 24, 2018 9:02 am

Sorry to disturb your discussion, guys, I would just like to pin out that in case you would like to keep the mark sticked on packet when it traverse across the network, you can translate it into dscp values and then you can translate it back on any of the routers you need. Just the dictionary has to be the same everywhere... Only an idea aside, keep going forward.

OriiOn · Sun Jun 24, 2018 12:39 pm

No. Mangle rules in this thread mark connections to both router and forwarded ports. But routing is marked only in prerouting, which covers only forwarded ports, not router's own output. If you'd get rid of your routing rule, you'd have to add another mangle rule, to mark routing for output (as in PCC example).

I see. So I can either keep this routing-rule:

/ip route rule
add action=lookup src-address=[IP-WAN2] table=WAN-2

Or add this mangle-rule:

/ip firewall mangle
add action=mark-routing chain=output connection-mark=MARK-WAN-2 new-routing-mark=WAN-2

Both has the same effect.
Which is better, or would you prefer? Honestly I like the mangle-rule better. I have some troubles wrapping my head around what the routing-rule exactly does.
Thanks a lot Sob

anav · Sun Jun 24, 2018 5:37 pm

... since we are not routing on the way in, the mark route is useless and has no effect ...
No no no no no. There's always some routing (well, unless there's bridging instead, but that's not our case). And nothing unexpected happens. What you set is what you get. We have two rules:

1) RULE1:
Code: Select all
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=WAN2 new-connection-mark=WAN2conn passthrough=yes
2) RULE2:
Code: Select all
/ip firewall mangle
add action=mark-routing chain=prerouting connection-mark=WAN2conn new-routing-mark=WAN2route passthrough=no
Initial packet from internet comes to interface WAN2. It's start of new connection, so it doesn't have any connection mark. Does RULE1 match? Yes, it does, so the connection gets WAN2conn mark. Next is RULE2, does it match? Yes, it does, because the connection was already marked a moment before by RULE1. So the packet gets routing mark WAN2route. And that's also the moment when it gets screwed, because our routing table WAN2route has only default route via ISP2's gateway and no route to LAN subnet.

Edit: In case it's not clear, the problem is with packets going in, not out.

Okay so lets follow the bouncing packet......
The DSTNAT rule states where the packet should go or is allowed to go is probably more accurate but my question is when is this rule applied in the order of things...........

BUT, your saying the router is confused because we have a route-mark rule on the packet and an IP route rule which says go out WAN2.
So how does the packet get moved after matching the second mangle rule to the LAN server seems to be the critical matter.
Okay, so your saying we need a special IP route rule for incoming packets to make sure they get to the LAN and NOT use/apply the mangle outgoing rule until later........
Funny here I thought the whole time IP Route was only for going out the router.......... another barrier broken......... (where is a pillow to cry into).

Okay so now I can look at this rule without putting a pen into my eye at least not yet.
/ip route rule
add action=lookup-only-in-table dst-address=<your LAN subnet> table=main

So lets be clear those two mangle rules were in prerouting, and we have established we have intentionally screwed up the incoming packets by attaching markings that tell the router that they should go out WAN2 but they have not reached the LAN server yet.
So how do they get to the LAN server? Yes I understand this is unique to unsolicited packets because they are new connections. Not an issue with return traffic origination from the LAN.
I should also assume the routing information is needed before FW rules are hit???

My question is how do packets even know to use the new funky IP route rule above, or does the router after applying the two prerouting chain mangle rules, then say okay its been marked where do I send it and the router checks the available route rules setup for guidance and thus sees this new funky rule (but so what) ? The problem I cant get past this point is , without hitting the FW (dstnat etc) there is no way this interim route rule makes sense because the router DOES NOT KNOW the packets are destined for the LAN (that is in a FW dstnat rule which has not been seen yet). ???????????????????????????????

CZFan · Sun Jun 24, 2018 8:48 pm

@Sob, won't it be easier if you mark connection / routing coming in on WAN2 in postrouting chain?

Sob · Sun Jun 24, 2018 11:27 pm

Which is better, or would you prefer? Honestly I like the mangle-rule better. I have some troubles wrapping my head around what the routing-rule exactly does.

I can't really say. I'm starting to like routing rules, and I think they should be generally more efficient, because they can be simple and stateless. But I don't think there could be noticeable difference with small setups, so use whatever you like better.

And what it does is simple. If packet's source address is IP address of WAN2, it will try to find destination in routing table WAN-2. And if you have route to 0.0.0.0/0 there, it will find it. So it's simple and elegant way how to make connections to router itself work with multiple WANs, because it doesn't need anything else. If you'd not forward any ports inside and just had the router as e.g. multihomed VPN server, this is enough, no connection tracking required.

The DSTNAT rule states where the packet should go or is allowed to go is probably more accurate ...

Dstnat rule only changes destination address or port, nothing else. Where the packet actually goes is decided after that by routing.

Funny here I thought the whole time IP Route was only for going out the router..

Brace yourself. For router, there it no "in" or "out" in terms of interfaces, no LAN or WAN. It's all just interfaces. None of them has any special meaning for router. Packets can go from any interface to any other (or even the same). It's only the things you define (routes, firewall rules, routing rules, ...) that make it into something.

Yes I understand this is unique to unsolicited packets because they are new connections. Not an issue with return traffic origination from the LAN.

In fact, no. Routing in this context is a low-level thing. It just showels packets between interfaces. It doesn't care about any connections, if they are new, old, invalid, or whatever. Only those fancy high-level things like conntrack care about such unimportant details (from routing's perspective).

About how everything fits together, there are some interesting diagrams. They will scare you at first, but don't worry, it's not just you, they really are scary, when you see them for the first time. But there's one which is less scary:

Prerouting on first line is everything that happens when packet comes to router. Routing (not in this image) comes after that. It checks routes, consults routing rules and comes up with decision where the packet should go. It can continue to either input or forward (if it went to forward, it will pass through postrouting next). So if we stick to current topic, you can see that mangle rules in prerouting and dstnat are before any routing happens, and they can prepare a packet for it.

@Sob, won't it be easier if you mark connection / routing coming in on WAN2 in postrouting chain?

You can mark routing only in prerouting, it doesn't make sense anywhere else, because it's only useful before the routing happens. You could mark connection in forward (not postrouting, it doesn't support in-interface), and it would save the first packet. But not any following ones. What you can do, if you don't like routing rules, and I did mention it in very first reply, is to mark routing only for outgoing packets:

/ip firewall mangle
add action=mark-routing chain=prerouting connection-mark=MARK-WAN-2 new-routing-mark=WAN-2 in-interface=LAN

anav · Sun Jun 24, 2018 11:50 pm

Okay Sob,
But mangling only outgoing packets doesnt help in this scenario because we are dealing with incoming packets on WAN2
What is funny is I now can make sense of that prerouting chain LOL. It was like Greek to me before........ Perhaps edible and drinkable but not understandable.

Okay so, after we mangle the packets it goes to dstnat where we tell the packets what IP and port to map too......
Great,
So you state after it knows what IP and port it should be mapped to, I assume it goes to postrouting............
So far it has not been shovelled to any interface yet........
So how does the router, route the packets to the lan interface and to the server?
The mangle route rules say GO OUT WAN2............. this conflicts with going input or forward???
Is this where the funky new IP route rule you made comes in............... BIG GAP to address here.

CZFan · Mon Jun 25, 2018 1:26 am

@Sob, won't it be easier if you mark connection / routing coming in on WAN2 in postrouting chain?
You can mark routing only in prerouting, it doesn't make sense anywhere else, because it's only useful before the routing happens. You could mark connection in forward (not postrouting, it doesn't support in-interface), and it would save the first packet. But not any following ones. What you can do, if you don't like routing rules, and I did mention it in very first reply, is to mark routing only for outgoing packets:
Code: Select all
/ip firewall mangle
add action=mark-routing chain=prerouting connection-mark=MARK-WAN-2 new-routing-mark=WAN-2 in-interface=LAN

Did not test or look at packet flow diagrams, but thinking was if you mark packet already completed inward routing, it will reach internal destination, being marked for outward routing will ensure correct use of wan interface

CZFan · Mon Jun 25, 2018 1:30 am

Is this where the funky new IP route rule you made comes in............... BIG GAP to address here.

The rule as per Sob will use main table for the internal destination cause there will be a direct connected route in main table

Sob · Mon Jun 25, 2018 2:12 am

But mangling only outgoing packets doesnt help in this scenario because we are dealing with incoming packets on WAN2

It does. Remember that routing works with packets (not connections), and doesn't care about any fancy things like connection state. So if you don't mark routing for incoming packets, routing will use main routing table for them, and proper route to address in LAN is there. If you only mark routing for packets from LAN, based on WAN2 connection mark, routing will use WAN2 routing table for them. And everything will work.

So why the hell do we need any routing rules? Like with other things, options, different ways to reach same goal. Plus I'm sure there are scenarios, where routing rules work much better. But I don't have any example (sorry) that would blow your mind and make you realize how cool routing rules are.

So you state after it knows what IP and port it should be mapped to, I assume it goes to postrouting............

It's always in this order (unless it's for router itself, then it goes to input instead of forward and there's no postrouting):

[incoming packet] -> prerouting -> [routing decision] -> forward -> postrouting -> [outgoing packet].

The mangle route rules say GO OUT WAN2............. this conflicts with going input or forward???

It passes through forward chain like any other routed packet.

Is this where the funky new IP route rule you made comes in............... BIG GAP to address here.

I'd swear I already did that.

Did not test or look at packet flow diagrams, but thinking was if you mark packet already completed inward routing, it will reach internal destination, being marked for outward routing will ensure correct use of wan interface

Yes. But without either in-interface=LAN or routing rule, you will have the same problem with next packet belonging to given connection that comes from internet to router. Because it will have connection mark for WAN2, mangle rule will thus give it a routing mark for WAN2... and we're back where we started.

anav · Mon Jun 25, 2018 4:30 am

No capiche

I still have no idea how to ensure incoming traffic (new connections on WAN2) will go back out WAN2 when WAN1 is the primary WAN.
I am more confused on packet flow then ever.
I will stop assisting others as everything I thought I knew is wrong..............
My posting days are over for now. chow!

Sob · Mon Jun 25, 2018 7:20 am

Ok, and what seems to be the problem? I mean, by now, you must have some idea, what happens to packets, how they pass through router, etc. If I remember correctly, you even have dual WAN config at home, so you can experiment. Pick some exotic port number and forward it to some internal server in LAN, surely you have something, or you can start some little temporary webserver on your own computer. Then test different configs (only change things related to this port, or one remote address you'll use for testing, and you don't need to break anything that otherwise works), and see what happens. Add logging rules for this specific port in every possible chain, and you will see exactly what happens to packets. And if you'll have any questions, you can ask.

Who is online