OK here goes. I setup PCC with 2 internet connections.

wan1 is on ether1 with 72.24.205.34/24 as it's ip
wan2 is on ether2 with 216.161.237.204/29 as it's ip
lan is on ether5 with 172.16.5.12/22 as it's ip

PCC is working great but now I need to get to step 2. I have a server behind the firewall and I would like to dst-nat from each wan interface so that no matter which wan interface I come in from I can get to the server. So from a remote browser I should be able to browse to either ip and get to the server behind the firewall.


# ADDRESS NETWORK BROADCAST INTERFACE
0 172.16.5.12/22 172.16.4.0 172.16.7.255 ether5
1 216.161.237.204/29 216.161.237.200 216.161.237.207 ether2
2 D 72.24.205.34/24 72.24.205.0 72.24.205.255 ether1


/ip route
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
72.24.205.1 routing-mark=to_wan1 scope=30 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
216.161.237.206 routing-mark=to_wan2 scope=30 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
72.24.205.1 scope=30 target-scope=10
add check-gateway=ping disabled=no distance=2 dst-address=0.0.0.0/0 gateway=\
216.161.237.206 scope=30 target-scope=10


/ip firewall filter
add action=accept chain=input comment="" connection-state=established disabled=no
add action=accept chain=input comment="" connection-state=related disabled=no
add action=accept chain=forward comment="" connection-state=established disabled=no
add action=accept chain=forward comment="" connection-state=related disabled=no
add action=accept chain=output comment="" connection-state=established disabled=no
add action=accept chain=output comment="" connection-state=related disabled=no
add action=accept chain=input comment="" connection-state=new disabled=no icmp-options=8:0-255 limit=1,0 protocol=icmp
add action=accept chain=input comment="" connection-state=new disabled=no dst-port=22 limit=2/1m,0 protocol=tcp
add action=accept chain=input comment="" connection-state=new disabled=no dst-port=8291 limit=2/1m,0 protocol=tcp
add action=accept chain=forward comment="" disabled=no dst-address=172.16.5.4 dst-port=80 out-interface=ether5 protocol=\
tcp
add action=accept chain=forward comment="" connection-state=new disabled=no in-interface=ether5 src-address=\
172.16.4.0/22
add action=drop chain=input comment="" disabled=no
add action=drop chain=forward comment="" disabled=no


/ip firewall mangle
add action=mark-connection chain=input comment="" disabled=no in-interface=ether1 new-connection-mark=wan1_conn \
passthrough=yes
add action=mark-connection chain=input comment="" disabled=no in-interface=ether2 new-connection-mark=wan2_conn \
passthrough=yes
add action=mark-routing chain=output comment="" connection-mark=wan1_conn disabled=no new-routing-mark=to_wan1 \
passthrough=yes
add action=mark-routing chain=output comment="" connection-mark=wan2_conn disabled=no new-routing-mark=to_wan2 \
passthrough=yes
add action=accept chain=prerouting comment="" disabled=no dst-address=72.24.205.0/24 in-interface=ether5
add action=accept chain=prerouting comment="" disabled=no dst-address=216.161.237.200/29 in-interface=ether5
add action=mark-connection chain=prerouting comment="" disabled=no dst-address-type=!local in-interface=ether5 \
new-connection-mark=wan1_conn passthrough=yes per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting comment="" disabled=no dst-address-type=!local in-interface=ether5 \
new-connection-mark=wan2_conn passthrough=yes per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting comment="" connection-mark=wan1_conn disabled=no in-interface=ether5 \
new-routing-mark=to_wan1 passthrough=yes
add action=mark-routing chain=prerouting comment="" connection-mark=wan2_conn disabled=no in-interface=ether5 \
new-routing-mark=to_wan2 passthrough=yes


/ip firewall nat
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=80 in-interface=ether1 protocol=tcp to-addresses=\
172.16.5.4
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=80 in-interface=ether2 protocol=tcp to-addresses=\
172.16.5.4
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ether1
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ether2


so here's what I got going on at the moment. When I try to hit the interfaces on from the web I get mixed results. Sometimes it loads and sometimes it doesn't.

make a custom chain and create a jump rule if packet comes from each interface, this way every change you make in custom chain will apply to all connections

Do you think you could give me a little example. I'm struggling a little bit. Once I'm in the custom chain I have to mark my connections right?

OK a little more info. Here's what happening. When I'm comming in wan1 sometimes it's get's marked with wan2 marking instead so obviously the return packets are getting sent out the wrong interface on the reply. So how do I fix this??

Looking at the Packet Flow diagram (http://wiki.mikrotik.com/wiki/Packet_Flow), it looks as if a packet dst-nat'd to a server behind the firewall hits the following chains in sequence, listing only the relevant parts:

prerouting (connection tracker, prerouting mangle, dst-nat) -> make routing decision, at this point it would choose 'forward' instead of 'input' since the destination IP is no longer the router itself.

Your rules only mark packets in the input chain, or in the prerouting chain if they are from the LAN interface - so that's the first time a mark is applied, and it is chosen somewhat randomly, resulting in packets potentially going out the wrong interface.

Insert two rules like this:
to mark new incoming connections that are forwarded through the router. I think you'll also have to change:
and add 'connection-mark=no-mark' (available in 4.x only, I think?) to make sure that you don't overwrite the connection-mark in prerouting on the reply from the server.

That's off the top of my head - maybe I'll have time at work today to try it in a lab.

The mangle rules I have in the firewall at the moment are apart of the tutorial for PCC load balancing but I also need my dst-nat working for both wans. I will try a couple things to today to see if I can get it to work. I think the hardest part is walking through the packet flow and making sure you don't violate or break rules that are already in place.

Did you try the adding the rules (and changing the two) I posted? It should do the trick.

I added the new mangle rules for the forward table but the other PCC rules are exactly the same as what I have in there already so I didn't change them. I will try this now and see if it works.

ok so with those 2 forward rules in place 2 things have happened. I can't get to the router from the internet now and it always marks the connections from either interface was wan1_conn so it only works from the 72.24.x.x ip.

From the conntrack table ---->

11 S protocol=tcp src-address=199.253.130.17:39516 dst-address=216.161.237.204:80 reply-src-address=172.16.5.4:80
reply-dst-address=199.253.130.17:39516 tcp-state=syn-recv timeout=0s connection-mark="wan1_conn" p2p=none

As you can see I tried to hit it form wan2's ip 216.161.237.204

Alright, more explicitly. Change:
to
That will require 4.x

Works. I also see what this does. I will keep playing around but I really appreciate your help on this. Many thanks!!!

If I think I got this right what the no-mark does it prevent the dst-nat return traffic from getting tagged by the PCC rule which causes it to be sent out the wrong interface.

Yes. The connection gets marked by the new rules in the forward chain when it is established (connection-state=new) depending on which interface the request came in from. You need to make sure that the PCC rules don't overwrite the connection-mark on the outbound return traffic as with your former rules they somewhat randomly choose an outbound interface for _all_ traffic coming in via the LAN interface, that's done by PCC marking only connections that have not been marked yet (connection-mark=no-mark).

Yep that makes sense. That was the link I have been missing the whole time. Thank you for explaining it.