Dual WAN PCC with port forwards

PeteH · February 14, 2017, 7:41pm

All,

I am having a bear of a time getting PCC to work properly with a dual WAN configuration. When both links are up, connections seem to work as defined except for port forwarding. As you can see in my configuration export below, I have port 22 forwarded from both WANs to an internal host. If I connect to port 22 over the primary WAN (ETH1-CEO), packets are returned as normal; but if I attempt to connect over port 22 from WAN2 (ETH2-TBB), the connection never succeeds unless WAN1 has been disabled or is unreachable. Do routing marks need to be added to my ip firewall filter commands so that the packet knows which WAN to 'reply' to? Here is the ip firewall filter section:

/ip firewall filter
add chain=input comment="accept ICMP" protocol=icmp
add chain=input comment="accept ssh" dst-port=22 protocol=tcp
add chain=input comment="accept http" dst-port=80 protocol=tcp
add chain=input comment="accept cups" dst-port=631 protocol=tcp
add chain=input comment="accept ssh2" dst-port=2222 protocol=tcp
add chain=input comment="accept vnc" dst-port=5900 protocol=tcp
add chain=input comment="accept established,related" connection-state=
established,related
add action=drop chain=input in-interface=ETH1-CEO
add action=drop chain=input in-interface=ETH2-TBB
add action=fasttrack-connection chain=forward comment=
"this speeds up local connections" connection-state=established,related
add chain=forward comment="forward established,related" connection-state=
established,related
add action=drop chain=forward comment="drop invalid" connection-state=invalid
add action=drop chain=forward comment="drop all from WAN not DSTNATed"
connection-nat-state=!dstnat connection-state=new in-interface=ETH1-CEO
add action=drop chain=forward comment="drop all from WAN not DSTNATed"
connection-nat-state=!dstnat connection-state=new in-interface=ETH2-TBB

Second question has to do with PCC and how it determines if a WAN link is up or down. If I disable a WAN port and/or unplug the network cable, PCC does exactly as it is supposed to and will only use the active link. But, if the link is down past the gateway, PCC never recognizes this event. As seen in my ip route section:

/ip route
add check-gateway=ping distance=1 gateway=10.100.100.1 routing-mark=to_ETH1-CEO
add check-gateway=ping distance=1 gateway=10.200.100.65 routing-mark=
to_ETH2-TBB
add check-gateway=ping distance=1 gateway=10.100.100.1
add check-gateway=ping distance=1 gateway=10.200.100.65

What I would like to do is instead of checking the gateway status, I would rather check a host further along the chain or some host on the internet - Google DNS as an example: 8.8.8.8 and/or 8.8.4.4. Is RouterOS - RouterOS - MikroTik Documentation applicable when using PCC? I have tried multiple forms of the section 'Multiple host checking per Uplink' and have not had any success. Finally, how do 'scope' and 'target-scope' factor in to the configuration? Also, I do not find the 'distance' value to be clear in the ip route section with respect to PCC. Does distance matter, if so, how/why?

Any pointers would be greatly appreciated. For completeness here is my entire export:

[admin@gold] > /export

feb/14/2017 11:52:57 by RouterOS 6.34.6

software id = 9ZHE-Z49D

/interface bridge
add admin-mac=E4:8D:8C:1E:8A:8B auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] name=ETH1-CEO
set [ find default-name=ether2 ] name=ETH2-TBB
set [ find default-name=ether3 ] name=ETH3-LAN
set [ find default-name=ether4 ] master-port=ETH3-LAN name=ETH4-LAN
set [ find default-name=ether5 ] master-port=ETH3-LAN name=ETH5-LAN
set [ find default-name=ether6 ] name=ether6-master
set [ find default-name=ether7 ] master-port=ether6-master
set [ find default-name=ether8 ] master-port=ether6-master
set [ find default-name=ether9 ] master-port=ether6-master
set [ find default-name=ether10 ] master-port=ether6-master
/ip neighbor discovery
set ETH1-CEO discover=no
set ETH2-TBB discover=no
set bridge comment=defconf
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=static ranges=192.168.0.10-192.168.0.100
add name=dhcp ranges=192.168.0.101-192.168.0.254
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge lease-time=4h name=dhcp
/interface bridge port
add bridge=bridge comment=defconf interface=ether6-master
add bridge=bridge comment=defconf interface=sfp1
add bridge=bridge interface=ETH3-LAN
/ip address
add address=10.100.100.254/24 interface=ETH1-CEO network=10.100.100.0
add address=10.200.100.126/26 interface=ETH2-TBB network=10.200.100.64
add address=192.168.0.1/24 interface=bridge network=192.168.0.0
/ip dhcp-server network
add address=192.168.0.0/24 dns-server=192.168.0.1 gateway=192.168.0.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip dns static
add address=192.168.0.1 name=router
/ip firewall filter
add chain=input comment="accept ICMP" protocol=icmp
add chain=input comment="accept ssh" dst-port=22 protocol=tcp
add chain=input comment="accept http" dst-port=80 protocol=tcp
add chain=input comment="accept cups" dst-port=631 protocol=tcp
add chain=input comment="accept ssh2" dst-port=2222 protocol=tcp
add chain=input comment="accept vnc" dst-port=5900 protocol=tcp
add chain=input comment="accept established,related" connection-state=
established,related
add action=drop chain=input in-interface=ETH1-CEO
add action=drop chain=input in-interface=ETH2-TBB
add action=fasttrack-connection chain=forward comment=
"this speeds up local connections" connection-state=established,related
add chain=forward comment="forward established,related" connection-state=
established,related
add action=drop chain=forward comment="drop invalid" connection-state=invalid
add action=drop chain=forward comment="drop all from WAN not DSTNATed"
connection-nat-state=!dstnat connection-state=new in-interface=ETH1-CEO
add action=drop chain=forward comment="drop all from WAN not DSTNATed"
connection-nat-state=!dstnat connection-state=new in-interface=ETH2-TBB
/ip firewall mangle
add action=mark-connection chain=input in-interface=ETH1-CEO
new-connection-mark=ETH1-CEO_conn
add action=mark-connection chain=input in-interface=ETH2-TBB
new-connection-mark=ETH2-TBB_conn
add action=mark-routing chain=output connection-mark=ETH1-CEO_conn
new-routing-mark=to_ETH1-CEO
add action=mark-routing chain=output connection-mark=ETH2-TBB_conn
new-routing-mark=to_ETH2-TBB
add chain=prerouting dst-address=10.100.100.0/24 in-interface=bridge
add chain=prerouting dst-address=10.200.100.64/26 in-interface=bridge
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=bridge new-connection-mark=ETH1-CEO_conn
per-connection-classifier=src-address:3/0
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=bridge new-connection-mark=ETH1-CEO_conn
per-connection-classifier=src-address:3/1
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=bridge new-connection-mark=ETH2-TBB_conn
per-connection-classifier=src-address:3/2
add action=mark-routing chain=prerouting connection-mark=ETH1-CEO_conn
in-interface=bridge new-routing-mark=to_ETH1-CEO
add action=mark-routing chain=prerouting connection-mark=ETH2-TBB_conn
in-interface=bridge new-routing-mark=to_ETH2-TBB
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ETH1-CEO
add action=masquerade chain=srcnat out-interface=ETH2-TBB
add action=dst-nat chain=dstnat dst-address=10.100.100.254 dst-port=22
in-interface=ETH1-CEO log=yes protocol=tcp to-addresses=192.168.0.30
add action=dst-nat chain=dstnat dst-address=10.100.100.254 dst-port=80
in-interface=ETH1-CEO protocol=tcp to-addresses=192.168.0.1
add action=dst-nat chain=dstnat dst-address=10.100.100.254 dst-port=631
in-interface=ETH1-CEO protocol=tcp to-addresses=192.168.0.70
add action=dst-nat chain=dstnat dst-address=10.100.100.254 dst-port=2222
in-interface=ETH1-CEO protocol=tcp to-addresses=192.168.0.1 to-ports=22
add action=dst-nat chain=dstnat dst-address=10.100.100.254 dst-port=5900
in-interface=ETH1-CEO protocol=tcp to-addresses=192.168.0.30 to-ports=
0-65535
add action=dst-nat chain=dstnat dst-address=10.200.100.126 dst-port=22
in-interface=ETH2-TBB log=yes protocol=tcp to-addresses=192.168.0.30
add action=dst-nat chain=dstnat dst-address=10.200.100.126 dst-port=80
in-interface=ETH2-TBB protocol=tcp to-addresses=192.168.0.1
add action=dst-nat chain=dstnat dst-address=10.200.100.126 dst-port=631
in-interface=ETH2-TBB protocol=tcp to-addresses=192.168.0.70
add action=dst-nat chain=dstnat dst-address=10.200.100.126 dst-port=2222
in-interface=ETH2-TBB protocol=tcp to-addresses=192.168.0.1 to-ports=22
add action=dst-nat chain=dstnat dst-address=10.200.100.126 dst-port=5900
in-interface=ETH2-TBB protocol=tcp to-addresses=192.168.0.30
/ip route
add check-gateway=ping distance=1 gateway=10.100.100.1 routing-mark=to_ETH1-CEO
add check-gateway=ping distance=1 gateway=10.200.100.65 routing-mark=
to_ETH2-TBB
add check-gateway=ping distance=1 gateway=10.100.100.1
add check-gateway=ping distance=1 gateway=10.200.100.65
/lcd
set default-screen=stat-slideshow read-only-mode=yes
/lcd interface
set sfp1 disabled=yes
set ETH4-LAN disabled=yes
set ETH5-LAN disabled=yes
set ether6-master disabled=yes
set ether7 disabled=yes
set ether8 disabled=yes
set ether9 disabled=yes
set ether10 disabled=yes
/system clock
set time-zone-autodetect=no time-zone-name=America/Chicago
/system identity
set name=gold
/system package update
set channel=bugfix
/tool mac-server
set [ find default=yes ] disabled=yes
add interface=bridge
/tool mac-server mac-winbox
set [ find default=yes ] disabled=yes
add interface=bridge

Thanks to all.

-Pete

Sob · February 15, 2017, 1:54am

I don’t have much time right now, so just few quick pointers:

For port forwarding, you need to mark new incoming connections from WANs with connection marks. And then for outgoing packets of these previously marked connections, you need to mark routing, to make them use the right WAN.
You’re already marking connections, so just don’t do it in input chain, do it in prerouting instead and it will cover both traffic to router and forwarded ports.
PCC couldn’t care less about state of WANs. It just marks connections. The only reason why it works when one WAN goes down is that when you set routing mark for a packet, and there’s no route in given routing table, the main one is searched instead.
Route distance matters when you have more than one route to given destination. If you mark routing for all outgoing connections, you’re telling router to use different routing tables with only one route in each - router can’t pick any other route.

dgnevans · February 15, 2017, 12:11pm

Have a look at his https://www.youtube.com/watch?v=67Dna_ffCvc&list=PLt3aE2eGS5P99VGblqX2SGrkpRq4qn5nb&index=4
also http://gregsowell.com/?p=2395

You need to ensure that traffic coming in that is new is marked so it goes out the same connection. You need to have the routes in place to ensure that marked traffic goes out correct interface. Then you need to ensure you have dstnat for for each wan interface to your internal servers.

Here is a copy of my Nat, mangle and ip routes to give idea. Note I am not doing PCC on this 1 wan is purely failover, but all services accesible though both wan connections

ip firewall nat
add action=masquerade chain=srcnat comment="Masq WAN 1" dst-address=0.0.0.0/0 out-interface=vlan2459 src-address=172.17.0.0/16
add action=masquerade chain=srcnat comment="Masq WAN 2" dst-address=0.0.0.0/0 out-interface=vlan12 src-address=172.17.0.0/16
add action=dst-nat chain=dstnat comment="SMTP Server WAN1" dst-address=154.xxx.xxx.xxx dst-address-type=local dst-port=25 protocol=tcp to-addresses=\
    172.17.0.126 to-ports=25
add action=dst-nat chain=dstnat comment="SMTP Server WAN2" dst-address=66.xxx.xxx.xxx dst-address-type=local dst-port=25 protocol=tcp to-addresses=\
    172.17.0.126 to-ports=25
add action=dst-nat chain=dstnat comment="Autodiscovery WAN1" dst-address=154.xxx.xxx.xxx dst-address-type=local dst-port=80 protocol=tcp to-addresses=\
    172.17.0.126 to-ports=25
add action=dst-nat chain=dstnat comment="Autodiscovery WAN2" dst-address=66.xxx.xxx.xxx dst-address-type=local dst-port=80 protocol=tcp to-addresses=\
    172.17.0.126 to-ports=25
add action=dst-nat chain=dstnat comment="Outlook Anywhere WAN1" dst-address=154.xxx.xxx.xxx dst-address-type=local dst-port=443 protocol=tcp \
    to-addresses=172.17.0.126 to-ports=443
add action=dst-nat chain=dstnat comment="Outlook Anywhere WAN2" dst-address=66.xxx.xxx.xxx dst-address-type=local dst-port=443 protocol=tcp to-addresses=\
    172.17.0.126 to-ports=443
add action=dst-nat chain=dstnat comment="Directory Services WAN1" dst-address=154.xxx.xxx.xxx dst-address-type=local dst-port=3268 protocol=tcp \
    to-addresses=172.17.0.125 to-ports=3268
add action=dst-nat chain=dstnat comment="Directory Services WAN2" dst-address=66.xxx.xxx.xxx dst-address-type=local dst-port=3268 protocol=tcp \
    to-addresses=172.17.0.125 to-ports=3268 
ip firewall mangle
add action=mark-routing chain=prerouting comment=" pfw WAN1, out WAN1" connection-mark=WAN1_pfw in-interface=vlan2 new-routing-mark=WAN1_traffic \
    passthrough=no
add action=mark-routing chain=prerouting comment=" pfw WAN2, out WAN2" connection-mark=WAN2_pfw in-interface=vlan2 new-routing-mark=WAN2_traffic \
    passthrough=no
add action=mark-connection chain=input comment=" in WAN1,out WAN1" in-interface=vlan2459 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=input comment=" in WAN2,out WAN2" in-interface=vlan12 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=forward comment=" pfw WAN1, out WAN1" connection-state=new in-interface=vlan2459 new-connection-mark=WAN1_pfw \
    passthrough=yes
add action=mark-connection chain=forward comment=" pfw WAN2, out WAN2" connection-state=new in-interface=vlan12 new-connection-mark=WAN2_pfw passthrough=\
    yes
add action=mark-routing chain=output comment=" in WAN1,out WAN1" connection-mark=WAN1_conn new-routing-mark=WAN1_traffic passthrough=no
add action=mark-routing chain=output comment=" in WAN2,out WAN2" connection-mark=WAN2_conn new-routing-mark=WAN2_traffic passthrough=no

ip route

add check-gateway=ping comment="In WAN1 out WAN1" distance=1 gateway=154.xxx.xxx.xxy routing-mark=WAN1_traffic
add check-gateway=ping comment="In WAN2 out WAN2" distance=1 gateway=66.xxx.xxx.xxy routing-mark=WAN2_traffic
add check-gateway=ping comment="Primary out WAN1" distance=1 gateway=154.xxx.xxx.xxy
add check-gateway=ping comment="Secondary out WAN2" distance=2 gateway=66.xxx.xxx.xxy