Help with NAT and Mangle - PCC on 3 ISPs & 7 IPs

munitech · May 6, 2016, 9:35pm

Hey all,

Per the forum choice, I’m new to RouterOS, and working on replacing our old load balancer/firewall/router/coffee maker with a CCR-1036. I’ve read, and tested, and worked on this, but with my PCC mangle rules enabled, I’m losing traffic - slow and spotty to establish connections. Additionally, struggling to get DST-NAT traffic to respond via appropriate gateway

Network looks vaguely like:
Ether1 - Connection to independent Fibre Carrier’s ethernet handoff. Traffic is tagged:

VLAN2 - “L3VPN” to ISP1 (1.1.1.234-238 GW 1.1.1.233) - 5mbps bandwidth
VLAN3 - “L3VPN” to ISP2 (2.2.2.249 GW 2.2.2.250) - 20mbps bandwidth
VLAN4 - “L3VPN” to our rather large SCADA network | Bridged to VLAN4 on Ether12 (192.168.111.226, GW 111.225, local subnet is 224/28)
VLAN5 - Future L3VPN to a remote site, will only communicate with local SCADA network (192.168.98.6 GW 98.5)

Ether 2 - ISP3 (3.3.120.3 [Static IP given through DHCP]) - 2.5mbps bandwidth
Ether 12 - LAN, split into varying subnets, self-explanatory (see config). Guest VLAN traffic filtered from rest of network.

I’m trying to implement PCC per the Wiki, as well as recursive routing based failover without scripting. Relevant configuration below. I hope. I thought the configuration was solid before I deployed it, but I can’t get traffic to go out reliably unless I disable all the mangle rules, but even then NAT’d traffic isn’t being returned if I don’t have the default gateway of that IP active.

Code is a wee bit big, tried to filter what I could. Sorry. Any help would be vastly appreciated.

may/04/2016 14:52:01 by RouterOS 6.30.4

/interface bridge
add name=br-SCADA
/interface ethernet
set [ find default-name=ether1 ] name=e1-FibreCarrier
set [ find default-name=ether2 ] name=e2-ISP3
set [ find default-name=ether12 ] name=e12-LAN
/interface vlan
add interface=e1-FibreCarrier l2mtu=1576 name=v2e1-ISP1 vlan-id=2
add interface=e1-FibreCarrier l2mtu=1576 name=v3e1-ISP2 vlan-id=3
add interface=e1-FibreCarrier l2mtu=1576 name=v4e1-SCADA vlan-id=4
add interface=e12-LAN l2mtu=1576 name=v4e12-SCADA vlan-id=4
add interface=e1-FibreCarrier l2mtu=1576 name=v5e1-RemoteOffice vlan-id=5
add interface=e12-LAN l2mtu=1576 name=v10e12-Backbone vlan-id=10
add interface=e12-LAN l2mtu=1576 name=v20e12-Servers vlan-id=20
add interface=e12-LAN l2mtu=1576 name=v30e12-Printers vlan-id=30
add interface=e12-LAN l2mtu=1576 name=v50e12-VoIP vlan-id=50
add interface=e12-LAN l2mtu=1576 name=v60e12-Guest vlan-id=60
add interface=e12-LAN l2mtu=1576 name=v70e12-Workstations vlan-id=70
/interface bridge port
add bridge=br-SCADA interface=v4e1-SCADA
add bridge=br-SCADA interface=v4e12-SCADA
/ip address
add address=1.1.1.234/29 interface=v2e1-ISP1 network=1.1.1.232
add address=1.1.1.235/29 interface=v2e1-ISP1 network=1.1.1.232
add address=1.1.1.236/29 interface=v2e1-ISP1 network=1.1.1.232
add address=1.1.1.237/29 interface=v2e1-ISP1 network=1.1.1.232
add address=1.1.1.238/29 interface=v2e1-ISP1 network=1.1.1.232
add address=192.168.98.6/30 interface=v5e1-RemoteOffice network=192.168.98.4
add address=2.2.2.249/30 interface=v3e1-ISP2 network=2.2.2.248
add address=192.168.111.226/28 interface=br-SCADA network=192.168.111.224
add address=192.168.0.254/24 interface=e12-LAN network=192.168.0.0
add address=10.10.20.254/24 interface=v20e12-Servers network=10.10.20.0
add address=10.10.60.254/24 interface=v60e12-Guest network=10.10.60.0
add address=10.10.70.254/24 interface=v70e12-Workstations network=10.10.70.0
add address=192.168.1.254 disabled=yes interface=v50e12-VoIP network=**
192.168.1.254
add address=10.10.10.1/24 interface=v10e12-Backbone network=10.10.10.0
add address=10.10.30.254/24 interface=v30e12-Printers network=10.10.30.0
/ip dhcp-client
add add-default-route=no dhcp-options=hostname,clientid disabled=no *
interface=e2-ISP3 use-peer-dns=no use-peer-ntp=no
/ip firewall filter
add chain=forward comment=“Allow Inbound NAT” connection-nat-state=dstnat
add chain=input protocol=icmp
add action=jump chain=forward comment=“Allow packets from established connections” connection-state=established,related jump-target=Established
add chain=Established in-interface=v2e1-ISP1
add chain=Established in-interface=v3e1-ISP2
add chain=Established in-interface=e2-ISP3
add chain=Established in-interface=v60e12-Guest
add action=jump chain=forward comment=“Alow guest to WAN only” in-interface=v60e12-Guest jump-target=v60-Guest
add chain=v60-Guest out-interface=v2e1-ISP1
add chain=v60-Guest out-interface=v3e1-ISP2
add chain=v60-Guest out-interface=e2-ISP3
add chain=forward comment=“Allow LAN communications” in-interface=e12-LAN
add chain=forward in-interface=v20e12-Servers
add chain=forward in-interface=v30e12-Printers
add chain=forward in-interface=v50e12-VoIP
add chain=forward in-interface=v70e12-Workstations
add action=jump chain=forward comment=“Allow SCADA to WAN and RemoteOffice” in-interface=br-SCADA jump-target=SCADA
add chain=SCADA out-interface=v2e1-ISP1
add chain=SCADA out-interface=v3e1-ISP2
add chain=SCADA out-interface=e2-ISP3
add chain=SCADA out-interface=v5e1-RemoteOffice
add chain=forward in-interface=v5e1-RemoteOffice out-interface=br-SCADA
add action=drop chain=input comment=“Drop all else”
add action=drop chain=forward
/ip firewall mangle
add action=jump chain=prerouting comment=“Strictly define routing for directly connected networks” *
in-interface=e12-LAN jump-target=Connected
add action=jump chain=prerouting in-interface=v10e12-Backbone jump-target=**
Connected
add action=jump chain=prerouting in-interface=v20e12-Servers jump-target=**
Connected
add action=jump chain=prerouting in-interface=v30e12-Printers jump-target=**
Connected
add action=jump chain=prerouting in-interface=v50e12-VoIP jump-target=**
Connected
add action=jump chain=prerouting in-interface=v60e12-Guest jump-target=**
Connected
add action=jump chain=prerouting in-interface=v70e12-Workstations *
jump-target=Connected
add chain=Connected dst-address=1.1.1.232/29
add chain=Connected dst-address=2.2.2.248/30
add chain=Connected dst-address=3.3.120.0/21
add chain=Connected dst-address=192.168.0.0/24
add chain=Connected dst-address=10.10.0.0/16
add action=mark-connection chain=prerouting comment=**
“Tag Inbound to return through same link” connection-mark=no-mark *
in-interface=v2e1-ISP1 new-connection-mark=v3e1-ISP2_conn
add action=mark-connection chain=prerouting connection-mark=no-mark *
in-interface=v3e1-ISP2 new-connection-mark=v3e1-ISP2_conn
add action=mark-connection chain=prerouting connection-mark=no-mark *
in-interface=e2-ISP3 new-connection-mark=e2-ISP3_conn
add action=jump chain=prerouting comment=“Jump to add connection mark” *
connection-mark=no-mark dst-address-type=!local in-interface=e12-LAN *
jump-target=AddConnectionMark
add action=jump chain=prerouting connection-mark=no-mark dst-address-type=*
!local in-interface=v20e12-Servers jump-target=AddConnectionMark
add action=jump chain=prerouting connection-mark=no-mark dst-address-type=**
!local in-interface=v50e12-VoIP jump-target=AddConnectionMark
add action=jump chain=prerouting connection-mark=no-mark dst-address-type=**
!local in-interface=v60e12-Guest jump-target=AddConnectionMark
add action=jump chain=prerouting connection-mark=no-mark dst-address-type=**
!local in-interface=v70e12-Workstations jump-target=AddConnectionMark
add action=jump chain=prerouting connection-mark=no-mark dst-address-type=**
!local in-interface=v30e12-Printers jump-target=AddConnectionMark
add action=mark-connection chain=AddConnectionMark comment=**
“Add Connection & Routing Mark” new-connection-mark=v3e1-ISP2_conn *
per-connection-classifier=both-addresses:11/0
add action=mark-connection chain=AddConnectionMark new-connection-mark=**
v3e1-ISP2_conn per-connection-classifier=both-addresses:11/1
add action=mark-connection chain=AddConnectionMark new-connection-mark=**
v3e1-ISP2_conn per-connection-classifier=both-addresses:11/2
add action=mark-connection chain=AddConnectionMark new-connection-mark=**
v3e1-ISP2_conn per-connection-classifier=both-addresses:11/3
add action=mark-connection chain=AddConnectionMark new-connection-mark=**
v3e1-ISP2_conn per-connection-classifier=both-addresses:11/4
add action=mark-connection chain=AddConnectionMark new-connection-mark=**
v3e1-ISP2_conn per-connection-classifier=[color=black]both-addresses[/color][color=#B4B80A]:[/color][color=black]11/5[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]mark-connection [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]AddConnectionMark [/color][color=green]new-connection-mark[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] v3e1-ISP2_conn per-connection-classifier[/color][color=#B4B80A]=[/color][color=black]both-addresses[/color][color=#B4B80A]:[/color][color=black]11/6[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]mark-connection [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]AddConnectionMark [/color][color=green]new-connection-mark[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] v3e1-ISP2_conn per-connection-classifier[/color][color=#B4B80A]=[/color][color=black]both-addresses[/color][color=#B4B80A]:[/color][color=black]11/7[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]mark-connection [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]AddConnectionMark [/color][color=green]new-connection-mark[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] v2e1-ISP1_conn per-connection-classifier[/color][color=#B4B80A]=[/color][color=black]both-addresses[/color][color=#B4B80A]:[/color][color=black]11/8[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]mark-connection [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]AddConnectionMark [/color][color=green]new-connection-mark[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] v2e1-ISP1_conn per-connection-classifier[/color][color=#B4B80A]=[/color][color=black]both-addresses[/color][color=#B4B80A]:[/color][color=black]11/9[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]mark-connection [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]AddConnectionMark [/color][color=green]new-connection-mark[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] e2-ISP3_conn per-connection-classifier[/color][color=#B4B80A]=[/color][color=black]both-addresses[/color][color=#B4B80A]:[/color][color=black]11/10[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]mark-routing [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]AddConnectionMark [/color][color=green]connection-mark[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] v2e1-ISP1_conn [/color][color=green]disabled[/color][color=#B4B80A]=[/color][color=black]yes [/color][color=green]new-routing-mark[/color][color=#B4B80A]=[/color][color=black]to_v2e1-ISP1[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]mark-routing [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]AddConnectionMark [/color][color=green]connection-mark[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] v3e1-ISP2_conn [/color][color=green]disabled[/color][color=#B4B80A]=[/color][color=black]yes [/color][color=green]new-routing-mark[/color][color=#B4B80A]=[/color][color=black]to_v3e1-ISP2[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]mark-routing [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]AddConnectionMark [/color][color=green]connection-mark[/color][color=#B4B80A]=[/color][color=black]e2-ISP3_conn [/color][color=blue][/color]
[color=green]disabled[/color][color=#B4B80A]=[/color][color=black]yes [/color][color=green]new-routing-mark[/color][color=#B4B80A]=[/color][color=black]to_e2-ISP3[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]mark-routing [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]output [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=black]“Balance router’s traffic” [/color][color=blue][/color]
[color=green]connection-mark[/color][color=#B4B80A]=[/color][color=black]v2e1-ISP1_conn [/color][color=green]disabled[/color][color=#B4B80A]=[/color][color=black]yes [/color][color=green]new-routing-mark[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] to_v2e1-ISP1[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]mark-routing [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]output [/color][color=green]connection-mark[/color][color=#B4B80A]=[/color][color=black]v3e1-ISP2_conn [/color][color=blue][/color]
[color=green]disabled[/color][color=#B4B80A]=[/color][color=black]yes [/color][color=green]new-routing-mark[/color][color=#B4B80A]=[/color][color=black]to_v3e1-ISP2[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]mark-routing [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]output [/color][color=green]connection-mark[/color][color=#B4B80A]=[/color][color=black]e2-ISP3_conn [/color][color=green]disabled[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] yes [/color][color=green]new-routing-mark[/color][color=#B4B80A]=[/color][color=black]to_e2-ISP3[/color]
[color=#0080FF]/ip [/color][color=#0080FF]firewall[/color] [color=#0080FF]nat[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]jump [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]dstnat [/color][color=green]in-interface[/color][color=#B4B80A]=[/color][color=black]v2e1-ISP1 [/color][color=green]jump-target[/color][color=#B4B80A]=[/color][color=black]Inbound[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]jump [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]dstnat [/color][color=green]in-interface[/color][color=#B4B80A]=[/color][color=black]v3e1-ISP2 [/color][color=green]jump-target[/color][color=#B4B80A]=[/color][color=black]Inbound[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]jump [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]dstnat [/color][color=green]in-interface[/color][color=#B4B80A]=[/color][color=black]e2-ISP3 [/color][color=green]jump-target[/color][color=#B4B80A]=[/color][color=black]Inbound[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]jump [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]dstnat [/color][color=green]in-interface[/color][color=#B4B80A]=[/color][color=black]v60e12-Guest [/color][color=green]jump-target[/color][color=#B4B80A]=[/color][color=black]Inbound[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]dst-nat [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]Inbound [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=black]“DC HTTP(S)” [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] 1.1.1.234 [/color][color=green]dst-port[/color][color=#B4B80A]=[/color][color=black]80[/color][color=#B4B80A],[/color][color=black]443 [/color][color=green]protocol[/color][color=#B4B80A]=[/color][color=black]tcp [/color][color=green]to-addresses[/color][color=#B4B80A]=[/color][color=black]192.168.0.7[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]dst-nat [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]Inbound [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=black]“RDS HTTP(S)” [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] 1.1.1.235 [/color][color=green]dst-port[/color][color=#B4B80A]=[/color][color=black]80[/color][color=#B4B80A],[/color][color=black]443 [/color][color=green]protocol[/color][color=#B4B80A]=[/color][color=black]tcp [/color][color=green]to-addresses[/color][color=#B4B80A]=[/color][color=black]10.10.20.13[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]dst-nat [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]Inbound [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=black]“Mail HTTP(S)” [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] 1.1.1.236 [/color][color=green]dst-port[/color][color=#B4B80A]=[/color][color=black]80[/color][color=#B4B80A],[/color][color=black]443 [/color][color=green]protocol[/color][color=#B4B80A]=[/color][color=black]tcp [/color][color=green]to-addresses[/color][color=#B4B80A]=[/color][color=black]10.10.20.11[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]dst-nat [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]Inbound [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=black]“Mail HTTP(S)” [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] 3.3.126.215 [/color][color=green]dst-port[/color][color=#B4B80A]=[/color][color=black]80[/color][color=#B4B80A],[/color][color=black]443 [/color][color=green]protocol[/color][color=#B4B80A]=[/color][color=black]tcp [/color][color=green]to-addresses[/color][color=#B4B80A]=[/color][color=black]10.10.20.11[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]dst-nat [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]Inbound [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=black]“IIS HTTP(S)” [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] 2.2.2.249 [/color][color=green]dst-port[/color][color=#B4B80A]=[/color][color=black]80[/color][color=#B4B80A],[/color][color=black]443 [/color][color=green]protocol[/color][color=#B4B80A]=[/color][color=black]tcp [/color][color=green]to-addresses[/color][color=#B4B80A]=[/color][color=black]10.10.20.14[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]dst-nat [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]Inbound [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=black]“Mail SMTP” [/color][color=green]dst-port[/color][color=#B4B80A]=[/color][color=black]25 [/color][color=green]protocol[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] tcp [/color][color=green]to-addresses[/color][color=#B4B80A]=[/color][color=black]10.10.20.11[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]dst-nat [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]Inbound [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=black]Spiceworks [/color][color=green]dst-port[/color][color=#B4B80A]=[/color][color=black]9876-9877 [/color][color=blue][/color]
[color=green]protocol[/color][color=#B4B80A]=[/color][color=black]tcp [/color][color=green]to-addresses[/color][color=#B4B80A]=[/color][color=black]192.168.0.9[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]dst-nat [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]Inbound [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=#0080FF]GPS[/color] [color=green]dst-port[/color][color=#B4B80A]=[/color][color=black]60366 [/color][color=green]protocol[/color][color=#B4B80A]=[/color][color=black]tcp [/color][color=blue][/color]
[color=green]to-addresses[/color][color=#B4B80A]=[/color][color=black]10.10.20.16[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]dst-nat [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]Inbound [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=black]Access Control [/color][color=green]dst-port[/color][color=#B4B80A]=[/color][color=black]18802[/color][color=#B4B80A],[/color][color=black]8801-8802 [/color][color=blue][/color]
[color=green]protocol[/color][color=#B4B80A]=[/color][color=black]tcp [/color][color=green]to-addresses[/color][color=#B4B80A]=[/color][color=black]10.10.20.12[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]dst-nat [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]Inbound [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=black]Access Control [/color][color=green]dst-port[/color][color=#B4B80A]=[/color][color=black]18001[/color][color=#B4B80A],[/color][color=black]18801-18803 [/color][color=blue][/color]
[color=green]protocol[/color][color=#B4B80A]=[/color][color=black]udp [/color][color=green]to-addresses[/color][color=#B4B80A]=[/color][color=black]10.10.20.12[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]dst-nat [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]Inbound [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=black]ERP/Timesheets [/color][color=green]dst-port[/color][color=#B4B80A]=[/color][color=black]500-502[/color][color=#B4B80A],[/color][color=black]48620 [/color][color=blue][/color]
[color=green]protocol[/color][color=#B4B80A]=[/color][color=black]tcp [/color][color=green]to-addresses[/color][color=#B4B80A]=[/color][color=black]192.168.0.109[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]dst-nat [/color][color=green]chain[/color][color=#B4B80A]=[/color][color=black]Inbound [/color][color=green]comment[/color][color=#B4B80A]=[/color][color=#0080FF]GPS[/color] [color=green]dst-port[/color][color=#B4B80A]=[/color][color=black]60366[/color][color=#B4B80A],[/color][color=black]51001 [/color][color=green]protocol[/color][color=#B4B80A]=[/color][color=blue][/color]
[color=black] udp [/color][color=green]to-addresses[/color][color=#B4B80A]=[/color][color=black]10.10.20.16[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]masquerade chain=srcnat [/color][color=green]out-interface[/color][color=#B4B80A]=[/color][color=black]v3e1-ISP2[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]masquerade chain=srcnat [/color][color=green]out-interface[/color][color=#B4B80A]=[/color][color=black]v2e1-ISP1[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]masquerade chain=srcnat [/color][color=green]out-interface[/color][color=#B4B80A]=[/color][color=black]e2-ISP3[/color]
[color=#B9005C]add[/color] [color=green]action[/color][color=#B4B80A]=[/color][color=black]masquerade chain=srcnat [/color][color=green]out-interface[/color][color=#B4B80A]=[/color][color=black]v60e12-Guest[/color]
[color=#0080FF]/ip [/color][color=#0080FF]route[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]192.168.1.0/24 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]192.168.111.225 [/color][color=blue][/color]
[color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_SCADA[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]192.168.99.0/24 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]192.168.111.225 [/color][color=blue][/color]
[color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_SCADA[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]192.168.100.0/24 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]192.168.111.225 [/color][color=blue][/color]
[color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_SCADA[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]192.168.104.0/24 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]192.168.111.225 [/color][color=blue][/color]
[color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_SCADA[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]192.168.110.0/24 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]192.168.111.225 [/color][color=blue][/color]
[color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_SCADA[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]192.168.111.0/24 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]192.168.111.225 [/color][color=blue][/color]
[color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_SCADA[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]192.168.112.0/24 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]192.168.111.225 [/color][color=blue][/color]
[color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_SCADA[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]192.168.112.224/28 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]192.168.98.5 [/color][color=blue][/color]
[color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_SCADA[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]192.168.113.0/24 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]192.168.111.225 [/color][color=blue][/color]
[color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_SCADA[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]192.168.114.0/24 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]192.168.111.225 [/color][color=blue][/color]
[color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_SCADA[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]172.31.255.2 [/color][color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_v2e1-ISP1[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]2 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]172.31.255.1 [/color][color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_v2e1-ISP1[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]3 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]172.31.255.3 [/color][color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_v2e1-ISP1[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]172.31.255.1 [/color][color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_v3e1-ISP2[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]2 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]172.31.255.2 [/color][color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_v3e1-ISP2[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]3 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]172.31.255.3 [/color][color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_v3e1-ISP2[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]172.31.255.3 [/color][color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_e2-ISP3[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]2 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]172.31.255.1 [/color][color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_e2-ISP3[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]3 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]172.31.255.2 [/color][color=green]routing-mark[/color][color=#B4B80A]=[/color][color=black]to_e2-ISP3[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]172.31.255.1[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]2 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]172.31.255.2[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]3 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]172.31.255.3[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]4.2.2.1/32 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]2.2.2.250 [/color][color=green]scope[/color][color=#B4B80A]=[/color][color=black]10[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]4.2.2.2/32 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]2.2.2.250 [/color][color=green]scope[/color][color=#B4B80A]=[/color][color=black]10[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]4.2.2.3/32 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]1.1.1.233 [/color][color=green]scope[/color][color=#B4B80A]=[/color][color=black]10[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]4.2.2.4/32 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]1.1.1.233 [/color][color=green]scope[/color][color=#B4B80A]=[/color][color=black]10[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]4.2.2.5/32 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]3.3.120.1 [/color][color=green]scope[/color][color=#B4B80A]=[/color][color=black]10[/color]
[color=#B9005C]add[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]4.2.2.6/32 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]3.3.120.1 [/color][color=green]scope[/color][color=#B4B80A]=[/color][color=black]10[/color]
[color=#B9005C]add[/color][color=black] check-gateway[/color][color=#B4B80A]=[/color][color=#B9005C]ping[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]172.31.255.1/32 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]4.2.2.1 [/color][color=blue][/color]
[color=green]scope[/color][color=#B4B80A]=[/color][color=black]10[/color]
[color=#B9005C]add[/color][color=black] check-gateway[/color][color=#B4B80A]=[/color][color=#B9005C]ping[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]2 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]172.31.255.1/32 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]4.2.2.2 [/color][color=blue][/color]
[color=green]scope[/color][color=#B4B80A]=[/color][color=black]10[/color]
[color=#B9005C]add[/color][color=black] check-gateway[/color][color=#B4B80A]=[/color][color=#B9005C]ping[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]172.31.255.2/32 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]4.2.2.3 [/color][color=blue][/color]
[color=green]scope[/color][color=#B4B80A]=[/color][color=black]10[/color]
[color=#B9005C]add[/color][color=black] check-gateway[/color][color=#B4B80A]=[/color][color=#B9005C]ping[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]2 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]172.31.255.2/32 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]4.2.2.4 [/color][color=blue][/color]
[color=green]scope[/color][color=#B4B80A]=[/color][color=black]10[/color]
[color=#B9005C]add[/color][color=black] check-gateway[/color][color=#B4B80A]=[/color][color=#B9005C]ping[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]1 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]172.31.255.3/32 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]4.2.2.5 [/color][color=blue][/color]
[color=green]scope[/color][color=#B4B80A]=[/color][color=black]10[/color]
[color=#B9005C]add[/color][color=black] check-gateway[/color][color=#B4B80A]=[/color][color=#B9005C]ping[/color] [color=green]distance[/color][color=#B4B80A]=[/color][color=black]2 [/color][color=green]dst-address[/color][color=#B4B80A]=[/color][color=black]172.31.255.3/32 [/color][color=green]gateway[/color][color=#B4B80A]=[/color][color=black]4.2.2.6 [/color][color=blue][/color]
[color=green]scope[/color][color=#B4B80A]=[/color][color=black]10[/color]

munitech · May 9, 2016, 9:31pm

I should also note - I’m not sure if it’s an issue with my config or my switch, but my switch ended up blocking Eth12 due to STP. I disabled STP on the switch port for the time being.

munitech · May 20, 2016, 3:43pm

Not sure if the rules of the forum allow a bump, but I’m still looking for guidance on my mangle rules.

ZeroByte · May 20, 2016, 6:11pm

One thing I noticed:

I don’t know if this was an error introduced while you were sanitizing/formatting your post, or if it’s really configured that way in your box, but if so, that’s one issue…

I started going blind in the meat of the mangle rules, but I follow the basics of what you’re doing, and lots of people do their policy-routing exceptions the way you’ve done (lots of “accept” rules for targets that shouldn’t get policy-routed).

May I humbly suggest a different, and (in my opinion) simpler method:

Use the /ip route rules to make the policy exceptions, and just do big-picture stuff in the mangle table…

Basically, set up all of your routes in the main routing table - all of the SCADA stuff, etc.
Then set up some route rules as follows:

/ip route rule
add dst-address=192.168.110.0/23 action=lookup-only-in-table table=main
add dst-address=192.168.112.0/23 action=lookup-only-in-table table=main
add dst-address=192.168.114.0/24 action=lookup-only-in-table table=main
...etc

You can paint these ranges with a broad brush because all they do is specify which dst ranges should be looked up only in the main table - the main routing table can have all of the more-specific subnet routes that you like.

Also you need to add the ping test targets and force them to the main routing table
add dst-address=4.2.2.1/32 action=lookup-only-in-table table=main
add dst-address=4.2.2.2/32 action=lookup-only-in-table table=main
etc.

Then your mangling is a lot simpler and should basically look like this:
Prerouting:

connection-mark=no-mark : jump to “classify”
connection-mark=wan1 : mark-routing=wan1
connection-mark=wan2 : mark-routing=wan2
connection-mark=wan3 : mark-routing=wan3

Output:
(copy rules 2-4 from Prerouting)
It would be nice to jump into the same classify chain from Output, but you can’t because classify is all going to be based on in-interface which is a no-no in the output context. It’s not horribly important though, because if it’s to the LAN, then who cares (exempted by route rules) and if it’s to the WAN, just let the Mikrotik go out whatever interface, and then when the reply comes back, it’s going to come back in the same interface because it got masqueraded accordingly - and at that point, the classify chain will mark the connection properly anyway

Classify:
in-interface=wan1 → mark-connection=wan1
in-interface=wan2 & no-connection-mark → mark-connection=wan2
in-interface=wan3 & no-connection-mark → mark-connection=wan3
connection-mark=!no-mark → return
PCC rules here

Final changes:
add your locally-attached WAN IP ranges to the route rules exceptions as well.
Sort out the ping test logc:

/ip route
add distance=1 dst-address=4.2.2.1/32 gateway=2.2.2.250 scope=10
add distance=1 dst-address=4.2.2.2/32 gateway=2.2.2.250 scope=10
add distance=1 dst-address=4.2.2.3/32 gateway=1.1.1.233 scope=10
add distance=1 dst-address=4.2.2.4/32 gateway=1.1.1.233 scope=10
add distance=1 dst-address=4.2.2.5/32 gateway=3.3.120.1 scope=10
add distance=1 dst-address=4.2.2.6/32 gateway=3.3.120.1 scope=10
add distance=1 dst-address=4.2.2.1/32 type=blackhole scope=10 distance=2
add distance=1 dst-address=4.2.2.2/32 type=blackhole scope=10 distance=2
add distance=1 dst-address=4.2.2.3/32 type=blackhole scope=10 distance=2
add distance=1 dst-address=4.2.2.4/32 type=blackhole scope=10 distance=2
add distance=1 dst-address=4.2.2.5/32 type=blackhole scope=10 distance=2
add distance=1 dst-address=4.2.2.6/32 type=blackhole scope=10 distance=2

And make sure that all of your SCADA routes are in the main routing table.

Your meta-default-GW logic looks fine as it is. You just want to make sure the ping tests are black-holed if their specific interface goes down because if it starts answering pings on the wrong interface, you’ll get flapping routes.

munitech · May 24, 2016, 2:42pm

Thanks for the input. I’ll work on these changes then try and schedule a maintenance window later this week to implement.

The routing rules method looks a lot cleaner, which obviously this configuration could use some simplification. I’ll let you know how it goes.

I don’t know if this was an error introduced while you were sanitizing/formatting your post, or if it’s really configured that way in your box, but if so, that’s one issue…

Looked at the config pre-sanitation, and the issue is present there as well. Thanks for the good eye on that one. Would definitely explain the NAT behaviour, as the majority of my inbound is currently via ISP1.

munitech · May 27, 2016, 3:05pm

Well I got a chance for a maintenance window last night to implement, and everything seems to be working smoothly.

The Route Rules method is far easier to read, which makes me quite happy, and the NAT seems to be working fine. Since I originally posted, one of my ISP’s switch me from a /32 to a /28, so I’ve got quite a few more IPs as well, but I got dst and src NAT working in conjunction with the load balancing.

Speed tests show my WAN interfaces all being saturated simultaneously.

ZeroByte · May 27, 2016, 9:23pm

Glad you got it all tuned to your satisfaction. When I first saw how route rules work, I was impressed at how much simpler the mangle logic/routing tables get to be. Now I always recommend that over these “early exit” rules in mangle chains.

munitech · August 5, 2016, 3:39pm

Several months later, as the complexity of my network continues to grow - the simplicity of the routing rules has kept my config readable.

Just wanted to follow up and say this continues to be a huge help.