I have 3 backbone providers. I NAT all the clients to a public IP address, and some clients have their own src-nat set up to get their own public IP address. I am using ECMP in the routing table to balance out the connections across all the links. When it was just 1 provider I had zero issues with the clients who were routed to their own public IP but once I started using ECMP I began having issues.
With the top mangle rules it does force the clients with a public ip address to src-nat out the proper interface so they show up with the proper public ip address. The problem I have is trying to get back in to their equipment from behind the router with the rule enabled. Once I enable the top mangle rule I am unable to do so from behind the router, but I can do so just fine from outside my network. I have spend a lot of time trying to come up with a mangle solution that would allow back through one of dst-nat ip's to the clients router to no avail. Any help would be appreciated.
/ip firewall mangle
add action=mark-routing chain=prerouting comment=\
"Make sure our StaticIP customers go out over the proper connection." new-routing-mark=to_ether1 passthrough=\
add action=mark-connection chain=input connection-mark=no-mark in-interface="ether1 - Provider 1" log-prefix="" new-connection-mark=ether1_conn passthrough=no
add action=mark-connection chain=input connection-mark=no-mark in-interface="ether2 - Provider 2" log-prefix="" new-connection-mark=ether2_conn passthrough=no
add action=mark-connection chain=input connection-mark=no-mark in-interface="ether3 - Provider 3" new-connection-mark=ether3_conn passthrough=no
add action=mark-routing chain=output connection-mark=ether1_conn log-prefix="" new-routing-mark=to_ether1 passthrough=no
add action=mark-routing chain=output connection-mark=ether2_conn log-prefix="" new-routing-mark=to_ether2 passthrough=no
add action=mark-routing chain=output connection-mark=ether3_conn new-routing-mark=to_ether3 passthrough=no