We've two internet connection on diferent ISPs and we're load balancing these connections using ECMP. Both have dst-natted ports
to internal servers (ie: web, smtp).
ISP1 ---> | MK RB 532 | ---> LAN (Servers, Workstations).
ISP2 ---> | |
Yesterday we've noticed that if we ping (from far far away on the Internet) an ISP2 address, the echo reply *may* go through ISP1 (it's up to what the ECMP route wants). Fortunately both ISP1 and ISP2 allow to send packets whose source address is
outside the address space they gave us and this works, even when it's wrongly configured.
Our goal is to make sure that packets that arrived from ISP1, go back to ISP1 and vice-versa.
1) route-mark the packets in mangle/prerouting
2) use a default gateway route to ISP1 for packets marked as comingo from ISP1 and vice-versa.
Didn't work because it tried to send packets destined to our LAN to ISP1. A solution would be to duplicate the entire routing table, but i think is quite messy.
We also tried this with policy routing, with the same results.