I have two routers placed at remote locations, both places have 2 ISP available. What I would like to do is to connect both ISP in each unit, then set up VPNs between units, and then I need to create some routing and mangle rules to set these routers to use links the best way. Behind each box there'll be small network with NATed Internet access.

What I want to accomplish, is to have at each box one link to be primarily loaded with Internet traffic (general Internet access), and the other link to be primary used for VPN traffic (so, each link used for one primarily aim and so VPN won't be affected by Internet consumption of user behind box). I also try to create fault-tolerance so if one of these two links goes down then its traffic will go over second (alive) link.

I think I need to establish 4 VPNs (box1-isp1 <-> box2-isp1, box1-isp1 <-> box2-isp2, box1-isp2 <-> box2-isp1, box1-isp2 <-> box2isp2) - and yet I don't know how to "bind" VPN clients to given source interface - and then set up some rules so it'll be loaded one by one as links become down in a case ISPs become offline. And this is where my problem starts: I need to send traffic on source- and destination-based manner to the way I need.

May I please ask you for an advice, which way would you implement such a scheme?