We're a small ISP owning two /22 networks. Our upstream ISP provides us with two separate peering endpoints located in the same datacenter.
We've been running so far with one Edge CCR router which terminated both those peers via eBGP in a failover scenario (not loadbalance). From that edge, we have iBGP to all our secondary downstream routers advertising the default route.
We recently moved to a 2 x CCR2004-1G-12S+2XS routers acting as edges and we wanted to terminate each peer in a different router possibly loadbalancing them as well. Similarly, each edge has iBGP peering with each of our secondary downstream routers.
We attempted to try this setup, but only works reliably if only one edge is up and the other down. Having them both up at the same time results in loss of some hosted services, and we suspect that's because inbound & outbound traffic might get mixed up inbetween peers. Our downstream routers receive two default routes via iBGP as expected with equal distance, hence only one active at a time.
We also attempted an iBGP link between the two Edges with default originate ON, but that did not seem to work as expected and made no difference. We've read a few similar posts on the forums suggesting AS prepends, OSPF and local prefs, but we're not 100% sure how to proceed given the context.
We don't advertise anything else other than those two /22 upstream via "Networks" (no redistribute connected or any filters).
Any suggestions would be most welcome!