We have a link from 1 ISP running BGP and we have advertised 4 x /24 over this ISP. Now we are adding a second ISP and need to multihome with the following conditions. Both ISP links will be used at the same time. It will not be a failover configuration, but a load-balancing one. I have read through the documentation available in the wiki and forums regarding BGP Multihoming but I am still missing a few crucial pieces.
a) ISP A is connected and BGP is working fine. Our prefixes are being advertised without a hitch. An IP from our IP pool is assigned as a loopback IP on the core router doing BGP.
b) ISP B is connected and the BGP peer is configured and enabled. The peer gets established and our prefix becomes unreachable.
c) My prefixes show up in my BGP advertisements and both ISP A and ISP B confirm that they are receiving the prefixes via my peer.
I check the routes and see that in the routing table, the default route for ISP A is showing as active and that for ISP B is inactive. I change the distance of the ISP A route to 2 so the ISP B route becomes active and our prefix becomes reachable again. I have checked using traceroute from outside the network and the packet traverses ISP B to reach my router so no issues there. My queries are
Why do my advertised prefixes lose connectivity to the outside as soon as I enable the BGP peer for ISP B?
If only one ISP remains reachable and operational at any given point in time, how is it possible to do load sharing using something like nth or PCC?
Also, make sure you are either forcing a change of the next hop as the routes are learned into your AS or you will need to advertise the public /30 addresses you peer with your ISPs on so that it is reachable on the iBGP peering between A and B
Both ISP A and ISP B are connected to one router at my NOC and they are my upstream ISPs so they are different ASNs and hence are not connected to each other via iBGP.
The /30 ip address blocks to connect to both the ISPs are provided to me by the ISPs
Do you have a “strict” reverse path filter in /ip settings?
Have both your upstream providers made sure that their upstreams in turn have sorted out BGP prefix filters, ACLs, etc?
A clue of AS number and/or the prefixes you’re announcing would help a bit - or you could use a tool like BGPlay to see how far the announcements are getting via each of your upstreams. And then have a look at a tool like RIPE’s Atlas to do a distributed traceroute from a few hundred devices to see whether you’re being filtered (by accident) somewhere.
The load balancing scenario that you’re looking can be accomplish using those prepends (recommended by ahmadzai), but it can be done by publishing a most specific prefix by one provider and a summarized version of that by the other (as example: 1.1.0.0/24 advertised to ISP A / 1.1.1.0/24 advertised to ISP B / 1.1.0.0/23 advertised to both service providers); the interesting part of it is that you’ll be able to balance the download traffic, but the upload traffic (which can be traduced at the default-route) will not be able to do the same load balancing scenario.
If you want to do an upload load-balancing scenario, you’ll need two full routing tables or a default route / partial-route table at your edge borders; also an important advice if you want to deploy this scenario (do not use the rule drop invalid-tcp-connections at the forward/input firewall rule).
This is the answer you are looking for. You can’t install defaults from both ISPs at the same time. If you pull full tables, your router will send traffic to the provider closest (smallest AS path) to your traffic’s destination. This will not be true load balancing, but will keep traffic flowing out both providers.
If your router will not handle full tables from both providers, you can accept default routes (weighted to the provider with more bandwidth if they are not matched) and then filter out anything further than so many AS’s. Just an FYI: at an AS Path length of 0-5, our table was around 5,500 routes from both of our two providers and can take your router some time to dig through and fully establish.