What we are doing is using a Ubiquity AirFiber to get our secondary fiber connection from a building about .5km away. To make this simplier I’m showing both Mikrotik routers at the same location.
I am trying to configure BGP with Level3 & TW using my own IP and AS from ARIN. We currently have it up and working but we are having design problems with handling router (hardware) failures and just simply handling an ISP going down.
This is a Active/Standby setup I am trying to achieve and looking for help on the design for the following:
My firewall WAN gateway is 192.48.x.1 which is the first public IP in my /24. This is configured on the Mikrotik with VRRP. This way my firewalls do not have to do any routing changes
The router on-site (left in diagram) is the Master. In the event of this hardware going out the VRRP should go to the Standby router and go out the TWTelecom ISP (bgp will fail over due to hardware going out)
Now the problem I have is when I shut off the Level3 interface. The VRRP IP still remains on the Master device but the master device still tries to send it out the Level3 ISP. What I guess i’m looking for is it to be able to send it to the standby unit over the iBGP link and out the TWTelecom side which is the only active peer at this point (because Level3 is down).
Other than the routes received from Level 3, what other routes are present on that left hand router? I’m asking because from your description it sounded as if that router is still passing traffic to Level 3 in the absence of a Level 3 BGP session.
I suggest taking this as two issues - getting the routing resolved and then looking at how you want to use VRRP once the routing is clear.
You can use Netwatch script to disable VRRP interface when ISP is down. Then You can remove iBGP cross link. This solution isn’t perfect, but should works.
If your firewalls support OSPF, then you should use that for redundancy, not VRRP.
It will give you much more fault tolerance than just hardware failure protection (which is what VRRP mainly does)
If you use OSPF to conditionally originate default-information from the two BGP routers, then the firewall(s) will have both routers as default GW candidates in its OSPF. If R1 fails, ospf will remove it from the routing table and R2 will become the default GW. IF your BGP peers are not sending a default route to you, then you will have to create a recursive static default GW to something like 8.8.8.8 so that OSPF will see a default route to use for its “distribute-default=if-installed” condition.
Finally, tune the default-route metrics so that the firewall will primarily use the router who has more acive outbound routes - this is mostly an efficiency tuning step because a router won’t advertise default into OSPF it it can’t get out to the Internet.
Another important thing that this design fixes: if the inward-facing links on R1 are down but the Internet connection is up, R1 will withdraw its BGP announcements so the Internet will stop trying to come to your network on R1.
To make this last item work, you should use the firewall(s) to originate your master network prefixes that BGP will use to originate. Remove the static blackhole route(s) on R1/R2 so that the BGP advertisements are no longer “nailed up” locally.
So your routing table will look something like this under normal conditions: