Hello !

I have an issue with my local setup using BGP and multiple uplink peers. So far I have two uplink ISP providers peering with BGP, one its the main ISP (ISP 1) which I'm connected through MPLS and the other (ISP 2) its a new provider which I prefer because I have a direct last mile with them so latency is a lot better than with ISP 1, However ISP 2 peers directly with ISP 1 and its their main provider, they will only use another upstream if their connection to ISP 1 fails.

So far I have a single BGP instance where I have two BGP peers and I have the following filters setup for ISP1 and ISP 2, IPv4 and IPv6 IN/OUT filters
However I have these issues and I'm looking for some recommendation to fix it:

1. When the two BGP peers are UP I have two 0.0.0.0/0 routes and ISP 1 its preferred because AS Path is smaller than ISP 2, but I need to force using ISP 2 as gateway for some of my customer segments

2. When the two BGP peers are UP I see some packets going out ISP 1 but came back using ISP 2, I know with static routing I can force a specific interface using routing marks, however I'm not sure how to do it using BGP

3. If one of the BGP peers goes down my core router remain accessible from the other BGP Peer, however my customers starts reporting they loose internet connection, I have the feeling its because the TCP connection remains established using the old route. I can't disable the Connection tracking on my border router because some of my customer segments are going out to internet through NAT


I'm attaching my network diagram, any help will be highly appreciated

However I have these issues and I'm looking for some recommendation to fix it:

1. When the two BGP peers are UP I have two 0.0.0.0/0 routes and ISP 1 its preferred because AS Path is smaller than ISP 2, but I need to force using ISP 2 as gateway for some of my customer segments

This is normal. If you need to direct traffic based on source address, you will need to use routing rules or some kind of mangle rules to do policy/source based routing into a 2nd route table where you do use the preferred route

2. When the two BGP peers are UP I see some packets going out ISP 1 but came back using ISP 2, I know with static routing I can force a specific interface using routing marks, however I'm not sure how to do it using BGP

This is just BGP. You can't necessarily decide the path a packet takes to get back to you, as its decided in each network along the way based on many factors. You can try engineer traffic by using more specific subnets, and/or AS prepending, but its not a 100% success rate all the time.

3. If one of the BGP peers goes down my core router remain accessible from the other BGP Peer, however my customers starts reporting they loose internet connection, I have the feeling its because the TCP connection remains established using the old route. I can't disable the Connection tracking on my border router because some of my customer segments are going out to internet through NAT

How long after the failure are people still reporting internet down? When your BGP link goes down, it can take time for the change to be reflected globally, so for a short time after the service went down a lot of networks globally might still be trying to route via that service. They'll eventually learn the alternate path, unless for some reason your upstream continues advertising for some reason. This would have to be diagnosed at point of outage to see where the stoppage is occurring.

1. Put NAT on a separate router - separation of network functions is key to good network design (https://stubarea51.net/2022/05/02/webin ... functions/)
2. Does your provider support BGP communities for traffic engineering? This is the best way to influence traffic and then you can use longest match, etc.

I agree with the separation of concerns.

Furthermore, with the symptoms you're describing, It sounds like you're NATting to a border interface on the router. If you're NATting to an IP that's advertised through both peers, i.e. to a public IP assigned to a loopback interface on the router, then when your BGP session times out, the incoming traffic should automatically switch to the other provider.

I personally route (blackhole) small subnets to my CGNAT router, then have SRCNAT rules that change a block of customers' IPs to one of the IPs within the blackholed subnet(s). That way, inbound traffic without a NAT session doesn't hit the router's input interface, but all outbound and related inbound traffic gets properly routed. The CGNAT's default gateway is the border router, and it doesn't know or care about the outside world's routes.

hello @jrest

well, running eBGP is more like playing *mind games* with your friends. no one knows what the other up to for traffic forwarding. ended up tweaking the MED : prepend, filters etc.

but, usually, *zero config eBGP* the winner path is that which upstream gave you their ip blocks, then you both having directly connected route - because the path should be having no policy at all.

for your question
1, pbr. vrf. etc. your choice you decide.

2. every dynamic routing protocol has their own hold on timer, before switching to the other path. which must match with your peers.

so, you have your answer for the customers. and your upstream sla should be reviewed.

3. +1 with the others, cgnat-ed as close to your customers. don't do that on your ebgp routers. core - distribution - access. don't touch those core and distribution if you don't have urgent need. try to stick with it, it will make your design and operation easier.

hth.

We are seeing that when multiple local prefs exist then ROS7 has 1 core running constantly 100%. I guess its evaluating the routes all the time, so needs to be optimized from MT side. We dont do load balancing due to that right now. We have 4 transits and 7 IXs with routeservers that we peer on.

1. Put NAT on a separate router - separation of network functions is key to good network design (https://stubarea51.net/2022/05/02/webin ... functions/)
2. Does your provider support BGP communities for traffic engineering? This is the best way to influence traffic and then you can use longest match, etc.

I agree with you, right now we are considering improving our network design and include new Border / Core routers, however I think this won't happen in less than six months due to we still need to cover our current operational expenses

I've been trying to agree to create BGP communities with our upstream providers but so far we haven't had so much luck due to we have less than 5Gbps in BGP traffic

I agree with the separation of concerns.

Furthermore, with the symptoms you're describing, It sounds like you're NATting to a border interface on the router. If you're NATting to an IP that's advertised through both peers, i.e. to a public IP assigned to a loopback interface on the router, then when your BGP session times out, the incoming traffic should automatically switch to the other provider.

I personally route (blackhole) small subnets to my CGNAT router, then have SRCNAT rules that change a block of customers' IPs to one of the IPs within the blackholed subnet(s). That way, inbound traffic without a NAT session doesn't hit the router's input interface, but all outbound and related inbound traffic gets properly routed. The CGNAT's default gateway is the border router, and it doesn't know or care about the outside world's routes.

I have two different scenarios going on here: Right now our leased IPv4 block has been blacklisted by a couple CDN providers like Imperva before it was assigned to us and we still don't have a NAT64 config in place to freely use our IPv6 blocks so sometimes we need to NAT customers through our upstream providers IPs (they assign a /29 to us). So I'm trying to figure out a way of having the src-nat in place and having the automatic failover when one of the BGP peers fails without having to manually go and change the NAT rules.

Our current NAT configuration looks like this:

Customer Segment 1 -> src-nat (172.16.XX.XX/30) ISP 1 IPv4 address
Customer Segment 2 -> src-nat (172.16.XX.YY/30) ISP 2 IPv4 address
Customer Segment 3 -> src-nat (206.1.XX.XX/24) Our leased IPv4 Block

Actually when one of our BGP goes down our IPv4 block doesn't loose connection to outside internet, our external monitoring system doesn't detect our 206.1.XX.XX/24 block being offline so BGP routing its working fine. I think part of the issue is having to Manually change the src-nat from one provider's IPv4 to another and I suspect when this is made Mikrotik still leaves the TCP connections established to the previous src-nat IP, when this happens I've seen that If I manually delete the connection track related to the old IP the connection is established again.

hello.

I've been trying to agree to create BGP communities with our upstream providers but so far we haven't had so much luck due to we have less than 5Gbps in BGP traffic

well, i think it's not about your bgp communities proposal being rejected with 5gig traffic. afaik, it's all about what benefits you offered for them. mostly, being rejected because your network is a single leaf node ie. not a transit network.

nat64 is rather out of context for your internal network problem ie. your nat failure.

Actually when one of our BGP goes down our IPv4 block doesn't loose connection to outside internet, our external monitoring system doesn't detect our 206.1.XX.XX/24 block being offline so BGP routing its working fine.

yes. you are correct. but that was from the internet bgp reachability perspective. you have 2 upstream, then yes your network reachable from the internet.

but, your actual problem is inside your own nat network, which is how did you route your internal network? static? igp? before they were sent to bgp.

if you have to change your to change your nat setting, maybe it means you just having a distribute connected bgp advertisment setting?

nat is local access router significant, they don't care about path picking. as long as they have the config they are good to go. the traffic got sent or not, nat don't care.

that is why you should at least put nat router on different devices, so your nat routers know which gateway is really active for traffic forwarding.

hth.