Good Day Users!!

I am an ISP in India. Although I am not new to Mikrotik but despite a lot of tinkering and research, I failed to come up with a solution.

I currently have only one WAN link from ISP1 and I am planning to get another one from ISP2

Core Router : CCR1016-12G
Edge routers : All of them are RB1100AHx2
RouterOS version on all routers : 6.30

To go with specs, here they are.
ether1=WAN1 (ISP1)
ether2=WAN2 (ISP2)
ether3=LAN1
ether4=LAN2
ether5=LAN3
ether6=LAN4........ether10=LAN10

There are a total of 10 x /24 public IP Pools hosted on my AS number and I am running a BGP session with ISP1

The second WAN link from ISP2 will also be on a BGP session with same bandwidth capacity as that of ISP1.

Each of my LAN port is connected to 1 or more Edge routers which distribute connectivity to end users. Each edge router has a public IP for routing and management purposes. Customers who do not have public IPs are masqueraded at edge routers.

I want to load balance on per connection basis between two WAN links.

WAN1 IP = 125.1.1.2/30
WAN2 IP = 122.1.1.2/30

Public IP Pools allotted to my AS number
14.1.0.0/23
103.1.0.0/22
45.1.0.0/22


I searched to and fro for solutions. Every solution I found on internet assumes that I have only 1 LAN interface. This is where I am confused as I have more than 1 LAN interfaces with Public IPs distributed to Edge routers and customers on 'as required' basis.

I request Mikrotik support team and forum users to guide me for a solution on this.

BGP is not designed to offer ---> load balance on per connection basis between two WAN links.

BGP will help you select the best path to a destination aka shortest, but will not do per connection load balancing for you. You might though still be able to do such things if you consult the mikrotik wiki. It is not so recommended on an ISP level. It breaks some applications in the end and customers will not be that happy.

Hi,

Similar problem with BGP, please tell me if is correct what I do.
I have 2 BGP connections with 2 providers on a CCR1036:
ISP1 - 300Mbps - send me 816 prefixes
ISP2 - 400Mbps - send me 88 prefixes
with guarantied speeds.

How can I make some load balance the traffic if one of the connection is used 99%?
I have made some Route Filters to cut some prefixes, but I don't know if it's good solution.
I set also distance 21 for default route of route1 to be route2 default route used more.

Right now is used 100% the ISP1(300Mbit) connection and 25% the ISP2(400Mbit), can you advice what to do?
I don't want to upgrade ISP1 as ISP2 isn't used at least 50%.

I thing same problem appears to many bgp users, please give me some advice.

thanks
ronn

Increase local preference for prefixes you get from ISP2

Increase local preference for prefixes you get from ISP2

thanks for your replay, sorry but I don't have experience on tuning BGP

I made a filter accept role with BGP Local Pref = 200
And I applied to peer ISP2(less used) in-filter.
That bgp-peers prefixes now have 200 local pref value.
But I don't see immediate big trafic changes, maybe at evening i will see the graphs.
If the trafic still not going to ISP2, I need to cut more prefixes from ISP1?

thanks

Increase local preference for prefixes you get from ISP2

thanks for your replay, sorry but I don't have experience on tuning BGP

I made a filter accept role with BGP Local Pref = 200
And I applied to peer ISP2(less used) in-filter.
That bgp-peers prefixes now have 200 local pref value.
But I don't see immediate big trafic changes, maybe at evening i will see the graphs.
If the trafic still not going to ISP2, I need to cut more prefixes from ISP1?

thanks

Setting local preference on a received route only affects outbound traffic or from the customer's perspective, an upload. If you want to influence inbound traffic, talk to your upstreams and see if they offer BGP communities that you can utilize to influence inbound traffic.

First of all - are we talking about outbound traffic or inbound traffic?

Local_Pref (and other such metrics) will tune your OUTBOUND preference, but only on the prefixes you receive. If you're getting only a few dozen or few hundred prefixes, then you're going to get almost zero load-balancing from this, as the global routing table is around 660k routes. The vast majority of your outbound traffic is therefore being routed to the default GW prefix.

Inbound traffic engineering is not granular like many people here want to do.

As for inbound traffic engineering, you accomplish this by a few menthods:
- Announcing more specific prefix to certain ISPs (the most effective method)
- BGP Communities - if your ISPs offer community functionality, then you can see what policy actions they offer and leverage those
- AS-Prepending your advertisements (longer AS-Path tends to cause traffic to choose another path)

Remember, though - just as easily as you can set your own local_pref, weight, etc to control your outbound routing policy, so too can the other networks out there. If their policies route traffic such that you get inbound traffic in ways you don't want, then that may just be too bad for you.

Thanks for the answers.
Yes, my problem is with inbound trafic - my clients download, they use 100%of ISP1 bgp peer bandwidth.

bgp peerings:
ISP1 - 300Mbps - send me 816 prefixes = default route + one comunity xxx:100
ISP2 - 400Mbps - send me 88 prefixes = default route + 88 prefixes with no comunity

ISP1 download used 95%
ISP2 download used 25%

Right now the default route is from ISP2, with local pref higher, the upload is going to ISP2, but the downoad is coming from ISP1.
It can be changed if made a filter to discard half of ISP1 prefixes?

It can be changed if made a filter to discard half of ISP1 prefixes?

If you do that, and ISP2 goes down, then any IP within those prefixes not being advertised to ISP1 will not have Internet connectivity.

Let's say that you have a /20 of IP space, all contiguous as a single /20 block, e.g. 100.64.8.0/20

You could announce 100.64.8.0/20 to both ISPs, and then any sub-prefixes (e.g. 100.64.10.0/23) where you want routing to favor one particular ISP, announce the sub-prefix only to the preferred ISP. If that link fails, then the master /20 prefix will act as a backup route for all of the IPs in the sub-prefix. So with these examples, say you wanted 100.64.10/23 to come in ISP2. You would announce 100.64.10.0/23 to ISP2 only, but also announce 100.64.8.0/20 to both ISPs. If ISP2 fails, then 100.64.10.0/23 will retain service due to the fact that it is a sub-prefix of 100.64.8.0/20 via ISP1. When ISP2 comes back up, the hosts in that /23 will return to ISP2 for downstream traffic.

However, I'd say that when dealing in BGP, you probably need to learn to think differently. If you have two connections, then you probably want them to be the same size, and you don't want your total traffic to exceed either of the individual circuits' capacity. If you do, then during a failure, you will be over-budget and experience slow Internet.

It can be changed if made a filter to discard half of ISP1 prefixes?

If you do that, and ISP2 goes down, then any IP within those prefixes not being advertised to ISP1 will not have Internet connectivity.

No, here I mean the route prefixes that I receive from ISP1 and
not my advertised networks prefixes.

I have a /22 and 2x /24 prefixes, advertised to both providers.

It can be changed if made a filter to discard half of ISP1 prefixes?

No, here I mean the route prefixes that I receive from ISP1 and
not my advertised networks prefixes.

This would only affect your outbound traffic path selection. In fact, the better thing to do about prefixes received from upstream carriers is to weight them with metrics, usually local_preference.

It can be changed if made a filter to discard half of ISP1 prefixes?

If you do that, and ISP2 goes down, then any IP within those prefixes not being advertised to ISP1 will not have Internet connectivity.

No, here I mean the route prefixes that I receive from ISP1 and
not my advertised networks prefixes.

I have a /22 and 2x /24 prefixes, advertised to both providers.

Have you checked with your upstreams to see what BGP communities they offer? I've worked with Reliance and TATA before in India and I believe both of them have BGP communities for traffic management.

You can also split the the /22 and /24 advertisements up and advertise more specific prefixes out of the higher capacity link but this is generally a last resort when BGP communities aren't available from the upstream ISP.

You could try AS path prepending to ISP1. That should make more of your prefixes prefer to reach you over ISP2 as you will appear closer in terms of AS hops.

Unless you connect to the same ISP on both links, setting communities is only of limited value, unless you can get both ISPs to agree to using each others communities for traffic engineering, or that they will pass on any communities that you set to each other (communities can be wiped and re-written).

You could try AS path prepending to ISP1. That should make more of your prefixes prefer to reach you over ISP2 as you will appear closer in terms of AS hops.

Unless you connect to the same ISP on both links, setting communities is only of limited value, unless you can get both ISPs to agree to using each others communities for traffic engineering, or that they will pass on any communities that you set to each other (communities can be wiped and re-written).

The communities IParchitechs is talking about work in almost all scenarios, the ones that allow to selectively make your upstreams broadcast routes based on different things like:

• Geo-Locations of their upstreams (usually by continental separation, APAC, Americas, Europe, etc.)
• Type of the peering relationship (customer, paid-transit, IX, etc.)
• Geo-Location of where their peer (IX SF, IX LA, IX DE, etc.)

Some providers also allow you to manipulate routes towards certain specific peers, doing things like:

• Set As-path prepend 5X to EU peers but not to APAC peers, thus making EU peers more likely to use your backup link
• Broadcast everywhere but APAC peers, thus if APAC traffic is heavy inbound it won't use this link rather it'll use your backup

And of course communities that allow you to de-pref your broadcasts.

Sadly, not all providers have this functionality (the vast majority of providers don't offer this AFAIK).

Nevertheless back to the topic, the main issue here is the lack of knowledge about routing at the internet level, we could offer guidance but the best thing OP could do in my honest opinion (and everyone that has the same kind of questions) is to either hire someone that knows about this to guide him step by step, or take the path of learning about how BGP and routing at the internet works (which is what I recommend), it'll help you grasp easily all the excellent recommendations here provided, it can be quite a big hit when you start a WISP business small with all the NAT involved then go big with BGP, the reasoning is quite different here, the way to approach/tackle problems is different as well.