Community discussions

MikroTik App
 
changeip
Forum Guru
Forum Guru
Topic Author
Posts: 3819
Joined: Fri May 28, 2004 5:22 pm

bgp - level3 / cogent

Fri Oct 21, 2005 1:42 am

Does anyone have any BGP configs they'd like to share? We are working on setting up with level3 and cogent (to dodge the nov9th de-peer again) and are thinking of using MT as the bgp server. I'm not sure if this will be done on the same box or another one in front of it. I'm interested to see if anyone has gone down this path with level3 / cogent before. Is MT's bgp implementation compatible with both providers? are there limitations or problems that I am going to run into even with the new 2.9 routing-test package.

AS number was approved today. IP block from ARIN is being requested but you 'the book' states you have to have an egg before you have the chicken : ) Pain in the butt these people are!

Thx,
Sam
 
User avatar
maximan
Trainer
Trainer
Posts: 549
Joined: Sat May 29, 2004 12:10 am
Location: Rio Cuarto, Argentina
Contact:

Mon Oct 24, 2005 9:48 pm

i want it too.
there are example or someone will share the bgp configuration for ASN with 2 or more carrier provider?.
MKE Solutions > Professional Support IT (Spanish / English)
FastNetMon / FNM Manager: DDoS Detection Tools.
 
changeip
Forum Guru
Forum Guru
Topic Author
Posts: 3819
Joined: Fri May 28, 2004 5:22 pm

Mon Oct 24, 2005 10:08 pm

We succesfully got 2.9.6 connected via BGP to Cogent using their Peer A & B setup. Today I am hoping to get Level3 connected. Once done I will post full configs.

I had cogent drop us down to a non-full transit list of routes. 167,000 seems to really tax the mikrotik box and really isn't needed for us. We ended up with 12000 routes from Cogent, and we'll see about level3 later today.

Sam
 
Instructor
just joined
Posts: 9
Joined: Wed Sep 14, 2005 4:35 pm
Location: Germany

Tue Oct 25, 2005 8:36 am

We succesfully got 2.9.6 connected via BGP to Cogent using their Peer A & B setup. Today I am hoping to get Level3 connected. Once done I will post full configs.

I had cogent drop us down to a non-full transit list of routes. 167,000 seems to really tax the mikrotik box and really isn't needed for us. We ended up with 12000 routes from Cogent, and we'll see about level3 later today.

Sam
Hi Sam,

are you using routing-test package?

if ~170k Routes is really an issue for the MT box, then BGP implementation has to be redesigned. with every peer that supplies full-table, the amount of total routes doubles in the router. if just a single full-table with 170k routes can exceed MT's routing capabilities, there must be a design flaw anywhere in the context.

-Frank
 
changeip
Forum Guru
Forum Guru
Topic Author
Posts: 3819
Joined: Fri May 28, 2004 5:22 pm

Thu Oct 27, 2005 6:54 pm

if just a single full-table with 170k routes can exceed MT's routing capabilities, there must be a design flaw anywhere in the context.
Exactly. I have emailed support with many supout files and asked that they take a look to see if it can be optimized. I have a good feeling they will figure out whats wrong and fix it. I don't think its a bug, i think its simply optimization that needs to be done - as well it is a routing-test package so its still beta at this point. I have full confidence in these people at Mikrotik!

So the status of our BGP ... we've got a router upgraded to 2.9.6 with routing-test package and connections to both Cogent and Level3 via 100mbps pipes. BGP is accepting routes from both providers. We started with full transit routes and within a few minutes I realized it was impossible. I had them both start sending customer routes only from each provider. Cogent is sending about 12,500 routes and Level3 is about 75,000 routes. Cogents route set is small enough that mikrotik handles it no problem, it takes about 30 seconds and it's synced up and CPU goes to idle after being at 100% during sync. No problems there. Level3 peer starts up and after about 3 minutes seems to just spin out of control - BGP sessions start terminating, hold timers expire, etc. The CPU is pegged at 100% the entire time. If I run a /ip route print count-only during the sync I see it crawling up and once it gets over about 20,000 it seems to die a slow death. We turned off Cogent peering to see if it could handle just the level3 peering session but no luck. The only way I can get the router under control is if I filter out the routes as they come in. I believe in the beginning we actually had them accepting with a routing-mark using the filter and that worked, I believe because there werent overlapping routes causing it to work hard to decide where to send things.

I think the problem stems from the way RouterOS injects routes into the table. I noticed when I filter out all routes coming in that the BGP session takes about 5 seconds to sync up and its done... I see a huge 5-10mbps download while it syncs up. Without the filter I see them trickle in because it seems to be applying and processing as each route comes in.

Here is our config:



/ routing filter 
add chain=cogent-a prefix=38.XX.XX.0/24 prefix-length=24 action=accept \
    comment="" disabled=no 
add chain=cogent-a prefix=38.101.160.XXX prefix-length=32 action=accept \
    comment="" disabled=no 
add chain=cogent-a prefix=0.0.0.0/0 action=discard comment="" disabled=no 
add chain=cogent-b prefix=0.0.0.0/0 action=discard comment="" disabled=no 
add chain=cogent-in set-nexthop=38.XX.XXX.1 comment="" disabled=no 
The above is required for Cogent's peer A & B setup. The 38.101.x.x address is their loopback IP required to get peer B up and running.
add chain=level3-in action=discard comment="" disabled=no 
add chain=level3-out prefix=38.XX.XX.0/24 prefix-length=24 action=accept \
    comment="" disabled=no 
add chain=level3-out prefix=0.0.0.0/0 action=discard comment="" disabled=no 
Notice that we 'action=discard' all rules coming in. If we simply change that to 'accept' or anything else it craps out and the router hits 100% cpu for hours.
/ routing bgp instance 
set default as=19557 router-id=38.XX.XX.2 redistribute-static=no \
    redistribute-connected=yes redistribute-rip=no redistribute-ospf=no \
    redistribute-other-bgp=no name="default" out-filter="" disabled=no 
add as=19557 router-id=XX.XX.XX.18 redistribute-static=no \
    redistribute-connected=yes redistribute-rip=no redistribute-ospf=no \
    redistribute-other-bgp=no name="level3" out-filter="" disabled=no 
We setup 2 instances because there might have been problems using the same router-id with both peers. If set to 0.0.0.0 its supposed to figure it out, but i was being cautious and forced the right ip with each peer. I didn't want to create a peering session using the other providers pipe ... and I think it might have only done that because we use ECMP on the way back out.
/ routing bgp peer 
add remote-address=38.XX.XX.1 remote-as=174 multihop=no in-filter="" \
    out-filter=cogent-a keepalive-time=0s hold-time=3m ttl=60 disabled=no 
add remote-address=38.101.160.XXX remote-as=174 multihop=yes \
    in-filter=cogent-in out-filter=cogent-b keepalive-time=0s hold-time=3m \
    ttl=6 disabled=no 
add remote-address=63.210.XX.XX remote-as=3356 multihop=no \
    in-filter=level3-in out-filter=level3-out keepalive-time=0s hold-time=3m \
    instance=level3 ttl=60 disabled=no 
Cogent has 2 peering sessions, one to send routes and one to accept routes. Level3 does all within a single peering session.

Anyhow, I think Mikrotik will (hopefully) improve their routing table injections process and all will be good. We plan on now splitting our border routes up into 2, 1 for each connection ... I'm hoping that we don't have to use Crisco or other as we really have faith in the RouterOS to do the job well. I have been looking into NAPI with the intel cards - hoping its just part of the OS already and its being used. I won't really know until we get a dev environment setup and can actually test pps ratings.

Thx
Sam
 
changeip
Forum Guru
Forum Guru
Topic Author
Posts: 3819
Joined: Fri May 28, 2004 5:22 pm

Fri Oct 28, 2005 2:10 am

I setup zebra on another test box on our dev environment just experiment and troubleshoot. here is some debug output from a zebra to mikrotik box that I captured using debug on mikrotik:
route,calc Tag destination for recalculation in 27-Oct 15:59:18.17 from 10.20.1.1
route,calc     DstAddress=10.120.251.0/24 in 27-Oct 15:59:18.17 from 10.20.1.1
route,event Added candidate route in 27-Oct 15:59:18.17 from 10.20.1.1
route,event     DstAddress=10.120.250.0/24 in 27-Oct 15:59:18.18 from 10.20.1.1
route,event     Attributes in 27-Oct 15:59:18.20 from 10.20.1.1
route,event         type=BGP in 27-Oct 15:59:18.20 from 10.20.1.1
route,event         dst=10.120.250.0/24 in 27-Oct 15:59:18.21 from 10.20.1.1
route,event         targetscope=10 in 27-Oct 15:59:18.21 from 10.20.1.1
route,event         nexthop=weight=0address=10.20.1.147 in 27-Oct 15:59:18.23 from 10.20.1.1
route,event         origin-id=12 in 27-Oct 15:59:18.23 from 10.20.1.1
route,event         bgp-origin=0 in 27-Oct 15:59:18.23 from 10.20.1.1
route,event         bgp-aspath=19557 in 27-Oct 15:59:18.25 from 10.20.1.1
route,event         bgp-aspath-len=1 in 27-Oct 15:59:18.25 from 10.20.1.1
route,event         bgp-nexthop=10.20.1.147 in 27-Oct 15:59:18.25 from 10.20.1.1
route,event         bgp-med=0 in 27-Oct 15:59:18.26 from 10.20.1.1
route,calc Tag destination for recalculation in 27-Oct 15:59:18.26 from 10.20.1.1
route,calc     DstAddress=10.120.250.0/24 in 27-Oct 15:59:18.26 from 10.20.1.1
route,event Added candidate route in 27-Oct 15:59:18.28 from 10.20.1.1
route,event     DstAddress=10.120.249.0/24 in 27-Oct 15:59:18.28 from 10.20.1.1
route,event     Attributes in 27-Oct 15:59:18.28 from 10.20.1.1
route,event         type=BGP in 27-Oct 15:59:18.28 from 10.20.1.1
route,event         dst=10.120.249.0/24 in 27-Oct 15:59:18.29 from 10.20.1.1
route,event         targetscope=10 in 27-Oct 15:59:18.29 from 10.20.1.1
route,event         nexthop=weight=0address=10.20.1.147 in 27-Oct 15:59:18.29 from 10.20.1.1
route,event         origin-id=12 in 27-Oct 15:59:18.29 from 10.20.1.1
route,event         bgp-origin=0 in 27-Oct 15:59:18.31 from 10.20.1.1
route,event         bgp-aspath=19557 in 27-Oct 15:59:18.31 from 10.20.1.1
route,event         bgp-aspath-len=1 in 27-Oct 15:59:18.31 from 10.20.1.1
route,event         bgp-nexthop=10.20.1.147 in 27-Oct 15:59:18.32 from 10.20.1.1
route,event         bgp-med=0 in 27-Oct 15:59:18.32 from 10.20.1.1
I'm guessing the slowness comes from the 'Tag destination for recalculation' processes that runs after each individual update. I don't know much about BGP and routing tables - maybe it has to run that way, but it seems like it would be nice to pause 'recalcs' until the bgp feed is in the established state or something ... Who knows. Anyhow, hoping something can get resolved with this or else we gotta spend a bunch of money on hardware that i hate. I only like to spend money on Mikrotik stuff : )

Sam
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24608
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Fri Oct 28, 2005 9:44 am

actually we ARE working on improving BGPs speed.
 
changeip
Forum Guru
Forum Guru
Topic Author
Posts: 3819
Joined: Fri May 28, 2004 5:22 pm

Fri Oct 28, 2005 9:53 am

I knew you would : ) RouterOS is our choice becuase you guys know what you are doing. Thanks!
 
nikhil
Member Candidate
Member Candidate
Posts: 262
Joined: Wed Dec 22, 2004 5:04 pm
Location: US

Fri Oct 28, 2005 2:00 pm

Nice to hear they are looking at this. Im taking twin feeds full bgp on 2.8.28 (no routing-test of course) and it works fine.

Who is online

Users browsing this forum: amala, DanMos79, eworm, Google [Bot], GreySer, pauulschadauer, RackKing, sindy and 72 guests