Community discussions

 
User avatar
mrz
MikroTik Support
MikroTik Support
Topic Author
Posts: 5934
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

What BGP setups need to be optimized

Fri Feb 20, 2015 2:25 pm

Before v7 is released we would like to optimize v6 routing.
Anyone who have performance issues with BGP, please describe in details what setup you are running and detailed problem description:
* how many peers;
* how many routes in routing table;
* is there also OSPF,MPLS, VPLS, RIP etc running on the router;
* what are the hardware specs;
* are there routing filters;
* average BGP update rate from upstream peers;
* average amount of traffic forwarded over the router;
* are there frequent BGP bgp peer flaps,
* any other info that is relevant to routing and can be useful for optimization.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2946
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: What BGP setups need to be optimized

Fri Feb 20, 2015 4:48 pm

CCR1036-12G-4S
RouterOS: 6.20 -> 6.22 -> 6.23 -> 6.24 -> 6.25 SFP not working at all -> 6.24 again -> 6.27
No one single problem or performance issue on BGP.

The only problem I haved (for me fixed on 6.27) with the machine (and also with it's VRRP mirror) is the SFP driver not auto-sensing rightly.
Too much often BGP peer freeze not for the BGP itself, but for the poor quality of SFP link with S-85DLC05D 850nm

Now, with the 6.27 this problem is vanished (I never try 6.26).
I'm Italian, not English. Sorry for my imperfect grammar.
 
User avatar
Pawel
just joined
Posts: 8
Joined: Tue Feb 10, 2015 8:02 pm
Location: Poland
Contact:

Re: What BGP setups need to be optimized

Fri Feb 20, 2015 9:20 pm

Hi Mrz,
Any chance to have support for SNMP BGP4-MIB in 6.x?


Thanks.
Regards,
Pawel Rybczyk
Regional Manager
Border 6 - Non-Stop Internet
Intelligent BGP Controller
 
keefe007
Member Candidate
Member Candidate
Posts: 124
Joined: Sun Jun 25, 2006 3:01 am

Re: What BGP setups need to be optimized

Sat Feb 21, 2015 2:59 am

I'd like to see the BGP4 MIB too!
 
cheeze
Member Candidate
Member Candidate
Posts: 146
Joined: Tue Jul 31, 2012 7:44 am

Re: What BGP setups need to be optimized

Sat Feb 21, 2015 4:36 am

MRZ, I'd say other than the single threaded process of evaluating routes/loading routes from the BGP RIB-IN to RIB LOCAL I don't think I've had any problem come up.

Obviously more BGP address families would be nice but, that's not really to the point of your question :)
 
joegoldman
Member
Member
Posts: 491
Joined: Mon May 27, 2013 2:05 am

Re: What BGP setups need to be optimized

Sat Feb 21, 2015 11:23 am

Hi,

CCR1036's, 11~ BGP peers, about 15k~ routes, although Im purposely not taking full table as the time to LOAD the table takes too much, and filtering through the routetable takes too long. This limits my ability to have multiple upstreams (the extra peers are IX's and bi-lateral peering)

Run about average 15 routing filters (some are shared chains) per peer.

Really its just about time to import routes and time to filter through route table. Some filters are a bit quicker but, for instance, if I need to know what route 8.8.8.8 is taking, even on 15k routes takes too long to be worthwhile for troubleshooting. (/ip route pr where 8.8.8.8 in dst-address)

Thanks,
Joe
 
User avatar
mandrade
Member Candidate
Member Candidate
Posts: 104
Joined: Fri Sep 08, 2006 12:35 pm
Location: Brazil
Contact:

Re: What BGP setups need to be optimized

Sat Feb 21, 2015 12:51 pm

Hi Mrz,
Any chance to have support for SNMP BGP4-MIB in 6.x?


Thanks.

+1 in this list.....
 
patrick7
Member Candidate
Member Candidate
Posts: 298
Joined: Sat Jul 20, 2013 2:40 pm

Re: What BGP setups need to be optimized

Sat Feb 21, 2015 1:07 pm

Hi Mrz,
Any chance to have support for SNMP BGP4-MIB in 6.x?


Thanks.

+1 in this list.....
+1
 
User avatar
mandrade
Member Candidate
Member Candidate
Posts: 104
Joined: Fri Sep 08, 2006 12:35 pm
Location: Brazil
Contact:

Re: What BGP setups need to be optimized

Sat Feb 21, 2015 1:21 pm

Here is a REALLY NEEDED change on BGP : http://forum.mikrotik.com/viewtopic.php?f=1&t=93298
 
bronx
newbie
Posts: 39
Joined: Wed Feb 11, 2015 1:04 am
Location: Turin, Italy

Re: What BGP setups need to be optimized

Sat Feb 21, 2015 1:53 pm

* how many peers;
69 (some only IPv4, some only IPv6 )

* how many routes in routing table;
645748 ipv4
64762 ipv6

* is there also OSPF,MPLS, VPLS, RIP etc running on the router;
Also OSPF on the router,

* what are the hardware specs;
CCR1036-12G-4S
* are there routing filters;
175 routing filter

* average BGP update rate from upstream peers;
70@19day uptime

* average amount of traffic forwarded over the router;
330 Mbps

* are there frequent BGP bgp peer flaps,
HoldTimer expired is the cause of peer flaps

* any other info that is relevant to routing and can be useful for optimization.


You can connect to our system if you need.
 
lmm
just joined
Posts: 1
Joined: Sat Feb 21, 2015 4:14 pm

Re: What BGP setups need to be optimized

Sat Feb 21, 2015 4:35 pm

Hello, we tried to install CCR1036-8G-2S+ with 6.27 and connect it to our peers, the main trouble is performance of the BGP process even we had just two sessions with our route reflectors sending full internet BGP table (no filters attached) and one external peer sending full tables with easy filtering.

It takes about ~5-7mins to remove routes from mikrotik routing table if the external peer is disconnected.

* how many peers;
3 with full internet tables, 7 just sending default route and receiving about 20 prefixes

* how many routes in routing table;
about 500k
*
is there also OSPF,MPLS, VPLS, RIP etc running on the router;
no

* what are the hardware specs;
CCR1036-8G-2S+, peers connected over 10Gb ports

* are there routing filters;
just for BGP, there is no firewall

* average BGP update rate from upstream peers;
about 10-20 per min

* average amount of traffic forwarded over the router;
400 Mbps, tested more than 1500Mb/s in one stream with very low CPU usage (about 20-30% on one cpu)

* are there frequent BGP bgp peer flaps,
no, if the Cisco RR is pushing the full table to CCR1036-8G-2S+, CCR1036-8G-2S+ is not able to send keep alives and prefixes back to Cisco RR, cisco output buffer is full for about 5mins

* any other info that is relevant to routing and can be useful for optimization.
make route processing parallel please and it may be a good box for ISP usage

lm
 
User avatar
lyberis1979
Frequent Visitor
Frequent Visitor
Posts: 58
Joined: Mon Oct 20, 2014 2:43 am
Location: Athens, Greece
Contact:

Re: What BGP setups need to be optimized

Sat Feb 21, 2015 6:09 pm

Hi Mrz,
Any chance to have support for SNMP BGP4-MIB in 6.x?


Thanks.

+1
 
User avatar
paoloaga
Member Candidate
Member Candidate
Posts: 222
Joined: Tue Mar 08, 2011 2:52 am
Location: Vaprio d'Agogna (NO) - Italy
Contact:

Re: What BGP setups need to be optimized

Sun Feb 22, 2015 8:40 am

Hello Mrz,
Anyone who have performance issues with BGP, please describe in details what setup you are running and detailed problem description:
We have several BGP speaking routers on our network:

* how many peers;

8 - 70

* how many routes in routing table;

100 to 550.000

* is there also OSPF,MPLS, VPLS, RIP etc running on the router;

BGP & OSPF

* what are the hardware specs;

CCR 1036 - CCR 1016

* are there routing filters;

Of course :-)

* average BGP update rate from upstream peers;

~25k/hour

* average amount of traffic forwarded over the router;

200 - 500 Mbit/s

* are there frequent BGP bgp peer flaps,

not frequent

* any other info that is relevant to routing and can be useful for optimization.

Routes are received very quickly in latest releases (~ 1 min to load the full bgp table). But once loaded the table, routes become active much later (up to 15 minutes!). Also the updates received take a lot of time to become working (several minutes). I would like to see a more rapid network topology change.

The other very important issue is recursive next-hop lookup in IPv6 which is actually causing a lot of problems in deploying v6 on the network.

Access to routers is available if needed.

Thanks.
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: What BGP setups need to be optimized

Mon Feb 23, 2015 3:32 am

BGP performance hasn't been too much of an issue. The things talked about at MUM for v7 sound great. As far as BGP related requests, these are some:
Hi Mrz,
Any chance to have support for SNMP BGP4-MIB in 6.x?

Thanks.
+1 BGP4-MIB

Also, adding the BGP AS source to netflow
Cisco offers either peer or origin:
The origin-as keyword specifies that export statistics include the originating autonomous system for the source and destination.
The peer-as keyword specifies that export statistics include the peer autonomous system for the source and destination.
If traffic flow is being improved, I would also request that fast path support isn't negated with its use.

I already checked with support on the ability to use v7 as a pure route reflector and was happy to hear that functionality will be included.

Using multiple peers, especially on an internet exchange, both of these features become very important in managing many peers. The CCR series is becoming more and more powerful and RouterOS is improving software performance, but without these two important features, many will be stuck with the competition.

Thanks for checking in mrz. I'm looking forward to v7
Last edited by alexjhart on Tue Feb 24, 2015 10:38 pm, edited 1 time in total.
-----
Alex Hart

The Brothers WISP
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1818
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: What BGP setups need to be optimized

Mon Feb 23, 2015 10:05 am

+1 BGP4-MIB

Also, adding the BGP AS source to netflow http://forum.mikrotik.com/viewtopic.php ... 3&p=438631
http://www.cisco.com/c/en/us/td/docs/io ... 82ECF20EDA
Cisco offers either peer or origin:
The origin-as keyword specifies that export statistics include the originating autonomous system for the source and destination.
The peer-as keyword specifies that export statistics include the peer autonomous system for the source and destination.
+1 for both of these. I have requested the Netflow v9 origin-as and peer-as features directly to Mikrotik support, they notified me that they will try and bring these to RouterOS v7 as it requires the new-routing engine.
I already checked with support on the ability to use v7 as a pure route reflector and was happy to hear that functionality will be included.
This is already possible:

- Create ip/vrf
- Create bgp/instance
- Assign bgp/instance to ip/vrf
- Create bgp/vrf
- Connect to peers and see their routes only in the VRF table you created above
- Success!

We have been running this in production for many years now. Or maybe you mean something else ?
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA
 
robertpenz
Frequent Visitor
Frequent Visitor
Posts: 80
Joined: Mon Oct 10, 2011 8:41 am

Re: What BGP setups need to be optimized

Tue Feb 24, 2015 10:59 am

Answer is per router:

* how many peers;
10

* how many routes in routing table;
/ip route print count-only
1978010
/ipv6 route print count-only
43565

* is there also OSPF,MPLS, VPLS, RIP etc running on the router;
OSPF

* what are the hardware specs;
CCR1036-8G-2S+

* are there routing filters;

yes

* average BGP update rate from upstream peers;

20-30k / h

* average amount of traffic forwarded over the router;

200Mbit

* are there frequent BGP bgp peer flaps,

no

* any other info that is relevant to routing and can be useful for optimization.
no really

Feature requests:

* something like "show ip bgp summary" integrated ... using the script from the forum is ok, but it is slow
* Faster search for prefixes .. e.g. I want to check where a route goes it takes minutes not seconds
* better monitoring via SMNP
* Netflow v9 origin-as and peer-as
 
Muqatil
Trainer
Trainer
Posts: 574
Joined: Mon Mar 03, 2008 1:03 pm
Location: London - UK
Contact:

Re: What BGP setups need to be optimized

Tue Feb 24, 2015 5:23 pm

* how many peers;
/routing bgp peer print count-only 
24
* how many routes in routing table;
/ip route print count-only 
1575550
* is there also OSPF,MPLS, VPLS, RIP etc running on the router;
OSPF with around 2k routes
* what are the hardware specs;
/system resource> pr
             uptime: 2w4d14h38m46s
            version: 6.26
         build-time: Feb/03/2015 15:18:36
        free-memory: 2948.7MiB
       total-memory: 3968.9MiB
                cpu: tilegx
          cpu-count: 36
      cpu-frequency: 1200MHz
           cpu-load: 4%
     free-hdd-space: 903.2MiB
    total-hdd-space: 1024.0MiB
  architecture-name: tile
         board-name: CCR1036-8G-2S+
           platform: MikroTik
* are there routing filters;
/routing filter print count-only 
20
* average BGP update rate from upstream peers;
Most of em stay up as router uptime
* average amount of traffic forwarded over the router;
300Mbit/s
* are there frequent BGP bgp peer flaps,
It's in a IX so there are some flappings, but rare.
* any other info that is relevant to routing and can be useful for optimization.
When i make some change on filters, router sometimes becomes not responsive, but the traffic keeps forwarded. Looks like the CPU is topped and winbox suffers it.
Route changes converge very slowly if one peer with Full route table is involved in some changes
CPU never goes up more than 5% but one core is always 100% (bgp process?)

If i open a winbox window of /ip route, after a while winbox freezes, and when i close and reopen the connection, winbox tries to load the 1M routes again.
/bgp advertisement in winbox is not usable at all
Renato Bernardi

skype: medtech5
 
User avatar
mandrade
Member Candidate
Member Candidate
Posts: 104
Joined: Fri Sep 08, 2006 12:35 pm
Location: Brazil
Contact:

Re: What BGP setups need to be optimized

Tue Feb 24, 2015 8:28 pm

Here it is :

# Peers Counter :
/routing bgp peer print count-only
27
# IPv4 Routes Counter :
 /ip route print count-only 
639431
# IPv6 Routes Counter :
/ipv6 route print count-only 
26376
# Running OSPF too :
/routing ospf route print count-only 
12530
# Hardware Specs :
 /system resource print 
             uptime: 1d3h47m38s
            version: 6.27
         build-time: Feb/11/2015 13:24:13
        free-memory: 2960.6MiB
       total-memory: 3968.9MiB
                cpu: tilegx
          cpu-count: 36
      cpu-frequency: 1200MHz
           cpu-load: 40%
     free-hdd-space: 906.0MiB
    total-hdd-space: 1024.0MiB
  architecture-name: tile
         board-name: CCR1036-12G-4S
           platform: MikroTik
# Filter Counter :
 /routing filter print count-only
78
# average BGP update rate from upstream peers : Most of em stay up as router uptime

# average amount of traffic forwarded over the router : Our 95th Percentile is 2297.58 mbit (in+out)

# relevant info : we really need a better multitask to BGP process and BGP-MIB implementation .
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: What BGP setups need to be optimized

Tue Feb 24, 2015 10:32 pm

I already checked with support on the ability to use v7 as a pure route reflector and was happy to hear that functionality will be included.
This is already possible:

- Create ip/vrf
- Create bgp/instance
- Assign bgp/instance to ip/vrf
- Create bgp/vrf
- Connect to peers and see their routes only in the VRF table you created above
- Success!

We have been running this in production for many years now. Or maybe you mean something else ?
I am talking about this: propagate BGP route updates without installing them in IP route table
-----
Alex Hart

The Brothers WISP
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1818
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: What BGP setups need to be optimized

Wed Feb 25, 2015 3:09 am

OK. That would indeed be useful.

The method we use is really a dirty hack, we only get away with it by having dedicated x86 VM's as route reflectors.
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1818
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: What BGP setups need to be optimized

Wed Feb 25, 2015 4:36 am

Mrz

Thank you for soliciting feedback from the community.

Sorry for turning this into a v7 wish list.

It is however good to see common requests.
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA
 
User avatar
wulfgard
Frequent Visitor
Frequent Visitor
Posts: 86
Joined: Wed Oct 17, 2012 1:06 pm
Location: France
Contact:

Re: What BGP setups need to be optimized

Thu Feb 26, 2015 1:22 am

Hello

our network is made of 8 border routers and 2 route server
total number of bgp session is about 500
we are using x86 xeon router with ROS v6.27 because CCR are too slow for route calculation

standard load of a router is about
- 1000000 routes
- between 10 and 200 bgp session per router ( half of them ipv6 )
- 300 filters per router
- between 500Mbps up to 4Gbps of trafic in+out per router
- ospf, firewall

standard time to setup all session at startup on a router is about 30 seconds
total time to propagate full table on all routers through route server is about 60 seconds
issue is on CCR which can take up to 2 minutes to get full table because using only one processor
processor load is about 20 percent
we can notice that wrong ospf configuration can harvest cpu

we would like to see

- BGP multipath
- enforce deterministic med switch on/off
- fix ipv6 loopback problem especialy for ibgp
- have dampening feature
- support for AS number in sflow
- BGP mib
- multicore for BGP updates on CCR
- duplicate feature in filter list ( all the filter not only one rule )
- enforce BGP origin ( igp - egp - incomplete ) with filters
- match chain for AS number ( like for prefix )

also relevant to BGP

- MPLS with multi tag per router
- MPLS not killing ECMP
- MPLS load balancing
- colored TE Tunnels

Thank You
Thierry
System and Network Engineer
Mikrotik Trainer - MTCNA MTCRE
Official French Mikrotik Distributor
 
Poundbury
just joined
Posts: 14
Joined: Mon Feb 11, 2013 6:51 pm

Re: What BGP setups need to be optimized

Thu Feb 26, 2015 3:27 pm

We have around 15 CCR-1036's with full internet BGP tables.
We tend to limit the number of peers per router because the convergence time is so high. There must be something different in the way that MT loads the routes into the FiB, because we don't see this on x86 hardware running Quagga, or on Cisco routers.
We use route filtering on every peer, and no more than 4 full tables I think per router. The biggest route count I can find in our network is 1945127
(from /ip route print count-only).

The only problem is the convergance time really. We see real world peaks of around 2Gbps of small symmetrical voip packets without problems. A simple improvement would be to get the OS to drop routes more quickly when a peer dies. It's less important that the route table is populated quickly, but it's vital that dead routes are removed as soon as possible.

I'd like to see vlan interfaces on ethernet be fast-path enabled. Many carriers deliver to us with a tagged vlan.

We love the products!
Eagerly awaiting the 72 core 10G models!

Mike
 
User avatar
mrz
MikroTik Support
MikroTik Support
Topic Author
Posts: 5934
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: What BGP setups need to be optimized

Thu Feb 26, 2015 5:44 pm

Thank you all for the input.
We hear your feature requests and most of them are already in development.

As for x86 quagga compared to CCR, please do not compare completely different systems.
Any single core for such calculations on modern x86 CPU is approximately 10 times faster than single CCR core.

We also made the tests comparing ROS v6 and quagga on the same x86 hardware, performance are almost identical except that quagga uses more memory.
 
hzdrus
newbie
Posts: 37
Joined: Mon May 14, 2012 3:58 pm

Re: What BGP setups need to be optimized

Mon Mar 02, 2015 2:13 am

Apart from features already requested, it would be very useful if Mikrotik had a real Cisco-like "neigh x.x.x.x received-routes" command. Currently it is only possible to see routes after filters.
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1818
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: What BGP setups need to be optimized

Mon Mar 02, 2015 11:13 am

Apart from features already requested, it would be very useful if Mikrotik had a real Cisco-like "neigh x.x.x.x received-routes" command. Currently it is only possible to see routes after filters.
This has been requested by me multiple times, as well as advertised-routes & routes, and working from instances within a VRF. Mikrotik support have responded that these features will be in RouterOS v7 :)
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA
 
bikey
just joined
Posts: 2
Joined: Wed Mar 04, 2015 3:30 pm

Re: What BGP setups need to be optimized

Wed Mar 04, 2015 3:34 pm

Hello all ;)

I'm new on the forum. I take this thread on the road.

It would be cool if we can automate BGP tuning with the IRP solution of "Noction" (www.noction.com) directly with CCR.

And yes, time of convergence is *really* problematic.
 
omago
just joined
Posts: 5
Joined: Mon Mar 11, 2013 1:30 pm

Re: What BGP setups need to be optimized

Fri Mar 06, 2015 1:33 pm

BGP ECMP Support.
Will we see it in v7 ?

Regards.
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: What BGP setups need to be optimized

Thu Mar 12, 2015 9:31 pm

My list...

BGPv4 MIB
BGP ECMP
BGP Conditional Advertisements
Ability to see received routes vs routes after filters
BGP multi threaded on CCR
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
Begetan
Frequent Visitor
Frequent Visitor
Posts: 97
Joined: Mon Jul 11, 2011 11:49 am

Re: What BGP setups need to be optimized

Sat Mar 14, 2015 10:21 pm

I've tried to use CCR wit ROS up to 6.5 version. I was really disappointed with BGP configuration on CCR1036 with multihomed customers.

If you are runing high demand multihomed customers you have to implement

1. Complicated routing filters based on AS_PATH. Such filters not always working good on Mikrotik ROS. Sometimes it has to be clear BGP session or even ro reboot router to fix issues with routing filters. I have to disconnect one of my upstream which had border based on CCR1036 too because he not succeed in maintaining apropriated routing filters.

2. Fast BGP convergence. It was really bad experience to run BGP process on one CCR cpu with slow response.

3. Really stable software. Developming cycle of ROS is really pain in ass for me. Each release has some new bugs or broken features. I would like to see completelley new approach in software deveopment for ROS 7.xxx. It should be splited for two branch. One branch, which is stable should be released two times in a year. Exeption is only security updates which should be released immedeatelly. Second branch is development version, which should be processed us usually. Each hallf year development version shold be stoped for development and pushed for hard bug testing for 2-4 week and go to stable release candidate. So everyone in community can upgrade and spend a time to full test of new stable release.

Currently I am using used Cisco 72xx for BGP borders. I am very happy with uptime from 300 to 700 days. No one user of CCR shows uptime more than 2 month. It's unnacceptable for hi-demand customers.

Hope my advise can help in developing of new ROS version.
 
spite
just joined
Posts: 7
Joined: Sat Jun 16, 2012 5:23 am

Re: What BGP setups need to be optimized

Tue Mar 17, 2015 1:23 pm

Just dropping some input into this thread. I use a CCR as a test edge for a peering fabric I don't care much about. For the most part it works great but have had a couple of reboots with silly problems (Enabling netflow etc).

What I would REALLY REALLY like to see from Mikrotik is the ability to configure filters based on REGEX of BGP communities.

Our edge configurations on Cisco/Brocade might be a 40 line route-map, but to mirror (and worse, maintain) this on Mikrotik is something like 180+ lines with a rabit hole of of filtering references.


+1 Votes to Multi-threaded route information processing.
 
Nissarin
just joined
Posts: 18
Joined: Fri Feb 20, 2015 4:01 pm

Re: What BGP setups need to be optimized

Tue Mar 17, 2015 5:10 pm

* how many peers;
* how many routes in routing table;

2x full bgp feed

* is there also OSPF,MPLS, VPLS, RIP etc running on the router;

Not at the moment

* what are the hardware specs;

CCR1036-12G-4S

* are there routing filters;

~15 per peer (in/out) - basic filtering + prepends

* average BGP update rate from upstream peers;

~15k/h (based on total updates/uptime)

* average amount of traffic forwarded over the router;

500Mbit total (up to 1Gbit)

* are there frequent BGP bgp peer flaps,

None

* any other info that is relevant to routing and can be useful for optimization.

The same machine is actually running queues as well - we have 2 CCRs and swap them on RouterOS updates, this will probably continue until 1072 is released and will work as dedicated BGP router.


Anyway, this is out current setup, it works (TM). Most of the things I would want to see are already mentioned, besides that I would want to have more control over route filtering. Some time ago we wanted to setup peering with another local ISP (also using CCR) and it was quite hard to test various rulesets, at some point my colleague was not sure if the rules were applied or not (and which ones). The main problem is that some rules are applied automatically, it would be nice if there was information on whatever current rules are in effect (perhaps as simple counter mentioned by IPANetEngineer) and there was an option to disable auto update.
 
Invity
just joined
Posts: 1
Joined: Wed Mar 18, 2015 9:08 am
Contact:

Re: What BGP setups need to be optimized

Wed Mar 18, 2015 9:21 am

This is Useful information.. Interesting Ideas Thanks..
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: What BGP setups need to be optimized

Sat Mar 21, 2015 7:39 am

I would also like to add having faster response on /ip route print where xxxxx commands would be appreciated. Many are very slow with full routing tables.

I guess this was already offered up:
Image
-----
Alex Hart

The Brothers WISP
 
bryantabb
just joined
Posts: 13
Joined: Thu Aug 16, 2012 7:42 am

Re: What BGP setups need to be optimized

Mon Mar 23, 2015 2:24 am

+1 for SNMP monitoring & fixing the problem for ipv6 ospfv3 where it ignores cisco loopback routes.

This is the fault with the work around is to use redistribute connected and no ospfv3 config on the interface.


BGP related as we use loopbacks to build ibgp sessions across.

Thanks :D
 
User avatar
slarner
newbie
Posts: 30
Joined: Wed Jul 18, 2007 10:54 am
Location: UK
Contact:

Re: What BGP setups need to be optimized

Mon Mar 30, 2015 2:10 pm

Here is our Core Router which sits in our London Datacentre :-

CCR1036-12G-4S

Moves around 700Mbits at peak times with 20% cpu
/system resource print
             uptime: 5w4d16h26m42s
            version: 6.20
         build-time: Oct/01/2014 10:06:12
        free-memory: 2507.2MiB
       total-memory: 3969.2MiB
                cpu: tilegx
          cpu-count: 36
      cpu-frequency: 1200MHz
           cpu-load: 14%
     free-hdd-space: 895.7MiB
    total-hdd-space: 1024.0MiB
  architecture-name: tile
         board-name: CCR1036-12G-4S
           platform: MikroTik
BGP Peers (Most are private peers on our local exchange)
/routing bgp peer print count-only
39
/ip route print count-only
604178
/ipv6 route print count-only
53576
/routing ospf route print count-only
53
/routing filter print count-only
173
I would like to see a huge improvement in v7 on the BGP cpu usage and the time it takes to load all the routes in.

Another thing that really annoys me is the amount of time it takes for a route to change in the table. (OSPF)

I would also like to see L2TP LAC for termination of ADSL. Its looking like we have to buy a cisco router just to be able to do this!

+1 on BGP SNMP. We could then monitor BGP state in observium!
http://www.countybroadband.net
-----------------------------------------------
Provider of wireless broadband in Essex and Suffolk UK
 
shdwmstff
newbie
Posts: 43
Joined: Wed May 30, 2012 12:00 am

Re: What BGP setups need to be optimized

Fri Apr 03, 2015 10:41 pm

Anyway to Bind more cores to the routing processes?
Currently see only core dealing with this... would like to see more CPU for this.
 
Nissarin
just joined
Posts: 18
Joined: Fri Feb 20, 2015 4:01 pm

Re: What BGP setups need to be optimized

Sun Apr 05, 2015 1:13 am

Anyway to Bind more cores to the routing processes?
Currently see only core dealing with this... would like to see more CPU for this.
If I remember correctly from the presentation about v7, the way routing (at least some parts of it) is implemented in Linux makes it impossible to take advantage of multiple cores without redesigning and reimplementing it from the scratch and that's just too much work.
 
User avatar
armandfumal
Member Candidate
Member Candidate
Posts: 125
Joined: Wed Apr 25, 2012 5:50 pm
Location: Huldange,LUX
Contact:

Re: What BGP setups need to be optimized

Wed Apr 15, 2015 11:57 pm

+1 Votes to Multi-threaded route information processing.

and real tools for IT to manage BGP prefixe in real time...
Armand Fumal
________________________________________________________________
MikroTik & Ubiquity
 
argeorge
Trainer
Trainer
Posts: 9
Joined: Sun Dec 03, 2006 9:14 pm
Location: Greece

Re: What BGP setups need to be optimized

Mon Apr 27, 2015 3:50 pm

Before v7 is released we would like to optimize v6 routing.
Anyone who have performance issues with BGP, please describe in details what setup you are running and detailed problem description:
* how many peers;
* how many routes in routing table;
* is there also OSPF,MPLS, VPLS, RIP etc running on the router;
* what are the hardware specs;
* are there routing filters;
* average BGP update rate from upstream peers;
* average amount of traffic forwarded over the router;
* are there frequent BGP bgp peer flaps,
* any other info that is relevant to routing and can be useful for optimization.
->Please add RPKI support
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: What BGP setups need to be optimized

Mon Apr 27, 2015 5:04 pm

Anyway to Bind more cores to the routing processes?
Currently see only core dealing with this... would like to see more CPU for this.
If I remember correctly from the presentation about v7, the way routing (at least some parts of it) is implemented in Linux makes it impossible to take advantage of multiple cores without redesigning and reimplementing it from the scratch and that's just too much work.
I talked to MikroTik about this at the USA MUM last week and they replied that although multi-core is off the table for right now, they have been able to significantly increase the performance of loading a single full table as well as using some indexing to accelerate the loading of subsequent tables.

Can't wait for v7 beta to come out so we can do some perf testing.
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
shahbazian
Trainer
Trainer
Posts: 166
Joined: Fri Sep 09, 2011 6:22 pm
Location: Iran
Contact:

Re: What BGP setups need to be optimized

Thu Apr 30, 2015 1:59 am

Hi Dear,
Please add support for BGP as label distribution protocol for mpls setups.
Learn MikroTik to improve your network.
( MTCNA, MTCRE, MTCWE, MTCTCE, MTCUME, MTCIPv6E, MTCINE )
MikroTik Certified Trainer & Consultant
RIPE NCC Trainer
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8308
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: What BGP setups need to be optimized

Thu Apr 30, 2015 12:48 pm

Please add support for
this topic is about optimization, new features will be in version 7
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
jmorby
just joined
Posts: 13
Joined: Sat Jan 04, 2014 6:06 pm

Re: What BGP setups need to be optimized

Thu Apr 30, 2015 6:34 pm

Hi

We have over 1,000 peers which we'd like to try and move onto a CCR if it ever goes multithreaded ... but for the time being in our lab we have just 4 or 5 peers throwing a full table towards the CCR and 3-5 minute load times (at least) make it impossible to use in anything like a production environment

I had 10 x CCR 1072's on back order specifically because of the promise of multi threaded BGP / etc and already have 10-15 CCR1036's in test / doing some MPLS / etc

Problems today are already well reported ... and MT support have acknowledged them time and again saying "no head room to fix" (basically eluding to all efforts going on ROS7)

OSPF with the random incorrect flag state on sessions causing reloads and drops

Load times on a full table with only a handful of peers (I hate to think what it would be like with 500-1000 peers each with 500-15,000 prefixes and some with a full table)

Lack of a full BGP community string matching capability (match a community exactly, but you can't filter based on one community in a list) (ie 8282:100,8282:101,8282:1010,3356:1044,3356:1012 and say export anything with 8282:101 - doesn't seem to work / let along saying export to these peers if 8282:1010 and 3356:1012 / etc)

and so on :(
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8308
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: What BGP setups need to be optimized

Thu Apr 30, 2015 7:12 pm

full table towards the CCR and 3-5 minute load times
huh... what is full view load time on cisco routers?..
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
jmorby
just joined
Posts: 13
Joined: Sat Jan 04, 2014 6:06 pm

Re: What BGP setups need to be optimized

Sun May 10, 2015 4:23 am

huh... what is full view load time on cisco routers?..
No idea ... we don't use them. Juniper is generally much quicker

And tbh I was being generous with 3-5 min ....
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1818
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: What BGP setups need to be optimized

Tue May 12, 2015 7:46 am

Problems today are already well reported ... and MT support have acknowledged them time and again saying "no head room to fix" (basically eluding to all efforts going on ROS7)

OSPF with the random incorrect flag state on sessions causing reloads and drops

Lack of a full BGP community string matching capability (match a community exactly, but you can't filter based on one community in a list) (ie 8282:100,8282:101,8282:1010,3356:1044,3356:1012 and say export anything with 8282:101 - doesn't seem to work / let along saying export to these peers if 8282:1010 and 3356:1012 / etc)
100% agree on these two points.

The OSPF issues affect us on a weekly basis. Lack of string/regex matching on communities in route filters is also making life difficult as we try and deliver more and more complex solutions with RouterOS.

RouterOS v7 is eagerly awaited by us :)
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA
 
wispwest
Member
Member
Posts: 477
Joined: Tue May 19, 2009 3:48 am

Re: What BGP setups need to be optimized

Fri May 15, 2015 12:03 am

We have (2) upstream providers, and (1) client, so (3) BGP instances being used.

At night, our Cloud Core Router won't process more than 900Mb aggregate, and I've narrowed it down to our CCR router.

All was fine until a recent surge in bandwidth usage from all our "snowbird" customers who came home and started using the net. We need a fix for this ASAP, or we'll have to dump MT and go with another solution.

Clearly, its a BGP problem that will be fixed in ROS v7? How long until we can test?

Also, is the amount of RAM important with running BGP? I was told it wasn't, then I seen in this thread that it is. We are using the CCR1036-12G-4S model, with 16Gb of RAM

It looks like one or two of the cores is being maxed out to 100% while all the rest are barely used at all. WHY isn't it spreading the processing power over all 36, rather than hammering one core for BGP???
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1818
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: What BGP setups need to be optimized

Fri May 15, 2015 4:05 am

Clearly, its a BGP problem that will be fixed in ROS v7? How long until we can test?

Also, is the amount of RAM important with running BGP? I was told it wasn't, then I seen in this thread that it is. We are using the CCR1036-12G-4S model, with 16Gb of RAM

It looks like one or two of the cores is being maxed out to 100% while all the rest are barely used at all. WHY isn't it spreading the processing power over all 36, rather than hammering one core for BGP???
This does not sound like it is related to BGP at all.

IP forwarding is already spread across all cores on CCR's, for UDP everything is, for TCP it is spread on a session by session basis. If you have one or two cores maxing out when traffic increases, you must be doing something to the traffic that is no optimised for the Tilera architecture. Typically this is queueing or tunnelling.

What version of RouterOS are you running ?
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA
 
aeg
just joined
Posts: 4
Joined: Wed Dec 17, 2014 2:55 am

Re: What BGP setups need to be optimized

Mon May 25, 2015 7:12 am

Hi Mrz,
Any chance to have support for SNMP BGP4-MIB in 6.x?


Thanks.

That would be of great help indead
 
User avatar
mrz
MikroTik Support
MikroTik Support
Topic Author
Posts: 5934
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: What BGP setups need to be optimized

Mon May 25, 2015 11:48 am

There will be no new routing feature in ROS v6.
 
Lonecrow
Member Candidate
Member Candidate
Posts: 129
Joined: Mon Dec 24, 2007 4:58 am

Re: What BGP setups need to be optimized

Tue Jun 02, 2015 5:41 pm

Hi Mrz,
Any chance to have support for SNMP BGP4-MIB in 6.x?


Thanks.

YES PLEASE
 
Lonecrow
Member Candidate
Member Candidate
Posts: 129
Joined: Mon Dec 24, 2007 4:58 am

Re: What BGP setups need to be optimized

Tue Jun 02, 2015 5:42 pm

There will be no new routing feature in ROS v6.
You have to be kidding...
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1818
Joined: Mon Jan 14, 2008 1:53 pm
Location: Straya
Contact:

Re: What BGP setups need to be optimized

Wed Jun 03, 2015 8:39 am

There will be no new routing feature in ROS v6.
You have to be kidding...
They have to draw a line in the sand somewhere.

Keeping on adding features to version 6 is a waste of developer time, it is better to let the developers work on v7 which will be a massive release.
http://thebrotherswisp.com/ | Mikrotik MTCNA, MTCRE, MTCINE | Fortinet FTCNA, FCNSP, FCT | Extreme Networks ENA
 
wispwest
Member
Member
Posts: 477
Joined: Tue May 19, 2009 3:48 am

Re: What BGP setups need to be optimized

Wed Jun 10, 2015 5:59 am

Clearly, its a BGP problem that will be fixed in ROS v7? How long until we can test?

Also, is the amount of RAM important with running BGP? I was told it wasn't, then I seen in this thread that it is. We are using the CCR1036-12G-4S model, with 16Gb of RAM

It looks like one or two of the cores is being maxed out to 100% while all the rest are barely used at all. WHY isn't it spreading the processing power over all 36, rather than hammering one core for BGP???
This does not sound like it is related to BGP at all.

IP forwarding is already spread across all cores on CCR's, for UDP everything is, for TCP it is spread on a session by session basis. If you have one or two cores maxing out when traffic increases, you must be doing something to the traffic that is no optimised for the Tilera architecture. Typically this is queueing or tunnelling.

What version of RouterOS are you running ?

Well, I'm at 6.29.1 now. Check out this Cacti graph attached. If there is any expert MIkrotik / BGP consultants out there, please let me know, as I really would like to assure things are running at maximum/best config.
You do not have the required permissions to view the files attached to this post.
 
hedele
Member
Member
Posts: 338
Joined: Tue Feb 24, 2009 11:23 pm

Re: What BGP setups need to be optimized

Wed Jun 10, 2015 3:56 pm

What you have seen is actually the BGP table scan hitting one of the CPUs for 100%. Unfortunately, once you get a full BGP table (550k routes) on a CCR, the single CPU core can barely finish one run of the table scan before the next one is due, which is the reason why one core stays more or less 100% loaded all the time. So yes, this is related to BGP, and no, there is nothing you can do about it (other than filter away /24s to reduce table size), and no, Mikrotik will not fix this behavior with ROS 6. One more reason why people are really hoping for ROS 7 to drop soon.
 
roadracer96
Forum Veteran
Forum Veteran
Posts: 714
Joined: Tue Aug 25, 2009 12:01 am

Re: What BGP setups need to be optimized

Thu Jun 11, 2015 8:54 pm

Multicore
SNMP monitoring/traps
Graceful restart
Flap dampening
Native multipathing


Would be nice:
IRR lookups
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8308
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: What BGP setups need to be optimized

Fri Jun 12, 2015 1:44 am

these are feature requests, not optimizations of current features ;)
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
Werbes
just joined
Posts: 1
Joined: Tue Oct 20, 2009 3:14 pm

Re: What BGP setups need to be optimized

Fri Aug 14, 2015 10:18 am

/system resource print
uptime: 4w7h32m34s
version: 6.30
build-time: Jul/08/2015 09:07:45
free-memory: 1266.1MiB
total-memory: 1956.3MiB
cpu: tilegx
cpu-count: 9
cpu-frequency: 1200MHz
cpu-load: 12%
free-hdd-space: 77.7MiB
total-hdd-space: 128.0MiB
architecture-name: tile
board-name: CCR1009-8G-1S-1S+
platform: MikroTik

/routing bgp peer print count-only
4

/ip route print count-only
975212

/ipv6 route print count-only
39318

/routing filter print count-only
5

No ospf / vrp

Who is online

Users browsing this forum: No registered users and 8 guests