What BGP setups need to be optimized

mrz · February 20, 2015, 12:25pm

Before v7 is released we would like to optimize v6 routing.
Anyone who have performance issues with BGP, please describe in details what setup you are running and detailed problem description:

how many peers;
how many routes in routing table;
is there also OSPF,MPLS, VPLS, RIP etc running on the router;
what are the hardware specs;
are there routing filters;
average BGP update rate from upstream peers;
average amount of traffic forwarded over the router;
are there frequent BGP bgp peer flaps,
any other info that is relevant to routing and can be useful for optimization.

rextended · February 20, 2015, 2:48pm

CCR1036-12G-4S
RouterOS: 6.20 → 6.22 → 6.23 → 6.24 → 6.25 SFP not working at all → 6.24 again → 6.27
No one single problem or performance issue on BGP.

The only problem I haved (for me fixed on 6.27) with the machine (and also with it’s VRRP mirror) is the SFP driver not auto-sensing rightly.
Too much often BGP peer freeze not for the BGP itself, but for the poor quality of SFP link with S-85DLC05D 850nm

Now, with the 6.27 this problem is vanished (I never try 6.26).

Pawel · February 20, 2015, 7:20pm

Hi Mrz,
Any chance to have support for SNMP BGP4-MIB in 6.x?

Thanks.

anon40853160 · February 21, 2015, 12:59am

I’d like to see the BGP4 MIB too!

cheeze · February 21, 2015, 2:36am

MRZ, I’d say other than the single threaded process of evaluating routes/loading routes from the BGP RIB-IN to RIB LOCAL I don’t think I’ve had any problem come up.

Obviously more BGP address families would be nice but, that’s not really to the point of your question

joegoldman · February 21, 2015, 9:23am

Hi,

CCR1036’s, 11~ BGP peers, about 15k~ routes, although Im purposely not taking full table as the time to LOAD the table takes too much, and filtering through the routetable takes too long. This limits my ability to have multiple upstreams (the extra peers are IX’s and bi-lateral peering)

Run about average 15 routing filters (some are shared chains) per peer.

Really its just about time to import routes and time to filter through route table. Some filters are a bit quicker but, for instance, if I need to know what route 8.8.8.8 is taking, even on 15k routes takes too long to be worthwhile for troubleshooting. (/ip route pr where 8.8.8.8 in dst-address)

Thanks,
Joe

mandrade · February 21, 2015, 10:51am

+1 in this list…

patrick7 · February 21, 2015, 11:07am

+1

mandrade · February 21, 2015, 11:21am

Here is a REALLY NEEDED change on BGP : http://forum.mikrotik.com/t/bgp-multicore-load-balancing/84753/1

bronx · February 21, 2015, 11:53am

how many peers;
69 (some only IPv4, some only IPv6 )
how many routes in routing table;
645748 ipv4
64762 ipv6
is there also OSPF,MPLS, VPLS, RIP etc running on the router;
Also OSPF on the router,
what are the hardware specs;
CCR1036-12G-4S
are there routing filters;
175 routing filter
average BGP update rate from upstream peers;
70@19day uptime
average amount of traffic forwarded over the router;
330 Mbps
are there frequent BGP bgp peer flaps,
HoldTimer expired is the cause of peer flaps
any other info that is relevant to routing and can be useful for optimization.

You can connect to our system if you need.

lmm · February 21, 2015, 2:35pm

Hello, we tried to install CCR1036-8G-2S+ with 6.27 and connect it to our peers, the main trouble is performance of the BGP process even we had just two sessions with our route reflectors sending full internet BGP table (no filters attached) and one external peer sending full tables with easy filtering.

It takes about ~5-7mins to remove routes from mikrotik routing table if the external peer is disconnected.

how many peers;
3 with full internet tables, 7 just sending default route and receiving about 20 prefixes
how many routes in routing table;
about 500k

is there also OSPF,MPLS, VPLS, RIP etc running on the router;
no

what are the hardware specs;
CCR1036-8G-2S+, peers connected over 10Gb ports
are there routing filters;
just for BGP, there is no firewall
average BGP update rate from upstream peers;
about 10-20 per min
average amount of traffic forwarded over the router;
400 Mbps, tested more than 1500Mb/s in one stream with very low CPU usage (about 20-30% on one cpu)
are there frequent BGP bgp peer flaps,
no, if the Cisco RR is pushing the full table to CCR1036-8G-2S+, CCR1036-8G-2S+ is not able to send keep alives and prefixes back to Cisco RR, cisco output buffer is full for about 5mins
any other info that is relevant to routing and can be useful for optimization.
make route processing parallel please and it may be a good box for ISP usage

lm

lyberis1979 · February 21, 2015, 4:09pm

+1

paoloaga · February 22, 2015, 6:40am

Hello Mrz,

We have several BGP speaking routers on our network:

how many peers;

8 - 70

how many routes in routing table;

100 to 550.000

is there also OSPF,MPLS, VPLS, RIP etc running on the router;

BGP & OSPF

what are the hardware specs;

CCR 1036 - CCR 1016

are there routing filters;

Of course

average BGP update rate from upstream peers;

~25k/hour

average amount of traffic forwarded over the router;

200 - 500 Mbit/s

are there frequent BGP bgp peer flaps,

not frequent

any other info that is relevant to routing and can be useful for optimization.

Routes are received very quickly in latest releases (~ 1 min to load the full bgp table). But once loaded the table, routes become active much later (up to 15 minutes!). Also the updates received take a lot of time to become working (several minutes). I would like to see a more rapid network topology change.

The other very important issue is recursive next-hop lookup in IPv6 which is actually causing a lot of problems in deploying v6 on the network.

Access to routers is available if needed.

Thanks.

alexjhart · February 23, 2015, 1:32am

BGP performance hasn’t been too much of an issue. The things talked about at MUM for v7 sound great. As far as BGP related requests, these are some:

+1 BGP4-MIB

Also, adding the BGP AS source to netflow
Cisco offers either peer or origin:

The origin-as keyword specifies that export statistics include the originating autonomous system for the source and destination.
The peer-as keyword specifies that export statistics include the peer autonomous system for the source and destination.

If traffic flow is being improved, I would also request that fast path support isn’t negated with its use.

I already checked with support on the ability to use v7 as a pure route reflector and was happy to hear that functionality will be included.

Using multiple peers, especially on an internet exchange, both of these features become very important in managing many peers. The CCR series is becoming more and more powerful and RouterOS is improving software performance, but without these two important features, many will be stuck with the competition.

Thanks for checking in mrz. I’m looking forward to v7

nz_monkey · February 23, 2015, 8:05am

+1 for both of these. I have requested the Netflow v9 origin-as and peer-as features directly to Mikrotik support, they notified me that they will try and bring these to RouterOS v7 as it requires the new-routing engine.

This is already possible:

Create ip/vrf
Create bgp/instance
Assign bgp/instance to ip/vrf
Create bgp/vrf
Connect to peers and see their routes only in the VRF table you created above
Success!

We have been running this in production for many years now. Or maybe you mean something else ?

robertpenz · February 24, 2015, 8:59am

Answer is per router:

how many peers;
10
how many routes in routing table;
/ip route print count-only
1978010
/ipv6 route print count-only
43565
is there also OSPF,MPLS, VPLS, RIP etc running on the router;
OSPF
what are the hardware specs;
CCR1036-8G-2S+
are there routing filters;

yes

average BGP update rate from upstream peers;

20-30k / h

average amount of traffic forwarded over the router;

200Mbit

are there frequent BGP bgp peer flaps,

no

any other info that is relevant to routing and can be useful for optimization.
no really

Feature requests:

something like “show ip bgp summary” integrated … using the script from the forum is ok, but it is slow
Faster search for prefixes .. e.g. I want to check where a route goes it takes minutes not seconds
better monitoring via SMNP
Netflow v9 origin-as and peer-as

Muqatil · February 24, 2015, 3:23pm

how many peers;

/routing bgp peer print count-only 
24

how many routes in routing table;

/ip route print count-only 
1575550

is there also OSPF,MPLS, VPLS, RIP etc running on the router;
OSPF with around 2k routes
what are the hardware specs;

/system resource> pr
             uptime: 2w4d14h38m46s
            version: 6.26
         build-time: Feb/03/2015 15:18:36
        free-memory: 2948.7MiB
       total-memory: 3968.9MiB
                cpu: tilegx
          cpu-count: 36
      cpu-frequency: 1200MHz
           cpu-load: 4%
     free-hdd-space: 903.2MiB
    total-hdd-space: 1024.0MiB
  architecture-name: tile
         board-name: CCR1036-8G-2S+
           platform: MikroTik

are there routing filters;

/routing filter print count-only 
20

average BGP update rate from upstream peers;
Most of em stay up as router uptime
average amount of traffic forwarded over the router;
300Mbit/s
are there frequent BGP bgp peer flaps,
It’s in a IX so there are some flappings, but rare.
any other info that is relevant to routing and can be useful for optimization.
When i make some change on filters, router sometimes becomes not responsive, but the traffic keeps forwarded. Looks like the CPU is topped and winbox suffers it.
Route changes converge very slowly if one peer with Full route table is involved in some changes
CPU never goes up more than 5% but one core is always 100% (bgp process?)

If i open a winbox window of /ip route, after a while winbox freezes, and when i close and reopen the connection, winbox tries to load the 1M routes again.
/bgp advertisement in winbox is not usable at all

mandrade · February 24, 2015, 6:28pm

Here it is :

Peers Counter :

/routing bgp peer print count-only
27

IPv4 Routes Counter :

 /ip route print count-only 
639431

IPv6 Routes Counter :

/ipv6 route print count-only 
26376

Running OSPF too :

/routing ospf route print count-only 
12530

Hardware Specs :

 /system resource print 
             uptime: 1d3h47m38s
            version: 6.27
         build-time: Feb/11/2015 13:24:13
        free-memory: 2960.6MiB
       total-memory: 3968.9MiB
                cpu: tilegx
          cpu-count: 36
      cpu-frequency: 1200MHz
           cpu-load: 40%
     free-hdd-space: 906.0MiB
    total-hdd-space: 1024.0MiB
  architecture-name: tile
         board-name: CCR1036-12G-4S
           platform: MikroTik

Filter Counter :

 /routing filter print count-only
78

average BGP update rate from upstream peers : Most of em stay up as router uptime

average amount of traffic forwarded over the router : Our 95th Percentile is 2297.58 mbit (in+out)

relevant info : we really need a better multitask to BGP process and BGP-MIB implementation .

alexjhart · February 24, 2015, 8:32pm

I am talking about this: propagate BGP route updates without installing them in IP route table

nz_monkey · February 25, 2015, 1:09am

OK. That would indeed be useful.

The method we use is really a dirty hack, we only get away with it by having dedicated x86 VM’s as route reflectors.