Hey guys, what are your real-world results on how many routes / peers a CCR2216 can handle? I know there are some BGP bugs between 7.16-7.18, which hopefully are fixed now.
Full tables and almost unlimited peers. How many do you need?
I’m looking for at least 4 Full tables, and around 50 peer at most.
I have one CCR2216 that I want to BGP Peer with Cogent twice, and Hurricane Electric twice, and then like 40 small peers, just CDN’s, etc. My Cogent and HE peers, like one is in Seattle, and the other is in Boise, ID so they’re in two different physical locations.
4 full tables shouldn’t be an issue with 16 GB RAM. Just curious, though: why do you need 4 full BGP tables? Are you running a big exchange or acting as a transit network?
We’re just a WISP up in North-Central Idaho.
The idea is to have redundancy, and as much connectivity as I can. I want to peer with Cogent in Seattle and Boise, and HE in Seattle and Boise, which in my experience would give me four full tables? Or is that not the case? I guess I’m assuming that I’d get full tables with each BGP peer?
Got it, redundancy makes sense. And yep, each BGP session with Cogent and HE at each location will give you a full table if you request it.
Hi,
Mikrotik advises the limits as shown here https://help.mikrotik.com/docs/spaces/ROS/pages/62390319/L3+Hardware+Offloading#L3HardwareOffloading-CCR2xxx,CRS3xx,CRS5xx:SwitchDX8000andDX4000Series.
I have peering with the local Internet exchange and Hurricane Electric which results in about 174k IPv4 routes on my CCR2216 running 17.6.2. I have a default route via another upstream provider.
This high number of routes has disabled the L3HW offloading and I get the following stats and you can see the ipv4-routes-hw has gone to zero. I also had a table full event in the event log:
/interface/ethernet/switch/l3hw-settings/monitor
ipv4-routes-total: 174137
ipv4-routes-hw: 0
ipv4-routes-cpu: 174136
ipv4-shortest-hw-prefix: 24
ipv4-hosts: 0
route-queue-size: 0
fasttrack-ipv4-conns: 0
fasttrack-hw-min-speed: 0
nexthop-cap: 8192
nexthop-usage: 85
If I stop some BGP sessions so that the total number of routes reduces to below 120k, the monitor shows that the HW routes start to climb, starting with the default route.
As I don’t have many customers, the CPU load is still trivial but as the OP said in the original post, hopefully this is fixed in a later release.
Just be aware of the documented limits.
Use version 19.1, many problems with version 17 in BGP are solved.
I strongly advise against using v7.19.1 in production since it’s just been released. We’re sticking with v7.15.3 until the 19.x branch proves stable enough for production use.
We have been running the CCR2116s with about 2 full tables not facing any issues so far on 7.15.x. I have to check on the L3HW offloading on some routes though. The routers did handle around 10 Gbit/s of traffic at peak without any issues / high cpu as well. We will connect one of them to a large European ixp soon as well.
A quick and easy way to get some significant l3hw gains is to suppress l3hw on bgp in and only offload your prefixes that you are advertising, or your upstream is advertising for you. This will result in all download packets (~70% of pps for typical eyeball network) being handled in hardware, and only uplink packets going through the CPU. Depending on your public IP space, it might only need one route in hardware.
Well, there are a bunch of other ways to hardware offload, but as long as you don’t throttle the CPU at 100%, there’s no need to worry about it.
But if you do find yourself hitting 100% all the time, it might be worth looking into route selection for hardware offload. There’s a thread about this from a while back where someone from MikroTik support explains how everything works and shares some useful tips and tricks in a really clear way.
Hi Larsa, do you think you could share the link?
Thanks.
I’m thinking about doing what it mentions on the Wiki, which is offloading TCP connections and not UDP connections, what are your thoughts?
Could you give some specifics on how to accomplish this?
Nope, not that one. I’ll see if I can find the thread that was a bit more recent.
Specifics will vary depending on how you get your routes, but a route filter rule with {set suppress-hw-offload yes} will prevent matching routes from being hardware offloaded.
In my case, I put this in my BGP-IN filter chains to suppress l3hw on all routes learned from my transit providers:
if (dst in 0.0.0.0/0) {set suppress-hw-offload yes}
My BGP border router gets my internal routes from OSPF, so I have the following filters in the ospf in chain to block l3hw for everything, and then to reenable it for routes in our public address space (we currently have a single /21):
if (dst in 0.0.0.0/0) {set suppress-hw-offload yes}
if (dst in x.x.x.0/21) {set suppress-hw-offload no}
I didn’t want to leave it with no upload packets getting hardware offloaded, so I also added some rules to my BGP-IN chains to offload routes originating from specific ASNs (Mostly from CDNs) via a num-list, and also another rule to offload prefixes that match a specific firewall address list. The result in my case is ~20k routes hardware offloaded, and about 90% of total pps getting hardware offloaded.
if (bgp-as-path [[:allow_l3offload:]]$) {set suppress-hw-offload no}
Do this to add ASN 16509 to num-list:
/routing filter num-list add disabled=no list=allow_l3offload range=16509
if (dst in l3hw_offload) {set suppress-hw-offload no}
Add prefixes to firewall list. Exact prefix has to match, in this example a route for 173.183.128.0/24 would not match the above filter rule, only the full /17 will:
/ip firewall address-list add address=173.183.128.0/17 list=l3hw_offload
This is a pretty long thread about CCR2216 L3HW offloading, but do check out @EdPa’s explanation, which sums things up spot on: http://forum.mikrotik.com/t/isp-ccr2216-l3hw-offloading-issues/182539/30 If you’re interested in deeper technical stuff at the chip level, you might also find @raimondsp’s explanation interesting: http://forum.mikrotik.com/t/how-does-l3hw-actually-works/155752/99 Also, don’t forget the MT docs about HW offload: RouterOS > Bridging and Switching > Route Configuration
And as I mentioned earlier, unless you’re pushing all cores on your 2216 to 100% constantly, there’s really no need to start fine-tuning with hardware offload for routing.
beyond L3 HW matter which is very relevant, keep in mind that with several million of routes on RIB, routing table related operations will take some time, forget about using GUI for routing table related tasks, you will need to rely on CLI and be patient for any command you issue
if you have high cpu usage this will even more noticiable