CCR2004 performance?

Hardware MikroTik hardware general
1

Hello!

I am running an small ISP and we are rebuilding basically our entire network.

Our current design is of no importance at all as we have decided on the new design topology, what we are trying to figure out is what device to place where.

We have decided on running a pair of servers with ROSX86 as service routers for our datacenters on each site we have, these routers will handle things like: Receive full BGP table from multiple transits and distribute to different service such as: Cloud hosting, Co-location services and handle any route selection for any of these services.

On the ISP side we have and are going with two CCR2116 to handle basically the same as above but instead the downstream is fiber ISP customers and these two devices also handles NAT for anyone not having an public IP.

Now here is the main question: I am seeing a LOT of conflicting information regarding the performance of the CCR2004 and what they are actually useful for and not but here is what we want to use them for and we want to ask you all, Is this a good usecase?

Basically we want on every transit have a single CCR2004 whose job only acts as an peering router towards a SINGLE upstream, If we have 2 locations then we will have 2 CCR2004, if we have 10 then we will have 10 of them. The job for these will be ultra simple. Recieve the full BGP table from the transit provider of the datacenter it is located in (We have L2 between all sites so we can go out on other sites transits if needed) and then provide this to all the service routers down stream, so for example the CCR2116 for the fiber ISP stuff, The X86 for the datacenter services and so on will all connect to these CCR2004 only to get the full tables from them and to advertise their services prefixes back to the internet.

THATS IT, no nat, No DHCP no PPOE, Just pure routing and providing a single full BGP table downstream.

There will be no communication between the two CCR2004 for BGP so they will not provide tables to eachother either, If a single CCR2004 fails then the service routers will just pick whichever other “Transit/Peering” router is available and best path in any other datacenter and exit that way instead.

Does anyone else do this?

What kind of performance do you see? We currently have 10Gbit per transit and are looking at dubbling that but after that we will rebuild the transit design, so the two Sfp+ ports of the lower end 2004 has more than enough linerate as we will NEVER see more than 20Gbit passing through these devices on a single site.

I know the CCR2004 is capable of this looking at the spec sheet for the tests but a LOT of people keep stating they only see 5 or 8 Gig on them which sounds VERY odd.

Money is a BIG question for us and just the default answer of “Go with 2116/2216 and solve all problems” Is not really welcome as it does not contribute at all as we would rather put that power and money where it matters more, Such as more service routing for additional datacenters.

Regards, Seneram.

2

Pass on CCR. You will start to hit limitations > 10Gbps. Most (and I say most) of what Tik does, is CPU bound. The CCRs simply does not have the CPU required. On the CRS devices, a lot of it is hardware offloaded, unfortunately that is not the case on the CCRs.

They make great edge devices, but that is about it.

3

This really does not make any sense, The CCRs dont have enough for what? Also, the CCR2116 and 2216 have the top end L3HW offloads far better than CRS. And CRS can ABSOLUTELY not do anything of what i describe above…..

Please explain your point of view as it goes against everything that is best practice and common knowledge.

4

CRS3xx, CRS5xx, CCR2116, CCR2216 switch chip features - RouterOS - MikroTik Documentation - read it, carefully.

In terms of Layer 3 (i.e. routing, not switching), the CCR2116 / 2216 does the following in hardware:

  • 32K or 128K (depending on model) IPv4 & IPv6 routes in hardware (a fraction of a full table)
  • Some bonding, not all
  • IGMP/DHCP Snooping (although, this is Layer 2, not Layer 3)

This is exactly the problem. You are talking about a SWITCH chip. There is a vast difference between SWITCHING, and ROUTING.

We are talking ROUTING, and you will be hard pressed to get > 10Gbps ROUTING on a CCR Everything, is processed in CPU, in terms of routing… Well, 99% of it.

5

I know the difference very well between an router and a switch. And the limitations thereof, But you are just plain wrong about your asumptions… the CCRs have really good processors for routing and ESPECIALLY the CCR2116 and the CCR2216 can hit very hight numbers on ROUTING, the CCR2116 has 16 gig of memory (can easily hold 8-12 full tables) and the 16 core CPU can easily route that at linerate, with proper configuration the CCR2216 can easily hit 100gig routing. The CRS have shittier stats, They have either the exact same limitations as the switch chip in the CCRs or worse in almost all cases… and the CPUs on the CRS can then do absolutely nothing in comparison…. i have 2116’s in use already on our service routers if you would have read my actual first post, I score easily 50+ Gbit on these. So maybe you should read your link carefully, you clearly do not know what you are talking about with it.

The topic has NOTHING to do with CRS vs CCR (Which is not even a topic to be had) And only about what peoples EXPERIENCE with the 2004 is.

6

Yep. Exactly. Over promise and they under deliver. But to each his own…

7

Something tells me you might be in the wrong forum….. Please if you do not have anything useful to add to the actual topic then refrain from writing…

8

Mate… you asked for an opinion - you got one.

Mikrotik’s own test results for the CCR2116 - https://mikrotik.com/product/ccr2116_12g_4splus#fndtn-testresults

As you will be passing 1500 frames, and you will have fw rules in order to protect the router, and you will be routing, not bridging / switching… You are looking at best, 36Gbps across all interfaces through the router, because that is all that the CPU can process.

https://mikrotik.com/product/ccr2216_1g_12xs_2xq#fndtn-testresults when you look at the CCR2216, their flagship 16 x 100G router. 1500 frames, < 25 fw rules to protect the control plane, in routed environment… 194Gbps at best, out of 1,200Gbps port capacity (1/10th basically).

But don’t mind me. I’ve only been using these devices for 25+ years. Search the forums, there’s plenty of stories about this topic. successes, as well as failures.

9

And that is already against what you said (That you wont easily get more than 10G on a CCR), I am not new to CCR nor CRS as i said, Have been using them for quite a while. You are way off topic here and i ask again to get back on topic or refrain from answering. Noone questions how long you have been using the devices. But you are making a lot of assumptions that are plain wrong and applying things that does not matter at all to the topic.

10

sigh.

At most 36Gbps. Add natting / firewall rules / mangle / etc, your performance will drastically drop.

Yes - BGP has grately improoved in v7. It’s not where it must be yet, but it is improoving. Your issue isn’t BGP, you can handle millions of routes. Your issue is PPS (Packets per Second). A CCR2004 will give you max ~14Gbps https://mikrotik.com/product/ccr2004_1g_12s_2xs#fndtn-testresults

So you wanting to deploy this for 10Gbps currently… Sure, it may work depending on how much firewalling / natting you are doing. Upgrading to 20Gbps… The devices will not handle the traffic.

Memory, is used for things like bGP, FIBs, Routing tables, etc. Actual packet processing, unfortunately is done by CPU. And some of those processes, even on v7, is also still single threaded, meaning it only uses 1 CPU core, irrespective of how many CPUs the device has.

This is ironically also, why a CHR will almost always outperform a CCR - faster cores, and more cores.

I hope this answers your exact questions. Look at the devices test results, you can form your own opinion based on how much traffic the device can push from Mikrotik’s official test results. Thats why they publish them.

11

Here you are wrong, even with NAT and FW rules, the device can achieve 50Gbps routing, simply because it supports L3HW with NAT (if you keep the number of tracked entries below the 2.25K limit). And @Seneram who has the devices and could measure the real performance already told you about the 50+ Gbps number.

On the devices with the chipset from this table L3 Hardware Offloading - RouterOS - MikroTik Documentation you can use the fasttrack rule with hw-offload=yes to let L3HW process the bulk of the packets (if the number of tracked connections is below the hardware limit, that's why the doc tells you to try to selectively apply the rule on important traffics), while keeping your firewall filter rules. The switch chip can even do NAT. The CPU will have to process those FW filter rules, and also setup NAT, but it only has to process them for the first packets of each connection only. If the connection can be fasttracked, with hw-offload=yes, the switch chip will handle the rest of the connection's packets. So you can take and apply the 50Gbps number of the Routing none (L3HW) row from the table you quoted.

But that is a feature that the CCR2004 doesn't have.

1 Like
12

I don’t think a 2004 is going to be a good fit for your intended use. According to what you’re planning, you’ll want to use fasttrack (or fastpath), and the relevant rule to establish throughput is to take the 512 byte / 25 rule test number and multiply it by 3-5.

This gives you 10 gbps - for sure, with a large margin; 15 gbps - almost surely; 20 gbps - expect occasional problems. So this doesn’t meet your reqs.

You should be aiming for the 2116. That can handle 3-4 times the above numbers, even disregarding hw offloading, which actually does help significantly. And it actually doesn’t cost all that much more.

What would be ideal in your situation (and lots of others) would be a 2116 with some additional higher speed ports. Alas, this isn’t available.

It would also be very useful if Mikrotik chose to allow filter/input rules with fastpath. I’ve been meaning to ask for this. Input packets are already always punted to the slow path, so the only change required for this would be a bit of additional nuance in the code for checking if fastpath can be enabled. One can hope, and this would be very useful in SP scenarios…

13

Heya, i was with you until you stated to take the results for using 25 rules, normally i agree on that but here not so much. There will not be any Firewall rules, no nat nothing like that. These will be as pure of a router as possible.

They only take packets from the uplink provider/transit and shuffle them down to the core where all the service routers/core routers can have fun. And even then there is no Firewall involved, that wont happen until after the service/datacenter/core routers when we hit the different customers/services Firewalls.

With this information, what would you say then regarding performance?

14

Your incredulity is merited and well received. As you might have gathered from my rant, I have quite strong views on this.

The numbers I gave were from memory, and I have reviewed the relevant product page. The 25 rules / 512 byte measurement is 4.85 Gbps. Multiply this by 5, that’s 24 Gbps. That you don’t intend to have any firewalling, just straight-up routing, you probably intend to run it in fastpath. For this, the figure published is 27 Gbps.

I have actually reproduced Mikrotik’s published stats for a number of devices and scenarios, and they are generally reproducible. However only with their stated methodology.

24 vs 27 Gbps is not so out of line to warrant incredulity. I simply base my estimates on some facts: 1% packet loss is generally not acceptable; packet sizes in real life differ somewhat; TCP connections (due to the congestion control) heavily throttle back even for small packet losses; balancing between cores is often imperfect, especially if you’re running BGP.

In short, I’m simply saying that either 24 or 27 Gbps is not enough overhead if you have definite plans for sustaining 20 Gbps. This combined with the fact that you may plan on replacement, this may not materialize because of external factors. This led to my suggestion. And the 2116 is really not that much more expensive.

15

It all depends, how to configure and use devices. For the context, we using CCR2004 with L3VPN over L2TP with firewall, and it could forward 920Mbps+ over this tunnel in each direction. However this is CPU intensive task. If you really using only pure IP routing in main routing table, then you can achieve 10Gbps through CCR2004. However there should be some tests between SW versions before going to prod. There should be tests before mass upgrades also.

16

The guy wants to start an “ISP” - He said he will nat. You really, realistically think it will be < 2.25K sessions tracked? There’s datasheets, and then there is being realistic.

17

You are wrong again.. i said i already run an ISP, i said these devices will NOT NAT. Our actual NAT routers for the fiber ISP are 2116 and they handle millions of tracked connections just fine.

18

As @lurker888 mentioned,

the CCR2004 12S+2XS can actually route 20-22 Gbps (aggregate) with FastPath, but the requirements are a bit complicated.

  • Stateful Firewall/FastTrack, VRF, queues, and hotspots cannot be used.

Without FastPath, throughput drops significantly to less than 10 Gbps.

19

Yeah this is what i read out of the spec sheet and what i am hoping someone could confirm or deny. Our use case as described above will not use any of those “CPU eating” functions. Just bgp and packet in packet out.

20

I can confirm this. What I also advise: you’ll probably have intermittent issues running them at 20 Gbps. There’s simply too little margin.

Just an example: there are all sorts of driver or other software changes that are pushed by MT, and these sometimes result in performance decreases (and also improvements.) Look at the posts on this forum, where while fixing some bug, there was a 10-20% decrease in performance in certain scenarios between two versions.

And a question: are you keen on using the 2004 because it has 25G ports? It’s what I would guess, and it’s really a shame that there is no 2116 available with the non-switch switch chip and post config of the bigger 2004.