Greetings. I’m about to implement a new network architecture. I’d like your help regarding the experience of using the equipment. I need to decide between a CCR1036 or a CCR2116 to use as CGNAT. Which would be the best option and why?
i think (if properly implemented) ccr2116 can get a noticiable boost by using
Offloading Fasttrack Connections
https://help.mikrotik.com/docs/spaces/ROS/pages/62390319/L3+Hardware+Offloading#L3HardwareOffloading-OffloadingFasttrackConnections
How much bandwidth?
I moved to 2116’s from 1036’s last year and haven’t had any issues. Only pushing about 5Gbps at peak, but the 2116’s are handling it fine.
HW offload for NAT kind of works. The few times I’ve tried enabling it, I get complaints from customers. I’m also syncing to a second NAT router and using VRRP in case one fails or needs maintenance.
The CCR1036 has been discontinued. I hear that they are still available second hand… only this would make me buy the CCR2116.
I have no experience with the CCR1036, but I have used the 2116 for CGNAT. They can easily push 10Gb+ even without fasttrack. Even with bonding the interface limit is practically around 20Gb, and I believe (though haven’t tested above 15Gb) that they can do it with fasttrack. My only complaint with this device is that Mikrotik didn’t include 25G ports.
The usual suggestion is to separate the roles, so you would want one device for queuing (with PPPoE termination, if that’s your architecture) with conntrack off and another device for CGNAT, where you enable fasttrack. For NAT, if you find performance lacking, you can always add another one.
Regarding hardware offloading: basically forget it. Only around 2K connections can be offloaded, which is way too few for CGNAT. Reports are that there is some development needed still (cough) in managing the offloading.
2k connections at 2mbps each connection can add up around 4gbps, for a device like ccr2116 with an estimated throughput of 20gbps thats around 20%, it is not something insignificant
i have seen a significant reduction in cpu usage with this offload feature in ccr2116 and also in ccr2216
For me it was unstable, meaning that connections were mysteriously interrupted. It could do what I wanted purely in software so I haven’t turned it back on since.
This was admittedly on a much earlier software version, and things around L3 offloading are steadily improving. So if it works well, why not use it 
There’s some hint in the docs that the software tries to offload the most intensive connections, so the effect may be greater than your (modest) calculation would suggest. Just out of curiosity, how much did your CPU usage drop?
This was admittedly on a much earlier software version, and things around L3 offloading are steadily improving. So if it works well, why not use it >
maybe, i only tested this since v7.14
how much did your CPU usage drop?
real scenarios at peak traffic hour, each router is on a different network (no vrrp):
ccr2116 v7.16.2
8.4gbps total traffic
3.5gbps offloaded (estimated from changes on interface/monitor-traffic aggregate values)
cpu 14-15% with offload enabled
disabling fasttrack-hw offload leads cpu to 22-24%
re-enabling fasttrack-hw offload cpu returns to 14-15%
300.000 active connections (taken from ip/firewall/connection/tracking/print values)
ccr2216 v7.16.2
12gbps total traffic
8gbps offloaded (estimated from changes on interface/monitor-traffic aggregate values)
cpu 16-18% with offload enabled
disabling fasttrack-hw offload leads cpu to 33-36%
re-enabling fasttrack-hw offload cpu returns to 16-18%
370.000 active connections (taken from ip/firewall/connection/tracking/print values)
Thanks for the detailed stats!
So the software actually makes quite good choices about what to offload. In one case 2.25k / 300k unloads 40% of traffic, in the other 4k / 370k unloads 65%. Nice. You were right, that’s absolutely worthwhile.
It’s good to hear some success stories here sometimes.
You said: estimated from changes on interface/monitor-traffic aggregate values
Sorry, but how did you estimate the amount offloaded?
Would you mind at some point repeating these stats checks, but also at the same time watching System > Resources > CPU > sort by CPU usage in descending order, and report back on what the average highest single core load is (how much load is the top-used core under on average at all times, both with and without hw offload)? Thanks!
I posted a different thread about this a few weeks back, but I’m currently tracking an issue with ROS 7.x where connection tracking suddenly breaks the forwarding of IPv4 fragments after some as-yet-undetermined threshold is crossed (can’t tell yet if it is throughput, PPS, # of tracked connections, or what). Since CCR2xxx series cannot run ROS 6.x, this bug prevents us from using ARM64-based CCRs for any kind of NAT. Even though it is EOL, this means 1036 is the better solution for us.
Those of you who are using 2116 or 2216 to do CGNAT & with large amounts of throughput (somewhere around 10Gbit) and tracked connections (~300K), would you mind running a test: Send larger-than-MTU pings (e.g. 2000 bytes) to one of your CGNAT customer IPs from the “internet” side of your CGNAT box, and see if you get back responses or not? (So e.g., “ping -l 2000 100.64.100.100”, going through the CGNAT box.) It would also be interesting to see if the results differ when you have Fasttrack HW offload enabled or disabled. (I had it disabled in all of my tests. Also, this bug is reproducible on all RouterOS hardware platforms when running 7.x, not just CCR2xxx, and of course that would seem to indicate the bug is in the pure-software implementation of connection tracking. I’m wondering if fasttrack hw offload “works around” the issue for any offloaded connections. Since it’s impossible to offload 100% of connections, though, this would of course not be a “fix”.)