I used to have RB2011 as primary router between workstations. However I wanted to use firewall on bridge so fasttrack didn't work and I didn't want to use switch. Okay. Even with fasttrack RB2011 was bottlenecking at around 600-700mbps. Without fasttrack - it barely exceeded 120-150mbps when CPU was maxed (downclocked to 500mhz).
Okay so I decided i need better router capable of routing gigabit, makes sense right? Right. The next model after RB2011 is RB3011. Buuuuut... there's no PC case version and I don't have rack so it didn't work for me - out. Next one in PC case is CCR1009. Quite expensive beast for home router but okay - at least it does gigabit right? People around the internet say it's capable of routing full gigabit speed. Cool. It arrived and I immediately set it up. First thing was to copy firewall settings from RB2011. So it's almost 1:1 firewall wise. Also enabled bridge firewall. I checked fasttrack counters bilion of times - 0B fasttracked. Fasttrack doesn't work - just like on RB. So everything seems to be the same as on RB, just hardware upgraded. Cool. I thought gigabit is gonna be at least a bit challenging for CCR so I clocked it to 1200 mhz to see how it performs.
I started some initial benchmarks - fired up simple dd backup of Samsung SSD over LAN - it achieves gigabit easily ofc. I look at CPU load in /system resource and what I see?
sometimes 2, but typically 0 or 1. Okay well... Data for sure is transferred because I see that both machines perform I/o and in fact load on both is quite high despite them being conventional x86 PCs. So I downclocked CCR1009 to 400mhz to see how much it bottlenecks then. Test repeat, CPU load reported:
WTH. Okay. I entered /system/resource/irq and set all the stuff to 1st core. So I guess it realistically makes this machine more or less single core traffic wise. Test repeat, CPU load:
So wait - RB2011 being 500mhz is capable of routing locally 120 mbps at 100% cpu load. And CCR1009 being 400mhz is capable of routing gigabit at 1% cpu load?... Okay even lets say this "single core" setting didn't work and it's still 9 cores. Then 9x 400mhz * 1% is still MUCH less than 1x 500mhz * 100%. Why is that? I'm not sure now if RB performs SOOOOO badly or CCR is some magic device that performs routing of established connections and connection tracking at wire speed with 0 cpu overhead. I checked it bilion times - there IS firewall between those 2 machines and ther IS NO fasttrack working.
I tried with both /dev/zero and /dev/sdg (ssd) but it's about the same. It's not like I'm complaining but... I feel a bit confused by both benchmarks shown on routerboard.com and math and common sense... In fact I feel confused by everything it just doesn't make sense to me at all...