I used to have RB2011 as primary router between workstations. However I wanted to use firewall on bridge so fasttrack didn't work and I didn't want to use switch. Okay. Even with fasttrack RB2011 was bottlenecking at around 600-700mbps. Without fasttrack - it barely exceeded 120-150mbps when CPU was maxed (downclocked to 500mhz).

Okay so I decided i need better router capable of routing gigabit, makes sense right? Right. The next model after RB2011 is RB3011. Buuuuut... there's no PC case version and I don't have rack so it didn't work for me - out. Next one in PC case is CCR1009. Quite expensive beast for home router but okay - at least it does gigabit right? People around the internet say it's capable of routing full gigabit speed. Cool. It arrived and I immediately set it up. First thing was to copy firewall settings from RB2011. So it's almost 1:1 firewall wise. Also enabled bridge firewall. I checked fasttrack counters bilion of times - 0B fasttracked. Fasttrack doesn't work - just like on RB. So everything seems to be the same as on RB, just hardware upgraded. Cool. I thought gigabit is gonna be at least a bit challenging for CCR so I clocked it to 1200 mhz to see how it performs.

I started some initial benchmarks - fired up simple dd backup of Samsung SSD over LAN - it achieves gigabit easily ofc. I look at CPU load in /system resource and what I see?

1%

sometimes 2, but typically 0 or 1. Okay well... Data for sure is transferred because I see that both machines perform I/o and in fact load on both is quite high despite them being conventional x86 PCs. So I downclocked CCR1009 to 400mhz to see how much it bottlenecks then. Test repeat, CPU load reported:

2%

WTH. Okay. I entered /system/resource/irq and set all the stuff to 1st core. So I guess it realistically makes this machine more or less single core traffic wise. Test repeat, CPU load:

1%

...

So wait - RB2011 being 500mhz is capable of routing locally 120 mbps at 100% cpu load. And CCR1009 being 400mhz is capable of routing gigabit at 1% cpu load?... Okay even lets say this "single core" setting didn't work and it's still 9 cores. Then 9x 400mhz * 1% is still MUCH less than 1x 500mhz * 100%. Why is that? I'm not sure now if RB performs SOOOOO badly or CCR is some magic device that performs routing of established connections and connection tracking at wire speed with 0 cpu overhead. I checked it bilion times - there IS firewall between those 2 machines and ther IS NO fasttrack working.

SS: https://dl.dropbox.com/u/44131220/susecap325.png
I tried with both /dev/zero and /dev/sdg (ssd) but it's about the same. It's not like I'm complaining but... I feel a bit confused by both benchmarks shown on routerboard.com and math and common sense... In fact I feel confused by everything it just doesn't make sense to me at all...

it's a completely different CPU on those.
A new Intel i7 CPU does more work per Hz than an old Pentium..

it's a completely different CPU on those.
A new Intel i7 CPU does more work per Hz than an old Pentium..

I thought Tile Gx is just another power efficiency optimized CPU like some ARM or MIPS. I totally didn't expect it to be anything high performance. It's... wow. I can't wait for metaROUTER support on those machines. I'm going to immediately set up some gentoo on it and test how it performs against x86 in different network related workloads because it's impressive asf. Also I'm starting to wonder... If it does gigabit at 1% cpu... ._. 10G... 10G everywhere! 10G + 4G bonding and beyond!

It's true that CCR1009 is crazy fast, but the RB2011 could get more than you have. Maybe send your supout.rif file and test method to support@mikrotik.com and maybe there is something to optimise.

tile gx is generally tailored for HPC(super-computers, High-grade multi-chip, many-core workstations, etc) and especially in military(radars, sonars, satellite(imagery, radar, whatever) processing)
and thus it rely on built-in VLIW co-process/DSP/offloading thing HEAVILIY, completely unused by ROS.
its had neat fabric/bus within, but in terms of "generic apps IPC" its quite Meh, compared to say Phezy (among other MIPS64-alikes, it derived from, initially).
new, ARM-based things is more interesting, they shifted(much like MIPS do, or intel/AMD before)to SIMD most stuff in shuffled/updated silicon/arch ISA from proprietary DSP/offloading
and thats in result was making it more silicon/hz/power-efficient for networking, but sub-optimal, yet, esp compared to other vendors ManyCore SOC. but such changes(further that line)is about time and it will become more neat for routers.
problem is: Tilera rise prices for chips to quite ridiculous amounts, and nothing gonna change that, sadly.
(atleast until their major clients remain in US military/intel)