MikroTik

Which model can get me 500Mb/s or more at about 40K PPS over a GRE tunnel? Any of them? I assume CCRs are out because GRE can't use multiple cores, right?

No encryption, No VPN, just a GRE tunnel.

viewtopic.php?t=87892
I cannot find the forum post that discusses a fix to a re-ordering problem on tunnels using mutli-core. I believe it is resolved.

as long as there is no fragmentation in the game, packet reordering should have no practical effect, especially if fastrack is in play. it's just about stripping off the outer IP and GRE header and forwarding the encapsulated IP packet as it is towards its destination address. as far as i know routerOS does not support gre sequencing (at least it is not exposed to the cli, and i did not see any traces in my packet captures), so should be packets arriving in any particular order, they will be forwarded in the very same sequence - practically the router does not recognise they are not arriving in order. so it's the IP destination's job to buffer and handle the OO traffic.

but that's all theory and it depends how the code is written in RouterOS.

So yesterday I was given an opportunity to test this. Our provider to one of our PoPs dropped our 1Gb/s metro-e link after a manhole fire. In the building, we were able to find someone with a GPON service that's setup for 500/500. We only have an L3 switch there, and the switch cannot do anywhere near line-rate GRE tunneling since it is totally in software and is an old PowerPC 405 chip (cisco 3750). We had a spare RB2011, so I configured that and set it up to tunnel between this location with the dead metro-e connection and our main PoP using this GPON connection. Our customers all have public IPs, so I couldn't simply NAT them to this GPON connection... Routing is now just static - at our main PoP I point the subnets to the tunnel, at the Mikrotik, the same statics are set to point to the switch. For the other direction, the best I could come up with was policy routing - I mark everything inbound on the port facing the building switch and then have a route that matches that mark to push that traffic back to the tunnel. This works, but I feel like there's probably a better way. Normally we use OSPF, but I've only used that on Cisco so this was the "I fully understand how this works, so I'm doing it this way method".

So performance - not horrible, but not great. Impressive for the hardware I guess though. Getting roughly 90/90 and the cap here seems to be packets/second, which is maxing around 10K.

Firewall rules are minimal - just management access rules for 8291, 22, 80, 443. The packet marking is a single mangle rule. Profiling tells me this:
The other measurement I know of shows that almost all my CPU is in interrupt, which I'll admit I don't know what to make of on Mikrotik. Any thoughts on this? Any hardware recommendations, as I do want to setup a permanent backup using this tunnel method.

I'm especially curious about my packet marking - if that's really a pig, I can probably figure out OSPF.

90/90 on a 2011 is impressive! RB1100AHx2/4 or CCR1009 should easily get you line rate, probably with encryption too.