Hi there.
I have deployed quite a few rb5009 devices and in fact I have used one for home use for a few years. I am very pleased with it. I think your device choice is spot on, this device is probably the best bang for buck if you have a multi-gig setup.
As to what you can expect from this device:
- in pure software routing (including NAT, stateful firewalling, policy routing, queueing - so all features available): around 4 Gbps
- with what Mikrotik calls “fasttrack” - a software acceleration technique, that precludes the use of some features (policy routing, some queueing, etc.) but does NAT and stateful firewalling, expect 7-8 Gbps (what is different in the way Mikrotik does it compared to other vendors is that it can be applied to traffic selectively - so e.g. you can choose to accelerate inter-VLAN, but apply ACM queues for internet traffic, etc.)
- with no stateful firewalling (e.g. the “notrack” thingy that was mentioned) you can get quite close to 10 Gbps
These speeds can be achieved on any combination of internet/vlan-tagged/untagged/inter-vlan etc. paths.
So basically your routing is most probably somehow set up incorrectly. If you would give a detailed diagram, including the L3 switch, and include the setup of the L3 device, we could probably find out what your exact problem is or suggest an appropriate configuration. (Lots of other devices include/apply some settings “by default” or “automatically” - basically Mikrotik exposes the “ip” commands in Linux through a simplified interface, but the settings are just as detailed. You have to spell out things exactly - so simply saying that it works with something else configured somehow doesn’t really help. Maybe the rb5009 has to be configured differently, maybe another device, maybe both…)
One such “default” that can very easily be the source of your problems is that many manufacturers include a default “hairpin nat” rule, that is if an internal host addresses another internal host (sometimes only on the same network, sometimes for all internal networks… the exact form varies), then the source IP of the forwarded packet is changed from the sender to the address of the router. This does provide quite a bit of help in making otherwise incorrect configurations work, but it’s best to fix what’s broken instead of papering over it. (Mikrotik of course supports hairpin nat, it just has to be configures explicitly. In some cases it is considered necessary - and easier than correcting the network design - but generally networks designed from first principles do not need it.) Such a rule for example would solve the “triangular routing” problem . but again: it’s better practice to configure routing properly.
Please pay attention to the fact that the rb5009 comes with a default firewall configuration. This is only suitable for an “I have one internet and one home subnet” type configuration, so for yours it has to be modified. Did you do these modifications? Did you so them correctly?
Posting a full config helps a lot. (A config export can be created in the terminal with the /export file=whatever.rsc command. This has to be downloaded to your computer afterwards for further use. Please review what you post, and mask any details that you don’t wish to make public.) If you post a config please do it in a code block (the 5th icon in the forum interface.) You may - as you wish - post excerpts from your config (to accentuate a given part), but please always include the full one as well, because some configurations done in different parts of the config can easily affect each other.
The “notrack” suggestion is not bogus. It instructs the OS not to lookup or create connection tracking entries for given packets, and as such has performance benefits. Of course without connection tracking only stateless filtering of traffic (what many manufacturers call ACLs) is available, and of course no NAT (only for the packets that are notracked). This is a firewall feature, so that is where you will find it (IP->Firewall) and it has to be placed in the RAW table and prerouting chain. (The option is available both in winbox, webfig and cli.)
Just a suggestion: first fix the routing problem, and only after you reach the limitations that I have written above set out to optimize the design, e.g. use fasttrack, notrack, etc. While you have more serious problems these only confuse things.
Sorry for the wall of text and good luck!