We operate a WISP and have experienced growth pretty fast in our area. Each year having to upgrade our gateway fiber, first we had 100 mbps then 250 mbps then 500 mbps now we are at 1 gbps. Peak time hours we stay at 500 mbps. We just got our fiber upgrade completed today. I am able to move 945 mbps of bandwidth through the gateway device symmetrical until I place our router inline. I am trying to figure out if we need a better edge router or maybe if our collision domain is too large and overwhelming the switches. I will say this. After I was able to perform a speed test at 945/945 mbps connected right to the fiber gateway simply adding the router then plugging in my pc directly and only my pc to the router I can only get about 550 mbps of throughput, it seems the router is the bottleneck.

Our internal network is a flat layer 2 network / bridged. We have a single public ip on the wan interface of our router and a /24 public subnet on the lan side of our router that is routed through the wan ip on interface 1. If anyone has any insight on what I can try or what would help it would be greatly appreciated. If someone wants to consult on this I would gladly pay for some help as this network has grown past my expertise.

We are using a Mikrotik CCR1009-8G-1S-1S+ router and Netonix switches.

What kind of speed test are you doing? A single TCP connection will be limited by the CCR per-core frequency, but multiple connections should max out the link no problem. Test with iperf3 through the router for best results. Check profiler to see where load is.

It's a multi thread test such as speedtest.net I have also worked with my upstream provider to setup a speedtest called truespeed this is a test from their data center to my gateway. Same poor results on both test. Hooked directly to their copper gateway port I get tcp throughput in the 940mbps area. My PC also sees 900 mbps range with speedtest.net I understand their is not routing this way. The iperf test I'll try that from my router through the switch and to my pc and see what I get. Thanks for the insight. I'll post back my results. I need a sfp connection from the router to my switch soon it seems.

It's a multi thread test such as speedtest.net I have also worked with my upstream provider to setup a speedtest called truespeed this is a test from their data center to my gateway. Same poor results on both test. Hooked directly to their copper gateway port I get tcp throughput in the 940mbps area. My PC also sees 900 mbps range with speedtest.net I understand their is not routing this way. The iperf test I'll try that from my router through the switch and to my pc and see what I get. Thanks for the insight. I'll post back my results. I need a sfp connection from the router to my switch soon it seems.

The important thing on the CCR router is that no single CPU is maxed out. Run Tools->Profile on all CPUs and sort the list by CPU. If you see any individual CPU going up higher than 80 or 90% you are likely hitting a CPU bottleneck, then it is a matter of determining which process is responsible.

Since you stated that you're using a CCR1009-8G-1S-1S+ it is important to know which interfaces you are using for your backhaul and LAN side.
That one still has a switch chip which aggregates ether1-ether4 to a single 1G pipe to the CPU. If you have both interfaces in that port range, it might be a hint for the bottleneck.
Ether5...8 have dedicated 1G pipes.

Have a look at the block diagram of your device:


-Chris

Chris, WOW! I think you have found the possible cause for my issue. We are in deed using eth 1 for wan and eth 2,3,4 for lan. How difficult is it to swap over to say ETH 8? How would this arrangement best be accomplished?

That should be fairly easy - of course it depends on the complexity of your setup.
I'd strongly advise to schedule a downtime for doing this b/c you never know...

I'd create a script that adjusts all necessary settings (i.e. IP address interface binding, firewall rules with in/out-interface matchers, possible queues, static ARP entries, etc)
Then fire the script and move the plug. Done.

-Chris

"best practice" (although it will in theory cost a little bit of performance) is to always use a bridge for every network or link connected to your router.

For your LAN you already have a bridge, with ports 2,3,4 in it. Now also create a bridge for internet, and put port ether1 in it for now.
Move over everything that you have now configured on ether1 to that bridge. interface list memberships, IP addreses or DHCP client, firewall rules, etc etc.
To be really sure not to miss something, make a /export to a file and search the file for ether1 to see what you need to adjust.

When you are completely finished with that, you can add e.g. ether8 to that same bridge, move the connector from 1 to 8, and then you can remove ether1
from that bridge and put it in the local network bridge so you have a new port there.

With this setup, from now on you will no longer be dependent on the physical ports in use. E.g. should your router fail you can buy another model
and use the same config only with ports differently allocated to bridges.

When following advice by @pe1chl, be sure to set admin-mac=xxxxx and auto-mac=no to the newly created bridge. The MAC address must be unique in your LAN/WAN combination, but it is quite easy to invent one ... can be any local MAC address. If you don't do it, then bridge will assume MAC address of first interface attached (in your case ether1) and when you remove "MAC donor" interface from bridge, its MAC address changes, which in turn disturbs traffic through it. If you set unique MAC address to bridge, then none of changes (moving cable between interfaces, removing interface) makes any wider-scale harm (surely unplugging and re-plugging WAN cable disturbs traffic, but if done quickly, not many connections will notice that).