Braintrust,
I just this week upgraded my internet via Frontier (US) to 2GB/2GB. They overprovision, so it's not uncommon to see 2200MB/sec transfers. I'm a 20 year MT user, so this upgrade with my new router should have been a snap.
They deliver on an ONT that has a 10GB RJ45 port on it. I added a Mikrotik "r2" SFP+ to RJ45 that shows temp and link advisements and negotiated speed to connect to the ONT from the CCR2004-16S-2S+ router on SFPPLUS2, and a Mikrotik S+DA0001 cable on SFPPLUS1 to link to the PC in test or the CRS328-4C-20S-4S+ CRS.
Default config from a factory reset (to rule out my normal config with a lot in it). No extra firewall rules, no queues, "factory". Grab DHCP from the ISP from sfpplus2, hand out 88.x IP's on the LAN sfpplus1 port. Fast track is enabled. Both ports are linked up at 10GB full duplex (no speed differences between the interfaces as i've seen discussed with the 5009 and the 2.5GB port).
Go do a speed test. I'm not quite getting full speed. About 10% less than if I go direct PC to the ONT with the RJ45 SFP and a Mellanox Pro 3 card. During speed tests, downloads hit about 13% cpu load overall, and uploads about 9% overall, looking at profile, nothing is spiking in usage, plenty of headroom.
Looking at the interface stats I think I have found the culprit. I see a that under RX Stats, RX Drop shoots up in lumps of 500-1500 at a pop, with a run of Ookla's speed test generating 30,000 or so drops on the interfaces each way per test.
I tried making a queue, using both cake and fc_codel at ranges starting at 1500 thru 1800MB/sec - just under my max speed. That resulted in another 20% speed loss, however the drops slowed down some (but didn't stop), so that didn't resolve it.
I turned on Flow Control and the drops reduced drastically (maybe 30 per speed test run), and I'd see TX pauses on the interfaces, but once again, I was slowed down to 20% of the speed I can get directly to the ONT.
Reading thru the forums, there seems to be some discussion regarding the CCR2004, buffer size on the hardware queue not being large enough, and other items, but nothing that's been a solid fix.
This only occurs under heavy load situations. Traffic under 1GB/sec doesn't generally create the drops, it does but just a few here and there.
Bufferbloat testing from Waveform direct to PC is an A usually, thru the router, B or C.
I realize this is only 200MB of speed loss so not a huge deal for a home lab, but the drops do cause some packet loss when it's loaded up.
Any thoughts or suggestions? More info I could provide? I was hoping this CCR2004 was going to be mature enough at this point to be solid. I'd post config, but again, it's the factory reset default config.
Thanks much,
Dave