Community discussions

MikroTik App
 
leo2kp
just joined
Topic Author
Posts: 1
Joined: Thu Jan 26, 2023 4:17 am

CRS354-48P-4S+2Q+ 1GB > 10GB slow internet performance

Sun Feb 05, 2023 6:19 pm

I've spent over a week troubleshooting an issue I've had since building out my new network several months ago.

TL:DR
The problem:
While both modem interfaces are set above 1GB linkspeed, gigabit devices (including those on my APs) only get ~200mb download on a 1.4GB internet plan. 10GB devices get full 1.4GB speed.
Work-around: Set router WAN/LAN interfaces from 10GB/10GB to 1GB/10GB, or 10GB/1GB. This gives all devices Gigabit internet performance, but of course I lose out on the other ~400mb.
The culprit: CRS35-48P switch in SwitchOS mode (v2.13)

Network:
Modem: Arris S33 (2.5G interface)
Router: Custom pfSense CE using Intel x550-T2 10GB NIC (capable of 1/2.5/5/10GBE) at v2.6, Snort, pfBlockerNG, OpenVPN
Switch: MikroTik CRS354-48P-4S+2Q+ in SWOS mode using 10GB SFP+ copper modules
10GBE devices: Gaming PC with AQtion AQC107 10GBE NIC on board, R710 server with Intel x540-T2. Both devices have stock settings (no Jumbo Frames, buffer or MTU changes)
Gigabit devices: Two Ubiquiti WIFI6-Pro APs, four laptops

More information:
This was a weird problem to troubleshoot. The strange thing is that in iPerf tests, all devices got full bandwidth between internal network nodes. What I've been able to track down is that while my modem NICs were BOTH at anything above 1GB linkspeed (for example 2.5G for the WAN port connected to the modem, 10GB on the LAN port...OR...adding another multi-speed 10G switch in between the modem and router so that both LAN and WAN ports can be 10GBE), my 10GBE devices could hit 1.4g download, but my 1GB devices, including WIFI, only clocked in at around 175-200mb.
In testing I discovered that putting EITHER of the two x550 ports to 1GBE allowed 1GB throughput out to the internet for any device, including 1GB devices. So the issue only presented itself while the x550 NIC had BOTH ports at above 1GBE linkspeed.
Again, iPerf tested great in all configurations between nodes and all the way to the x550 LAN port, even while it was at 10GBE linkspeed.
I spent most of my time troubleshooting the router, confirming buffers weren't being overrun, MTU was set properly, packet captures not telling me anything useful, CPU load was fine, disabled all of my plugins, and completely disabling flow-control. Nothing helped.
This morning I decided to make sure my switches had the latest firmware (SwitchOS was already at 2.13). They were running in SwitchOS mode because I didn't need anything above L2. I booted in to RouterOS and downloaded 7.7 to the CRS354, but didn't reboot right away. I wanted to test throughput in RouterOS.
Everything tested good - Gigabit and 10GB devices got all access to internet bandwidth. So I performed a reboot, updated the RouterBoard firmware, reboot again in to SwitchOS. I tested again in SwitchOS and the problem presented itself again - 1GB devices only got about 200mb throughput out to the internet, 10GB devices had full speed.
I rebooted back in to RouterOS, and performance was good again.

So anyway, I don't know what I can do to help make this in to a bug ticket. I work in technology as a sys admin but I'm not a very technical person, especially on the network side. I do what I can to get through this stuff ;)

Who is online

Users browsing this forum: No registered users and 13 guests