I’m new to the forums and would call myself average in the network space, so please be gentle. A quick overview of the issue, then I’ll go in to how I got there. My router, server, and gaming PC are on SFP+ ports at 10GbE over copper. I have Comcast Gigabit cable internet which clocks in at over 1,400mb down on a good day using SpeedTest with between 9ms and 12ms latency. Over SFP+ I can reach full 1400mb speed. But over my Gigabit ports, I’m lucky to get around 200mb down. Latency is the same across the board. But continue reading to see the weird part.
Testing tools: SpeedTest, iperf3
Hardware: CRS354-48P-4S+2Q+ switch running SWOS 2.13 with SFP-10G-SR SFP+ adapters
CAT6A ethernet throughout the home
CAT7 between network devices (don’t judge)
pfSense router with an Intel X550-T2 adapter to support my modem’s 2.5GB interface
Arris modem
2x Ubiquiti U6-Pro APs
Various Gigabit devices
10GB and 1GB on gaming PC, 10GB on server
I first noticed an issue when my WIFI devices would connect to the AP from 800mb to over 2 gigabit (when I opened up the channel width to max), so the link speed looked good. But when I performed a SpeedTest, I was hitting around 200mb. Usually 175mb or so. New house, so I thought my cable terminations were junk, so I wanted to test the cables.
I tested gigabit ethernet using my laptop and clocked in at about 200mb.
I thought I should make sure the cables were good, so I tested various ports in the room using a variety of cables. All around 200mb down.
I tested 10gigabit on my gaming rig and got 1,350mb down. Internet is good, cable is good.
Switched my gaming rig from the SFP+ port to a gigabit port on the switch after confirming the chain looked solid, 200mb down. That’s when I realized that anything on my gigabit ports was not getting full gigabit internet, regardless of cable or device.
So then I started wondering if it was a translation issue between the gigabit and 10gig ports, so I set up an iperf3 server on my R710 server (4x gigabit and 2x 10gb on an Intel X540-T2 adapter) and began testing with that. I have one IP on the 10GB and one IP on a 1GB port, so I test to the port I want to use that way.
Everything works fine over iperf. 1gb > 10gb registers about 950mb down, which is right where it should be. 10gb > 10gb coming in at closer to 5gb transfer single-threaded, just fine.
So then I thought what if it’s some kind of issue communicating to the router from 1gb to the router’s 10gb SFP+ port, so I install iperf server on pfSense
Gigabit devices get full 1gb speed all the way to the router. But STILL only about 200mb down using SpeedTest.
Finally I move my router to a gigabit port and run all the same tests. They all pass. My gigabit ports are now getting full gigabit internet, all my access points are able to do 950+mb down…
So… there is something fishy when I run SpeedTest on a Gigabit port to the 10GB SFP+ uplinked pfSense router, but have no issues with iperf over the same connection. To me it comes down to protocol or something else that seems to go away when I connect the LAN interface to a Gigabit connection. I don’t have any funky configuration on the router, no traffic shaping, no non-standard transmit buffer sizes, and didn’t change anything when I switched uplink ports. I just looked this up, but both iperf and SpeedTest use TCP protocol. I’m a bit at a loss. Is this a bug with SwOS 2.13?
Any thoughts or suggestions? I appreciate any help I can get!