Slow internet speed but high LAN speed through CRS326

nofdak · September 8, 2022, 2:27am

I’m seeing low bandwidth (~250Mbps) when connected to a 1G RJ-45 port on my CRS326, but not when connected to the 10G SFP+ port. This low speed is only when going to the internet, I’m able to get the full 1Gbps to talk to my firewall/router via iperf3.

Here’s a poor diagram of my setup:
Untitled Diagram.drawio.png
Connection 1: 10G SFP+ to CRS326 SFP+2
Connection 2: 1g CAT6 to CRS326 ether10

Connection 1 → CRS326 → CRS309 → speedtest.net:

Speedtest by Ookla

     Server: Spectrum - Austin, TX (id = 16965)
        ISP: Spectrum
    Latency:    14.73 ms   (4.29 ms jitter)
   Download:  1174.86 Mbps (data used: 1.6 GB )                               
     Upload:    40.59 Mbps (data used: 19.8 MB )                               
Packet Loss:     0.0%

Connection 1 → CRS326 → CRS309 → iperf3 to firewall:

Connecting to host 172.16.10.1, port 5201
[  5] local 172.16.10.192 port 38474 connected to 172.16.10.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   658 MBytes  5.52 Gbits/sec   35   1.02 MBytes       
[  5]   1.00-2.00   sec   551 MBytes  4.62 Gbits/sec  299   1.41 MBytes       
[  5]   2.00-3.00   sec   672 MBytes  5.64 Gbits/sec    3   1.36 MBytes       
[  5]   3.00-4.00   sec   672 MBytes  5.64 Gbits/sec  137   1.10 MBytes       
[  5]   4.00-5.00   sec   676 MBytes  5.67 Gbits/sec  127    757 KBytes       
[  5]   5.00-6.00   sec   674 MBytes  5.65 Gbits/sec    0   1.23 MBytes       
[  5]   6.00-7.00   sec   671 MBytes  5.63 Gbits/sec   10   1.18 MBytes       
[  5]   7.00-8.00   sec   677 MBytes  5.68 Gbits/sec  136    848 KBytes       
[  5]   8.00-9.00   sec   674 MBytes  5.65 Gbits/sec    0   1.29 MBytes       
[  5]   9.00-10.00  sec   675 MBytes  5.66 Gbits/sec   86   1.25 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.45 GBytes  5.54 Gbits/sec  833             sender
[  5]   0.00-10.00  sec  6.44 GBytes  5.53 Gbits/sec                  receiver

iperf Done.

Connection 2 → CRS326 → CRS309 → speedtest.net:

Speedtest by Ookla

     Server: Spectrum - Austin, TX (id = 16965)
        ISP: Spectrum
    Latency:    14.82 ms   (2.25 ms jitter)
   Download:   266.18 Mbps (data used: 332.2 MB )                               
     Upload:    40.49 Mbps (data used: 19.4 MB )                               
Packet Loss:     0.0%

Connection 2 → CRS326 → CRS309 → iperf firewall:

Connecting to host 172.16.10.1, port 5201
[  5] local 172.16.10.108 port 44780 connected to 172.16.10.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   956 Mbits/sec    0    491 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   942 Mbits/sec    0    491 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec    0    491 KBytes       
[  5]   3.00-4.00   sec   111 MBytes   934 Mbits/sec    0    491 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   942 Mbits/sec    0    491 KBytes       
[  5]   5.00-6.00   sec   111 MBytes   934 Mbits/sec    0    491 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   942 Mbits/sec    0    491 KBytes       
[  5]   7.00-8.00   sec   111 MBytes   934 Mbits/sec    0    491 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   942 Mbits/sec    0    491 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   942 Mbits/sec    0    491 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec                  receiver

iperf Done.

My 1Gbps copper connection is clearly capable of talking to my firewall at line rate through the CRS326. But somehow, when reaching the internet through my CRS326, my connection is drastically reduced. Both Mikrotik switches are operating at L2, not handling any L3 routing, and the CPU usage of both is 0% during all tests, so none of their CPUs are being utilized. This happens both with 7.5 stable and 7.6beta6. Neither switch shows any errors or packet loss. At this point I’m at a loss for what’s happening. I assume some of my config is wrong but I’m not sure what it would be.

In case it’s relevant, here’s the relevant sections of my CRS326 config:

/interface bridge
add admin-mac=18:FD:74:47:E9:57 auto-mac=no comment=defconf ingress-filtering=\
    no name=bridge vlan-filtering=yes
/interface vlan
add interface=bridge name=vlan10 vlan-id=10
/interface ethernet switch
set 0 l3-hw-offloading=yes
/port
set 0 name=serial0
/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=no interface=ether10 pvid=\
    10
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp-sfpplus1

/interface bridge vlan
add bridge=bridge comment=LAN tagged=\
    bridge,sfp-sfpplus1,ether23,ether7,ether9,sfp-sfpplus2 vlan-ids=10

biomesh · September 8, 2022, 10:50am

If iperf both ways between switches work (with and without the -R option) then the issue would be at the firewall.

Check to see what the load is on the firewall and also check to see if you have enabled jumbo frames elsewhere in your network that is possibly being fragmented at the firewall.

nofdak · September 8, 2022, 11:31am

iperf works at 1Gbps, both to and from the firewall. CPU load on the firewall is negligible during either a full speed test through Connection 1 or through the slow test via Connection 2. I have no jumbo frames enabled anywhere in my network, and I’ve confirmed that in both CRS switch configs.

Do you know what I should look for at my firewall? And if it was the firewall, do you know why it’d work at full speed when going through the SFP port in my CRS326 but not when going through the RJ45 port?

biomesh · September 8, 2022, 2:33pm

I would make sure you test iperf going from the server to the client (either using -d or -R option). The output you show is the “upload” on your ISP test.

If that shows 1Gbps, then the switch is working properly - it doesn’t do anything special for out of your local subnet traffic unless you were routing. I would look at your local device (drivers, anti-virus, firewall, etc ) to see what the issue is.

nofdak · September 8, 2022, 9:46pm

I tested with -R and it works as expected.

❯ iperf3 -c 172.16.10.1
Connecting to host 172.16.10.1, port 5201
[  5] local 172.16.10.108 port 54558 connected to 172.16.10.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   960 Mbits/sec    0    498 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   942 Mbits/sec    0    498 KBytes       
[  5]   2.00-3.00   sec   111 MBytes   935 Mbits/sec    0    498 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   942 Mbits/sec    0    498 KBytes       
[  5]   4.00-5.00   sec   111 MBytes   934 Mbits/sec    0    498 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   942 Mbits/sec    0    498 KBytes       
[  5]   6.00-7.00   sec   111 MBytes   934 Mbits/sec    0    498 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   942 Mbits/sec    0    498 KBytes       
[  5]   8.00-9.00   sec   111 MBytes   934 Mbits/sec    0    498 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   942 Mbits/sec    0    498 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec                  receiver

iperf Done.
❯ iperf3 -c 172.16.10.1 -R
Connecting to host 172.16.10.1, port 5201
Reverse mode, remote host 172.16.10.1 is sending
[  5] local 172.16.10.108 port 36572 connected to 172.16.10.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   111 MBytes   935 Mbits/sec                  
[  5]   1.00-2.00   sec   111 MBytes   928 Mbits/sec                  
[  5]   2.00-3.00   sec   112 MBytes   936 Mbits/sec                  
[  5]   3.00-4.00   sec   111 MBytes   935 Mbits/sec                  
[  5]   4.00-5.00   sec   112 MBytes   937 Mbits/sec                  
[  5]   5.00-6.00   sec   111 MBytes   934 Mbits/sec                  
[  5]   6.00-7.00   sec   112 MBytes   936 Mbits/sec                  
[  5]   7.00-8.00   sec   112 MBytes   938 Mbits/sec                  
[  5]   8.00-9.00   sec   112 MBytes   936 Mbits/sec                  
[  5]   9.00-10.00  sec   112 MBytes   937 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   936 Mbits/sec  450             sender
[  5]   0.00-10.00  sec  1.09 GBytes   935 Mbits/sec                  receiver

iperf Done.

I would look at your local device (drivers, anti-virus, firewall, etc ) to see what the issue is.

I agree that the switch should be working properly and shouldn’t be doing anything to the traffic due to the fact that it’s not routing. It’s very possible it’s a local device problem, but every single local device connected to the 1G RJ-45 ports has this issue. My primary workstation downloads at >1Gbps when connected to the SFP port, but the same machine gets <500Mbps bandwidth through the RJ-45 port. Do you have any suggestions on what local settings might affect this or what I could look at? I see no packet loss or anything with any diagnosis I can look at, nor any high CPU load anywhere.

nofdak · September 8, 2022, 10:51pm

It looks like the problem is somehow the SFP+ link between the CRS326 and the CRS309. I changed the link between the two from the SFP+ ↔ SFP+ to CRS309 ether1 ↔ CRS326 ether1 and my speed issues seem to have been resolved:

CRS326 and CRS309 connected via SFP+ DAC:

Speedtest by Ookla

     Server: Spectrum - Austin, TX (id = 16965)
        ISP: Spectrum
    Latency:    14.82 ms   (2.25 ms jitter)
   Download:   266.18 Mbps (data used: 332.2 MB )                               
     Upload:    40.49 Mbps (data used: 19.4 MB )                               
Packet Loss:     0.0%

CRS326 and CRS309 connected via Cat 6 ether1 ↔ ether1 (noisy network, so I’m not expecting full 1Gbps):

   Speedtest by Ookla

     Server: Spectrum - Austin, TX (id = 16965)
        ISP: Spectrum
    Latency:    13.89 ms   (2.01 ms jitter)
   Download:   847.29 Mbps (data used: 996.5 MB )                               
     Upload:    41.34 Mbps (data used: 33.0 MB )                               
Packet Loss:     0.0%

The only change between these two results is the CRS326 to CRS309 connection. Does anyone have any idea why that would be the case?

Update: I tried an S+RJ10 in the CRS326’s SFP+ port and connected that to ether1 at 1Gbps and I get the proper fast bandwidth. I only get the reduced speeds when that port is operating at 10Gbps.