Community discussions

MikroTik App
 
tonyd
newbie
Topic Author
Posts: 49
Joined: Fri Jul 20, 2012 3:31 pm

Router & Switch Upgrade and 10G SFP+ - Getting less than 1G

Sun Jan 29, 2023 1:25 am

Hello,

I am upgrading my existing home network/lab, a CRS109-8G-1S-2HnD. I wanted to set up 10G between my Router, Switch, NAS(s), Proxmox Servers and workstation. My physical connection needs don't require a router or switch with a lot of ports. So I just got a CCR2004-16G-2S+ and a CRS305-1G-4S+. I've installed 10G NICs in my servers (Broadcom) and workstation (Intel). The servers are connected with 1.5m DAC cables, my workstation with a 9m DAC and a 1m DAC to crossconnect between the CCR2004's sfp1 port and the CRS305's sfp1 port. All interfaces show 10G link w/ no errors or dropped links. I left the CRS305 running the RouterOS vs switching to the SwitchOS mode, mostly because I'm very familiar with RouterOS, not so much with SwitchOS. I know there are several ways w/ Mikrotik to set up vlans and trunking. How I've done it over the years is I set up a vlan_interface_bridge with the vlans I use on my network on that bridge. Then I set up the Bridge VLANs with the ID's and the interfaces as Tagged or Untagged. Generally I find I have to set up the Bridge Port PVID with the vlan ID I want when setting up a port as not tagged. I've done this on both devices and and effectively have the interfaces in trunking mode. I've set up a /30 between the two devices on one of the vlans. When I run a bandwidth test from the switch, TCP, Random Data, in receive mode, I'm only seeing as high as 401.5Mbps but avgs around 380Mbps. UDP avgs around 1160Mbps. For TCP, that's no better than my 1G network prior my upgrade. What have I missed or completely misunderstand? Have I made a mistake in getting the CRS305? CPU peaks to about 27% on the CCR but 100% on the CRS. So that tells me something. I DO NOT have "Use IP Firewall" enabled on the bridge and left Allow Fast Path enabled. I've also set Spanning Tree Protocol mode to "none" and left "Ingress Filtering" disabled. There's no Queues defined. I haven't upgraded the CRS to v7, didn't think at this point it would have any bearing. Would really appreciate your experiences. If I haven't provided enough info, I'm happy to provide any additional.

Thanks
[tonyd@er1.lajolla.ccr] > /system/resource/print
                   uptime: 1w6d6h46m20s
                  version: 7.7 (stable)
               build-time: Jan/12/2023 07:35:45
         factory-software: 7.4.1
              free-memory: 3765.9MiB
             total-memory: 4032.0MiB
                      cpu: ARM64
                cpu-count: 4
                 cpu-load: 1%
           free-hdd-space: 105.5MiB
          total-hdd-space: 129.0MiB
  write-sect-since-reboot: 2171
         write-sect-total: 31160
               bad-blocks: 0%
        architecture-name: arm64
               board-name: CCR2004-16G-2S+
                 platform: MikroTik

[tonyd@10G-Switch] > interface monitor-traffic sfp-sfpplus1-router without-paging 
append 
                         name:  sfp-sfpplus1-router
        rx-packets-per-second:               33 080
           rx-bits-per-second:            402.2Mbps
     fp-rx-packets-per-second:               30 378
        fp-rx-bits-per-second:            368.3Mbps
        tx-packets-per-second:               16 647
           tx-bits-per-second:             10.0Mbps
     fp-tx-packets-per-second:               15 573
        fp-tx-bits-per-second:              8.9Mbps
    tx-queue-drops-per-second:                    0

[tonyd@10G-Switch] /system resource> print
                   uptime: 2d6h58m42s
                  version: 6.48.6 (long-term)
               build-time: Dec/03/2021 12:15:05
         factory-software: 6.46.8
              free-memory: 478.5MiB
             total-memory: 512.0MiB
                      cpu: ARMv7
                cpu-count: 1
            cpu-frequency: 800MHz
                 cpu-load: 100%
           free-hdd-space: 2296.0KiB
          total-hdd-space: 16.0MiB
  write-sect-since-reboot: 2616
         write-sect-total: 2616
               bad-blocks: 0%
        architecture-name: arm
               board-name: CRS305-1G-4S+
                 platform: MikroTik
 
tdw
Forum Guru
Forum Guru
Posts: 1843
Joined: Sat May 05, 2018 11:55 am

Re: Router & Switch Upgrade and 10G SFP+ - Getting less than 1G

Sun Jan 29, 2023 2:57 am

The CPU on the CRS isn't capable of handling 10Gb, use suitably powerful external devices for bandwidth testing.
 
tonyd
newbie
Topic Author
Posts: 49
Joined: Fri Jul 20, 2012 3:31 pm

Re: Router & Switch Upgrade and 10G SFP+ - Getting less than 1G

Sun Jan 29, 2023 5:05 pm

Thanks @tdw A few questions if I may. First question, I've seen some youtube vids ( @TheNetworkTrip & @Inquirinity ) showing how to set up Vlans using the Switch instead of setting them up using the Bridge->VLANs which uses the cpu. This passes the traffic directly using the switch chip vs going through the cpu. You have to set up a Bridge w/o VLAN Filtering and add all the relative ports under the Bridge->Ports tab ensuring that each has "Hardware Offload" enabled. I've done that. But the Switch config box doesn't show a VLANs tab like I see in the vids. In both videos the switches were using v6.47.1, v6.48.2 respectively. BTW, I upgraded the CRS305 to v7.7. I have also referenced (https://wiki.mikrotik.com/wiki/Manual:I ... Offloading). Second question, should I switch to SwitchOS? And third question, if the cpu isn't capable, then why is it sold as a 10G switch? Am I to understand that the CRS305 will "PASS" 10G, but that you can't "TEST" it using the CRS305?

Thanks again
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11439
Joined: Thu Mar 03, 2016 10:23 pm

Re: Router & Switch Upgrade and 10G SFP+ - Getting less than 1G

Sun Jan 29, 2023 5:39 pm

bridge can offload certain functions to underlying hardware switch. And CRS3xx supported offloading VLAN filtering to underlying hardware right from beginning. So here's the reason not to follow random youtube videos without any considerations.
Many devices received L2 HW offload since, your CCR included. Your CRS1xx did not, so you have to know your hardware when considering options of configuring certain things, such as VLANs.

Answer to second question is subjective. SwOS has easier interface but lacks lots of advanced functionality.

Switch is fine for passing traffic between its ports (i.e. switching). Bandwidth test (function of RouterOS), however, uses CPU to generate (and consume) traffic and built-in test is not exactly performance optimized.
 
tonyd
newbie
Topic Author
Posts: 49
Joined: Fri Jul 20, 2012 3:31 pm

Re: Router & Switch Upgrade and 10G SFP+ - Getting less than 1G

Sun Jan 29, 2023 6:55 pm

Thanks @mkx You are absolutely correct in your comment regarding "random" youtube videos. You do have to know your hardware and the context of both your hardware and what you are trying to accomplish. But, we all have benefited by research and using youtube as a resource so I don't want to poo poo it as an effective resource to use and to avoid. And I don't think that's what you are saying, but just be careful and have a clear understanding not taking what you see verbatim. In may case I understood from the feature matrix that the CRS305 did support hardware offload, etc. I consider myself quite familiar with RouterOS; I've configured and managed 100's of mtik devices used in a number of WISPs our company has acquired over the years. In every case, I was using RouterOS and so only had brief experience with SwOS. And me personally, found SwOS a bit clunky to use. But that's only my opinion. So if I understand you, and now thinking about it it makes sense, that the Bandwidth Test tool uses the CPU. So even if I continue to use RouterOS to configure my vlans, I should see the increased bandwidth capability when simply passing traffic through the switch across the ports/vlans. I admit, I had not yet set up an iperf or perfsonar node/vm and tested through the switch. I will do that to confirm. Super appreciate the feedback.

Thanks :)

Who is online

Users browsing this forum: jvanhambelgium and 86 guests