switch that supports more than four 40-Gig ports

RouterOS General
1

What is a decent switch that supports more than four 40-Gig ports ?

Mikrotik tops out at two or four 40-Gig ports , and I need at least 8 or more 40-Gig ports.

If Mikrotik had any switch with qty eight 40-Gig ports, I would have an order for at least two of them right away.

North Idaho Tom Jones

2

Mikrotik does not even have a 4Q+ switch.

3

OK - what is a decent cost-effective switch with at least qty one 1-Gig port and qty 8 (or more) 40-Gig ports.

FYI - I am in the process of :

  • selecting a switch where my multiple VMware ESXi servers can use 40-Gig interfaces to my multiple FreeNAS servers ( all talking at 40 Gig )
  • I also want to enable my multiple VMware ESXi Mikrotik CHRs to be able to use a vmxnet-3 40-gig interface through a second 40-gig switch to my multiple networks.
4

I have an alternative question: would you spend more than 10K ?

5

At 10K , that is a larger budget than I was wanting. I have seen some from D-Link for around 5K. But I was hoping for an 8-port ( or more ports) 40-Gig switch under 2K.

At 10k, I think I would skip the switches and just order lots of 40-Gig cards and run direct 40-Gig links between servers ( without a switch )

6

The only cheaper things I had seen were a 32-port HP for 6K or some modules.

7

Have you seen the switches on fs.com?

8

There are plenty of Arista and Dell switches on Ebay with more than four QSFP+ ports for below $2k.

9

Used or new? I saw similar used stuff on amazon.

10

used of course, new would be ± $4000.

@op
I bought new Juniper EX4600 switches - 24 x 10Gbps SFP+ and 4 x QSFP+ for ± $3800.
They also support expansion models e.g an extra 4 x QSFP+ and/or 8 x 10Gbps SFP_

11

STH reminded me of the Atom bug.

12

I really do NOT know how Mikrotik 40-Gig interfaces work ( or 40-Gig interfaces on any device … )

However ,
Question #1 – Am I correct that 40-Gig interfaces are actually four 10-Gig interfaces that are bonded by the network card and not by the operating system ?

This gets me to thinking … about the 40-Gig to four 10-Gig fan-out cables …
Question #2 – How does a Mikrotik with a 40-Gig interface know there is a “40-Gig cable with 40-Gig connectors on both ends of the cable” verses a 40-Gig interface with a “40-Gig fan-out cable to four 10-Gig interfaces on it” ?

Question #3 — Could something like this work to build a (more than 2 or 4 port) 40-Gig switch out of existing Mikrotik products - using something like what I will describe next ?

  • Switch #1 , A Mikrotik CRS354-48G-4S+2Q+RM ( four 10-Gig SFP+ ports - and - two 40-Gig ports )
  • Switch #2 , A Mikrotik CRS317-1G-16S+RM ( sixteen 10-Gig SFP+ ports)
  • Switch #3 , A Mikrotik CRS317-1G-16S+RM ( sixteen 10-Gig SFP+ ports)
  • Switch #4 , A Mikrotik CRS317-1G-16S+RM ( sixteen 10-Gig SFP+ ports)
  • Switch #5 , A Mikrotik CRS317-1G-16S+RM ( sixteen 10-Gig SFP+ ports)

Connections:

  • A 10-Gig SFP+ connection between Switch #1 and Switch # 2
  • A 10-Gig SFP+ connection from Switch #1 to your existing 10-Gig network
  • Qty sixteen Mikrotik Q+BC0003-S+ ( 40 Gbps QSFP+ break-out cable to 4x10G SFP+ )
    *** connect 10-Gig fan-out cable #1 to Switch 2
    *** connect 10-Gig fan-out cable #2 to Switch 3
    *** connect 10-Gig fan-out cable #3 to Switch 4
    *** connect 10-Gig fan-out cable #4 to Switch 5
    **** Connect all of the 40-Gig sides of the fan-out cable to sixteen computers with 40-Gig interfaces

So the resulting switch network is

  • ability to switch sixteen 40-Gig computers at 40-Gig rates ( using switches 2 & 3 & 4 & 5 )
  • ability for any 40-Gig computers to communicate to existing/other 10-Gig computers ( where 10-Gig only devices connect to switch # 1 ).

— Or is my thinking way wrong and not even close ???
— I don’t want to build it , but I want to understand what is actually happening with multi-port 10-Gig and multi-port 40-Gig switches.

North Idaho Tom Jones

13

If it’s a breakout cable, the module will work as 4x 10Gbps links, otherwise it will act as a 40Gbps connection.

14

Yea - but - How does the device know it it a 40-Gig connection verses four 10-Gig connections ? Is there some code/firmware in the cable that says “I am a 40-Gig cable” or “I am a 10-Gig fan-out cable” ?
I’m trying to get a grasp on what is happening at the hardware level and at the OS interface level.

15

Need more than 4x 40Gb ports then check Cisco 9500 series out..
https://www.cisco.com/c/en/us/products/switches/catalyst-9500-series-switches/index.html

16

Hi. Attached is a screen shot of what RouterOS shows for 2xQSFP’s in a CRS326-24S+2Q+
CRS326-24S+2Q+.png

17

What it shows does not depend on what is connected?