Picking the right switch

Hardware MikroTik hardware general
1

I'm trying to find a cost-effective switch for use in a one-room office that currently uses 10GbE (mostly UTP) with a QNAP 4-port SFP+/RJ45 and 8-ports of 1G, and I added a NICGIGA 8-port 10GBASE-T switch when we added more 10G machines. This system can easily run at 10G wire speeds, but we're looking to go faster to make editing in Resolve and FCP smoother (and maybe allow multiple machines to access the file server at the same time).

We've found an issue with using an M4 Mac Studio as the SMB server with two DAS units on TB3 that affects network-share performance. So we're looking at upgrading to a NAS which will have an SFP+ port initially to connect to the QNAP switch <1m away, and with a pair of M.2 SSDs to cache data from the 8-drive RAID5, we can get >10G speeds. This would require either 25GbE or 40GbE, with the former having a couple of "compatible" PCIe cards available and the latter listing an EOL card.

So I need either a 25G switching solution or a 40G solution with ALSO has 10G connectivity back into the rest of the network. In other words, the Studio, the M4 MBP, and the NAS will all be connected to the switch at one of this link rates, BUT the switch needs to connect into the rest of the network [ROW] at 10G. The first MikroTek switch I found that comes close is the CRS504-4XQ-IN, but it does not have a [single] 10G port for the ROW interface. The QSFP28 ports can accommodate a breakout cable to give me four 25G ports each--which I would use for my three devices--but there is also the following statement:

In fact, you can mix and match all kinds of connections this way: 100 Gigabits on one port, 40 Gigabits on the other, 4x25 Gigabits on the third, and maybe even some 10G or 1G cables on the fourth.

I tried to verify this in the start up guide and the product manual, but they have no info (so I guess it's simply in the RouterOS manual?). Maybe a 10G/40G breakout cable with a 40G module is what's necessary? Or is it an oversight on the webpage?

As you can tell, I'm kind of a networking n00b at these link rates, and mostly a HW engineer. Thanks in advance for answers.

2

I was looking at those right before I fell into MikroTik land, and I'm glad I looked deeper.

Those are impressive on a price-per-port basis, but a RouterOS based switch will run circles around them on a per-feature basis.

If all you want is a switch, and never visit the management UI, these distinctions are not important. However, I do get the sense that you're outgrowing the "plug dem wahrs tugedda" stage of network engineering maturity.

…an issue…M4 Mac Studio as the SMB server…affects network-share performance

Care to be more specific?

Switching the Mac from SMB server to client might well make things worse.

First, you can't get around Amdahl's Law. People get over-focused on the 40 Gbit/sec bandwidth of Thunderbolt 3, but swapping in a 40 Gbit/sec Ethernet switch isn't 1:1. That top-line spec equivalence only comes in for bulk transfers.

Any time you have lots of small round-trip I/O, latency rears its head, and then you must contend with the fact that TB3 to local nVME SSDs is about as raw as you can hope for, access speed wise. There are multiple layers to push through, but they're all in kernel space, and heavily optimized; it's basically external PCIe. When you put those same SSDs on the other end of an Ethernet cable, you put a bunch of layers in the way:

OS → SMB → TCP → IP → Ethernet…

That's one end of the connection to get your data from user space onto the wire. Mirror that on the other end to get data back into user space, where the NAS SMB implementation can turn right back around and send your data into kernel space once more to get it onto/off the disk. (Kernel SMB is a thing, but it isn't as big a win in the face of all this as you'd hope for.)

There are alternatives like iSCSI and nVME/RDMA that can help by removing some of those I/O layers, but at a cost: you lose the "network-shared filesystem" abstraction, and concurrent access from multiple hosts becomes tricky.

All this absolutely can matter, in real-world situations. Let me tell you about my QNAP NAS, which takes minutes to list a ~10k entry directory over SMB on my macOS client after the first boot of the latter, but which will pull that list up in under a second over sshfs to a Linux box.

Why? The macOS SMB client and filesystem abstractions suck hard, for multiple reasons:

  • SMB rewritten in-house to avoid Samba's switch to GPL3, then not matched on performance
  • too much Classic MacOS baggage dragged forward over too many decades
  • too much tiny metadata written with the assumption that everything is DAS, bringing the latency problem in when it's NAS instead
  • other SMB implementations written for a Windows-first world, bringing in assorted pain like NFC vs NFD for Unicode file names, inability to handle certain ASCII characters out of a blind assumption that MS-DOS limitations still matter, etc.

You can find partial solutions to all this with a web search, but you'll never get NAS to achieve 1:1 speed with DAS except in the best possible case, the kind of thing you find in breathless benchmarketing articles.

I tried to verify [the claim that speed-mixing is fine]…

Switches do indeed allow each port to run at a different speed. What they do not do, however, is magically solve the queueing problems that result. If one host can send data faster than the other, it has to back up somewhere, often multiple somewheres.

The lowest level of this is Ethernet flow control, which tells the higher levels, "Hey, stop, you're choking me!"

Above that, you have TCP flow control, presuming you have a TCP-based protocol. UDP lacks built-in flow control, although some UDP-based protocols strap it on after the fact.

Then there's queueing, but that's largely a routing feature due to the need for big buffers and a big CPU to manage it. Even with L3HW, the best you will get on a switch are QoS controls, which are more about contention management than single-host throughput.

TANSTAAFL.

I'm not trying to talk you out of buying a big badass MT 40G switch. All I'm telling you is, don't expect miracles.

3

THAT was an AMAZING response. Thank you.

Those are impressive on a price-per-port basis, but a RouterOS based switch will run circles around them on a per-feature basis.

I've had the QNAP switch for a while (since June '22) and it seemed like the best choice for four 10G ports plus a bunch of 1G because that's all we needed at the time. It didn't need to be managed, and the upstream port back to my Netgear 24-port (8 POE) switch was still 1G. The QNAP was therefore adequate. It also gave me 10G wire speeds (~1000MB/s) locally.

Upon urging by my editor, I expanded 10G from that location to my office where the equipment rack is to a second identical QNAP switch which then fed (a) my Studio in my office, (b) the QNAP in his office, and (c) the port in his bedroom. This year I improved the topology by going 10G at the source with a UC-Fiber gateway (2.5G to the MODEM with 1G service), a USW-Aggregation at the main 10G switch, and a USW-Flex 2.5G POE switch fed 10G from the Agg. This latter is because 3 of the 7 APs were upgraded from WiFi5 to WiFi7 and used 2.5G POE+ connections. The 2 QNAPs are fed from the Agg.

Originally, when the house was built 7 years ago, my IT group used an EdgeRouter (ER-X) with a CloudKey (UCK-G2) and seven AC-Pro APs. I specified the LinkSys switch to handle almost the whole house with POE for the APs. (I had a security camera system that ran from its DVR to the ER-X.)

But as we started getting into 4K60 video and now 8K60 to get "the best possible quality until YouTube transcodes it", BW requirements increased. And to keep ballooning source material from eating us alive, we commissioned a server farm to transcode into H265 or AV1 outside of the normal editing paths. (We have 8 M4 Minis running 24-7 with their own little RAID box that does bulk transfers to the main storage. Handbrake is used to cue jobs on each, and one Mini acts as the file server. They all sit on a 1G switch with a 10G line running back to the Agg.)

Since we updated to Tahoe last fall, we've been seeing increasing pauses in Resolve during edits. We thought that it was a network issue, and ironically, this is where we uncovered the Mac SMB Fileshare anomaly which we call "The 8GB problem." [NOTE: We don't think it's the root cause of the Resolve problem as it is also happening on the Studio with the DASs.]

The 8GB problem is where we exceed the Studio's [SMB] buffer of ~8GB and it invokes flow-control to stop the remote Mac from writing data. This is exhibited by running AJA speed test with 16GB files and the test pausing for ~10s every 8GB (i.e., halfway through the 16GB test and again at the end). This issue does not occur ON the Studio when editing there and using its DASs, so its obviously a network issue. The MAC does not transfer data to the DAS immediately (no disk activity lights blinking) until just before the 8GB limit hits, so it looks like the Mac panics, begins to frantically send the data out the TB3 link, and then resumes once the buffer is empty (and then the activity LEDs stop blinking).

So, solution #1 is to take the Mac out of the path by using a NAS that does not exhibit the issue. (Note that was also have a SoftRAID DAS attached to the Studio that does not suffer from the 8GB Problem, but its also about half peak 10G speeds--my Studio to his Studio--at 500-550MB/s writes and ~450MB/s writes.) The NAS we tested--a PegasusPro R8 DAS/NAS--confirmed with "old" 4TB drives that the 8GB problem goes away when using the NAS: it gets 10G wire speeds with no pauses in AJA. Unfortunately, the Pro does not do everything we want, so we're looking at a different NAS solution.

This leads us to the original question: Can we find a cheap switch above 10G so we can take advantage of the caching SSDs in the new NAS. Also, how fast with the RAID5 8-drive spinning platters ever go, and by how much will they exceed the 10G wire speed at their outer diameter? (I thought they could hit around 1500MB/s, but I haven't been able to replicate that. On my DAS, I get 900-1050 writes and 1100 reads on the RAID5 with 16TB drives, and 1100-1200 write and 1350 reads on the 18TB equivalent. (So, faster than 10G wire speeds.)

The NAS I'm looking at can take a 10G card, and there's a 10G card with four ports that supports aggregation--so I'm leaning towards 40GbE because getting 40G/10G switches that also support aggregation seem to be cheaper than the other way (straight 25G or 100G/25G). I expect that I can get incremental improvements from the spinning disks similar to what I quoted above on my Studio to my DASs over TB3. I realize that there is more [and different] overhead over Ethernet, but I want at least the option to test that.

And adding the caching the data off the spinning disks in the NAS should double that performance once the entire video file has been read--which is kind of the goal. If I RAID0 a pair of 2TB M.2 SSDs which have 3000MB/s performance each, then I'm at ludicrous speed. This would probably definitely max out 25GbE (at least using one port) and may come close to maxing 40GbE.

So those are my expectations. And to make the tests work, I need to find gear that enables it. (I haven't mentioned finding the NICs for the Macs...]

4

Your exact choice of new NAS isn’t on-topic here, but do beware that they tend to have underclassed CPUs, and you absolutely need additional CPU grunt to increase data throughput. A 10-year-old Celeron design won’t push 40G no matter which NIC you attach.

I have no personal experience with these but they’re worth your consideration for not having a ridiculous CPU bottleneck, at least. Whether the rest holds up for your purposes, I could but speculate.

5

That’s not entirely true.

SMB Direct and NFS over RDMA keep the normal shared-filesystem semantics. What RDMA changes is the data path: bulk file data can move directly from the sending host’s memory to the target host through the NIC using RDMA, avoiding the standard TCP/IP processing and memory copying.

If the goal is genuinely high throughput for file sharing at that speed, RDMA is the only choice.

6

I have seen poor numbers for iSCSI in RAIDs, specifically in benchmarks for the Ubiquiti ENAS; but the excuse offered by the reviewer/tester was that this was such a new product that there might be teething pains in the OS that will make iSCSI performance increase in the near future. Otherwise, I have no knowledge about it.

Similarly, I have seen some AI clustering videos (notably by Alex Ziskind) using the TB ports on Studios to leverage the unified memory the gets clumped together with this technology. The Interwebs state that "Macs support RDMA over TB5 since MacOS 26.2" which was earlier this year. However, we won't be using TB5 for the NAS ports and only TB4 on the "DAS" port.

To kind of tie the two comments together (tangent's and Larsa's), the current choice for NAS is the QNAP TSV-h874T 8-bay model with a 12-gen i7 12-core CPU, which I hope is not "underclassed". All of my previous NASs [which will remain unnamed] have all been underwhelming for what we wanted to do with them, so we're not skimping this time. (Yes, we're NOT getting the i9 version because would be "overclassed" IMO.)

My understanding of RDMA is that it needs to be implemented on both sides. Is that correct? Do Macs support it over Ethernet, and does the choice of NIC matter, or is RDMA above that layer? Also, the goal is higher throughput, which could be considered lower than high throughput. In other words, is there an easy, simply way to significantly move beyond 10GbE single-link speeds by 20-100%. If I'm being ridiculous, let me know. This is all great education.

7

You’re mixing two different things here: link speed and protocol overhead.

Going from 10GbE to 25/40/100GbE only gives you a faster pipe. It does not fix the overhead in the normal SMB, NFS or iSCSI host endpoint (i.e NIC + network and storage stack). If you actually want storage protocols to scale cleanly into those speeds, RDMA is the only proper solution like SMB Direct for SMB, NFS/RDMA for NFS, iSER for iSCSI and NVMe/RDMA.

And yes, RDMA is end-to-end. Both ends need a compatible RDMA network stack and NIC. On Linux you can even use Soft-RDMA with a standard NIC, but macOS does not currently provide Soft-RoCE over ordinary Ethernet.

RoCEv2 and similar IP based storage protocols can run across normal Ethernet switches, but if you want it to behave properly like to minimize latency under heavy load, you need switches with PFC/ECN support. MikroTik supports exactly that on several of its higher-end Prestera-based switches.

TL;DR on macOS 26 and RDMA:

macOS 26 technically has RDMA, but not the kind you need for Ethernet storage.

Apple supports RDMA over Thunderbolt 5 between Apple Silicon Macs. macOS currently has no general-purpose Ethernet RoCE/iWARP, no SMB Direct, no NFS/RDMA, no iSER and no NVMe/RDMA.

So for a normal Mac > Ethernet > NAS, forget RDMA today. It simply isn’t there.

If you still want seriously fast storage on a Mac, you have to pay for specialized third party alternatives like for example:

  • ATTO 100GbE + SMB Overdrive: proprietary "RDMA like" optimization of ordinary SMB/TCP. ATTO claims >10 GB/s. Roughly $1,500 for a 100GbE NIC or around $2,700 for Thunderbolt 5 -> 100GbE, before the rest like switches, optics and storage.

  • 64Gb Fibre Channel: mature, fast and fully supported on macOS. About $5,500 just for a dual-port Thunderbolt FC adapter, before the SAN infrastructure. A complete setup can easily run into tens of thousands of dollars.

  • iSCSI:ATTO Xtend SAN is around $250, but it is still ordinary iSCSI/TCP. No iSER.

  • RoCEv2/InfiniBand: projects like Verbifrost are trying to make Mellanox ConnectX RDMA work on macOS, but today it is experimental, not production-ready.

So yeah, you can get 20–100% beyond 10GbE without RDMA with a really fast multi core CPU. But once you start talking about sustained 25/40/50/100GbE-class storage performance, there is no free lunch. On platforms with proper RDMA support, this is the way to go. On macOS, you currently have to work around the lack of it with third party solutions, like ATTO, Fibre Channel etc.