First of all let me clarify that I’m new to using fibre connections, but I’m trying to do my homework so that I can start off with the right foot.
Now to the actual question: I’m eying the CCR1016-12S-1S+ router and three CRS226-24G-2S+RM switches for my office. Assuming that all devices but the last switch are in the same rack (last switch being 6 floors away in the server room), would there be any downside to doing a linear connection to the switches so that I’d have CCR(SFP+)->CRS1(SFP+1), CRS1(SFP+2)->CRS2(SFP+1), and CRS2(SFP+2)->CRS3(SFP+1), or would I be better off using 1.25G SFP to connect all switches straight to the router’s SFP ports? Would the SFP+ DAC work in either scenario for the rack devices, or should I be using SFP transceivers and the appropriate LC cables for the latter scenario?
yes there is a downside. Maximum bandwidth you can transmit between the switches is 1.25Gb/s for SFP and 10Gb/s for SFP+ modules. So assuming you use SFP+ modules and more than 10 machines try to communicate at gigabit speeds you will get bottlenecks. It is very likely the switch would drop packets and the machine would have to retransmit. You will also need to create paths for them using RSTP or costs otherwise the traffic may end up going the wrong way first for the middle switch
You can use bonding on the CCR and one of the CRS and RSTP to create a redundant loop. Even if you dont connect 10 cables, having another path for packets to travel can help sometimes. Depending on how many ethernet cables you attach you will need to add the route cost higher than the SFP+ based on the difference in bandwidth. It all depends on how many ports you use for your machines and how many you have to spare. Just make sure to configure RSTP to check for loops as often as possible.
And also take in account that passing a switch doesn’t happen instantaneously, it takes time to receive the packet (or at least its L2 header), store it in the switch chip, get its vlan/mac address, make a switching decision and send it on the outgoing interfaces. A microsecond maybe, but it does add up when it comes to response times.