Any members using Mikrotik in long haul fiber? I’m curious what your experience is?
For the bulk of my fiber cores I will be using EDFAs at 10 points along my 250 kilomiter path, but i will be taking 1 pair of fiber for local distribution along that path. I am planning to use a CRS309 at each relay point with Chinese 10G 80 Kilomiter SFP Transducers.
What do you think about the CRS309 for this task? It seems to be a good compromise between price and performance.
What else have you seen used for a similar application?
You probably need to provide some more details to get a decent answer, such as: Is it for a business case using dark fiber? A single link or an aggregator of multiple 10G links to different locations (how many)? Is there a need for redundancy? Which compatible SFP+ modules? Need L2 filtering?
It’s actually a new fiber line which I am laying through rual Congo. Some of the relay points are at major crossroads which lead to larger towns and hold potential for future growth.
In general these switches provide a way for us to serve some smaller villages which the fiber already passes through, which would not otherwise make good business sense. This is on the social side of the business. The main business is just ip transit.
Redundancy is not important.
Layer 2 filtering could be helpful as time passes and the network grows.
The actual SFP+ modules are YXF-PP-S55L-80D which the manufacturer has said is compatible with Mikrotik brand devices.
First of all, I don’t think a passive (fanless) switch like the CRS309 is a good choice for a climate like Congo’s, especially with 10G SFPs that can generate a lot of heat. Secondly, you really need to do some serious hands-on testing to ensure those YXFiber China modules actually work with MikroTik.
Just curious, since it seems like a lot of digging in rural Congo: are you using trenching or regular excavation machines? What kind of ducts and fibers are you using? Is this a fiber-to-home project, or how do you plan to connect the consumers? Do you have a network architect involved in the project?
For high ambient temperature, CRS309 needs good air flow; assure 1U clearance above and below each unit; consider temperature controlled fan to vent the cabinet.
10G-Base-T modules get quite hot, be prepared to 3D print side fan mount for 120mm x 32mm centrifugal blower to move air sideways through existing vent holes.
Components are readily available to control blower speed for constant or temperature controlled speeds.
The suggested modifications and tips for cabinet ventilation probably work great, but I’d try to avoid DIY solutions as much as possible. When you’re building fiber solutions in rural areas with long distances between all tech hubs and distributed switch cabinets, you definitely want everything to be rock-solid.
I think CRS309 and his Bigger brother CRS317 have decent performance, specially with recent RouterOS releases, a good practice will be to avoid mixing different port speeds on backbone switch, if you mantain all your interfaces at 10g performance will be very good, if you need to provide 1g access interfaces while your backbone interfaces are at 10g switch will have to queue packets going towards 1g interfaces with risk of dropping packets, and if you saturate one 1g interface the problem will aggravate a lot, in that case will be better to add an aditional access switch conected at 10g to CRS309/317 to provide 1g access interfaces and contain all potencial queing and packet drop at access switch without impact to backbone switch
Recent RouterOS Releases offer QoS HW offload featureon CRS 3xx/5xx switches allowing to monitor and tune switch queuing to mitigate to some extent queuing issues, but this has a limited scope because as in most switches hardware queues have a limited size, this is expected because this queues are build in switch chip ASIC hardware and can operate at wire-speed so they are limited only to a few megabytes, because of this a big surge of traffic needing queuing (for example because is ingressing at 10g interface and egressing towards 1g interface) cannot be fully absorbed and can led to dropping packets
15-20 years ago we did something similar in the Utah desert using Cisco switches. We eventually migrated to a more expensive (and expansive) DWDM solution to handle increasing bandwidth needs and to overlay 1GbE with OC3/OC48 SONET, then eventually migrated everything to multiple 10GbE circuits. I’m sure they have 40 or 100Gbps on that backbone now.
If you’re strictly looking at Layer 2 performance, pretty much anything that’s a switch is going to switch at wire speed. It really comes down to features, such as additional ports, and whether you want to eventually route traffic, handle more bandwidth, encapsulate traffic in different formats, etc.
As mentioned, the 309 is passively cooled. They work fine in my environment, but they are all indoors and kept around 70°F.
CRS310(4S+5S-1G)/NetFiber 9 have four SFP+ ports and five SFP ports. The CRS310 has a fan, and the NF9 has a large heat sink (designed for outdoor use). They’d be a good fit for add-drop scenarios where you have a mix of 1G and 10G, and for routing, they can do L3HW offload just fine with DHCP, BGP, & OSPF (no firewall filters). I’m using five of them as site routers for roughly 70-100 customers each, all mounted on rooftops in summer heat and winter cold.
With their four SFP+ ports, you could use one for eastbound traffic, one for westbound traffic, and one or two for connecting to additional switches (for north- and southbound) and/or routers. For example, an RB5009 at each site would allow the backbone switches to remain strictly Layer 2 while the routers handle site-to-site traffic, along with VPNs, MPLS/VXLAN encapsulation, PPPoE termination, and other CPU-bound tasks.
As I mentioned in my previous post, I think a fanless desktop switch like the CRS309 isn’t a good choice for a climate like Congo’s, especially with 10G SFPs that generate a lot of heat. That said, the CRS317 would probably be a better fit as a backbone switch. It’s a shame MikroTik hasn’t implemented large buffers in ROS yet, considering it has 1 GB of RAM.
I don’t think it’s trivial to add or implement large buffers in a switch.
To start, you’ll need an interface with enough bandwidth between ASIC and those additional buffers, then the memory for those large buffers will need to also provide enough bandwidth.
The requirements are such that some implementations use GDDR memory with wide interfaces (like on a desktop GPU)
More advanced implementations resort to HBM memory due to the high bandwidth demands (Like on latest IA datacenter GPUs)
All this paraphernalia occupies silicon space and consume additional power, therefore increase dramatically the cost of the solution
The 98DX82xx in the CRS317 supports large packet buffers, just like other high-end switches that come with a large amount of RAM. My guess is we’ll see this implemented in the datacenter switches first.
If I may - for me, “rock solid” and “moving mechanical parts” like fans are direct opposites, the first thing to break is typically the fan/blower. So if the NetFiber 9 can handle the environment in Congo even if equipped with multiple 10 Gbps SFP+, it looks like the best choice to me. Any device with fans inside has to be installed in a closed cabinet that uses either a forced air flow with an air intake equipped with a filter that requires regular maintenance or uses a split A/C unit which also requires regular maintenance, so the dimensions and expenses grow significantly.
Sorry Sindy, that’s where our opinions go in completely different directions.
Heat management is a challenge even in colder climates, so I’m pretty sure you’ll never be able to go fanless in backbone installations in a country like Congo. Also, if you suspect that Mikrotik uses fans with poor bearings in their high-end products, it’s probably best to just avoid using Mikrotik products altogether.
I don’t think it is vendor related. I don’t suspect even much higher priced vendors to use fans with poor bearings, yet experience shows that the fine dust eventually does kill them. That’s why I’ve mentioned air intake filters or airtight containers with A/C. But I am definitely not going to argue regarding the other part, as I have no idea regarding heat loss on a long-distance 10G SFP+ running at full traffic capacity.
On 10g sfp+ 80km optical modules i have not seen a dramatic increase on heat or power consumption in comparison with other optical 10g sfp+ modules
10g baseT using twisted pair rj45 are another kind of beast mostly for LAN distances, S+RJ10 module at 10g officially supports distance up to 30 meters and has special power and temperature requeriments
regarding to operating in high temperatures there is also the option to find “industrial” sfp+ modules which are designed to support higher operating temperatures than standard modules
AFAIK in this kind of equipment longer links beyond 80km will rely on external amplifiers to achieve longer distances
I really appreciate this information. this has gone a long way in confirming what i was already thinking.
I too have concerns about the fine dust and moving parts. It is hot and humid where we are located and i am actually constructing small houses which the telecom cabinets will be installed into. This will eliminate direct sunlight and will create a buffer before external heat and dust can reach the Telcom cabinet.
For trenching, we will be hand digging all trenches as we will be moving directly through dense tropical forest and we want to minimize our impact on the surrounding area. We have used this technique previously and 3 months after finalizing the work, you cant even tell we were there.
In terms of installation method, we will be using armored, direct burial fiber. This project will be the first to bring non-Vsat internet to a town of 500,000 people. It will serve every type of customer including Residential, Business, Government, 4G MNOs and ISPs.
There is not, currently a network architect working on the job, but that will be coming shortly. The goal at this stage is to have a workable hardware solution for budgeting which the architect can finetune if needed, once selected.