RDMA hardware accelaration in a NIC is simply a performance optimization. It can be implemented in the driver using pure software if the NIC lacks hardware support for it.
If you do it with software, chances are you are still relying on the kernel, just like a normal NIC. The whole point of using it is to have hardware acceleration and bypass the kernel altogether.
Doing it in software is like having an EV and charge it using a Diesel generator 
Yeah, thatâs the major advantage of using hardware RDMA-enabled NICs
Okay, but have you specifically checked for a chip/SoC in an existing MT switch/router that supports flow and congestion control features like PFC/ECN?
Have at look at this: http://forum.mikrotik.com/t/ieee-802-1qbb/116274/1
AFAICT from public files, all the DX chips from 98DX8216 onward have support to it. Iâm not the first one actually asking for this around this forum apparently as this come all the way back to 2018. They actually got a reply from support saying âit is not supportedâ but they donât admit the chip support it and they just donât have it implemented on software.
It is hard to find the data sheets publicly but a quick search now got one chip which is âmore basicâ from the same family and it has it https://datasheet.datasheetarchive.com/originals/crawler/marvell.com/15f2603aee17ead4977c17196a11b4d5.pdf
But nonetheless, IF the chips on all the Mikrotik switches/routers actually are a crippled version from that family which doesnât support it, they should at minimum come to the customers and say that. Not keep silent or give half backed replies with ânot supportedâ. It is ok if it doesnât support it, no company is obligated to support everything. It is just a shame people get all over the place without any proper reply since we know Mikrotik employees are here in this forum as they reply many other threads, when they want.
Okay, but are you sure IEEE 802.1Qbb implements PCB as required by DCB? How about ECN, ETS and DCQCN?
It is important that all facts are available. Licensing costs must also be considered. Even if a SoC has the necessary hw support, activating a specific function may require additional licensing. This is becoming increasingly common because it is often cheaper to produce a single chip than several different variants with limited functionality.
With regards to the support, yeah I think itâs more than reasonable to expect at least a brief respons.
I agree. There are nasty practices like that where the hardware is there but just not enabled due to licensing. I agree that all facts must be available but that information is not publicly available and we can only assume what we have at hand and from what we read, the feature is there. But I may be wrong. But in all cases, a simple message from Mikrotik could clarify everything. Again, it is ok if the chip is not supported, or the license they acquired for the IP doesnât allow it. Really, it is ok. I just wish we had a closure on this so future questions like the one from 2018 wouldnât still be open and I wouldnât had to post this new question again.
I hope someday we will get the answer. Meanwhile, in order to support those environments, either I have to suggest buy a 3-5x more expensive switch or, do the switchless modes. Either way, would be good to be able to use Mikrotik.
I didnt read the whole threads, but there is no need for licensing rocev2 form nvidia right now. I know this function is missing, but even a open source implementation in the beginning would be better then nothing.
There are some on Github.
https://github.com/datenlord/open-rdma
And I am not sure if RoCE v1 even needs to be licensed.
Donât know what the fuss is about RDMA, RoCE, iWRAP, PCF, ETS and other stuff
What they will need to begin to support is BGP EVPN with VXLAN in hardware, or else itâs just stupid L2 switches.
Hello everyone!
After several months of conversation with the MT support team they got it implemented! Just got a reply that RoCE support was added on the current 7.17beta4 and the docs can be found here: https://help.mikrotik.com/docs/spaces/ROS/pages/189497483/Quality+of+Service#QualityofService-RDMAoverConvergedEthernet(RoCE)
Donât need to mention it still beta but it is a great start.
Thanks for the MT team for listening! Time to try it out
As usual zing above my head. I have not even used vxlan yet and DarkNate wants me to go udp4 lite! As always, amazed at the amount of experience, knowledge and practical advice here.
Also, just to point out DNate was clearly commenting on the functionality being crap nothing else. Trust me, if he wanted to rub you the wrong way it would be VERY CLEAR⌠
@galvesribeiro, yeah, this is great news and probably essential for MT if they are aiming to enter the data center market with their new 100G switches. It looks like thereâs support for both the older v1 L2 and current v2 L3 (UDP), with highly configurable ETS scheduling and bandwidth allocation. Overall, it seems feature-complete, which is promising even if itâs still in beta.
Itâs worth noting that to achieve wire-speed queuing and proper bandwidth allocation, especially during network traffic congestion, hardware offloading and large buffer support in the chipset are necessary. This applies to the entire data path, including the end points (NICs)
Unrelated to Mikrotik but I hope they add support for RDMA to the QNAP QSW-M7308R-4x 100GbE switch. I went with it over the 4 port 100G Mikrotik switch because of the additional 8 25G ports and itâs almost silent under full load. It uses a similar switch chip the Marvell Prestera 98DX7324. Either way itâs still a great switch full RDMA support would just be the cherry on top.
@galvesribeiro, thank you for that feature request! Year ago I was looking to buy CRS504, but lack of DCB support was stopping me, since itâs kinda strange to have 100G without DCB at enterprise level.
To others who was saying that âlol, just replace your architectureâ (at least I havenât seen âlol, just use Cisco/Dell/Aristaâ) or âCXL is futureâ - thatâs just terrible behavior.
@dante4; DCB is just a very vaguely defined umbrella term that says absolutely nothing about the underlying QoS protocols. If youâre wondering how Datacenter QoS protocols like ETS, PCP/PFC, AQM, DCBX, or CN are implemented in RouterOS v7, check out the Mikrotik docs, for example the links below:
https://help.mikrotik.com/docs/spaces/ROS/pages/189497483/Quality+of+Service
https://help.mikrotik.com/docs/spaces/ROS/pages/24805517/Neighbor+discovery
https://help.mikrotik.com/docs/spaces/ROS/pages/30474317/CRS3xx+CRS5xx+CCR2116+CCR2216+switch+chip+features
CXL is a device interconnect based on PCIe and has nothing to do with the above.
Are you sure we are talking about the same DCB?
https://1.ieee802.org/dcb/
Itâs quite literally defines that DCB consists of:
PFC IEEE 802.1Qbb
ETS IEEE 802.1Qaz
DCBx* (optional) IEEE 802.1AB
IP ECN IEEE 802.1Qau
So I donât see any âvaguely definedâ
Yes, I know, read the first posts that advise Topic Starter to switch to it, instead of pushing Mikrotik to add DCB support.
If you read reasonably carefully, youâll probably notice that DBC is really just a collective term for a wide range of use cases and protocols, which you can find in the project links for the protocols I mentioned in my previous post. In other words, there is no fixed way to define DBC in terms of what you have to use, it is more like a family of different solutions and a collection of QoS protocols.
A year later, howâs it looking? Iâm looking for RoCE v2 for lower CPU utilization for Ceph on a hyperconverged OKD+Kubvirt cluster.
I'm also looking forward for DCB support to be able to use high performance RDMA, and consequently NVMe over RDMA, for which Priority Flow Control 802.1Qbb) and Enhanced Transmission Selection are crucial. Fortunately for now I'll have switchless setup so I'm good to go.
Regarding âvaguenessâ of spec. Somehow Force10, Dell, Cisco, Aruba, Juniper and maybe others were able to produce working interoperable solutions out of these âvaguely defined specsâ, so this is not a valid argument.
Fingers crossed for DCB/PFC and ETS support in Mikrotik. We need âlossless Ethernetâ
Sorry folks, I forgot to update here.
I got from the Mikrotik support an year ago an answer. They pointed to new docs here: Quality of Service - RouterOS - MikroTik Documentation
Apparently it is working now. I didnât had a chance to test it myself.
Iâm working in an environment right now which is based on bare metal Kubernetes + Ceph + Kubevirt (for the legacy VM based apps). In this, Iâm trying to get RDMA on Ceph but (1) there is almost 0 documentation on how to do so, and (2) the little Iâve found about it, just doesnt work. The OSD processes will never start when RDMA is enabled either using ceph with Cephadm or thru Rook Ceph. If anyone had any past experiences with either Iâd appreciate some guidance.
Thanks!