Community discussions

MikroTik App
 
glennswest
just joined
Topic Author
Posts: 12
Joined: Wed Apr 07, 2021 4:00 pm

nvmeOf - NVME over TCP

Tue Mar 01, 2022 7:19 pm

With MikroTik moving to higher speed interfaces, I would love to see nvmeOf/Nvme Over TCP incorporated.
Even better if you have a "router" box with a butch of nvme slots.
Happy to help if needed on software or hardware.
 
mada3k
Long time Member
Long time Member
Posts: 682
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: nvmeOf - NVME over TCP

Tue Mar 01, 2022 8:47 pm

So Mikrotik should try to get into the market of enterprise storage now ?

Isn't that quite a saturated market by now.
 
glennswest
just joined
Topic Author
Posts: 12
Joined: Wed Apr 07, 2021 4:00 pm

Re: nvmeOf - NVME over TCP

Wed Mar 02, 2022 3:30 pm

So Mikrotik should try to get into the market of enterprise storage now ?

Isn't that quite a saturated market by now.
Isnt networking and access point saturated?
Actually there is a "hole", in existing nas - and storage has moved into "TCP", so networking and storage is merging.
All over ethernet.

Checkout https://www.marvell.com/content/dam/mar ... 018-08.pdf
Basically its a single chip you can put a nvme storage device on top of a switch. Mikrotik does switches? Add a bit of routeros, and your good to go.
Basically a higher performance (Alot) of your usb storage solution on existing mikrotiks
No need to do nfs or smb or anything. I would add some nice software management on top, but can start basic and go from there.

Cheap and fast, and simple.
Hardware wise you can vary it a bit for m.2 slots or ruler drives, and diff nic sizes.


I work alot with datacenters, and run a home lab, and want some of the performance without spending 10-30K. 4-8 m.2 drives would be great, a removable solution for m2 would be even better.
Last edited by glennswest on Wed Mar 02, 2022 5:18 pm, edited 1 time in total.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26287
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: nvmeOf - NVME over TCP

Wed Mar 02, 2022 3:39 pm

Maybe you have ideas, how you would use it, what features you would like to see?
 
glennswest
just joined
Topic Author
Posts: 12
Joined: Wed Apr 07, 2021 4:00 pm

Re: nvmeOf - NVME over TCP

Wed Mar 02, 2022 5:29 pm

Hardware
Phase 1 - A box of 4 m.2 drives with a 10 or 25Gig ethernet, routeros, a switch, and a upper range arm and memory
Phase 2 - A 1U chassis with ruler drives or other high capaciy and 25g/100g and multiple


Software
Phase 1 - Export Volume via NVMEof (Will go thru datasheet if you wish) and provide basic counters and monitoring (Jbod)
Phase 2 - Add ability to do virtual volumes
Phase 3 - Add thin provisioning
Phase 4 - Integration for Ceph to get all of ceph features on top (Lots of upstream work) (I work for a Open Source Company)

Certifications/Verifications
RHEL 8+/RHCOS/Openshift
Vmware

For me, I run my lab on 1 or more esxi nodes, so would plug it either directly to a nic in my server(s), or to a mikrotik switch.
Allocate vmfs on top of physical volume, if 4 drives, can combine them, and that works as phase 1.

As functionality grows can directly map and use drives from vm, and potentially boot from them to.

Also as we get Phase 2 software, adding the ability to use a kubernetes/openshift cluster to allocate drives makes it very cloud like.

Longer term setting up clustered read write storage would be interesting. (More of a host side issue).
 
glennswest
just joined
Topic Author
Posts: 12
Joined: Wed Apr 07, 2021 4:00 pm

Re: nvmeOf - NVME over TCP

Wed Mar 02, 2022 5:38 pm

Some examples:
https://github.com/ceph/ceph-nvmeof
https://openinfradays.kr/2020/files/track3/track3-1.pdf

Note, I would think there would be alot of interest in the Ceph community, and also fits the diagregation of storage.

Common problem I find in our environments is elastic search running in containers needing high performance storage, High Speed network to a PV on a nvmeof node would be great.
Failure of the pv is not such a great loss as its log data, so fast and throw away is pretty useful.
More important data may want to do a raid 1 (Software raid over nvmeof drives on two appliances)
Even more important data use ceph.

Also cluster file systems get alot more interesting when you have the bandwidth this looks like.
 
glennswest
just joined
Topic Author
Posts: 12
Joined: Wed Apr 07, 2021 4:00 pm

Re: nvmeOf - NVME over TCP

Wed Mar 02, 2022 6:36 pm

Im based in Central US (Chicago Time Zone), and open to do a call.
 
glennswest
just joined
Topic Author
Posts: 12
Joined: Wed Apr 07, 2021 4:00 pm

Re: nvmeOf - NVME over TCP

Fri Mar 11, 2022 10:07 pm

 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: nvmeOf - NVME over TCP

Fri Mar 11, 2022 10:55 pm

Be aware that the network path must mimic PCIe both in terms of speed and latency, ie it is not a standard L2/3 switch we are talking about here.

NVMe over fabrics using RDMA on IB/RoCE which is targeting datacenters requires highly specialised hardware that is able to xmit close to 300GB/s (equivalent to aprox 2 400 000 megabit/s) as well as at least one miljon IOPS (packets/s) with minimun latency PER CHANNEL. I doubt MT has something that can match that specification.

NVMe-oF Ethernet SSD and NVMe-oF JBOF
 
glennswest
just joined
Topic Author
Posts: 12
Joined: Wed Apr 07, 2021 4:00 pm

Re: nvmeOf - NVME over TCP

Wed Aug 10, 2022 11:45 pm

Note, its over TCP not RDMA.
So the timing is not so "deterministic".
Also generally the "protocol" conversion is done in hardware - so the only "software" is setup/provisioning/teardown.
Chip that connects to switch (25Gig) and creates a PCI-express bus(nvme).
One per drive.
the chip just creates IP frames/ethernet frames.
 
glennswest
just joined
Topic Author
Posts: 12
Joined: Wed Apr 07, 2021 4:00 pm

Re: nvmeOf - NVME over TCP

Thu Jan 12, 2023 6:59 am

Been digging, and found a perfect combo for this.
And this could work for alot of physical formats (m.2 u.2 u.3 and ruler drives)

https://dgway.com/NVMeTCP-IP_A_E.html

Basically nvme to 25Gig ethernet in a fpga. Wirespeed.
Add a pci express switch, and support commands in routeros, and be fantastic.
even a 4 drive u.2 or u.3 would be a wonderful start.
 
User avatar
Znevna
Forum Guru
Forum Guru
Posts: 1347
Joined: Mon Sep 23, 2019 1:04 pm

Re: nvmeOf - NVME over TCP

Thu Jan 12, 2023 8:47 am

And we have routers that route just fine with 16MB of flash storage.
How would NVMeOF (at least spell it right) help MikroTik developers fix the remaining bugs in RouterOS v7?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: nvmeOf - NVME over TCP

Thu Jan 12, 2023 3:46 pm

How would NVMeOF (at least spell it right) help MikroTik developers fix the remaining bugs in RouterOS v7?

I have a feeling that OP request / plea is another one from the department of "a farmer has a milk cow and now he wants her win on horse races ... cow's got 4 legs, so where's the problem?" ...
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: nvmeOf - NVME over TCP

Thu Jan 12, 2023 6:41 pm

I like the RB5009 design a lot. Some "storage add-on" in same form factor be pretty nifty, with some RAID protection against memory/disk failures. Now, it could be USB or some RouterOS-compatible SAN protocol for all I care. But with container, dude, etc., some kinda better storage option for the routers does make sense to me.

Today we end up using the RB1100AHx4 pretty much because it has a built in disk. But it kinda bulky and still only has a one disk inside. CHR certainly gets you all the disk options you'd want, but that's overkill in a lot of cases (and you'd likely already have some SAN solution if your at virtualization/CHR).

Basically RouterOS stuff needs some storage (container, user manager, dude, etc.), and it nice if that could be be protected again a local disk failure. Platform or protocol isn't as important.
 
User avatar
antonsb
MikroTik Support
MikroTik Support
Posts: 385
Joined: Sun Jul 24, 2016 3:12 pm
Location: Riga, Latvia

Re: nvmeOf - NVME over TCP

Fri Jan 20, 2023 3:57 pm

 
glennswest
just joined
Topic Author
Posts: 12
Joined: Wed Apr 07, 2021 4:00 pm

Re: nvmeOf - NVME over TCP

Fri Jul 21, 2023 4:30 pm

This is a good indicator of where things are going.
I see lot of the software already in routeros

https://www.servethehome.com/qnap-tbs-5 ... s-spotted/

Who is online

Users browsing this forum: Bing [Bot] and 25 guests