Hi all.

I'm going through my first foray into containers, and I have a question regarding internal vs external storage.

I'm aware that MikroTik strongly advises to use external storage for all things containers whenever possible, and my understanding is that this has to do with potential NAND degradation that could cause premature death of the router as opposed to, well, replacing a pendrive.

However, for now my intention is to run one thing on my RB5009 container, and one thing only: an Alpine Linux VM running only the cloudflared daemon. The reason I'm using a Linux VM instead o the cloudflared container directly is because I want the ability to easily SSH into it in case I ever need to modify any config.

My understanding is that cloudflared will just write the logs to disk and little else, but I'm unsure about any implications derived from running the Alpine Linux VM itself.

What's your opinion on this?

Should I absolutely put this on an external USB pendrive anyway to prevent NAND degradation in the long term? Or will the disk I/O of such a simple setup be insignificant realistically speaking compared to, say, the usual I/O that the RB5009 itself is doing during normal operation?

Needless to say, if I ever want to run a more complex setup on containers, I will 100% move everything to external storage. My question is specifically about the current setup I have in mind.

Appreciate any thoughts!

A RB5009 is around 220 $.
If the built in storage wears out, you need - besides the cost of replacing the chip, some 40-45 $ for a new licence, all in all 100 $, not counting the costs effects of the downtime..
A small USB stick of re-known/reliable brand is what? 5$ or so.

Even if you have less than 5% chance that the internal storage could be damaged, you would be ahead with a USB drive.

IMHO better be safe than sorry.

Do the right thing.

If container did recurring writes, like a database server, that be more problematic on NAND. I don't think cloudflared writes anything to disk when running, just uses stdio/stdout/sockets.

Also, I think there is distro-less version of cloudflared, that does not have Alpine. In RouterOS terms, the entrypoint is "cloudflared" and cmd is are its options. IDK, but Alpine may write something to disk (perhaps) on each boot, but even that I don't think be too problematic for NAND. I mention distroless, since it's easier to look at the "NAND effects" when it just one process in the container (cloudflared in this case).

If you do want temporary writes, like logs/etc, you can use a ramdisk as volume, with image on NAND.

Anyway, USB avoids worrying. But if you look at the "potential writes" a container might do that be a "good guide" to "how bad" it be for it to run on NAND.

A RB5009 is around 220 $.
If the built in storage wears out, you need - besides the cost of replacing the chip, some 40-45 $ for a new licence, all in all 100 $, not counting the costs effects of the downtime..
A small USB stick of re-known/reliable brand is what? 5$ or so.

Yes, I'm well aware of this. Like I said:

my understanding is that this has to do with potential NAND degradation that could cause premature death of the router as opposed to, well, replacing a pendrive.

Just wanted to understand if I'm "playing with fire" just by doing this temporarily while I make a decision, or if the effect will likely be negligible for now.

If container did recurring writes, like a database server, that be more problematic on NAND. I don't think cloudflared writes anything to disk when running, just uses stdio/stdout/sockets.

Thanks, this is my understanding as well based on what I could research.

Also, I think there is distro-less version of cloudflared, that does not have Alpine. In RouterOS terms, the entrypoint is "cloudflared" and cmd is are its options.

Yes, I guess you're talking about this one, which is the image they used in this video? The issue is, that package does not include a shell, so it becomes very clunky to manage and configure once deployed. That's the reason I went with Alpine.

Anyway, USB avoids worrying. But if you look at the "potential writes" a container might do that be a "good guide" to "how bad" it be for it to run on NAND.

Yep, I think I'm leaning towards using an external pendrive anyway. I'll probably reorganize my network a little bit so that I can get a bit more out of that pendrive, such as using it as a mini-NAS for some files I currently have kind of spread out across different places. That way I'll get the most out of the USB storage and it won't just be used for Cloudflared.

By the way, one more question if I may:

In the official container docs, they create a separate bridge for the containers, which lives in a separate subnet.

However, in the Cloudflared tutorial video I linked above, Normund seems to add the veth interface to the default LAN bridge used by the rest of his network.

Do you have any idea of why? Is it safe to simplify things by adding the interface to the default LAN bridge?

I found this article that explains some potential drawbacks if auto-mac is set to "yes" in the bridge. But I can see my default LAN bridge has that property set to "no", whereas it was set to "yes" in the "bridge_containers" I created for this, as it's the default value.

Is there any reason to set auto-mac to "yes" when using containers? If not, why can't I just add the veth interface to my LAN bridge and simplify my setup?

Thanks!

VETH are weird, and @tagent points out it be unclear what the actual "best practice" is from Mikrotik.

I don't think there are any issues with using VETH into "dumb" bridge, vlan-filtering=yes bridge, or unbridged VETH, per se.... Now... configuration of IP address, routing, firewall varies a LOT depending on which one you choose... I kinda view "auto-mac" as a separate topic that MIGHT be effected by configuration of VETH and a "consideration" not a rule. The take away be however auto-mac is configured, you probably don't want VETH being the admin-mac for a bridge interface be a quasi-rule...

Also if you enable logging, you should be able to see command line errors in the distroless version. Once you know the right command line, you probably don't need a shell. While I doubt an Alpine shell is significant hit on flash, just saying it at some point you'll know the right command for cloudflared...

I doubt experimenting with alpine + cloudflared is going to long term damage. What I see is that number of bad blocks does increase over years on devices, but even with 10 year old routers that used "graphing" and "DHCP leases on disk" (which both write to disk), I have not seen a complete failure of flash. Now on the 16MB flash ones... that were you would never want any container to touch the flash (which are generally pretty poor for containers)...

Anyway, my experience is that bad power is how Mikrotik's die, not failed flash due to writes. While I can't say never, an hour with Alpine shell trying to configure cloudflared running on flash is not something that could do "long term" damage.

Right, it's settled then. Once I finish playing around I'll just delete the current container, plug a tiny pendrive into my RB5009 and create the long term setup from scratch, this time inside the default LAN bridge.

Thanks again!