App inclusion request - Netbird

RouterOS Containers
1

Hi,

I like the new “App” feature and intend to actually use it on some routers.
From the current list of apps I’m guessing it’s primarily targeted at RDS.

Is there any chance Mikrotik could also add Netbird client to that apps list?
I’ve been using Netbird as a container on Mikrotik routers in production for 2 years across dozens of routers all over the world (mostly Chateau and RB5009). It works really well for me. Here are the docs.

But for wider adoption it would be really great to have it pre-configured as an “App”.

I believe that this would be beneficial for both Mikrotik and Netbird because the audience overlaps a lot.
Note that Netbird is already included in OpenWRT, TrueNAS, AUR, mobile playstores and everywhere else one would expect to find it.

2 Likes
2

+1, as maintainer of NetBird, we are happy to support the integration. Would love to discuss with the team to understand how to contribute.

1 Like
3

It would be great to see the server and client apps on my CCR2116. and HA on two CCRs :slight_smile:

4

One can add custom “/app’s” now, without having to wait for it. And app config host URL’s can be used to add the app config and autoupdate the app. @Amm0 has a couple of threads going about it, including one with his temporary instructions on how to generate /app YAML config files.

I hadn’t heard of Netbird before. It looks like it’ll scratch a few of the itches I have right now for projects I’m working on.

5

Me neither. In 15 minute test, it totally works. I started on Mac, then added /app as the 2nd peer per their cloud site. They do have interesting mix of features, like handling DNS and remote access things like builtin SSH/etc.

Internally, looks like they pull same trick (STUN/ICE + TURN/"Relay") as BackToHome with WG. It's actually similar to what I thought a "BackToWork" feature should look like as professional version of BTH.

But here is an /app for it... you just need a setup key for a peer from the Netbird website:

/app/add yaml="
name: netbird-client
services:
  netbird-client:
      environment:
          NB_SETUP_KEY: MUST_SET_FROM_NETBIRD
          NB_DISABLE_CUSTOM_ROUTING: 'true'
          NB_USE_LEGACY_ROUTING: 'true'
      volumes:
          - netbird-client:/var/lib/netbird
      image: docker.io/netbirdio/netbird:latest
"

You'll need the "setup id" from Netbird to connect. You can add it the YAML, or set it as an ENV variable in the /app.

With network id set, to enable the /app use:
/app/enable netbird-client
or click "Enable" in WinBox/WebFig. This will do the rest of container add, VETH, internal bridge, etc and auto-start it.

You can add Netbird options in the command: under the service in YAML. Or, once added, edit the "Container Command Lines" and fill in the 4th column next to netbird-client.

Now do more useful things, you'd need to think more about the network configuration.

Note: Netbird example uses some Docker cap_add stuff, that I ignored, since the RouterOS's container cgroups allow most of NET_ADMIN (AFAIK). And seems to connects. OP seems to have it working a /container so it must not be critical to set those (since you cannot on RouterOS).

Only issue was the builtin SSH did not work (even after adding policy and --allow-server-ssh --enable-ssh-root to /app command, but didn't troubleshoot it much. Now the cgroup might be involved here, but IDK. It could also be port forwarding since I'm using 22 already, but the remote SSH should use Netbird WG. As I said didn't research/test the remote SSH option much. More noting Netbird example uses SYS_ADMIN and SYS_RESOURCE previllege for Docker, in addition to more common NET_ADMIN... And MikroTik does not explain exactly how cgroups are mapped back to Docker capacities.

edit: added envs from: https://docs.netbird.io/use-cases/client-on-mikrotik-router

5 Likes
6

Thanks @sirbryan and @Amm0 !

I tested this as soon as apps appeared in testing channel and although it's somewhat more user friendly than rawdoggin it using /container, it's still quite clunky and not appropriate for Netbird to officially recommend doing it.

That's why I'm trying to get Mikrotik to notice this and include Netbird in their official catalog.

Being able to add custom catalogs would also help a lot but it's IMHO really most appropriate for Netbird to have a place in the official one, like it does in other router/nas products.

Note that I'm just a Netbird user, not staff. Maycon is their co-founder and CTO.

7

@Amm0 were you maybe able to figure out how to use “app-store-urls” ?
I can set it but no idea how to use it.

In case anyone else is playing with it….
It looks like a bunch of stuff from my howto is no longer true.
For example, netbird client now starts if I set NB_DISABLE_CUSTOM_ROUTING and NB_USE_LEGACY_ROUTING to false.
It also works as an exit node.
Note that I’ve allowed hardware access to network interface so that might have something to do with it; not sure.
I’m not gonna be able to get back to playing with this today so happy hacking!

8

Yes, you just wrap the YAML in an array.

So If you save this instead, and serve it from a webserver. And in app-store-urls= use the path to the file. IDK if it even checks MIME type, but I always set "Content-Type: text/yaml" in web server (which happens automatically in most defaults).

You can add more app by adding more element to the array.

-
  name: netbird-client
  services:
    netbird-client:
      environment:
        NB_SETUP_KEY: MUST_SET_FROM_NETBIRD
        NB_DISABLE_CUSTOM_ROUTING: 'true'
        NB_USE_LEGACY_ROUTING: 'true'
      volumes:
        - netbird-client:/var/lib/netbird
      image: docker.io/netbirdio/netbird:latest

See post 6 in https://forum.mikrotik.com/t/amm0s-manual-for-custom-app-containers-7-22beta/268036 for more details.

1 Like
9

I think that's right. I did not set them orginally. And things seem to work, but I didn't not test much, so did not do any custom routing.

So figure those might be needed for some feature, but IDK. I added them to example, so someone could decide without having to remember the ENV name later.

Overall, /app is pretty hot off-the-presses, and 7.22 release is still in beta. So waiting to update be a good idea to formally document this approach. More provided some YAML example as starting point.

And MikroTik have restructured internal permissions for containers in recent release, too. Which could also be why those settings aren't needed.

On the network, that part is especially unknown from my POV. They seem to working on most "custom" networking options, but not sure how there expose the YAML schema. But you should be able to pick in network= on the /app, either the "lan bridge" where VETH get added with DHCP to you LAN bridge, or the internal bridge with put all containers in a bridge and uses dst-nat for fixed port forwarding , which is similar to your doc Netbird's manual.

The internal bridge is used with YAML above (as that's the default)... but as I said you may need to set this manually to "bridge", or if you do create you own bridge in /interface/bridge, I think, you can select that to in /app/set netbird-client network=mycustombridge

1 Like
10

I get that it would be nice to add it from the stock setup. My point was that we don’t necessarily have to wait to test doing things the /app way vs. /container, which is cool for us because we can do some of those things now, instead of hoping to catch their eye.

To add to that, feature requests like this are best submitted as a ticket; their devs don’t all check out the forums. I have had good luck getting things like network drivers and kernel patches added. Support has also stated that they determine feature request priority by demand, so if enough of us open tickets, that gets their attention.

11

Been testing a bit more today.
Netbird client runs in Mikrotik containers (Apps) infinitely better than it used to.
No legacy env vars needed, no direct hardware access needed either. Only setup key and it just works.
It’s now got normal access to netfilter and iptables modules so it can do its thing.

NAT and masquerade work just fine
Direct routing works too
Netbird SSH server works too so netbird peer can be easily used as jumphost/bastion with integrated netbird auth out of the box !

I’ve been running the client on my test CHR for 4 days and had 0 disconnects. I wasn’t wise enough to set up smokeping 4 days ago to record any drops but Netbird in general is very stable so I doubt there were any issues.
For me this is totally production ready; I just gotta put logs in tmpfs to prevent any long-term issues.
I’ll test on rb5009 next.

1 Like
12

That's really nice to hear.

If you will be testing this on an actual physical router, could you please also collect some ballpark data on resource usage (e.g. how much disk space, ram, cpu for a mostly idle connection?)

13

I made a proper “App Store” for my RouterOS apps, currently only Netbird client.
Not only that - it’s automated GitLab CI/CD which publishes multiple app stores containing arbitrary app combinations.

Feel free to fork, contribute and donate if you find it useful.

https://gitlab.com/horzadome/mikrotik-apps

Readme contains instructions on how to use it; basically just add https://routeros.horza.org/ to your app store urls (and enable auto update).

I plan to improve it by versioning each store version so that users can just pin their app store URL for better security. So that I can’t push something malicious.
Entire pipeline is now fully auditable; just missing version pinning for perfect supply chain control.

Currently I’m also running Netbird client on rb5009, don’t have time to profile usage righ now; will do tomorrow-ish.

14

Here are some benchmarks of Netbird App on RB5009, aimed at figuring out CPU usage.
I don't know if anyone tried doing NAT and masquerade in mikrotik containers using OpenWRT or just normal linux.
I believe they'll have the same problem that I have with Netbird - high CPU usage.

https://gitlab.com/horzadome/mikrotik-apps/-/blob/main/docs/netbird_benchmark.md?ref_type=heads

In case anyone knows how to work around these issues, please let me know.
It really feels like this is not encryption or anything Netbird specific.

15

I’ve done even more testing of Netbird in RouterOS containers and apps something is weird.
Performance is excelent when I deploy Debian container and install Netbird in it.
But it is 10x worse when I deploy Netbird container as an App.
I’m thinking something is very wrong with my testing setup.
Results and conclusions are at

https://gitlab.com/horzadome/mikrotik-apps/-/blob/main/docs/netbird_benchmark.md?ref_type=heads#conclusion

16

Would love to see Netbird as a native app in Mikrotik.

It is easy to deploy another node, or even few Nodes, for redudancy and also for Exit Nodes, since it is a great price.