We are moving into a new 500m2 office building from 4 different smaller offices. We have around 50 employees, so guessing around 100 devices, computer and a mobile phone for everyone. The space already has a rack, a 48-port managed switch installed and all the wiring done. We have 8 Asus ZenWifi XT9 nodes laying around which we plan on using as WiFi APs. Obviously not everyone will be able to connect via LAN, but WiFi is good enough for most people, so no plans on expanding the local network right now.

We are planning to get two different providers with optical internet connection, one is 1000/500mbps, the other one is 1000/1000. We plan to use them for failover and load balancing. None will allow us to use our own ONT, but we can place the one we are getting from them in bridge mode. One of them will allow us to connect with PPPoE credentials, the other one uses DHCP. Both will have static IPv4 address. We will need a wireguard VPN server setup for remote work (probably not over 10 people at a time). The most connection intensive tasks for the employees would be teams calls, so we don't expect a very high load on the network. Currently we have 600/60 connection in each of the offices, and everything works all right. Now that we are moving in a single space, we would like to do it right.

What kind of hardware would you recommend us getting to accomplish these tasks? I do know some routerOS but we would definitely get someone to set everything up and maintain it afterwards, don't want that on my back. Budget is not a huge issue, but would like not to spend too much, maybe you can recommend the bare minimum to handle this, and then what would be the ideal hardware for the task?

Thanks in advance,
Danilo

What's the uplink speed on your switch?

Do you require any form of hardware redundancy?
And where is maintenance in your use case?
How long do you want to use this hardware?

The hardware question is, in my opinion, not that relevant and more of a result from all requirements.

@alibloke Sorry, forgot to put that info, it's a Cisco Business CBS350 (that's what I was told). Now that I'm looking it up, I see there are lots of versions with the same name. Let's say it's 1gbps, but can't be sure of course. Is that an issue? I'm guessing 10g would be better, but I don't believe that we will be investing in a new switch. In any case, right now all of the computers are connected to wifi, with 16 people in the largest office with 600/60 internet connection, and it's working well enough.

@erlinden redundancy isn't all that necessary, nothing will be lost if the hardware dies suddenly. In the worst case, we would take one of the ISP provided routers out of bridge mode and it would handle all the routing until we got the hw repaired or replaced. If the VPN server is not available, we would use remote desktop software to access a few servers (basically desktop computers with specialized software that we use) laying around the office. We got a 5 year contract on the building, so at least that long, after that time an upgrade could be possible if necessary. One reasonable assumption would be that we would get more people in those 5 years given that we are growing pretty quickly, but no more than 70-ish people (no room for more).

Didn't get the "where is maintenance" part? As in how far would someone have to come to do the maintenance? Because we have an IT company that we have worked with before, and they usually respond the same day once we contact them. If you're thinking what kind of maintenance do I expect, probably tweaking firewall rules if necessary, configuring new VPN peers, something like that. Not much of it expected, just noted we're aware it might happen and would be outsourced.

Right now the XT9s are used as the main router in each office, I've been maintaining them as in creating VPN connections, we have a few services running on the servers in our offices so that as well (I would probably keep maintaining this in the future as well), restarting the routers if the connection drops (happens very often with the current ISP, cable network, will use different ones in the new building).

It would be sensible to match the speed up the uplink port so be sure to check exactly which model switch you have. If you have 2x 1Gb internet connections and plan to load balance them then you will need >1Gb to your switch.

So if I understand correctly, I would use the router with 2 ports as WAN, and one 2.5g port to the switch? Could you do 2 1gbps ports to the switch? Never had to do stuff like this, don't know if it works that way.

This doesn't sound too expensive at all. How about CPU requirements, how much RAM or which level of ROS license do we need for the requirements I mentioned? I'm not that proficient as I said, I use HAP ac2 at home, but I'm guessing this needs to be a beefier device.

A quick look around the website, I see this one:

https://mikrotik.com/product/rb5009ug_s ... ifications

Would it be enough to do everything we need? I presume the load balancing puts a lot of weight on the router CPU.

Yes it's possible to use LACP in rOS if your uplinks are only 1Gb:
https://help.mikrotik.com/docs/display/ ... ng-802.3ad

The rb5009 is a fine router and from the sounds of things most likely to be ample for your needs. It will also do LACP in hardware. The next step up would be the CCR2004-16G-2S+ but if you don't have 10Gb uplinks on your switch there's probably no point.

Cool, if no one has a different proposal, I'll ask for that one in a couple of days. Still open for suggestions though

You should also consider whether you want to save $250 and then later find the router is under-performing and has to be upgraded.
I think the CCR2004-16G-2S+ is good advice even when you do not have the 10Gbps links, you will find that when doing more and more on the router the CPU performance of that model isn't really "overkill" for two 1Gbps links anyway.
Also, that model has dual (redundant) powersupplies, 19" rack mounting, and suitable cooling.

@pe1chl so the one you listed is has better hardware? Don't know how these CPU's compare, other than that the CCR2004 has a slightly higher CPU clock, and of course more RAM. NAND is not that important I believe. As you said, it's better to invest in better hardware and be safe for years, but I wouldn't like recommending getting more expensive gear and not get anything out of it.

How CPU intensive is load balancing and LACP? Does anyone have some real-world experience of a setup that might be similar as our own?

Load balancing is quite CPU intensive because it requires each packet to be marked with an appropriate route in such a way that packets for one connection always go the same route outside. That precludes certain acceleration tricks.
LACP costs nothing at all, it is a switch hardware feature managed by a little software on the CPU.

I am using both CCR2004 and RB5009 routers in the company network, and I like both of them, although they each have their limitations. Unfortunately I do not have a RB5009 in a place where it has 1Gbit internet so I cannot directly compare the performance.
However, I have seen that even a CCR2004 cannot fully saturate a pair of 1Gbit links when all kinds of nifty features are configured, such as a queue for prioritization of traffic, PCC load balancing, IPv4 and IPv6, etc. I had to optimize things a bit to get the full 1Gbit.
However the next step up (CCR2116) is quite a bit more expensive so I did not yet consider that.

An advantage of the CCR2004 over the RB5009 is that it can do hardware-accelerated L3 routing. That would be most interesting when you have several internal VLANs with a lot of traffic between them (e.g. your storage servers are on a different VLAN than your users), but I think it could in theory also be used for the internet routing. I have not yet tried that, it would require quite some reconfiguration in my case as I migrated from a CCR1009 with RouterOS v6.

Also consider how compact the RB5009 and the fact that although the CPU is a bit slower then CCR2004 you will probably never be CPU bound (Bottlenecked)

Also consider how compact the RB5009 and the fact that although the CPU is a bit slower then CCR2004 you will probably never be CPU bound (Bottlenecked)

Are you sure about that? He wants loadbalancing over 2 1Gbps links (ok one seems to be 500Mbps upload) and on one there is PPPoE.
There seems plenty of opportunity for being CPU bound!

An advantage of the CCR2004 over the RB5009 is that it can do hardware-accelerated L3 routing

Dokumentation does not list CCCR2004 as compatible with L3HW offloading. CCR2004 2x 88E6191X switch chip and RB5009 1x 88E6393X switch chip. None of which support L3 HW offloading.
https://help.mikrotik.com/docs/display/ ... iceSupport

Ok... I thought the CCR2004 had it, but apparently it was only for the CCR2116 which I also considered for that location.
(I had not studied to use L3HW in much detail because it requires quite some changes in the config that I use)

Also consider how compact the RB5009 and the fact that although the CPU is a bit slower then CCR2004 you will probably never be CPU bound (Bottlenecked)

Are you sure about that? He wants loadbalancing over 2 1Gbps links (ok one seems to be 500Mbps upload) and on one there is PPPoE.
There seems plenty of opportunity for being CPU bound!

Without Fasttrack on I can saturate the 2.5Gbps GPON connection I have by using 4xPPPoE and different IPs, and by mangle I load balance, never tried QoS as I don't think it's of much use for high bandwidth fiber.

Big Leaf BLR112 for the 2 ISPs
Mikrotik RB5009

This will provide you one PUBLIC IP. All your traffic will present as this one Public IP. The connection will dynamically shift traffic over ISPs. Not relying on round robin or connection times outs.

This will also stream line the issue of remote connection to servers or VPNs.

VoIP and wifi calling will be much improved too.

Hi all, thanks for all the inputs so far. I've been to the new offices and the switch in question is CBS350-48T-4G, so no 10g ports unfortunately. I've spoken with the person who will be deploying the network, and he's agreed that ccr2004 is the best way to go, the price difference isn't too big of an issue (the man-hours cost much more than the hardware). Additionally we'll probably end up connecting all the "APs" (the ASUS XT9s - a total of 8 of them) to the router itself, keeping the ports on the switch available for the computers.

We don't have much traffic inside the network, given that all the storage is kept in the cloud (onedrive and sharepoint), there is only one NAS that replicates the sharepoint daily and there is barely any traffic to the NAS (if at all - it's disconnected from the network and only comes online at night). It does have a 10g port, but the ISP connection speed makes it irrelevant. So no network storage, no VoIP, no wifi calling... We're not an IT company, we're an energy consultancy so most of our work is done on our own computers (laptops), and we have these threadripper machines which anyone can connect to via remote desktop and use the software with them. The Big Leaf thing does sound interesting but the person handling the network isn't familiar with it, and it doesn't feel it would bring any significant improvement to the network.

Thanks again for the inputs to everyone!

Regarding Bigleaf, RoS already has a built-in SD-WAN solution called ZeroTier, which is considerably cheaper.

With SD-WAN such as ZeroTier installed on your laptops and phones, you have constant access to your office anytime, but without having to "dial up your office VPN". You're always connected seamlessly (if you want) no matter what internet connection you're using, like your cell phone, hotel Wi-Fi, etc. It's able to use multiple connections at the same time or switch between them without dropping your connection, like when you jump on hotel Wi-Fi while still using your cell connection.

Minimal administration compared to traditional VPN connections. Everything is easily managed through a web page. Simply install the ZeroTier client on your laptop or phone, and then approve the client through the website. The same applies if you want to disconnect a device. The actual network traffic does not depend on the administrative web server.

It's a perfect fit for a consulting firm where a lot of people are on the move. You might test it for free for a limited number of users. Pricing is about $5 USD per user per month, but is negotiable.

Hmmm, I am aware of ZeroTier but I used to think about it as a VPN for those who don't have a public ip. Correct me if I'm wrong, but the traffic between devices is "coordinated" by a third party? I don't think the actual traffic goes through another server (I hope it doesn't), but there is "someone else" that is matching the two peers. I fail to see how this is better than a direct communication between peers (such as wireguard - keep in mind I'm not a network engineer, just like to fiddle with them).

ZeroTier is a "zero trust" solution, meaning it always uses end-to-end encryption. It works like DNS, with root servers (a.k.a ZeroTier "moons") for establishing the initial connection. Afterwards, all clients communicate directly to each other, like a giant mesh network, as long as they have IPv6, a public IPv4 address, or can use NAT "hole punching". Otherwise, traffic is relayed for that specific client. You can connect to several networks at once or set up site-to-site networking.

ZeroTier is super easy to set up and use with the standard service. It'll take you just a couple of minutes to configure ZeroTier on the Mikrotik router and a couple more minutes to download and install the ZeroTier client on your laptop or phone. If you want, you can of course run your own ZeroTier root server and admin controller, but that's way more hands-on and technical.

Larsa has completely skipped over the multiple ISP thing.

Zerotier uses UDP hole punching.

The system makes IP connections via the best route it can. Once the service knows where it's clients are... It makes connections from IP to IP then encrypts traffic between.

If you set up bridging... Zerotier can do full on later 2 broadcasts over those links... However...

It completely ignored the issue of having multiple ISPs. Shakey performance from one link or the other. Load balancing. Or rollover.

ZeroTier supports all of that, like most other SD-WANs do. Performance-wise, it all depends on the platform. There's not much any other SD-WAN solution can do about it, be it Bigleaf or others..

ZeroTier supports all of that, like most other SD-WANs do. Performance-wise, it all depends on the platform. There's not much any other SD-WAN solution can do about it, be it Bigleaf or others..

Because I am stupid...

Explain this too me.

Right now... I have 3 ISPs connected to the service I am paying a S--t TON for.
2 ISPs are public. the 3rd is CGNAT.

I am provided ONE public IP. I entered that into my Tik as my WAN connection and did the SRC-NAT to match.

Now ALL my traffic goes back to a data center over the 3 connections using various tricks to strype or push. When it hits the data center... the service combines the connections together and sends traffic where it needs to go. The data comes back to the data center when it makes decisions on how to send the traffic back over the 3 feeds. Then the data is reassembled and sent into my router.

Now if I am doing some sort of live time connection, and the connection from the ISP to the data center gets a little shakey, traffic will be shifted to the next feed that had a better connection. As the service is checking the connectivity down ALL feeds several times per second. So as the shift happens... more traffic goes over one ISP than the other to the data center. Where it then goes on to the interwebz. With NO change in the requesting IP.

When a circuit or ISP is performing better... traffic shifts to the better connections. Without SEVERING connections.

I have watched as ALL out going traffic went UP one circuit. But was returning down another circuit.

How EXACTLY... can I do this with Zerotier and not pay hundreds of dollars a month?

Most SD-WAN solutions does offer support for different kind of aggregation types. ZeroTier has several Standard Policies listed below but also offer Custom Policies as well as Segmentation. This allows you to aggregate multiple links of different types into different "circuits" using various policies.

• active-backup: Use only one primary link at a time and failover to another designated link.
• broadcast: Duplicate traffic across all available links at all times.
• balance-rr: Stripe packets across multiple links (not for use with TCP.)
• balance-xor: Hash flows to specific links.
• balance-aware: Auto-balance flows across links.

The big difference between for example Bigleaf and Zerotier is how link aggregation is administered. Bigleaf has a significantly simpler and more powerful user interface, whereas with Zerotier, one has to edit everything manually which can be challenging with complex configurations. Bigleaf also offers more granular built-in control of traffic shaping, QoS, etc.

Larsa...

So, it can't.

I'd say SD-WAN solutions like Netmaker, ZeroTier, Tailscale and similar, pretty much cover everything you need for small businesses, let's say up to 10-20 branch offices with people on the move or working from home. They're very easy to install and get going with great bang for your buck, with solid and reliable services and minimal need for administration. They don't offer all the bells and whistles for network administration which are seldom needed in smaller organizations.

When it comes to larger corporations and service providers there is a significantly greater need to granularly manage capacity allocation, link aggregation and segmentation with technologies like MPLS, QoS, etc, as well as security management, monitoring and troubleshooting. However, the underlying network principle remains the same. These services are often delivered as black box sw/hw 'appliances' with agreements for guaranteed uptime and service commitments which comes with a price tag. In combination with the fact that this usually requires dedicated IT staff, it's something smaller organizations often cannot afford.

As they say, you get what you pay for but if you don't know what you're doing you often pay way too much.

EDIT
Sorry, but I don't really know how to respond to your "So, it can't". Could you be a bit more specific about the details of the use case?

Larsa,

Certainly.

I described what I am using Big Leaf for.

I asked if Zerotier could do that, so I could stop paying a couple hundred bucks a month per site.

Okay, got it. Agregating 3 somewhat (intermittently) shaky wan links to a datacenter. Seems like load balansing using asymetict links tweaked with quality and capacity settings should do it. Check out Multipath Balance-Aware and beyond.

If you want to set up a testbed, it's not as fancy to configure as Bigleaf so you have to manually edit the policies config file.

Larsa

Where is the server that these links connect to?

Larsa

Where is the server that these links connect to?

And that's the reason you're paying a couple hundred bucks. Someone else has built the solution, hosts stuff in a datacenter, and has bandwidth/power/development costs associated with doing so.

I'd view ZeroTier as the tool to build something akin to what BigLeaf does. If you have another router or server in a well-connected datacenter, then ZeroTier would help your office/branch router find and connect (and stay connected) to that DC-hosted router.

The question is whether building the solution and hosting your own router in such a location is any cheaper than paying someone else to do it.

Hmmm, I am aware of ZeroTier but I used to think about it as a VPN for those who don't have a public ip. Correct me if I'm wrong, but the traffic between devices is "coordinated" by a third party? I don't think the actual traffic goes through another server (I hope it doesn't), but there is "someone else" that is matching the two peers. I fail to see how this is better than a direct communication between peers (such as wireguard - keep in mind I'm not a network engineer, just like to fiddle with them).

ZeroTier is software that creates and manages dynamic wireguard tunnels between endpoints, and, on occasion, relays traffic between those endpoints if they can't find each other directly.

If your devices have public IP's that don't change, making direct wireguard tunnels is ideal. ZeroTier helps with getting around NAT and some firewall issues, and makes adding new sites easier than manually creating a full mesh of VPN tunnels between sites.

@gotsprings, since the description of your use case is limited and you haven't specified what type of business it's used for, it's basically impossible to assess the best solution. It seems more like you need load balancing with redundancy to a central server solution and for that you don't really need SD-WAN as @Sirbryan indicated. A high-end SD-WAN solution is only really necessary if it involves very large volumes of WAN connections with a complex network topology which is where the solution works best and pays off financially.

If it's a small business with a few offices, Netmaker, Zerotier, etc. work just fine. If we assume that you have a service on a server you're already paying for there's no additional server costs to install a simple SD-WAN endpoint like Zerotier. Are you the dedicated resource to manage and administer the solution?

Generally, you have to pick two of the following: good, fast and cheap for it to reflect reality. You've only stated that you find the current solution expensive which indicates that you want to save money. What about the other factors?

I like Zerotier for connecting and reaching back into systems.

But and this is a big one...
I have a lot of systems that rely on having lots of connections that when your IP changes $#!+ Goes sideways.

Like for instance... My bars and restaurants rely on video services now. If your IP address is different from one device to another... You get flagged and one feed or the other stops working. So you need to get all your devices to use one Public IP as it presents to the video service provider. Big Leaf aggregates the ISPs into one IP at the data center. So you can use both feeds at once and one or the other goes out... No IP change.

In my business environment VoIP and wifi calling are a big deal. If some calls are going over one link and some over the other... If one feed "gets Shakey"... Too bad. You have to wait for the link to fail completely. Then wait for the client device to connect back to the cellular gateway and re-establish it's IPSec keys. Losing calls and so forth.

A small broadcast site where they live stream to the national provider... Yeah I need that connection to keep going live.

Etc