There are more than 60 routers (Mikrotik)… and yes usually it’s something like 22/6 and 24/7
About throughput there’s no requirements.
What type of business or location is this that has a network of 60 plus MT devices and no throughput requirements ???
THe only thing I can think of are factories with machinery that reports status (very low throughputs required) but they have specialized equipment for that.
We have a hobby network with more routers than that. There is a central router where about 60 routers connect, but there are wireless connections between them and to other routers that have no internet VPN connection.
We have “no requirements” but when something breaks the mailbox overflows so it is better to keep it working 
And at work I have such a network but it has only 8 sites. It uses various tunnel types (GRE/IPsec, GRE6/IPsec and L2TP/IPsec) and is a partial mesh.
If you read the whole thread, one of my answer was:
“Then I switched on OpenVPN TCP. A completely disaster.
Now I am in a hybrid situation between Gre/IPSEC and OpenVPN.”
This basically answers to your question: OpenVPN TCP used for Lan to Lan enviroments is terrible. Packets latency reaches 200/300ms (when over Gre/IPsec could be 10/20ms) but the customer is able to work, with some difficulties, even in this situation.
Throughtput is usually limited to few Mbps… not more than 10 Mbps to make you understand better.
The customer just can’t lose the connection to HQ servers
This is why, in my work network, the sites have 3 tunnels:
- a GRE/IPsec to ISP1 at the main office
- a GRE6/IPsec to ISP2 at the main office
- a L2TP/IPsec over 4G (USB stick) to a DNS name with IPv4 of both ISP1 and ISP2 at the main office.
And BGP routing on top of that, which prioritizes in this order.
So when something fails, there usually is a backup. And it is selected automatically. Without using up 4G data volume.
Well, then you have plenty of unpaid manpower that is quite something else compared to regular business operations with 22/6 or 24/7 requirements.
60 routers located over a large geographical area is a huge undertaking. To find a network protocol and architecture that fits your needs is just a small part of getting all together.
- Any requirements on L2 or just plain L3 networking?
- What’s most important, speed or stability?
- Do you have an existing management network in place?
- Tools in place for network monitoring, management and configuration?
- Are all the routers up and running thus you are forced to use remote configuration or are you able to configure and test units in a local lab env before sending it to customer?
- Any backup access (eg LTE) if configuration breaks?
- Manpower for network and configuration management during installation and later for day to day operations?
The challenge is that this manpower is largely uncoordinated and not expert in network configuration and operation.
Usually it is easy to get a new node running, the challenge is to prevent them from fouling it up later (e.g. by clicking in the “Quick Set” screen or upgrading to RouterOS v7).
Yeah, or if you are surrounded by self-appointed “network experts”, each with their own view on what’s the best network configuration. ;- )
Another thing that crossed my mind. If you choose to configure and operate ipsec/ike/ospf for that amount of tunnels and especially to keep everything up and running using DPD you will need plenty of manpower. And you will really need it, trust me. I’ve had my share of those …
This can be dealt with if the customer has an existing operating organisation in place with enough of time left over for new projects as well as adequate configuration, monitoring and automation tools.
But if not, it can sometimes be more beneficial when it comes to keeping operating costs down which also has a large impact on TCO (total cost of ownership) by replacing certain hardware to create a more long-term stable operating environment that is easier to install and maintain like for example wg or zerotier for business.
It depends more on the number of changes than on the size of the organisation. E.g. now at work we have to deal with change of IP addresses due to takeover of the ISP, and it is a nuisance as you have to make changes at a time that someone else decides. Fortunately it is only affecting the branch offices so the L2TP/IPsec over 4G backup remains working. Nice test for the failover.
Configuration and change management is/should be/ a part of any normal operations as so is planing of resources but I agree with you in general.
Regarding backup access. Unfortunately, you have to burn your fingers a few times before changing the approach to always have a working remote access as a backup. The bigger the crash, the sooner you realise that 4G is a necessary part of the standard installation. And it’s a really cheap insurance too.
I still remember my own f-ups as it was yesterday…
- Luckily only L3
- Stability for sure
- Yes I have a management network
- Zabbix for monitoring, Winbox and some .cfg templates for the configuration. On this part, every BO has the same configuration. It only changes ISPs and Network addresses.
- All router are already running. I am just planning to do tuning in order to use the best protocol
- I have a little lab for tests.
- We use some LTE backups, but not with carrier grade NAT. So we can handle them with DDNS - IP/Cloud features
When I saw “large” I expected thousands of spokes.
We using cleartext L2TP to aggregate IPv4 and IPv6 traffic from random/dynamic addressed Spokes (6000+, some behind NAT) on a single HUB.