Hello Forum,
I was wondering which is the best VPN protocol for a large Hub and Spoke topology based on your previous experiences.
In my scenario, I am talking about 50+ spokes. Every firewall site is a Mikrotik device. The HQ firewall is a CCR1036.

In order to make you understand better:
- HQ has two Internet links, each of them with a /28 public subnet.
- BO has three Internet links, each of them with a single IP address.

Basically, I want to realize every possible transport interface from BOs to HQ.
Here a stupid screenshot that can help. 1) Which is the best way to realize the transport layer?
- GRE/IPIP/EOIP over IPSEC ? If yes, in which way?
- RouterOS 7 OpenVPN over UDP?
- Wireguard?
... any other suggestion?

2) BO will have the VPN transport tunnel as default route. Is it suggested to use a Dynamic routing protocol like OSPF?
Be aware of the fact I want to be able to separate data and voice traffic over two different transport tunnels.

3) Any suggestion about things I should care about while realizing this kind of project?

Thanks !

That sounds like a management nightmare. Most people use dmvpn for such a use case. No matter what you do, your going to want to automate everything. if you can script then use your language of choice, if you can't then look into ansible. good luck, as that's a huge undertaking

Is this a typical network one sees in the field or is it a homework question??

Is this a typical network one sees in the field or is it a homework question??

Based on his other questions, its a relatively long standing problem, so probably not a homework question.

Is this a typical network one sees in the field or is it a homework question??

Based on his other questions, its a relatively long standing problem, so probably not a homework question.

But it is mostly one way, a taker not a giver.

Unfortunately it's a real scenario, any suggestion would be appreciated...

That sounds like a management nightmare. Most people use dmvpn for such a use case. No matter what you do, your going to want to automate everything. if you can script then use your language of choice, if you can't then look into ansible. good luck, as that's a huge undertaking

I agree.. I'll see what I can do...

It also depends on what functionality you expect from the two internet links at the HQ.
We have a structure like that, and we use L2TP/IPsec with a DNS name that resolves to the two addresses of the HQ.
On connection, the BO do a DNS request and get a "random" sequence of the two addresses (you need to check if your DNS provider does rotate them), and the BO connects via one of the two paths.
When that fails at some point in time, it retries and eventually reconnects to the working one. But at any time there is only one active L2TP link.

An advantage is that this solution scales quite well, you only need a user/password for each BO in the HQ. We assign it a fixed IP and use BGP to route the subnets for the BO.
This also means you can add additional links, e.g. a backup link via 4G or a link between two BO, and include them in the BGP so they will automatically be used in the routing table.
In fact you can also use this to setup more than one permanent link between BO and HQ and switchover the routing when one fails. However, in RouterOS it is infeasible to do balancing over these two links, so it will always be a primary/secondary (hot standby) solution.

You can also integrate fixed GRE tunnels in such a setup, but of course it requires manual configuration for each of them and of course fixed external IP addresses (or at least a fixed DNS name mapped to the variable address, e.g. via a DDNS service).

IPsec with GRE and OSPF with ECMP on top... Thats the classical solution to solve this.

If you have Mikrotiks on both ends, then EoIP with IPSEC should give you best "wireline compatible" link.

When you have a network like that, with 50+ nodes, EoIP (or L2TP ethernet bridging) is the LAST thing you want to use!

When you have a network like that, with 50+ nodes, EoIP (or L2TP ethernet bridging) is the LAST thing you want to use!

So inquiring minds what to know.

WHO is full of shit then

pe1chl or root. I should start a poll. My bet is on root, to be full of it!

I would pay the money for zerotier if I were in your shoes. Zerotier can seamlessly use all links and does all the hard work behind the scenes. It's a SD-WAN type solution which is exactly what your looking for.

zerotier is not available on all models and thus a cautionary offering.

zerotier is not available on all models and thus a cautionary offering.

True, but it can be spun up on, and ran off of almost any linux box. but then your adding complexity, cost and reliability issues.

zerotier is not available on all models and thus a cautionary offering.

True, but it can be spun up on, and ran off of almost any linux box. but then your adding complexity, cost and reliability issues.

As per the OPs notes; "Every firewall site is a Mikrotik device."

Nobody asked what is the current solution that he uses.
Also you can scratch OpenVPN from that list as OpenVPN and RouterOS aren't friends yet.

If you have Mikrotiks on both ends, then EoIP with IPSEC should give you best "wireline compatible" link.

I had Bad experiences with EoIP, I would not consider it.

zerotier is not available on all models and thus a cautionary offering.

True, but it can be spun up on, and ran off of almost any linux box. but then your adding complexity, cost and reliability issues.

It could be an idea but since I have also tile, mibpse devices and so on, it can't be a way.

Nobody asked what is the current solution that he uses.
Also you can scratch OpenVPN from that list as OpenVPN and RouterOS aren't friends yet.

Wanna laugh?
First setup, years ago, I had PPTP.
Not secure anymore, but not bad performances after all.
Then I switched on OpenVPN TCP. A completely disaster.
Now I am in a hybrid situation between Gre/IPSEC and OpenVPN.

Notes:
1) All the Mikrotiks have the last versions of routerOS 6
2) I am experiencing some problems with Gre/IPSEC... @pe1chl tried to help me a few months ago without success. (Still trying to fix them)

My hope was about OpenVPN with UDP with the new RouterOS but....
well... in testing environments on tile Mikrotiks, it causes a kernel panic....
I opened a ticket with the support and it will be fixed with 7.1.4...

Ok I use GRE/IPsec (and also without IPsec) and L2TP/IPsec in production on routers with TILE and MIPSBE on RouterOS v6 without any issues.
I use BGP on top of that for the routing, others use OSPF (may be better suited, I have no practical experience with it).

Ok I use GRE/IPsec (and also without IPsec) and L2TP/IPsec in production on routers with TILE and MIPSBE on RouterOS v6 without any issues.
I use BGP on top of that for the routing, others use OSPF (may be better suited, I have no practical experience with it).

Do you have any configuration with keepalive active?
What is your setup with DPD?

Cisco has a document about GRE keepalives over IPSec Encryption.
https://www.cisco.com/c/en/us/support/d ... .html#anc7
Is RouterOS afflicted the same?

...It could be my problem afterall...

Sounds like a real challenge. How many routers are there in total and how many are working full time with network operations? Any requirements on uptime like 24/7 operations or throughput?

Do you have any configuration with keepalive active?
What is your setup with DPD?

I always configure GRE without keepalive, it is known to be a problem in interoperability.
With IPsec I use the default DPD settings.
I never see the IPsec issues that sometimes occur when DPD does not work (link dead for the remainder of the 8 hour lifetime).
With plain GRE I sometimes see issues when the other end is behind a NAT router. Some of them get in incorrect state and can be recovered by a reboot or keeping the GRE tunnel silent for >5 minutes.
In such cases I add IPsec and it normally is solved. L2TP/IPsec is the least troublesome, although it is known to cause issues when clients are behind CGNAT (e.g. mobile users).

Sounds like a real challenge. How many routers are there in total and how many are working full time with network operations? Any requirements on uptime like 24/7 operations or throughput?

There are more than 60 routers (Mikrotik)... and yes usually it's something like 22/6 and 24/7
About throughput there's no requirements.

What type of business or location is this that has a network of 60 plus MT devices and no throughput requirements ????
THe only thing I can think of are factories with machinery that reports status (very low throughputs required) but they have specialized equipment for that.

There are more than 60 routers (Mikrotik)... and yes usually it's something like 22/6 and 24/7
About throughput there's no requirements.

We have a hobby network with more routers than that. There is a central router where about 60 routers connect, but there are wireless connections between them and to other routers that have no internet VPN connection.
We have "no requirements" but when something breaks the mailbox overflows so it is better to keep it working
And at work I have such a network but it has only 8 sites. It uses various tunnel types (GRE/IPsec, GRE6/IPsec and L2TP/IPsec) and is a partial mesh.

What type of business or location is this that has a network of 60 plus MT devices and no throughput requirements ????
THe only thing I can think of are factories with machinery that reports status (very low throughputs required) but they have specialized equipment for that.

If you read the whole thread, one of my answer was:

"Then I switched on OpenVPN TCP. A completely disaster.
Now I am in a hybrid situation between Gre/IPSEC and OpenVPN."

This basically answers to your question: OpenVPN TCP used for Lan to Lan enviroments is terrible. Packets latency reaches 200/300ms (when over Gre/IPsec could be 10/20ms) but the customer is able to work, with some difficulties, even in this situation.
Throughtput is usually limited to few Mbps... not more than 10 Mbps to make you understand better.
The customer just can't lose the connection to HQ servers

Throughtput is usually limited to few Mbps... not more than 10 Mbps to make you understand better.
The customer just can't lose the connection to HQ servers

This is why, in my work network, the sites have 3 tunnels:
- a GRE/IPsec to ISP1 at the main office
- a GRE6/IPsec to ISP2 at the main office
- a L2TP/IPsec over 4G (USB stick) to a DNS name with IPv4 of both ISP1 and ISP2 at the main office.
And BGP routing on top of that, which prioritizes in this order.
So when something fails, there usually is a backup. And it is selected automatically. Without using up 4G data volume.

We have a hobby network with more routers than that. There is a central router where about 60 routers connect, but there are wireless connections between them and to other routers that have no internet VPN connection.

Well, then you have plenty of unpaid manpower that is quite something else compared to regular business operations with 22/6 or 24/7 requirements.

There are more than 60 routers (Mikrotik)... and yes usually it's something like 22/6 and 24/7

60 routers located over a large geographical area is a huge undertaking. To find a network protocol and architecture that fits your needs is just a small part of getting all together.

- Any requirements on L2 or just plain L3 networking?
- What's most important, speed or stability?
- Do you have an existing management network in place?
- Tools in place for network monitoring, management and configuration?
- Are all the routers up and running thus you are forced to use remote configuration or are you able to configure and test units in a local lab env before sending it to customer?
- Any backup access (eg LTE) if configuration breaks?
- Manpower for network and configuration management during installation and later for day to day operations?

We have a hobby network with more routers than that. There is a central router where about 60 routers connect, but there are wireless connections between them and to other routers that have no internet VPN connection.

Well, then you have plenty of unpaid manpower that is quite something else compared to regular business operations with 22/6 or 24/7 requirements.

The challenge is that this manpower is largely uncoordinated and not expert in network configuration and operation.
Usually it is easy to get a new node running, the challenge is to prevent them from fouling it up later (e.g. by clicking in the "Quick Set" screen or upgrading to RouterOS v7).

The challenge is that this manpower is largely uncoordinated and not expert in network configuration and operation. Usually it is easy to get a new node running, the challenge is to prevent them from fouling it up later (e.g. by clicking in the "Quick Set" screen or upgrading to RouterOS v7).

Yeah, or if you are surrounded by self-appointed "network experts", each with their own view on what's the best network configuration. ;- )

Another thing that crossed my mind. If you choose to configure and operate ipsec/ike/ospf for that amount of tunnels and especially to keep everything up and running using DPD you will need plenty of manpower. And you will really need it, trust me. I've had my share of those ...

This can be dealt with if the customer has an existing operating organisation in place with enough of time left over for new projects as well as adequate configuration, monitoring and automation tools.

But if not, it can sometimes be more beneficial when it comes to keeping operating costs down which also has a large impact on TCO (total cost of ownership) by replacing certain hardware to create a more long-term stable operating environment that is easier to install and maintain like for example wg or zerotier for business.

It depends more on the number of changes than on the size of the organisation. E.g. now at work we have to deal with change of IP addresses due to takeover of the ISP, and it is a nuisance as you have to make changes at a time that someone else decides. Fortunately it is only affecting the branch offices so the L2TP/IPsec over 4G backup remains working. Nice test for the failover.

Configuration and change management is/should be/ a part of any normal operations as so is planing of resources but I agree with you in general.

Regarding backup access. Unfortunately, you have to burn your fingers a few times before changing the approach to always have a working remote access as a backup. The bigger the crash, the sooner you realise that 4G is a necessary part of the standard installation. And it's a really cheap insurance too.

I still remember my own f-ups as it was yesterday...

- Any requirements on L2 or just plain L3 networking?
- What's most important, speed or stability?
- Do you have an existing management network in place?
- Tools in place for network monitoring, management and configuration?
- Are all the routers up and running thus you are forced to use remote configuration or are you able to configure and test units in a local lab env before sending it to customer?
- Any backup access (eg LTE) if configuration breaks?
- Manpower for network and configuration management during installation and later for day to day operations?

- Luckily only L3
- Stability for sure
- Yes I have a management network
- Zabbix for monitoring, Winbox and some .cfg templates for the configuration. On this part, every BO has the same configuration. It only changes ISPs and Network addresses.
- All router are already running. I am just planning to do tuning in order to use the best protocol
- I have a little lab for tests.
- We use some LTE backups, but not with carrier grade NAT. So we can handle them with DDNS - IP/Cloud features

When I saw "large" I expected thousands of spokes.
We using cleartext L2TP to aggregate IPv4 and IPv6 traffic from random/dynamic addressed Spokes (6000+, some behind NAT) on a single HUB.