Hello Forum,
I was wondering which is the best VPN protocol for a large Hub and Spoke topology based on your previous experiences.
In my scenario, I am talking about 50+ spokes. Every firewall site is a Mikrotik device. The HQ firewall is a CCR1036.
In order to make you understand better:
HQ has two Internet links, each of them with a /28 public subnet.
BO has three Internet links, each of them with a single IP address.
Basically, I want to realize every possible transport interface from BOs to HQ.
Here a stupid screenshot that can help.
Which is the best way to realize the transport layer?
GRE/IPIP/EOIP over IPSEC ? If yes, in which way?
RouterOS 7 OpenVPN over UDP?
Wireguard?
… any other suggestion?
BO will have the VPN transport tunnel as default route. Is it suggested to use a Dynamic routing protocol like OSPF?
Be aware of the fact I want to be able to separate data and voice traffic over two different transport tunnels.
Any suggestion about things I should care about while realizing this kind of project?
That sounds like a management nightmare. Most people use dmvpn for such a use case. No matter what you do, your going to want to automate everything. if you can script then use your language of choice, if you can’t then look into ansible. good luck, as that’s a huge undertaking
It also depends on what functionality you expect from the two internet links at the HQ.
We have a structure like that, and we use L2TP/IPsec with a DNS name that resolves to the two addresses of the HQ.
On connection, the BO do a DNS request and get a “random” sequence of the two addresses (you need to check if your DNS provider does rotate them), and the BO connects via one of the two paths.
When that fails at some point in time, it retries and eventually reconnects to the working one. But at any time there is only one active L2TP link.
An advantage is that this solution scales quite well, you only need a user/password for each BO in the HQ. We assign it a fixed IP and use BGP to route the subnets for the BO.
This also means you can add additional links, e.g. a backup link via 4G or a link between two BO, and include them in the BGP so they will automatically be used in the routing table.
In fact you can also use this to setup more than one permanent link between BO and HQ and switchover the routing when one fails. However, in RouterOS it is infeasible to do balancing over these two links, so it will always be a primary/secondary (hot standby) solution.
You can also integrate fixed GRE tunnels in such a setup, but of course it requires manual configuration for each of them and of course fixed external IP addresses (or at least a fixed DNS name mapped to the variable address, e.g. via a DDNS service).
I would pay the money for zerotier if I were in your shoes. Zerotier can seamlessly use all links and does all the hard work behind the scenes. It’s a SD-WAN type solution which is exactly what your looking for.
I had Bad experiences with EoIP, I would not consider it.
It could be an idea but since I have also tile, mibpse devices and so on, it can’t be a way.
Wanna laugh?
First setup, years ago, I had PPTP.
Not secure anymore, but not bad performances after all.
Then I switched on OpenVPN TCP. A completely disaster.
Now I am in a hybrid situation between Gre/IPSEC and OpenVPN.
Notes:
All the Mikrotiks have the last versions of routerOS 6
I am experiencing some problems with Gre/IPSEC… @pe1chl tried to help me a few months ago without success. (Still trying to fix them)
My hope was about OpenVPN with UDP with the new RouterOS but…
well… in testing environments on tile Mikrotiks, it causes a kernel panic…
I opened a ticket with the support and it will be fixed with 7.1.4…
Ok I use GRE/IPsec (and also without IPsec) and L2TP/IPsec in production on routers with TILE and MIPSBE on RouterOS v6 without any issues.
I use BGP on top of that for the routing, others use OSPF (may be better suited, I have no practical experience with it).
Sounds like a real challenge. How many routers are there in total and how many are working full time with network operations? Any requirements on uptime like 24/7 operations or throughput?
I always configure GRE without keepalive, it is known to be a problem in interoperability.
With IPsec I use the default DPD settings.
I never see the IPsec issues that sometimes occur when DPD does not work (link dead for the remainder of the 8 hour lifetime).
With plain GRE I sometimes see issues when the other end is behind a NAT router. Some of them get in incorrect state and can be recovered by a reboot or keeping the GRE tunnel silent for >5 minutes.
In such cases I add IPsec and it normally is solved. L2TP/IPsec is the least troublesome, although it is known to cause issues when clients are behind CGNAT (e.g. mobile users).
