Only work 1 of 2 GRE VPNs

RouterOS General
1

Hello forum,

I did not want to create another topic because I have some topics pending, but I need to solve this before the others
I allways have issues with VPNs
This time I have the following issue:

  • 4 RB in 4 different sites
  • All 4 sites with 2 WAN interfaces working
  • Site 4 is new, before there was only 3 sites
  • Previously we had the following GRE tunnels, all working:
  • SITE1 WAN1 ↔ SITE2 WAN1
  • SITE1 WAN2 ↔ SITE2 WAN2
  • SITE1 WAN1 ↔ SITE3 WAN1
  • SITE1 WAN2 ↔ SITE3 WAN2
  • A day I dont know what happened but starting to work only 1 VPN per site, redundant GRE interfaces stop showing the letter “R” of running
  • First time in SITE4 we created another GRE VPN Between: SITE1 WAN1 and SITE4 WAN1, worked
  • I created a redundant tunnel between: SITE1 WAN2 and SITE4 WAN2, did not work (Did not show as running)
  • I tried to log all events with GRE protocol but all logged ítems were from the working VPN
  • I searched in firewall connections, filtering src-address and dst-address, something showed up but did not tell me anything
  • All VPNs are GRE without IPSEC
  • All RouterOS are V 6.42.6

I could attach the export, but the only thing I set in every VPN is remote address, local address and VPN name
It seems for some reason that I dont know, it is only working only 1 VPN between two different sites

Any idea?
Thanks in advance.
Regards!

2

EDIT: “It’s not what you don’t know that kills you, it’s what you know for sure that ain’t true.” - Mark Twain

I was sure about what I wrote below (now in gray), especially because I’ve run into the same situation described by the OP where everything was running smoothly when adding the tunnels one by one but after a reboot there were always issues, and adding the dedicated routes towards remote GRE peers with pref-src was the solution (which rules out the influence of src-nat/masquerade rules as suggested by @Sob below, because such hypothetical NAT rules would have beaten the pref-src settings). And worse than that, I’ve even come across a manual page which gave this explanation (which I’m obviously unable to google up right now).

However, a quick test on a lab device has now shown me that, at least using 6.42.6, the local-address in both /interface gre and /ip ipsec peer configurations is used to set up the source address of the outgoing transport packets. A slightly slower test has shown me that after a reboot, at least at the time when the /tool sniffer can be started, the packets also leave with the local-address as their source address. So I’m at my wit’s end, because a matter of fact is that what I suggest below was necessary to resolve that same problem on another pair of systems one of which is running 6.42.4 and the other one 6.42.6. In addition to 6.42.4 running on one of the ends, there is one more difference in the production case, though - the GRE tunnels are tunneled inside IPsec tunnels, so it takes some time after the reboot until the SAs get up and policies start stealing the packets from their normal routes, but that should not change anything about the packet’s source addresses. To say it all, the src-address and dst-address parameters of the policies are /32 addresses, i.e. they only match the local-address and remote-address parameters of the corresponding /interface gre configurations.

My shot into the darkness is that you have set local-address in /interface gre and assume that this address is automatically used as source one for outgoing GRE transport packets, which is unfortunately not true, as the local-address is only used to identify the tunnel to which received GRE transport packets belong.

The source address of GRE transport packets is chosen depending on the output interface chosen to send them in the initial step of routing, using the routing table. But if some routing adjustments are taken later due to policy routing and alike, so the actual output interface is different from the initially chosen one, the source address of the packet doesn’t change.

So a firewall somewhere on the way may not match packets in the public → private direction to an existing connection because they come from a different address than their counterparts in private → public direction, so it drops them and the tunnel doesn’t get up.

To override the default choice of source address, use the pref-src parameter of a route. So if you need to use a pair of GRE tunnels between two devices and you need that each of the tunnels uses a different WAN interface, configure it as follows:


Site 1:

/interface gre
add name=gre-1.1-2.1 local-address=site.1.wan.1 remote-address=site.2.wan.1
add name=gre-1.2-2.2 local-address=site.1.wan.2 remote-address=site.2.wan.2
/ip route
add dst-address=site.2.wan.1/32 gateway=site.1.gw.1 pref-src=site.1.wan.1
add dst-address=site.2.wan.2/32 gateway=site.1.gw.2 pref-src=site.1.wan.2

Site 2:

/interface gre
add name=gre-2.1-1.1 local-address=site.2.wan.1 remote-address=site.1.wan.1
add name=gre-2.2-1.2 local-address=site.2.wan.2 remote-address=site.1.wan.2
/ip route
add dst-address=site.1.wan.1/32 gateway=site.2.gw.1 pref-src=site.2.wan.1
add dst-address=site.1.wan.2/32 gateway=site.2.gw.2 pref-src=site.2.wan.2
3

That’s not what I see. Are you sure that it’s not just your default masquerade/srcnat changing the address? My GRE tunnel uses whatever I put in local-address as source.

What might be unexpected for some people is how the route is chosen. Just because you set the source address to WANx IP, it doesn’t mean that it will go out via WANx, by default it will still use whatever default route there is. But you can help it using:

/ip route rule
add src-address=<wan1address> table=<wan1table>
add src-address=<wan2address> table=<wan2table>
4

Sindy,
You did a shot into the darkness and you got the target.
You are right, this is the problem. Now both tunnels are working.

You are awesome!!
Thanks a lot
Regards!

5

Great to hear that, but now there is a bigger problem, how is it possible that it helps given that the source address of the GRE transports packets is determined by the local-address parameter?

I cannot do the reboot tests I did on the lab equipment on the production one, so the only possibility left is to look what my production configuration and your one have in common and the lab one is missing it, and try to change the lab configuration accordingly to see whether it makes the issue show up.

Would you mind exporting your configuration, taking the usual obfuscation steps as described in my automatic signature?

6

It is working, because it solves the routing problem. If you start with your GRE config and nothing for routing (either your /32 routes or my routing rules) then both tunnels will use default routing table. Let’s assume that current default route is via WAN1. First tunnel from site.1.wan.1 works great. But packets of second tunnel will go out via WAN1 too. Either with site.1.wan.2 as source (which ISP1 won’t like and will drop it) or site.1.wan.1 as source (if masquerade/srcnat applies also to router’s output, and it does with usual config; but second router doesn’t expect tunnel between site.1.wan.1 and site.2.wan.2).

7

The problem is that while Damián has not told us anything about his routing configuration before applying my voodoo (yet), in my production case the individual routes were there since the very start - except that they didn’t have the pref-src set. And having them there was not sufficient to make the second GRE tunnel get up after reboot.

8

My great theory is ruined, I hope you’re happy. :smiley: I’m trying some experiments and so far it looks like if those /32s are available from the start, they should do the trick, without the need for pref-source. And if GRE’s local-address is set, then /32’s pref-source doesn’t matter at all. Even if there’s masquerade, it takes address from pref-src of interface’s connected route.

9

Just to be clear,
The GRE interfaces were configured well, with local and remote address correctly
Both tunnels started to work at the same time when I added the static route as Sindy sugested:

/ip route rule
add src-address=<wan1address> table=<wan1table>
add src-address=<wan2address> table=<wan2table>

I thought that maybe setting the local address was enought to send the traffic out through the correct wan, but obviously it doesn’t
Masquerade/src-nat I think couldn’t solve the issue because both routers have public IPs in its WANs interfaces, so (I think, not sure) them does not masquerade when the traffic is generated in the router itself

I will make a monument to Sindy some day, I will contact you to know your face, hehehehehhe

Thanks a lot again :wink:

10

Try again, what exactly you added? You say sindy’s routes, but quote my rules. It’s not about the monument, but since we’re not 100% sure what’s happening, we need accurate info.

About srcnat/masquerade, it happens in postrouting phase and doesn’t care if connection came from router or through router, so unless you exclude local source, it will apply.

And there’s another factor. Assuming you have the usual connection/route marking to allow replies to connections from WANx go back from where the request came, there should be no problem when first GRE packet comes from the other router, because next outgoing packet from local router will be seen as part of established “connection”. It’s only when local router initiates the connection. So it can also complicate the testing a little.

11

Sorry, my mistake

/ip route
add dst-address=site.2.wan.1/32 gateway=site.1.gw.1 pref-src=site.1.wan.1
add dst-address=site.2.wan.2/32 gateway=site.1.gw.2 pref-src=site.1.wan.2

This is what I added, the Sindy’s routes
I think masquerade outgoing packets generated from the router does not have any sense, I ussualy leave masquerade, so it will masquerade the src-address and will replace the source address with the same address, anyway is good to know.

Routes are the default and there isn’t any mark, I just needed to add the Sindy’s routes and both VPNs started to work

Thanks both for your answers and work!!!

12

When speaking about monuments, it is actually @Sob who deserves one, as he too often has to come to correct my mistakes :slight_smile:

In this case, I have actually provided the correct solution by chance - as I didn’t expect you not to know how the routing works, I was supposing automatically that you do have individual routes in place (or use routing rules or routing-marks) to force each tunnel via its dedicated WAN, so I’ve concentrated on the issue I had in the past which was that the tunnels were not establishing, and I was supposing the reason to be wrong source address as I could see no way how the individual routes to /32 destinations could fail when handling the destination address. So adding the individiual routes which were initially missing in your configuration was the actual solution, but I’ve provided it as a kind of a side effect of a solution aimed to using the pref-src to rectify the source addresses in the simplest manner - which turned out to be redundant.

And again it was @Sob who had to kick in to prove wrong my false statement that the /interface gre’s local-address is only taken into account when processing incoming packets, which was the only explanation had for my own case (and I still have no other one, unfortunately).

13

I’m willing to share, let’s make it a statue and you can choose if you want to be on left or right. It’s same to me, my only request is for white marble, 'cause of pigeons. :slight_smile: And for the record, I don’t remember correcting you more than once or twice.

14

I would not say that I dont know how routing works, I know the basic, I just didn’t know that setting up the local address in the tunnel is not enought
Also, GRE manual does not say anything about this route, just about route to the remote network: https://wiki.mikrotik.com/wiki/Manual:Interface/Gre

Now, sorry about this, I created the following without any issue:

  • Site1 WAN1 ↔ Site4 WAN1
  • Site1 WAN2 ↔ Site4 WAN2
  • Site1 WAN1 ↔ Site2 WAN1
  • Site1 WAN2 ↔ Site2 WAN2
  • Site2 WAN1 ↔ Site3 WAN1
  • Site2 WAN2 ↔ Site3 WAN2

When I want to create the Site3 - Site4 GRE tunnel, it does not start. ¿Are there any kind of limitation about the amount of GRE VPNs?
I tried with WAN1-WAN1, WAN2-WAN2, WAN1-WAN2, WAN2-WAN1, no one started
The first time I created WAN1-WAN1 and WAN2-WAN2 gre tunnel, in site3 appeared the R letter on the first GRE, but in site4, it appeared in the second GRE (both GRE appeared as running in just 1 site)
Any idea shooting into the darkness?

15

Unless there is a mistake in some of the many addresses which need to be configured, I have no idea what could be wrong. Four tunnels per device is a ridiculously small number, I don’t know about any limitation at all for GRE tunnels, the license page doesn’t state anything for GRE, but for other types of tunnels the lowest limit is 200.

So it can be anything - incorrect addresses in the GRE configuration and/or in the individual routes, some srcnat or dstnat rules, firewall filter rules… hard to say without seeing your configuration exports from the two devices. And while preparing those exports for posting using the guidelines in my automatic signature below, you may find some issues when obfuscating, using a text editor, the IP addresses in the exports from both ends merged into a single text in order to maintain consistence of the data - each IP address must be substituted by exactly the same meaningful string in both exports, so if you do that and some address doesn’t get substituted at all or the substitution reveals an inconsistence between the two ends, you’ll see the mistake yourself.

16

I just spent some time playing with GRE and the only scenario without any routing adjustments where it works is when both local-address and keepalive are not set. Then both tunnels’ outgoing packets to remote router go out via same WAN (the primary one with active default route). They are simply mixed together and since GRE is stateless, it doesn’t matter. All GRE interfaces have “R” as running, but only because there are no actual checks, it’s just optimistic assumption. The result is routing like this:

+---------+                                    +---------+
|         | WAN1 gre1 -->>------<<-- gre1 WAN1 |         |
|         |      gre2 -->>-\  /-<<-- gre2      |         |
| Router1 |                 \/                 | Router2 |
|         |                 /\                 |         |
|         | WAN2 --<<------/  \------>>-- WAN2 |         |
+---------+                                    +---------+

Failover for traffic between remote subnets works, if you have routes like this on both routers:

/ip route
add dst-address=<remote subnet> gateway=<ip on the other end of tunnel 1> check-gateway=ping
add dst-address=<remote subnet> gateway=<ip on the other end of tunnel 2> check-gateway=ping

But if you either set local-address or enable keepalive, secondary tunnel fails. Only way I can see how you can have local-adress set, keepalive enabled (or both) and the secondary tunnel working is to make sure that it uses secondary WAN as outgoing interface. Correct routing is the key. But it won’t happen automatically, you need to help it. So how it could have worked for you before, if you didn’t, it’s mystery for me.

17

Hi,
Sorry about the delay.
I see the last week that those tunnels were all up, and those are still up, without changing anything
Now I have the 8 tunnels working, using a specific interface for each tunnel, using the routes of Sindy, using local and remote address in the interface properties and keeping the default keepalive, and using the pref-source in the routes as Sindy sugested

Thanks to both for your help, which is very useful.
Regards

18

So today I’ve come across that manual page again, and it turns out I’ve remembered what it said incorrectly or misinterpreted it when reading it for the first time - it’s not that the source address would not be set to local-address but it is really only the routing issue, i.e. that the WAN is not chosen based on the local-address automatically.

19

Hello Forum. sorry for re-open this post,

1 of all VPNs stopped working and I cannot find why
I left 8 VPNs working:
S of Site - W of WAN:
S1W1 - S2W1
S1W2 - S2W2
S2W1 - S3W1
S2W2 - S3W2
S3W1 - S4W1
S3W2 - S4W2
S4W1 - S1W1
S4W2 - S1W2

Previously I had VPNs among 3 sites, working only 3 of 6, the configuration was the basics, just set local and remote address in the interface, in both sides, a different local address in each side of the VPN but in the same network, and a route to Access to the other network throught the VPN, there were not marks configured and no route added to force one VPN to go out thought an specific WAN.
The solution was create a route to force each VPN to go out through the designated WAN (As Sindy sugested)
Now only 1 of the 8 VPNs is not working, in “interfaces” I can see it with the R of running, just in one side, in the other site it appears as not running

Now there is the following configuration, the same but inverted in the other side:

/interface gre
add disabled=yes local-address=S1W2Address name=S1W2-S4W2 remote-address=S4W2Address

/ip address
add address=1.1.1.1/29 interface="S1W2-S4W2" network=1.1.1.0  (I changed the IP)

/ip route
add check-gateway=ping comment="S1W2-S4W2 goes out through Wan2" distance=1 dst-address=S4W2Address gateway=S1W2Gateway pref-src=S1W2Address
add check-gateway=ping comment="Site 4 network" distance=20 dst-address=10.10.10.0/24 gateway=1.1.1.2

I have now in 1 of the sites mangle marks for other thinks, I just mark all inconming connections to go out through the same interface where it came up, and forced an specific network to go out throught WAN2

How can I check why this is not running?
Sob told me that he had less issues when the local address is not set and there is not any route added to force the VPN to go out throught the designated WAN, but the only difference with my previous settings (when just worked a half of the VPNs) is that I had the “Local address” in the GRE interface. Could remove the local address make it to work?
Any sugestion?

Thanks in advance.
Regards,
Damián

20

Show me /ip firewall connection print detail where protocol=gre from both ends while it does not work (obfuscate public IP addresses in a consistent way so that the relationship between the two ends remains visible).