Hello forum,
I did not want to create another topic because I have some topics pending, but I need to solve this before the others
I allways have issues with VPNs
This time I have the following issue:
- 4 RB in 4 different sites
- All 4 sites with 2 WAN interfaces working
- Site 4 is new, before there was only 3 sites
- Previously we had the following GRE tunnels, all working:
* SITE1 WAN1 <-> SITE2 WAN1
* SITE1 WAN2 <-> SITE2 WAN2
* SITE1 WAN1 <-> SITE3 WAN1
* SITE1 WAN2 <-> SITE3 WAN2
- A day I dont know what happened but starting to work only 1 VPN per site, redundant GRE interfaces stop showing the letter "R" of running
- First time in SITE4 we created another GRE VPN Between: SITE1 WAN1 and SITE4 WAN1, worked
- I created a redundant tunnel between: SITE1 WAN2 and SITE4 WAN2, did not work (Did not show as running)
- I tried to log all events with GRE protocol but all logged ítems were from the working VPN
- I searched in firewall connections, filtering src-address and dst-address, something showed up but did not tell me anything
- All VPNs are GRE without IPSEC
- All RouterOS are V 6.42.6
I could attach the export, but the only thing I set in every VPN is remote address, local address and VPN name
It seems for some reason that I dont know, it is only working only 1 VPN between two different sites
Any idea?
Thanks in advance.
Regards!
Re: Only work 1 of 2 GRE VPNs [SOLVED]
EDIT: "It's not what you don't know that kills you, it's what you know for sure that ain't true." - Mark Twain
I was sure about what I wrote below (now in gray), especially because I've run into the same situation described by the OP where everything was running smoothly when adding the tunnels one by one but after a reboot there were always issues, and adding the dedicated routes towards remote GRE peers with pref-src was the solution (which rules out the influence of src-nat/masquerade rules as suggested by @Sob below, because such hypothetical NAT rules would have beaten the pref-src settings). And worse than that, I've even come across a manual page which gave this explanation (which I'm obviously unable to google up right now).
However, a quick test on a lab device has now shown me that, at least using 6.42.6, the local-address in both /interface gre and /ip ipsec peer configurations is used to set up the source address of the outgoing transport packets. A slightly slower test has shown me that after a reboot, at least at the time when the /tool sniffer can be started, the packets also leave with the local-address as their source address. So I'm at my wit's end, because a matter of fact is that what I suggest below was necessary to resolve that same problem on another pair of systems one of which is running 6.42.4 and the other one 6.42.6. In addition to 6.42.4 running on one of the ends, there is one more difference in the production case, though - the GRE tunnels are tunneled inside IPsec tunnels, so it takes some time after the reboot until the SAs get up and policies start stealing the packets from their normal routes, but that should not change anything about the packet's source addresses. To say it all, the src-address and dst-address parameters of the policies are /32 addresses, i.e. they only match the local-address and remote-address parameters of the corresponding /interface gre configurations.
My shot into the darkness is that you have set local-address in /interface gre and assume that this address is automatically used as source one for outgoing GRE transport packets, which is unfortunately not true, as the local-address is only used to identify the tunnel to which received GRE transport packets belong.
The source address of GRE transport packets is chosen depending on the output interface chosen to send them in the initial step of routing, using the routing table. But if some routing adjustments are taken later due to policy routing and alike, so the actual output interface is different from the initially chosen one, the source address of the packet doesn't change.
So a firewall somewhere on the way may not match packets in the public -> private direction to an existing connection because they come from a different address than their counterparts in private -> public direction, so it drops them and the tunnel doesn't get up.
To override the default choice of source address, use the pref-src parameter of a route. So if you need to use a pair of GRE tunnels between two devices and you need that each of the tunnels uses a different WAN interface, configure it as follows:
Site 1:
Site 2:
I was sure about what I wrote below (now in gray), especially because I've run into the same situation described by the OP where everything was running smoothly when adding the tunnels one by one but after a reboot there were always issues, and adding the dedicated routes towards remote GRE peers with pref-src was the solution (which rules out the influence of src-nat/masquerade rules as suggested by @Sob below, because such hypothetical NAT rules would have beaten the pref-src settings). And worse than that, I've even come across a manual page which gave this explanation (which I'm obviously unable to google up right now).
However, a quick test on a lab device has now shown me that, at least using 6.42.6, the local-address in both /interface gre and /ip ipsec peer configurations is used to set up the source address of the outgoing transport packets. A slightly slower test has shown me that after a reboot, at least at the time when the /tool sniffer can be started, the packets also leave with the local-address as their source address. So I'm at my wit's end, because a matter of fact is that what I suggest below was necessary to resolve that same problem on another pair of systems one of which is running 6.42.4 and the other one 6.42.6. In addition to 6.42.4 running on one of the ends, there is one more difference in the production case, though - the GRE tunnels are tunneled inside IPsec tunnels, so it takes some time after the reboot until the SAs get up and policies start stealing the packets from their normal routes, but that should not change anything about the packet's source addresses. To say it all, the src-address and dst-address parameters of the policies are /32 addresses, i.e. they only match the local-address and remote-address parameters of the corresponding /interface gre configurations.
My shot into the darkness is that you have set local-address in /interface gre and assume that this address is automatically used as source one for outgoing GRE transport packets, which is unfortunately not true, as the local-address is only used to identify the tunnel to which received GRE transport packets belong.
The source address of GRE transport packets is chosen depending on the output interface chosen to send them in the initial step of routing, using the routing table. But if some routing adjustments are taken later due to policy routing and alike, so the actual output interface is different from the initially chosen one, the source address of the packet doesn't change.
So a firewall somewhere on the way may not match packets in the public -> private direction to an existing connection because they come from a different address than their counterparts in private -> public direction, so it drops them and the tunnel doesn't get up.
To override the default choice of source address, use the pref-src parameter of a route. So if you need to use a pair of GRE tunnels between two devices and you need that each of the tunnels uses a different WAN interface, configure it as follows:
Site 1:
Code: Select all
/interface gre
add name=gre-1.1-2.1 local-address=site.1.wan.1 remote-address=site.2.wan.1
add name=gre-1.2-2.2 local-address=site.1.wan.2 remote-address=site.2.wan.2
/ip route
add dst-address=site.2.wan.1/32 gateway=site.1.gw.1 pref-src=site.1.wan.1
add dst-address=site.2.wan.2/32 gateway=site.1.gw.2 pref-src=site.1.wan.2
Site 2:
Code: Select all
/interface gre
add name=gre-2.1-1.1 local-address=site.2.wan.1 remote-address=site.1.wan.1
add name=gre-2.2-1.2 local-address=site.2.wan.2 remote-address=site.1.wan.2
/ip route
add dst-address=site.1.wan.1/32 gateway=site.2.gw.1 pref-src=site.2.wan.1
add dst-address=site.1.wan.2/32 gateway=site.2.gw.2 pref-src=site.2.wan.2
Last edited by sindy on Sat Sep 22, 2018 11:11 am, edited 4 times in total.
Re: Only work 1 of 2 GRE VPNs
That's not what I see. Are you sure that it's not just your default masquerade/srcnat changing the address? My GRE tunnel uses whatever I put in local-address as source.My shot into the darkness is that you have set local-address in /interface gre and assume that this address is automatically used as source one for outgoing GRE transport packets, which is unfortunately not true, as the local-address is only used to identify the tunnel to which received GRE transport packets belong.
What might be unexpected for some people is how the route is chosen. Just because you set the source address to WANx IP, it doesn't mean that it will go out via WANx, by default it will still use whatever default route there is. But you can help it using:
Code: Select all
/ip route rule
add src-address=<wan1address> table=<wan1table>
add src-address=<wan2address> table=<wan2table>Re: Only work 1 of 2 GRE VPNs
Sindy,
You did a shot into the darkness and you got the target.
You are right, this is the problem. Now both tunnels are working.
You are awesome!!
Thanks a lot
Regards!
You did a shot into the darkness and you got the target.
You are right, this is the problem. Now both tunnels are working.
You are awesome!!
Thanks a lot
Regards!
Re: Only work 1 of 2 GRE VPNs
Great to hear that, but now there is a bigger problem, how is it possible that it helps given that the source address of the GRE transports packets is determined by the local-address parameter?You are right, this is the problem. Now both tunnels are working.
I cannot do the reboot tests I did on the lab equipment on the production one, so the only possibility left is to look what my production configuration and your one have in common and the lab one is missing it, and try to change the lab configuration accordingly to see whether it makes the issue show up.
Would you mind exporting your configuration, taking the usual obfuscation steps as described in my automatic signature?
Re: Only work 1 of 2 GRE VPNs
It is working, because it solves the routing problem. If you start with your GRE config and nothing for routing (either your /32 routes or my routing rules) then both tunnels will use default routing table. Let's assume that current default route is via WAN1. First tunnel from site.1.wan.1 works great. But packets of second tunnel will go out via WAN1 too. Either with site.1.wan.2 as source (which ISP1 won't like and will drop it) or site.1.wan.1 as source (if masquerade/srcnat applies also to router's output, and it does with usual config; but second router doesn't expect tunnel between site.1.wan.1 and site.2.wan.2).
Re: Only work 1 of 2 GRE VPNs
The problem is that while Damián has not told us anything about his routing configuration before applying my voodoo (yet), in my production case the individual routes were there since the very start - except that they didn't have the pref-src set. And having them there was not sufficient to make the second GRE tunnel get up after reboot.It is working, because it solves the routing problem.
Re: Only work 1 of 2 GRE VPNs
My great theory is ruined, I hope you're happy. 
I'm trying some experiments and so far it looks like if those /32s are available from the start, they should do the trick, without the need for pref-source. And if GRE's local-address is set, then /32's pref-source doesn't matter at all. Even if there's masquerade, it takes address from pref-src of interface's connected route.
I'm trying some experiments and so far it looks like if those /32s are available from the start, they should do the trick, without the need for pref-source. And if GRE's local-address is set, then /32's pref-source doesn't matter at all. Even if there's masquerade, it takes address from pref-src of interface's connected route.Re: Only work 1 of 2 GRE VPNs
Just to be clear,
The GRE interfaces were configured well, with local and remote address correctly
Both tunnels started to work at the same time when I added the static route as Sindy sugested:
I thought that maybe setting the local address was enought to send the traffic out through the correct wan, but obviously it doesn't
Masquerade/src-nat I think couldn't solve the issue because both routers have public IPs in its WANs interfaces, so (I think, not sure) them does not masquerade when the traffic is generated in the router itself
I will make a monument to Sindy some day, I will contact you to know your face, hehehehehhe
Thanks a lot again
The GRE interfaces were configured well, with local and remote address correctly
Both tunnels started to work at the same time when I added the static route as Sindy sugested:
Code: Select all
/ip route rule
add src-address=<wan1address> table=<wan1table>
add src-address=<wan2address> table=<wan2table>Masquerade/src-nat I think couldn't solve the issue because both routers have public IPs in its WANs interfaces, so (I think, not sure) them does not masquerade when the traffic is generated in the router itself
I will make a monument to Sindy some day, I will contact you to know your face, hehehehehhe
Thanks a lot again
Re: Only work 1 of 2 GRE VPNs
Try again, what exactly you added? You say sindy's routes, but quote my rules. It's not about the monument, but since we're not 100% sure what's happening, we need accurate info.
About srcnat/masquerade, it happens in postrouting phase and doesn't care if connection came from router or through router, so unless you exclude local source, it will apply.
And there's another factor. Assuming you have the usual connection/route marking to allow replies to connections from WANx go back from where the request came, there should be no problem when first GRE packet comes from the other router, because next outgoing packet from local router will be seen as part of established "connection". It's only when local router initiates the connection. So it can also complicate the testing a little.
About srcnat/masquerade, it happens in postrouting phase and doesn't care if connection came from router or through router, so unless you exclude local source, it will apply.
And there's another factor. Assuming you have the usual connection/route marking to allow replies to connections from WANx go back from where the request came, there should be no problem when first GRE packet comes from the other router, because next outgoing packet from local router will be seen as part of established "connection". It's only when local router initiates the connection. So it can also complicate the testing a little.
Re: Only work 1 of 2 GRE VPNs
Sorry, my mistake
This is what I added, the Sindy's routes
I think masquerade outgoing packets generated from the router does not have any sense, I ussualy leave masquerade, so it will masquerade the src-address and will replace the source address with the same address, anyway is good to know.
Routes are the default and there isn't any mark, I just needed to add the Sindy's routes and both VPNs started to work
Thanks both for your answers and work!!!
Code: Select all
/ip route
add dst-address=site.2.wan.1/32 gateway=site.1.gw.1 pref-src=site.1.wan.1
add dst-address=site.2.wan.2/32 gateway=site.1.gw.2 pref-src=site.1.wan.2I think masquerade outgoing packets generated from the router does not have any sense, I ussualy leave masquerade, so it will masquerade the src-address and will replace the source address with the same address, anyway is good to know.
Routes are the default and there isn't any mark, I just needed to add the Sindy's routes and both VPNs started to work
Thanks both for your answers and work!!!
Re: Only work 1 of 2 GRE VPNs
When speaking about monuments, it is actually @Sob who deserves one, as he too often has to come to correct my mistakes 
In this case, I have actually provided the correct solution by chance - as I didn't expect you not to know how the routing works, I was supposing automatically that you do have individual routes in place (or use routing rules or routing-marks) to force each tunnel via its dedicated WAN, so I've concentrated on the issue I had in the past which was that the tunnels were not establishing, and I was supposing the reason to be wrong source address as I could see no way how the individual routes to /32 destinations could fail when handling the destination address. So adding the individiual routes which were initially missing in your configuration was the actual solution, but I've provided it as a kind of a side effect of a solution aimed to using the pref-src to rectify the source addresses in the simplest manner - which turned out to be redundant.
And again it was @Sob who had to kick in to prove wrong my false statement that the /interface gre's local-address is only taken into account when processing incoming packets, which was the only explanation had for my own case (and I still have no other one, unfortunately).
In this case, I have actually provided the correct solution by chance - as I didn't expect you not to know how the routing works, I was supposing automatically that you do have individual routes in place (or use routing rules or routing-marks) to force each tunnel via its dedicated WAN, so I've concentrated on the issue I had in the past which was that the tunnels were not establishing, and I was supposing the reason to be wrong source address as I could see no way how the individual routes to /32 destinations could fail when handling the destination address. So adding the individiual routes which were initially missing in your configuration was the actual solution, but I've provided it as a kind of a side effect of a solution aimed to using the pref-src to rectify the source addresses in the simplest manner - which turned out to be redundant.
And again it was @Sob who had to kick in to prove wrong my false statement that the /interface gre's local-address is only taken into account when processing incoming packets, which was the only explanation had for my own case (and I still have no other one, unfortunately).
Re: Only work 1 of 2 GRE VPNs
I'm willing to share, let's make it a statue and you can choose if you want to be on left or right. It's same to me, my only request is for white marble, 'cause of pigeons. And for the record, I don't remember correcting you more than once or twice.
And for the record, I don't remember correcting you more than once or twice.Re: Only work 1 of 2 GRE VPNs
I would not say that I dont know how routing works, I know the basic, I just didn't know that setting up the local address in the tunnel is not enought
Also, GRE manual does not say anything about this route, just about route to the remote network: https://wiki.mikrotik.com/wiki/Manual:Interface/Gre
Now, sorry about this, I created the following without any issue:
- Site1 WAN1 <-> Site4 WAN1
- Site1 WAN2 <-> Site4 WAN2
- Site1 WAN1 <-> Site2 WAN1
- Site1 WAN2 <-> Site2 WAN2
- Site2 WAN1 <-> Site3 WAN1
- Site2 WAN2 <-> Site3 WAN2
When I want to create the Site3 - Site4 GRE tunnel, it does not start. ¿Are there any kind of limitation about the amount of GRE VPNs?
I tried with WAN1-WAN1, WAN2-WAN2, WAN1-WAN2, WAN2-WAN1, no one started
The first time I created WAN1-WAN1 and WAN2-WAN2 gre tunnel, in site3 appeared the R letter on the first GRE, but in site4, it appeared in the second GRE (both GRE appeared as running in just 1 site)
Any idea shooting into the darkness?
Also, GRE manual does not say anything about this route, just about route to the remote network: https://wiki.mikrotik.com/wiki/Manual:Interface/Gre
Now, sorry about this, I created the following without any issue:
- Site1 WAN1 <-> Site4 WAN1
- Site1 WAN2 <-> Site4 WAN2
- Site1 WAN1 <-> Site2 WAN1
- Site1 WAN2 <-> Site2 WAN2
- Site2 WAN1 <-> Site3 WAN1
- Site2 WAN2 <-> Site3 WAN2
When I want to create the Site3 - Site4 GRE tunnel, it does not start. ¿Are there any kind of limitation about the amount of GRE VPNs?
I tried with WAN1-WAN1, WAN2-WAN2, WAN1-WAN2, WAN2-WAN1, no one started
The first time I created WAN1-WAN1 and WAN2-WAN2 gre tunnel, in site3 appeared the R letter on the first GRE, but in site4, it appeared in the second GRE (both GRE appeared as running in just 1 site)
Any idea shooting into the darkness?
Re: Only work 1 of 2 GRE VPNs
Unless there is a mistake in some of the many addresses which need to be configured, I have no idea what could be wrong. Four tunnels per device is a ridiculously small number, I don't know about any limitation at all for GRE tunnels, the license page doesn't state anything for GRE, but for other types of tunnels the lowest limit is 200.
So it can be anything - incorrect addresses in the GRE configuration and/or in the individual routes, some srcnat or dstnat rules, firewall filter rules... hard to say without seeing your configuration exports from the two devices. And while preparing those exports for posting using the guidelines in my automatic signature below, you may find some issues when obfuscating, using a text editor, the IP addresses in the exports from both ends merged into a single text in order to maintain consistence of the data - each IP address must be substituted by exactly the same meaningful string in both exports, so if you do that and some address doesn't get substituted at all or the substitution reveals an inconsistence between the two ends, you'll see the mistake yourself.
So it can be anything - incorrect addresses in the GRE configuration and/or in the individual routes, some srcnat or dstnat rules, firewall filter rules... hard to say without seeing your configuration exports from the two devices. And while preparing those exports for posting using the guidelines in my automatic signature below, you may find some issues when obfuscating, using a text editor, the IP addresses in the exports from both ends merged into a single text in order to maintain consistence of the data - each IP address must be substituted by exactly the same meaningful string in both exports, so if you do that and some address doesn't get substituted at all or the substitution reveals an inconsistence between the two ends, you'll see the mistake yourself.
Re: Only work 1 of 2 GRE VPNs
I just spent some time playing with GRE and the only scenario without any routing adjustments where it works is when both local-address and keepalive are not set. Then both tunnels' outgoing packets to remote router go out via same WAN (the primary one with active default route). They are simply mixed together and since GRE is stateless, it doesn't matter. All GRE interfaces have "R" as running, but only because there are no actual checks, it's just optimistic assumption. The result is routing like this:
Failover for traffic between remote subnets works, if you have routes like this on both routers:
But if you either set local-address or enable keepalive, secondary tunnel fails. Only way I can see how you can have local-adress set, keepalive enabled (or both) and the secondary tunnel working is to make sure that it uses secondary WAN as outgoing interface. Correct routing is the key. But it won't happen automatically, you need to help it. So how it could have worked for you before, if you didn't, it's mystery for me.
Code: Select all
+---------+ +---------+
| | WAN1 gre1 -->>------<<-- gre1 WAN1 | |
| | gre2 -->>-\ /-<<-- gre2 | |
| Router1 | \/ | Router2 |
| | /\ | |
| | WAN2 --<<------/ \------>>-- WAN2 | |
+---------+ +---------+Code: Select all
/ip route
add dst-address=<remote subnet> gateway=<ip on the other end of tunnel 1> check-gateway=ping
add dst-address=<remote subnet> gateway=<ip on the other end of tunnel 2> check-gateway=pingRe: Only work 1 of 2 GRE VPNs
Hi,
Sorry about the delay.
I see the last week that those tunnels were all up, and those are still up, without changing anything
Now I have the 8 tunnels working, using a specific interface for each tunnel, using the routes of Sindy, using local and remote address in the interface properties and keeping the default keepalive, and using the pref-source in the routes as Sindy sugested
Thanks to both for your help, which is very useful.
Regards
Sorry about the delay.
I see the last week that those tunnels were all up, and those are still up, without changing anything
Now I have the 8 tunnels working, using a specific interface for each tunnel, using the routes of Sindy, using local and remote address in the interface properties and keeping the default keepalive, and using the pref-source in the routes as Sindy sugested
Thanks to both for your help, which is very useful.
Regards
Re: Only work 1 of 2 GRE VPNs
So today I've come across that manual page again, and it turns out I've remembered what it said incorrectly or misinterpreted it when reading it for the first time - it's not that the source address would not be set to local-address but it is really only the routing issue, i.e. that the WAN is not chosen based on the local-address automatically.I've even come across a manual page which gave this explanation (which I'm obviously unable to google up right now).
Re: Only work 1 of 2 GRE VPNs
Hello Forum. sorry for re-open this post,
1 of all VPNs stopped working and I cannot find why
I left 8 VPNs working:
S of Site - W of WAN:
S1W1 - S2W1
S1W2 - S2W2
S2W1 - S3W1
S2W2 - S3W2
S3W1 - S4W1
S3W2 - S4W2
S4W1 - S1W1
S4W2 - S1W2
Previously I had VPNs among 3 sites, working only 3 of 6, the configuration was the basics, just set local and remote address in the interface, in both sides, a different local address in each side of the VPN but in the same network, and a route to Access to the other network throught the VPN, there were not marks configured and no route added to force one VPN to go out thought an specific WAN.
The solution was create a route to force each VPN to go out through the designated WAN (As Sindy sugested)
Now only 1 of the 8 VPNs is not working, in "interfaces" I can see it with the R of running, just in one side, in the other site it appears as not running
Now there is the following configuration, the same but inverted in the other side:
I have now in 1 of the sites mangle marks for other thinks, I just mark all inconming connections to go out through the same interface where it came up, and forced an specific network to go out throught WAN2
How can I check why this is not running?
Sob told me that he had less issues when the local address is not set and there is not any route added to force the VPN to go out throught the designated WAN, but the only difference with my previous settings (when just worked a half of the VPNs) is that I had the "Local address" in the GRE interface. Could remove the local address make it to work?
Any sugestion?
Thanks in advance.
Regards,
Damián
1 of all VPNs stopped working and I cannot find why
I left 8 VPNs working:
S of Site - W of WAN:
S1W1 - S2W1
S1W2 - S2W2
S2W1 - S3W1
S2W2 - S3W2
S3W1 - S4W1
S3W2 - S4W2
S4W1 - S1W1
S4W2 - S1W2
Previously I had VPNs among 3 sites, working only 3 of 6, the configuration was the basics, just set local and remote address in the interface, in both sides, a different local address in each side of the VPN but in the same network, and a route to Access to the other network throught the VPN, there were not marks configured and no route added to force one VPN to go out thought an specific WAN.
The solution was create a route to force each VPN to go out through the designated WAN (As Sindy sugested)
Now only 1 of the 8 VPNs is not working, in "interfaces" I can see it with the R of running, just in one side, in the other site it appears as not running
Now there is the following configuration, the same but inverted in the other side:
Code: Select all
/interface gre
add disabled=yes local-address=S1W2Address name=S1W2-S4W2 remote-address=S4W2Address
/ip address
add address=1.1.1.1/29 interface="S1W2-S4W2" network=1.1.1.0 (I changed the IP)
/ip route
add check-gateway=ping comment="S1W2-S4W2 goes out through Wan2" distance=1 dst-address=S4W2Address gateway=S1W2Gateway pref-src=S1W2Address
add check-gateway=ping comment="Site 4 network" distance=20 dst-address=10.10.10.0/24 gateway=1.1.1.2How can I check why this is not running?
Sob told me that he had less issues when the local address is not set and there is not any route added to force the VPN to go out throught the designated WAN, but the only difference with my previous settings (when just worked a half of the VPNs) is that I had the "Local address" in the GRE interface. Could remove the local address make it to work?
Any sugestion?
Thanks in advance.
Regards,
Damián
Re: Only work 1 of 2 GRE VPNs
Show me /ip firewall connection print detail where protocol=gre from both ends while it does not work (obfuscate public IP addresses in a consistent way so that the relationship between the two ends remains visible).
Re: Only work 1 of 2 GRE VPNs
Site 1:
Site 4:
It is consistent, I used "Replace all" option in Notepad
Thanks in advance.
Regards
Code: Select all
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat
0 S C protocol=gre src-address=S1W1Address dst-address=S3W1Address reply-src-address=S3W1Address reply-dst-address=S1W1Address gre-key=0 timeout=9m56s connection-mark="Wan1_conn" orig-packets=877 690 427 orig-bytes=1 001 257 292 449
orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=2 229 962 repl-bytes=185 661 182 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=0bps repl-rate=0bps
1 S C protocol=gre src-address=S1W2Address dst-address=S4W2Address reply-src-address=S4W2Address reply-dst-address=S1W2Address gre-key=0 timeout=9m58s connection-mark="Wan2_conn" orig-packets=649 999 orig-bytes=78 577 614
orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=2 058 repl-bytes=135 970 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=384bps repl-rate=0bps
2 S C protocol=gre src-address=S1W2Address dst-address=S2W2Address reply-src-address=S2W2Address reply-dst-address=S1W2Address gre-key=0 timeout=9m59s connection-mark="Wan2_conn" orig-packets=1 020 222 441 orig-bytes=1 259 894 333 183
orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=2 066 857 repl-bytes=337 386 397 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=4.3Mbps repl-rate=0bps
3 S C protocol=gre src-address=S1W1Address dst-address=S4W1Address reply-src-address=S4W1Address reply-dst-address=S1W1Address gre-key=0 timeout=9m59s connection-mark="Wan1_conn" orig-packets=248 474 961 orig-bytes=109 747 382 886
orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=2 017 335 repl-bytes=805 042 103 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=411.5kbps repl-rate=0bps
4 S C protocol=gre src-address=S1W2Address dst-address=S3W2Address reply-src-address=S3W2Address reply-dst-address=S1W2Address gre-key=0 timeout=9m59s connection-mark="Wan2_conn" orig-packets=206 618 062 orig-bytes=249 947 475 728
orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=207 788 repl-bytes=17 175 583 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=5.7Mbps repl-rate=0bps Site 4:
Code: Select all
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat
0 C s protocol=gre src-address=S4W2Address dst-address=S1W2Address reply-src-address=S1W2Address reply-dst-address=S4W1Address gre-key=0 timeout=9m52s orig-packets=191 344 orig-bytes=10 770 687 orig-fasttrack-packets=0
orig-fasttrack-bytes=0 repl-packets=0 repl-bytes=0 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=0bps repl-rate=0bps
1 S C protocol=gre src-address=S4W1Address dst-address=S2W1Address reply-src-address=S2W1Address reply-dst-address=S4W1Address gre-key=0 timeout=9m52s orig-packets=148 083 orig-bytes=11 880 582 orig-fasttrack-packets=0
orig-fasttrack-bytes=0 repl-packets=3 137 repl-bytes=276 218 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=0bps repl-rate=0bps
2 C s protocol=gre src-address=S4W2Address dst-address=S2W2Address reply-src-address=S2W2Address reply-dst-address=S4W1Address gre-key=0 timeout=9m57s orig-packets=191 725 orig-bytes=10 770 044 orig-fasttrack-packets=0
orig-fasttrack-bytes=0 repl-packets=0 repl-bytes=0 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=0bps repl-rate=0bps
3 S C protocol=gre src-address=S4W1Address dst-address=S1W1Address reply-src-address=S1W1Address reply-dst-address=S4W1Address gre-key=0 timeout=9m59s orig-packets=87 494 788 orig-bytes=78 139 897 288 orig-fasttrack-packets=0
orig-fasttrack-bytes=0 repl-packets=987 024 repl-bytes=310 273 340 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=2.5Mbps repl-rate=0bps Thanks in advance.
Regards
Re: Only work 1 of 2 GRE VPNs
I just see, all ítems on Site 4 has as reply-dst-address: "S4W1Address", even the ítems which have "S4W2Address" as src-address
This means that everything is going out through WAN1, right?
Should I need to use mangle?
Thanks in advance.
Regards
Damián
This means that everything is going out through WAN1, right?
Should I need to use mangle?
Thanks in advance.
Regards
Damián
Re: Only work 1 of 2 GRE VPNs
An action=src-nat (or action=masquerade) rule on Site 4 is too wide so it src-nats packets from S4W2Address to S4W1Address as you've found yourself. So check your chain=srcnat at Site 4. It only happens if Site 4 sends the first GRE packet first; if Site1 is faster in sending the first GRE packet, chain=srcnat at Site4 is not used for the connection. Which explains why it sometimes works and sometimes not.
Re: Only work 1 of 2 GRE VPNs
What may actually happen is that when WAN2 on Site 4 goes down, the individual route with dst-address=S1W2Address becomes inactive, so the packets for S1W2Address take the default route via WAN1, and a masquerade rule says "src-nat whatever has out-interface=WAN1 to its IP address". So to prevent this from happening, you need to add a type=blackhole route towardsS1W2Address, with higher value of the distance parameter than the real route, which will make sure that if the WAN2 on Site 4 is down, the packets for S1W2Address which should have left through WAN2 won't leave at all.
Re: Only work 1 of 2 GRE VPNs
Thanks Sindy,
I never pay atention to the route type, allways created unicast routes, should read about it, I will
About the masquerade, there is just one rule for both WAN interfaces:
Do you think that when S4W2 fail and come back the masquerade rule keep changing src-address with S4W1Address?
Creating one diferent masquerade rule for each interface should work better?
Thanks again
Regards,
Damián
I never pay atention to the route type, allways created unicast routes, should read about it, I will
About the masquerade, there is just one rule for both WAN interfaces:
Code: Select all
/ip firewall nat
add action=masquerade chain=srcnat comment="Masquerade General" out-interface-list=WANCreating one diferent masquerade rule for each interface should work better?
Thanks again
Regards,
Damián
Re: Only work 1 of 2 GRE VPNs
Any NAT rules (in dstnat or scrnat chain) are only consulted for the initial (first ever) packet of a connection, and the resulting treatment is noted down into the context of that connection to be applied on subsequent packets belonging to the same connection, with respect to their direction. What is equally important, connection context doesn't store the initerface(s) used, it only works at L3 and above.
So setting a NAT rule for WAN2 wouldn't help because a src-nat rule which has out-interface=WAN2 among its conditions cannot match if the packet actually doesn't go via WAN2.
So the only way to prevent connections which should go via WAN2 from actually establishing via WAN1 and inheriting its address (which then remains assigned to the connection even after WAN2 becomes available again and the subsequent packets of the same connection start using it) is to do what I wrote (or to use routing-mark and /ip route rule action=lookup-only-in-table if you prefer).
So now you have a route with dst-address=S1W2Address/32 pref-src=S4W2Address distance=1 gateway=S4W2.gateway.ip
And you need to do the following:
/ip route add dst-address=S1W2Address/32 type=blackhole distance=2
The background is that routing always chooses only among routes most exactly matching the destination, but only takes active routes into account. Most exactly means with the longest prefix (=mask size). So once WAN2 goes down, all routes which use it become inactive; if no other route with dst-address exactly matching S1W2Address exists, the route search can be done among less precisely matching ones. Which, in your particular case, means that the default route via WAN1 is chosen. So by adding the blackhole route, which doesn't depend on state of any interface, you prevent the above from happening. And the higher value of distance on the blackhole route ensures that the blackhole route is shadowed by the "real" one whenever the latter is active.
So setting a NAT rule for WAN2 wouldn't help because a src-nat rule which has out-interface=WAN2 among its conditions cannot match if the packet actually doesn't go via WAN2.
So the only way to prevent connections which should go via WAN2 from actually establishing via WAN1 and inheriting its address (which then remains assigned to the connection even after WAN2 becomes available again and the subsequent packets of the same connection start using it) is to do what I wrote (or to use routing-mark and /ip route rule action=lookup-only-in-table if you prefer).
So now you have a route with dst-address=S1W2Address/32 pref-src=S4W2Address distance=1 gateway=S4W2.gateway.ip
And you need to do the following:
/ip route add dst-address=S1W2Address/32 type=blackhole distance=2
The background is that routing always chooses only among routes most exactly matching the destination, but only takes active routes into account. Most exactly means with the longest prefix (=mask size). So once WAN2 goes down, all routes which use it become inactive; if no other route with dst-address exactly matching S1W2Address exists, the route search can be done among less precisely matching ones. Which, in your particular case, means that the default route via WAN1 is chosen. So by adding the blackhole route, which doesn't depend on state of any interface, you prevent the above from happening. And the higher value of distance on the blackhole route ensures that the blackhole route is shadowed by the "real" one whenever the latter is active.
Re: Only work 1 of 2 GRE VPNs
Sindy, sorry because I did not explain,
I know all what you said in your last post and I did undestand why is the blackhole route
I just tried to avoid to create the blackhole route, just in case, someone in Site4 maybe is using S1W2Address without the VPN (Accessing directly through a dst-nat on Site1)
I disabled the GRE interface in both sides, re enable it, and watch the connection: Allways the same result
If I restart the tunnel when the WAN2 is working, I think this should use the route I added to S1W2. Am I wrong?
Thanks
Regards
I know all what you said in your last post and I did undestand why is the blackhole route
I just tried to avoid to create the blackhole route, just in case, someone in Site4 maybe is using S1W2Address without the VPN (Accessing directly through a dst-nat on Site1)
I disabled the GRE interface in both sides, re enable it, and watch the connection: Allways the same result
If I restart the tunnel when the WAN2 is working, I think this should use the route I added to S1W2. Am I wrong?
Thanks
Regards
Re: Only work 1 of 2 GRE VPNs
In that case, useI just tried to avoid to create the blackhole route, just in case, someone in Site4 maybe is using S1W2Address without the VPN (Accessing directly through a dst-nat on Site1)
/ip route rule
add src-address=S4W2Address/32 dst-address=S1W2Address/32 action=lookup-only-in-table table=gre-via-w2
/ip route set [find dst-address~"S1W2Address"] routing-mark=gre-via-w2
This will make sure that the individual route to S1W2Address will be used only for packets sent by the Mikrotik itself from the address of S4W2 before the NAT can eventually take place.
If you would like to further restrict it to GRE packets only, you would have to assign the routing-mark using /ip firewall mangle rules, as /ip route rule cannot match on protocol.
The firewall's connection tracker remembers other than TCP and ICMP connections for 3 minutes by default. So after disabling the gre interfaces at both sides, re-enable them only after 5 minutes "cooldown". Or instead you can manually remove the connections at both ends using /ip firewall connection remove [find protocol=gre srcnat] and re-enable the gre interfaces immediately.I disabled the GRE interface in both sides, re enable it, and watch the connection: Allways the same result
If I restart the tunnel when the WAN2 is working, I think this should use the route I added to S1W2. Am I wrong?
Re: Only work 1 of 2 GRE VPNs
Sindy, like allways, you saved me
I forgot about the route rules.
I will create one route rule for each VPN in each site
So far, after disabling the GRE interface for more than 5 minutes and re-enable it, worked fine.
Thanks a lot for all your help
Regards
Damián
I forgot about the route rules.
I will create one route rule for each VPN in each site
So far, after disabling the GRE interface for more than 5 minutes and re-enable it, worked fine.
Thanks a lot for all your help
Regards
Damián
Re: Only work 1 of 2 GRE VPNs
Sorry Sindy,
I am not sure if it could solve the issue.
As I think (not sure), the first command mark a route and the second modify the preiously created route to use only marked routes.
If the WAN2 link goes down again, which is the difference with the previous settings?
This route with dst-address=S1W2Address will be inactive too, then, because there is an only route in this routing table, will use the default routing table (With no marks)
When the WAN2 link return up, the S1W2-S4W2 GRE will use again the route with marks just because it has a mark?
Thanks again
Regards
I am not sure if it could solve the issue.
As I think (not sure), the first command mark a route and the second modify the preiously created route to use only marked routes.
If the WAN2 link goes down again, which is the difference with the previous settings?
This route with dst-address=S1W2Address will be inactive too, then, because there is an only route in this routing table, will use the default routing table (With no marks)
When the WAN2 link return up, the S1W2-S4W2 GRE will use again the route with marks just because it has a mark?
Thanks again
Regards
Re: Only work 1 of 2 GRE VPNs
The handling of routing-mark is a bit more complex. Normally yes - if no route with a routing-mark can be found, the packet is routed using the main routing table. However, if the action of /ip route rule says lookup-only-in-table, this default behaviour is overridden, and if no marked route is available, the packet is dropped. So btw the route marked with gre-via-w2 may actually be a default one for that routing-mark (i.e. dst-address=0.0.0.0/0 or none at all as 0.0.0.0/0 is the default value if dst-address is not provided).
Re: Only work 1 of 2 GRE VPNs
Ahh, ok, I got it,
I need to think how to handle it with the routing marks in one site, but you don't care about it
Thanks a lot, you are a genius
Regards,
Damián
I need to think how to handle it with the routing marks in one site, but you don't care about it
Thanks a lot, you are a genius
Regards,
Damián
Re: Only work 1 of 2 GRE VPNs
Thank you sindy, this small paragraph gave me the needed idea to solve a problem that bothered me for quite a while, where GRE/IPSEC would re-establish just fine after router reboot, but would not pass traffic traffic properly. As it turned out -- precisely for the reason you mentioned -- first traffic from impatient clients hits GRE tunnel interface before IPSEC policy is activated, gets routed through the default WAN route (essentially blackholing it, because tunnel IPs are dummies), and src-nat claims that connection and never releaes it (since there's always some activity that keeps it from expiring).What may actually happen is that when WAN2 on Site 4 goes down, the individual route with dst-address=S1W2Address becomes inactive, so the packets for S1W2Address take the default route via WAN1, and a masquerade rule says "src-nat whatever has out-interface=WAN1 to its IP address". So to prevent this from happening, you need to add a type=blackhole route towardsS1W2Address, with higher value of the distance parameter than the real route, which will make sure that if the WAN2 on Site 4 is down, the packets for S1W2Address which should have left through WAN2 won't leave at all.
Only I couldn't use "blackhole route" solution, and instead just told src-nat to stay away from tunnel dummy IPs, and told Filter to drop anything that goes to dummy tunnel IP but is not IPSEC:
Code: Select all
/ip firewall chain=srcnat action=masquerade dst-address-list=!IPSEC-DUMMY-DST out-interface=bridge-uplink ipsec-policy=out,none
/ip firewall chain=output action=drop dst-address-list=IPSEC-DUMMY-DST out-interface=bridge-uplink ipsec-policy=out,none