I want your kind help, this issue drives me crazy.
I’m sure I’m missing something very basic and elemental, but I could not find what it is.
I have the following topology, one router is on my office side, and the other is at my home:
I followed the example setup from help.mikrotik.com for “Site to Site GRE tunnel over IPsec (IKEv2) using DNS”. The difference is only I’m using a self-signed certificate for the authentication between the two peers.
The office side has a fixed IP address, the home side has a dynamic IP therefore I set up a dynamic DNS service for that station.
Office details are:
Local address: 10.77.0.0/24
IPsec tunnel (bridge): 172.16.10.1
GRE tunnel: 172.16.100.1
Home details are:
Local address: 10.1.1.192/28
IPsec tunnel (bridge): 172.16.10.2
GRE tunnel: 172.16.100.2
The IPSec tunnel is working well, the connection is established, and the policy is also established between 172.16.10.1 and 172.16.10.2.
I can ping the IP from the router (but I cannot ping the other side’s IP from one of the clients behind the router).
I created the GRE tunnel, it is established and working well - at least looking well. I created the static routes on the routers.
The result is that I can ping everything from the router itself - I can reach any client from the other side -, and I found the source address is always the local GRE tunnel. Once I change the source address to the router’s local address, for example in my home I try to ping from 10.1.1.193 (which is the IP of the router) the ping is lost, and there is a timeout. This is happening from any clients behind the router, I cannot reach any stations on the other side, except the other side’s GRE interface’s IP address. I can ping that.
So, I can ping 172.16.100.1 from home, and I can ping 172.16.100.2 from the office, but nothing more, but I can ping all addresses from routers where the source is always the GRE interface.
I set in the NAT table for masquerade that IPSec Policy: out and none to not masquerade the packet which must go over IPSec tunnel.
What do you think, about why I cannot ping the clients between the two routers?
Much appreciated for your ideas.
Additional information, it seems, if the source address is not the local GRE interface, but one of the LAN IPs (for example 10.1.1.193 at home), I can see the ICMP packet on the firewall’s out chain (logged) in the direction of the other router’s network - 10.77.0.160, but once I created a pcap (with sniffer) and checked on WireShark, I do not see any packet where the destination is 10.77.0.160.
If anyone can point me in the right direction, it would be much appreciated.
In order that ping from IP address A to IP address B could succeed, all devices in the path from A to B (includung device with IP address A itself) must have a correct route to B, and all devices in the path from B to A must have a correct route to A. So the first question is whether the Mikrotik in your office is the main router for all devices there, or whether there is another router and the Mikrotik is there only to provide the VPN tunnel.
Thank you very much for your reply.
The Mikrotik routers are the main routers both in the office and both in my home.
These are the only routers and these are providing the VPN tunnel.
Dear rplant,
Thank you very much for your reply too.
Yes, this is what I’ve done, and I tried your solution as well, where I pointed out the gre interface directly as the gateway.
The symptoms are the same.
If I use /ping 10.77.0.1 from the home router, the ping is working, I can check in the Wireshark, the source address is my GRE tunnel in this case, which is 172.16.100.2. I can ping any computer from the other side with this method, it is replying.
But once, I use /ping src-address=10.1.1.193 10.77.0.1 (where 10.1.1.193 is my LAN IP address for the router) the ping is not working - it runs to timeout. Even, I cannot see in Wireshark that the pack left the router, there is no any record where the destination is 10.77.0.1 .
Interesting
Thank you very much for all your help and ideas!
Gergely
You have only posted the parts of configurations you assume to be related to your issue, but the issue is typically caused by a part of configuration you wouldn’t even dream of being related.
What were the exact parameters of the sniffer filter when you were pinging from 10.1.1.193?
In the capture, there are 20 ping attempts in the beginning from 10.1.1.193 (local) to 10.77.0.1 (external):
ping src-address=10.1.1.193 10.77.0.1
→ I could not see any packet in the capture file, where the destination is 10.77.0.1 for this, even though there is no shown the ICMP. It has just disappeared, or the pack has not tried to leave the router. I don’t know, but this is not in the file. There was a “timeout” on the Mikrotik terminal.
In the end, there were 20 ping attempts from 10.1.1.193 to 10.77.0.1, where the source address was the GRE interface:
ping 10.77.0.1
→ I can see the packs, there are 21 packets at the end of the day, and all have a correct reply. The ping was succeeded from the Mikrotik terminal.
Really appreciate your help and ideas, I’m so grateful!
I cannot see anything in your configuration export that would explain the behaviour you encounter.
You assign a connection-mark (ipsec) to packets that go from 10.1.1.192/28 to 10.77.0.0/24, but you do not match on any connection-mark anywhere so that cannot be the explanation. But it will cause no harm if you disable this mangle rule.
There is no statically configured IPsec policy to steal the traffic with this source and destination, and the ipsec policy template does not allow such policy to get created dynamically.
There are also no rules at all in filter chain output.
Just a fun fact that won’t help you much - some months or even years ago another user from Hungary had a similar issue, on a much older version of RouterOS.
So unless you have some secret info you don’t want to share with anyone, I’d file a support ticket with Mikrotik about this behaviour, which requires to send them a supout.rif file.
You can try to replace the GRE tunnel by an IPIP one, but I only suggest this because I don’t like GRE tunnels as I had other kinds of issues with them, however these were related to handling of their transport packets, not the payload ones.
You need to read again what rplant did and understand it.
You can not “IP route” thru the GRE addresses they do not form a proper network they are just tunnel ends even if they have what looks like a network between them. You will find you can change to /32 non connected network IPs and they will work exactly the same. This is your confusion and its probably safer to write them as /32 so you aren’t tempted to believe they form a network.
The only thing you can ping from a gre tunnel IP is the other tunnel end or something your end.
To IP ROUTE thru the tunnel you need to put a proper network each end ON THE GRE INTERRFACE which is what rplant forming a proper /30 network.
Now you can IP ROUTE thru that network so going back to rdplant setup
HOME END
/ip address
add address=192.168.40.2/24 interface=gre-tunnel1
As Sindy said it you are tik to tik it’s much easier to use an IP-Tunnel just know that the two phases for ipsec on these are set by
/ipsec/profiles/default (phase 1)
/ipsec/proposals/default (phase 2)
NO. Just no. This is misleading (and has nothing to do with @kissge83’s issue).
Any L3 tunnel interface can be used as a gateway of a route even if no IP address is attached to it. Point-to-point interfaces do not need any MAC address because there is only one possible destination - “the other end”. So the local router sends a packet for x.x.x.x to the tunnel, the remote router receives that packet and eventually routes it further using its own routing tables.
If you do attach an IP address other than a /32 one to such a tunnel interface, the system dynamically adds exactly that, a route with that subnet as dst-address and that interface as a gateway:
[me@myTik] > ip address print where interface=gre-test
Flags: X - disabled, I - invalid, D - dynamic
# ADDRESS NETWORK INTERFACE
0 192.168.148.1/24 192.168.148.0 gre-test
[me@myTik] > ip route print where 192.168.148.3 in dst-address dynamic
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 DS 0.0.0.0/0 192.0.2.193 5
1 ADC 192.168.148.0/24 192.168.148.1 gre-test 0
If you add another route that uses any address from the subnet attached to an L3 interface as a gateway, the system will send packets whose destination addresses match that route’s dst-address to that tunnel as well.
You can ping anything from any own IP address of the router, it doesn’t matter to which interface that address is attached in particular - of course you will only receive responses if that address is routable from the outside.
Recommendation: don’t trust my words, try it on your own.
Dear Sindy,
Thank you so much for your effort to help me and for your kind reply.
Yeah, I do not like the GRE from now on , so I will not use it…
To be honest, I spent so much time investigating this and I decided to stop because I have the latest V7.12 on both of my routers, I decided to try WireGuard instead of IPSec (and GRE).
I successfully set up everything with it, and it is working like a charm, with no issues.
I only tried to install IPSec, because I would like to use a certificate instead of the pre-shared public key (I know it is safe enough), but I much like certificates more as authentication.
So, finally, I’m using WireGuard now and let go of IPSec and GRE.
Again, thank you so much for everything!
It is interesting that you mentioned somebody had the same problem in Hungary, later - once I have more time -, I will put this setup into my EVE-NG lab and test the settings there, I’m curious, about what I will experience there.
You are one of few Certificates are essential for proxied trust (i.e. where you have to rely on a 3rd party to guarantee authenticity of your peer because it is impossible to exchange keys with the peer before entering into contact, or in full mesh VPNs where maintaining of a full mesh of keys would be too complicated), but for simple cases like this one, the two key pairs are simpler (and they also never expire).
With these strange issues, even the CPU architecture may play a role. So the issue may not exist in the lab environment because it is x64 whereas the real boxes are not.
@Sindy
You are usually correct but in this I can assure you on OS7.12 you can’t IP route thru the GRE tunnel IP’s. It is definitely not how a CISCO does it but having played with these since upgrading a pile of tiks to OS7 it seems to be a fact at least on ARM64 and MPSIBE architecture.
If you WireShark the transfers what happens is
The phase 1 (which mikrotik calls the profile) connects via the Public or Outside IPs
The phase 2 (which mikrotik calls the proposal) use those GRE IP to establish the tunnel.
After that I can assure you no packet ever leaves that end of the GRE tunnel via those IP’s if you try to use them they only seem to be used by the tunnel itself for phase 2 signalling.
You are suggesting that behaviour is wrong in which case it is then a bug but that is how it is on OS7.2
The fix is what rplant did and put an IP or the gre interface itself and the routing then seems to work.
Maybe there is some misunderstanding? Without IPsec, it works in ROS 7.12 as expected, see below.
Test setup: a /24 subnet is attached to the GRE tunnel interface on the DUT:
_[me@myTik] > /ip/address/print where interface=gre-test
Columns: ADDRESS, NETWORK, INTERFACE
ADDRESS NETWORK INTERFACE
3 192.168.129.21/24 192.168.129.0 gre-test_
On the device on the other end of the GRE tunel, 192.168.221.1/32 is attached to another interface (br-lo).
ether1 of both devices is in the same subnet, DUT has 192.168.227.21/24 on it, and the other device has 192.168.227.22/24. So when pinging 192.168.221.1 via the GRE tunnel with source address forced to 192.168.227.21, we know that the backward route to 192.168.227.21 exists on the remote device. There is no src-nat, when pinging with src-address=192.168.227.21, you can see only the requests to arrive via the GRE tunnel when sniffing on the remote device, the responses take the path via ether1.
Test 1: there is no route to 192.168.221.0/24 on the DUT: [me@myTik] > /ip/route/print where dst-address=192.168.221.0/24
— empty line here —
[me@myTik] > ping 192.168.221.1 count=3
SEQ HOST SIZE TTL TIME STATUS
0 192.168.221.1 timeout
1 192.168.221.1 timeout
2 192.168.221.1 timeout
sent=3 received=0 packet-loss=100%
Test 2: a route to 192.168.221.0/24 has been added via another address in the subnet attached to gre-test on the DUT:
_[me@myTik] > /ip/route/add dst-address=192.168.221.0/24 gateway=192.168.129.22
[me@myTik] > /ip route/print where dst-address=192.168.221.0/24
Flags: A - ACTIVE; s - STATIC
Columns: DST-ADDRESS, GATEWAY, DISTANCE
Test 3: the gateway of the route to 192.168.221.0/24 has been changed from an IP address to the interface on the DUT:
_[me@myTik] > /ip/route/set [find where dst-address=192.168.221.0/24] gateway=gre-test
[me@myTik] > ip route/print where dst-address=192.168.221.0/24
Flags: A - ACTIVE; s - STATIC
Columns: DST-ADDRESS, GATEWAY, DISTANCE
Conclusion: on a CHR running ROS 7.12, on a GRE tunnel without IPsec, both the interface name and an IP address from the subnet attached to the interface can be used as a gateway of a route, just like it always was.
This was all I said. Did you understand something I said in another way?
If the tests above give different results on ARM64 and/or MIPSBE, it is indeed a bug that needs to be reported and fixed.
I don’t understand which addresses you have exactly in mind here when saying “GRE IP”.
Terminologically, Mikrotik stores the parameters of the proposal used for Phase 1 in a configuration item named profile, and the parameters of the proposal used for Phase 2 in a configuration item named proposal, but that’s not important for the setup.
For a gre tunnel, you can specify a particular local-address for the transport packets to be sent from, or you can let the routing choose it automatically depending on the route chosen up to the destination address; the remote-address must be always specified of course. If the local-address is specified but does not match an own address of the router which is up, the router will simply not send the transport packets.
The Phase 1 SA uses the address of the configuration item peer as a destination; as source, it uses the local address specified in local-address if specified and is up on the router, or the source address chosen by routing if local-address is set to the default 0.0.0.0.
For a Phase 2 SA, the router uses the same addresses like Phase 1. In transport mode, the sa-src address of the SA and the src-address of the traffic selector are the same (and the same applies for sa-dst-address and sa-dst-address); in tunnel mode, the traffic selector addresses and the SA addresses may be different.
So if you use the WAN address as both the local-address of the peer and the local-address of the GRE tunnel, or leave both of them on default (0.0.0.0), and the address of the peer is the same like the remote-address of the GRE tunnel, you can use a transport mode of the policy. This is what RouterOS does automatically if you set the ipsec-secret in the GRE tunnel settings.
If you want to use a different local and/or remote address for the GRE transport packets than for the peer, the Phase 2 SA must be in tunnel mode, so you have to configure IPsec manually and you have to set the traffic selector of the policy to include the local and remote address of the GRE transport packets in the src-address and dst-address prefixes,respectively.
What is definitely NOT a good idea is to set the IP address assigned to the GRE tunnel interface as a local-address for the transport packets of that same GRE tunnel; is that what you had in mind? But this would not work even in ROS 6, because the address assigned to the tunnel interface is not up until the tunnel interface goes up, and the tunnel cannot go up until the address it uses as source for transport packets is up, so we have a deadlock here (unless you disable keepalive, but even that way it is still a bad idea).
Can you please explain again what part you claim to behave different than this in 7.12?
Once I change the source address to the router’s local address, for example in my home I try to ping from 10.1.1.193 (which is the IP of the router) the ping is lost, and there is a timeout. This is happening from any clients behind the router, I cannot reach any stations on the other side, except the other side’s GRE interface’s IP address. I can ping that.
You are stating and seeing exactly what I am seeing as well no packets exit the far end of the tunnel from the tunnel IP which is why you can’t ping pr route thru it. The only way I can fix the problem by simply putting a /30 on each end of the GRE interfaces and I can route and ping thru those and everything works as expected.
I am on CCR2004 and CCR2116 routers on OS 7.12 and 7.2 and that is how it behaves and so I just accepted it having nothing to compare with.
I guess from what Sindy said it’s a bug but yes I agree it’s weird on not how a gre tunnel works on a cisco.
As per what Sindy said it’s much easier with an IPIP tunnel and the only weird thing with that I found is all IPIP and EOIP tunnels seem to use the default profile and proposal so you can’t have multiple tunnels with different settings.
But that’s something else that what @kissge83 has described in the OP. You can see that the packets do not exit the far end which may have multiple reasons, but @kissge83 could not see them enter the local GRE interface on the sending side, and the behavior depended on choice of local source address of the payload packets, not of the transport ones (those encapsulating the payload ones).
If it works with /30, it must work the same with a /24 or even /8, of course taking eventual other routes into consideration.
Sindy this is what he says
[quote = @kissge83]
In the capture, there are 20 ping attempts in the beginning from 10.1.1.193 (local) to 10.77.0.1 (external):
ping src-address=10.1.1.193 10.77.0.1
→ I could not see any packet in the capture file, where the destination is 10.77.0.1 for this, even though there is no shown the ICMP. It has just disappeared, or the pack has not tried to leave the router. I don’t know, but this is not in the file. There was a “timeout” on the Mikrotik terminal.[/quote]
AFAIK he is getting the same result as me and he states no packet ever leaves the far end of the tunnel (he calls it external).
The funny part is if you look or put a torch on the GRE tunnel the packet goes thru the tunnel and arrives at the far router GRE interface and then just disappears it is like it has a drop firewall rule on it.
Yes it works with any valid /network on the GRE interfaces and that is what makes it doubly weird.
I have to say it behaves very much like IPIP tunnel just that is has those two tunnel IPs which you can’t actually use only the tunnel comms seem to be able to use them
There are multiple posts where @kissge83 describes it and I read all of them the same, that the packet does not leave the local router, i.e. the packet is not sent via the local GRE interface. Of course that also means that it never reaches the remote router, but while there are many reasons why a packet may not reach the remote router, there are not so many reasons why a packet should get lost between the output chain of the local router and the interface - namely, an IPsec policy can redirect or drop it or a bug can drop it.
As for the difference between IPIP and GRE, I’m not sure again what you are saying. For me, these two tunnels behave absolutely the same, except that the GRE one has a smaller payload MTU on the same transport MTU because the overhead of GRE protocol is longer, as it has space to indicate payload type and you can optionally specify a tunnel ID (which Mikrotik doesn’t use when tunneling IP packets using the “gre” tunnel, and uses in a broken way when tunneling Ethernet frames using the “eoip” tunnel, so firewalls cannot match the two directions of EoIP connections). The concept of local and remote address used to send the transport packets and some other addresses to be assigned to the tunnel interfaces is identical for both.
, so I will not use it…