Community discussions

MikroTik App
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

VPN with GCP

Tue Jun 02, 2020 8:47 pm

I gathered many information in huge GCP documentation about setting IPsec using IKEv2 with BGP.

What I found is that , and this is important
Important: When using IKEv2, your peer VPN gateway must accept all of the CIDRs in each traffic selector using a single Child SA. Not all VPN gateways support this. VPN gateways that create a unique Child SA per CIDR are not compatible with Cloud VPN. See traffic selector strategies for additional details.
https://cloud.google.com/vpn/docs/conce ... -ip-ranges

and also I was struggling with a issue which occurs no ping to remote networks. More about that was described here
viewtopic.php?f=15&t=147798&p=763200#p763200

I was speaking with GCP support for past few days and they notice I receive constantly
{
 insertId:  "1hkfx2ag100glgy"   
 labels: {…}   
 logName:  "projects/casino-front/logs/cloud.googleapis.com%2Fipsec_events"   
 receiveTimestamp:  "2020-06-02T16:56:32.894395202Z"   
 resource: {…}   
 severity:  "NOTICE"   
 textPayload:  "Warning: Remote traffic selectors narrowed for Child SA: vpn_94.237.xx.xx. Configured TS: [0.0.0.0/0  ], negotiated TS:[172.16.18.0/24  ]. Please verify configuration on the remote side."   
 timestamp:  "2020-06-02T16:56:32.831941608Z"   
}
So problem is that first of all I can't use `level-unique` because GCP requires single Child for SA negotiation. So I changed it to `use` but also it is required to set 0.0.0.0 src and dst in IPsec policy. When I do that I loose connectivity
Could someone advise me how to proceed ?
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Tue Jun 02, 2020 9:25 pm

So I changed it to `use` but also it is required to set 0.0.0.0 src and dst in IPsec policy. When I do that I loose connectivity
Could someone advise me how to proceed ?
I'd prefer require to use for level, but that's minor. To prevent losing connectivity by setting policy's src and dst to 0.0.0.0/0, place policies with action=none and dst-address which needs to stay accessible directly (not via IPsec) before (above) the one used for GCP.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Tue Jun 02, 2020 9:56 pm

So I changed it to `use` but also it is required to set 0.0.0.0 src and dst in IPsec policy. When I do that I loose connectivity
Could someone advise me how to proceed ?
I'd prefer require to use for level, but that's minor. To prevent losing connectivity by setting policy's src and dst to 0.0.0.0/0, place policies with action=none and dst-address which needs to stay accessible directly (not via IPsec) before (above) the one used for GCP.
require - drop packet and acquire SA; <= When I had that in installed-sa I had lot of dead SA which weren't cleared when the tunnel reconnect itself. After set to `use` it looks good.
I was referring to the Single Child SA

So let me understand that
you are talking about that?
/ip ipsec policy add dst-address=$here action=none place-before=0 ?

$here is <mikrotik_public_IP> all current networks I have on mikrotik side (leftside) ?

When I've set the 0.0.0.0/0 in src and dst the rule become invalid
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Tue Jun 02, 2020 10:25 pm

So let me understand that
you are talking about that?
/ip ipsec policy add dst-address=$here action=none place-before=0 ?

$here is <mikrotik_public_IP> all current networks I have on mikrotik side (leftside) ?
IPsec policies override all routes including those to connected subnets. So a 0.0.0.0/0 -> 0.0.0.0/0 policy overrides even the access to the Mikrotik via local subnets. Hence if by "all current networks" you mean all the LAN subnets of the Mikrotik, then yes, this is what I had in mind. But the reference to <mikrotik_public_IP> doesn't fit to this understanding, so maybe describe the environment at the Mikrotik end in more detail, and describe what exactly you mean by "losing connectivity". Depending on which remote subnets you really need to be routed through the IPsec tunnel to GCP and which ones to use the "normal" routing, you may even have to generate action=none policies to shadow that 0.0.0.0/0 -> 0.0.0.0/0 policy for all destinations except a single address or subnet.

But as you mention BGP, I still wonder whether policy-based traffic selection will be sufficient for the purpose or whether you would need a virtual tunnel interface for IPsec, which is not available at Mikrotik.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Tue Jun 02, 2020 10:47 pm

Still I can't manage to create a ipsec policy to connect with GCP
/ip ipsec policy
add action=none dst-address=172.16.1.0/24 src-address=0.0.0.0/0
add action=none dst-address=172.16.3.0/24 src-address=0.0.0.0/0
add action=none dst-address=172.16.18.0/24 src-address=0.0.0.0/0
add action=none dst-address=10.0.0.0/13 src-address=0.0.0.0/0
add action=none dst-address=10.13.50.0/24 src-address=0.0.0.0/0
add action=none dst-address=10.99.5.0/24 src-address=0.0.0.0/0
add action=none dst-address=10.254.254.0/24 src-address=0.0.0.0/0
add action=none dst-address=10.255.254.0/23 src-address=0.0.0.0/0
And tried this approach https://wiki.mikrotik.com/wiki/IKEv2_EA ... sec_tunnel

But logs in GCP are with no mercy here
Logs from GCP attached

I don't kniw what you mean routed through. I don't wanna route anything through. I want to connect both sides and allow networks in those sides to communicate each other
You do not have the required permissions to view the files attached to this post.
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Tue Jun 02, 2020 10:58 pm

The log from GCP side suggests that you use mode-config=request-only in the /ip ipsec identity at Mikrotik side, thus asking the GCP end to assign an IP address to the Mikrotik, but it doesn't have one on stock. Is requesting an address via mode-config required by their documentation?
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Wed Jun 03, 2020 12:15 am

The log from GCP side suggests that you use mode-config=request-only in the /ip ipsec identity at Mikrotik side, thus asking the GCP end to assign an IP address to the Mikrotik, but it doesn't have one on stock. Is requesting an address via mode-config required by their documentation?
No it doesn't and I don't request anything. I showed a tutorial with nordvpn where there is a way to configure policy with 0.0.0.0/0 and by they way it uses `template` . And no, I'm not using request-only I created a unnecessary mode-config with different name. I've just removed it. But the logs shows the same thing. So.. still no luck. by the way still traffic says that I'm using wrong networks .. it takes SA Addresses. I have a question? E'm I the only one who is trying to set Google Cloud Platform VPN with BGP on MikroTik? Did no one created a proper configuration and can't share with me ?
You do not have the required permissions to view the files attached to this post.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Wed Jun 03, 2020 11:18 am

I received answer from Emil, from MikroTik Supportm and he says
Hello,
RouterOS has policy based IPsec only. You can configure 0.0.0.0/0<->0.0.0.0/0 traffic selector, but you will not be able to route specific traffic over the tunnel, so that really is not an option at this time.

Emīls Z.
So right now BGP will not work with local and remote traffic selector set on 0.0.0.0/0

Which means Using GCP VPN along with BGP isn't possible right now with MikroTik. Maybe a case for MikroTik Developers to fix this.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Sat Jun 13, 2020 3:46 am

So let me understand that
you are talking about that?
/ip ipsec policy add dst-address=$here action=none place-before=0 ?

$here is <mikrotik_public_IP> all current networks I have on mikrotik side (leftside) ?
IPsec policies override all routes including those to connected subnets. So a 0.0.0.0/0 -> 0.0.0.0/0 policy overrides even the access to the Mikrotik via local subnets. Hence if by "all current networks" you mean all the LAN subnets of the Mikrotik, then yes, this is what I had in mind. But the reference to <mikrotik_public_IP> doesn't fit to this understanding, so maybe describe the environment at the Mikrotik end in more detail, and describe what exactly you mean by "losing connectivity". Depending on which remote subnets you really need to be routed through the IPsec tunnel to GCP and which ones to use the "normal" routing, you may even have to generate action=none policies to shadow that 0.0.0.0/0 -> 0.0.0.0/0 policy for all destinations except a single address or subnet.

But as you mention BGP, I still wonder whether policy-based traffic selection will be sufficient for the purpose or whether you would need a virtual tunnel interface for IPsec, which is not available at Mikrotik.
But I don't get. I changed cloud vpn to route based mode and it requires 0.0.0.0/0 in traffic selector. Setting 0.0.0.0/0 likewise on local and remote ends up with ping processing but Winbox hangs and I'm waiting when i disconnect me. When it happens I'm checking ping from VM in GCP to mikrotik.. and i works
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Sun Jun 21, 2020 2:57 am

@sindy
The only thing I manage to do this using template


Image
take a picture gif
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Sun Jun 21, 2020 1:04 pm

As @emils has told you, RouterOS currently does not support virtual interfaces for IPsec. I'm not sure whether BGP can work with a peer that is not accessible via a particular interface, but if such limitation exists, it might be possible to overcome it using a dedicated bridge interface with no member ports - I've never tried that, though.

Other than that, the whole issue with the absence of a virtual interface functionality is that IPsec policy always wins over regular routing results, whereas the remote peer requires an any->any traffic selector; the policy's traffic selector is not only used locally but also negotiated using IKE or IKEv2, and it is not possible to use one selector for negotiation and another one for the actual traffic selection. So to fulfil the requirement of the remote peer, we need to "save" everything we don't want to send down the IPsec tunnel to that peer by means of other policies (plus we cannot have more than one such peer, as two identical policies are not supported).

So you have three groups of destination subnets:
  1. the ones local to the GCP site
  2. the ones local to the Mikrotik site
  3. the rest of the whole 0.0.0.0 .. 255.255.255.255 range
And we need that packets for any destination except the first type are matched by some other policy's traffic selector before they reach the 0.0.0.0/0 => 0.0.0.0/0 one.
It is relatively simple to exclude the local subnets, but it is a headache to match all the networks except the first type, especially if there is more than one subnet. So if the router is not used for regular internet access, you can only take care of the second type using the action=none policies, and blackhole the rest in regular routing (packets to blackholed destinations do not make it to traffic selector matching); otherwise, you need to build a complete collection of policies which match everything except the first type.

In your example, you've configured exception policies with destinations 0.0.0.0/1 and 128.0.0.0/1, which shadow the whole 0.0.0.0/0 range, so nothing reaches the policy with destination=0.0.0.0/0.

So as an experiment, you can configure the exceptional policies only for the local subnets, disable the default route in regular routing, and add only routes to the GCP's public IP (so that the tunnel could establish) and to the IP at which the GCP's BGP instance is listening. In this setup, establishing the tunnel should not kick you out from the router management, and it should still be possible to ping through the IPsec tunnel. Only once that works properly, it makes sense to configure BGP and additional exceptional routes and policies.

If your Mikrotik is resourceful enough to support the Metarouter functionality, it will be much simpler to use one instance of RouterOS configured as above, effectively imitating the IPsec virtual interface functionality, and run the BGP and the rest of the routing functionality on the basic instance. If it doesn't, the complexity of creation of the exceptional policies depends on the number of subnets at the GCP end, and whether they are static or change over time - in that case, the exceptional policies would have to be generated using a complex script from the routes learned via BGP.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Sun Jun 28, 2020 2:11 am

what do you mean by first type?

I di literally studied your answer but it hard to understand without practical example

I would be very grateful if you could show examples
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Sun Jun 28, 2020 12:24 pm

what do you mean by first type?
There was a numbered list of destination subnet groups/types/categories right above that. So "first type" means "the ones local to the GCP site".

I would be very grateful if you could show examples
There is a simplified example in this post. If it still doesn't explain the idea:

Imagine you have two subnets, 172.16.3.0/24 and 172.16.8.0/24, at the GCP end, and your local subnets are 172.16.5.0/24 and 172.16.71.0/24, and that you don't care about internet destinations at all.

There are 256 /24 subnets within 172.16.0.0/16, and you need to prevent just two of them from being matched only by the "any=>any" policy, because those which are neither local to the Mikrotik nor local to the GCP end will never be used (unless you have a more complex topology of your internal network). So for this case, leaving internet aside, the following "exception policies" are sufficient:

/ip ipsec policy
add action=none dst-address=172.16.5.0/24 src-address=0.0.0.0/0
add action=none dst-address=172.16.71.0/24 src-address=0.0.0.0/0
add action=encrypt dst-address=0.0.0.0/0 src-address=0.0.0.0/0


But if you need that hosts in the two local subnets can access internet, you need additional policies with action=none, covering all the public address ranges, as follows:
dst-address=0.0.0.0/1 (0.0.0.0-127.255.255.255) (actually, this includes also 0.0.0.0/8, 10.0.0.0/8 and 127.0.0.0/8, which are not public address ranges, but it doesn't matter as we do not want the "any=>any" IPsec policy to match these anyway),
dst-address=128.0.0.0/3 (128.0.0.0-159.255.255.255)
dst-address=160.0.0.0/5 (160.0.0.0-167.255.255.255)
dst-address=168.0.0.0/6 (168.0.0.0-171.255.255.255)
dst-address=172.0.0.0/12 (172.0.0.0-172.15.255.255)
<--- here is the gap for the private address range 172.16.0.0/12 (172.16.0.0 - 172.31.255.255) --->
dst-address=172.32.0.0/11 (172.32.0.0-172.63.255.255)
dst-address=172.64.0.0/10 (172.64.0.0-172.127.255.255)
dst-address=172.128.0.0/9 (172.128.0.0-172.255.255.255)
dst-address=173.0.0.0/8 (173.0.0.0-173.255.255.255)
dst-address=174.0.0.0/7 (174.0.0.0-175.255.255.255)
dst-address=176.0.0.0/4 (176.0.0.0-191.255.255.255)
dst-address=192.0.0.0/2 (192.0.0.0-255.255.255.255) (like above, it doesn't matter that 192.168.0.0/16 is not a public address range, as it doesn't matter that we protect it from IPsec as well)

If it is not clear why the above looks the way it looks, you may be missing the concept how subnet mask works. Say, we have a range from 128.0.0.0 to 191.255.255.255. This can be expressed as 128.0.0.0/2, because this means "the most important three bits of the most important byte must be 10, the rest can be anything". So the most important byte may be anything from 1000 0000 (0x80, 128) to 1011 1111 (0xbf, 191). If you want 168.0.0.0/5 (168.0.0.0-175.255.255.255) not to be included, you have to split the above into several ranges. But instead of 8 individual /5 subranges within that /2 range, you can group them where possible:
128/5 \
       > 128/4 \
136/5 /         \
                 > 128/3
144/5 \         /
       > 144/4 /
152/5 /

160/5

(168/5)

176/5 \
       > 176/4
184/5 /
So you end up with 128/3, 160/5, and 176/4 as exceptions from the complete 128/2, and only 168/5 is not covered by any of those.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 10:22 am

Ok basically what bothers me are two things.
1. I've set it like this (so we exclude two IP classes. 10.x and 172.16-31)
/ip ipsec policy
set 0 disabled=yes
add action=none dst-address=10.99.6.0/24 src-address=0.0.0.0/0
add action=none dst-address=10.0.0.0/13 src-address=0.0.0.0/0
add dst-address=0.0.0.0/0 proposal=GCP_phase2 src-address=0.0.0.0/0 template=yes
Bu the tunnel isn't connecting
10:19:52 ipsec,info new ike2 SA (I): 94.237.xx.xx[4500]-35.204.xx.xx[4500] spi:9df288f54d6e746b:0d8da2bdab6b7d67
10:19:52 ipsec,info,account peer authorized: 94.237.xx.xx[4500]-35.204.xx.xx[4500] spi:9df288f54d6e746b:0d8da2bdab6b7d67
10:19:52 ipsec,info killing ike2 SA: 94.237.xx.xx[4500]-35.204.xx.xx[4500] spi:9df288f54d6e746b:0d8da2bdab6b7d67
I manage once to get this working but I don't remember what was it.
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Mon Jun 29, 2020 10:55 am

You'll need detailed IPsec logging to see why the tunnel doesn't come up.
  1. Disable the peer
  2. Switch detailed logging on:
    /system logging add topics=ipsec,!packet
  3. Start copying the log into a dedicated file:
    /log print follow-only file=ipsec-startup where topics~"ipsec"
  4. Enable the peer
  5. Wait until the connection fails
  6. Break the /log print ...
  7. Download the file and analyse it
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 11:31 am

I must say that logs don't give me much information what's going beside that there is a timeout
# jun/29/2020 11:27:13 by RouterOS 6.45.9
# software id =
#
11:27:17 ipsec ike2 init retransmit
11:27:17 ipsec,debug ===== sending 424 bytes from 94.237.xx.xx[4500] to 35.204.xx.xx[4500]
11:27:17 ipsec,debug 1 times of 428 bytes message will be sent to 35.204.xx.xx[4500]
11:27:22 ipsec ike2 init retransmit
11:27:22 ipsec,debug ===== sending 424 bytes from 94.237.xx.xx[4500] to 35.204.xx.xx[4500]
11:27:22 ipsec,debug 1 times of 428 bytes message will be sent to 35.204.xx.xx[4500]
11:27:27 ipsec ike2 init retransmit
11:27:27 ipsec,debug ===== sending 424 bytes from 94.237.xx.xx[4500] to 35.204.xx.xx[4500]
11:27:27 ipsec,debug 1 times of 428 bytes message will be sent to 35.204.xx.xx[4500]
11:27:32 ipsec ike2 init timeout
11:27:40 ipsec ike2 starting for: 35.204.xx.xx
11:27:40 ipsec adding notify: NAT_DETECTION_DESTINATION_IP
11:27:40 ipsec,debug => (size 0x1c)
11:27:40 ipsec,debug 0000001c 00004005 0f3e2c4d c2745b24 83e70b68 8a30dbc2 7782d6e7
11:27:40 ipsec adding notify: NAT_DETECTION_SOURCE_IP
11:27:40 ipsec,debug => (size 0x1c)
11:27:40 ipsec,debug 0000001c 00004004 428c292b def2043e d7d6f8dd e6b91a68 09e073b2
11:27:40 ipsec adding payload: NONCE
11:27:40 ipsec,debug => (size 0x1c)
11:27:40 ipsec,debug 0000001c d99c3f3a bbcbda75 f1b3c1d7 7b543984 30f99f55 4e42eb41
11:27:40 ipsec adding payload: KE
11:27:40 ipsec,debug => (first 0x100 of 0x108)
11:27:40 ipsec,debug 00000108 000e0000 ed668335 36c9d9cc b88ad1a8 e96e6a9d 1037f7a0 1869128a
11:27:40 ipsec,debug 91a07568 998f38d9 1a76efff c1657e93 62be84ab b57b5ef9 8a1bdde4 039b6312
11:27:40 ipsec,debug 22c25e34 3ed385f1 cabd3c2f b0e2831d 42ddddb6 4d421532 eab70fa6 695e4f50
11:27:40 ipsec,debug 02f81a4c 15885dfe 37d9d641 b0282f9a 7e620794 f6483a47 b2980ccd d7d585e2
11:27:40 ipsec,debug d29286d5 4ffe1355 c4322e1e 35fef7ed c8450460 79d3cdf1 11334dcd 314fc420
11:27:40 ipsec,debug bf9c435e 0a3c2a2e af7356da b2f44be8 a1dfa554 28300ae7 e3f49f06 4b6ce481
11:27:40 ipsec,debug d8c702ca 9bcf6cc2 59b63898 c0a3aaa9 47a53360 753e21ad 5595b7bc 217fdf68
11:27:40 ipsec,debug 0defde63 d4d43ab1 818143fa 6c64c7cb 1273172c ca370a4f cd3bfd5f b1187685
11:27:40 ipsec adding payload: SA
11:27:40 ipsec,debug => (size 0x30)
11:27:40 ipsec,debug 00000030 0000002c 01010004 0300000c 0100000c 800e0080 03000008 02000002
11:27:40 ipsec,debug 03000008 03000002 00000008 0400000e
11:27:40 ipsec <- ike2 request, exchange: SA_INIT:0 35.204.xx.xx[4500]
11:27:40 ipsec,debug ===== sending 424 bytes from 94.237.xx.xx[4500] to 35.204.xx.xx[4500]
11:27:40 ipsec,debug 1 times of 428 bytes message will be sent to 35.204.xx.xx[4500]
11:27:47 ipsec ike2 init retransmit
11:27:47 ipsec,debug ===== sending 424 bytes from 94.237.xx.xx[4500] to 35.204.xx.xx[4500]
11:27:47 ipsec,debug 1 times of 428 bytes message will be sent to 35.204.xx.xx[4500]
11:27:52 ipsec ike2 init retransmit
11:27:52 ipsec,debug ===== sending 424 bytes from 94.237.xx.xx[4500] to 35.204.xx.xx[4500]
11:27:52 ipsec,debug 1 times of 428 bytes message will be sent to 35.204.xx.xx[4500]
11:27:57 ipsec ike2 init retransmit
11:27:57 ipsec,debug ===== sending 424 bytes from 94.237.xx.xx[4500] to 35.204.xx.xx[4500]
11:27:57 ipsec,debug 1 times of 428 bytes message will be sent to 35.204.xx.xx[4500]
11:28:02 ipsec ike2 init timeout
11:28:10 ipsec ike2 starting for: 35.204.xx.xx
11:28:10 ipsec adding notify: NAT_DETECTION_DESTINATION_IP
11:28:10 ipsec,debug => (size 0x1c)
11:28:10 ipsec,debug 0000001c 00004005 a5cc0ab1 7dfedc3b 6fa7b1cb 4d6fe469 61c0622b
11:28:10 ipsec adding notify: NAT_DETECTION_SOURCE_IP
11:28:10 ipsec,debug => (size 0x1c)
11:28:10 ipsec,debug 0000001c 00004004 56d2815f 3c6ebb24 2e003140 20e47d9e e5122910
11:28:10 ipsec adding payload: NONCE
11:28:10 ipsec,debug => (size 0x1c)
11:28:10 ipsec,debug 0000001c f0eb685e 91029382 2b2fd701 54bb6824 6d078d5e 71f503f5
11:28:10 ipsec adding payload: KE
11:28:10 ipsec,debug => (first 0x100 of 0x108)
11:28:10 ipsec,debug 00000108 000e0000 6f7c20b1 f6471ccf fba103f9 7c1bd7be f29f2519 4a872134
11:28:10 ipsec,debug 85664a1f a9aa4e39 cee2e703 4e18c774 785f1bf3 616938fb 096ac1eb 007d98f2
11:28:10 ipsec,debug 48fcd14e 35fa6c78 51b28d29 f77abceb af777fbd cb1c03cd 7cafbec9 4a9e80e7
11:28:10 ipsec,debug 0f5bfb2d 7ceb1c1f 25f70b44 96dca1e1 c6d11b16 39f93464 fa81438a 6f41f3ca
11:28:10 ipsec,debug a09f2c65 d967094e 7379f3c5 30b0aecb af26d72d ca016b9e fb9eb749 473bb5a6
11:28:10 ipsec,debug 1b0aa968 a039ff69 254f9281 77e24132 6ddc6d34 de3e7cab b7196669 30a136cd
11:28:10 ipsec,debug f7b39258 64b2a194 283e2bc3 805fa1c5 dc85012c a6010f6a 6e49c03c dd536aa8
11:28:10 ipsec,debug 1f4c557c ca1d540e 838adb54 84ba8973 a424eeef 24daf19b 304310da 38e5c199
11:28:10 ipsec adding payload: SA
11:28:10 ipsec,debug => (size 0x30)
11:28:10 ipsec,debug 00000030 0000002c 01010004 0300000c 0100000c 800e0080 03000008 02000002
11:28:10 ipsec,debug 03000008 03000002 00000008 0400000e
11:28:10 ipsec <- ike2 request, exchange: SA_INIT:0 35.204.xx.xx[4500]
11:28:10 ipsec,debug ===== sending 424 bytes from 94.237.xx.xx[4500] to 35.204.xx.xx[4500]
11:28:10 ipsec,debug 1 times of 428 bytes message will be sent to 35.204.xx.xx[4500]
Ok I manage to change routing and suddenly more connectivity started to log as something fall out
11:40:02 ipsec peer selected tunnel mode
11:40:02 ipsec mode change not supported
And I see this.
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Mon Jun 29, 2020 11:41 am

I must say that logs don't give me much information what's going beside that there is a timeout
The "ipsec,info,account peer authorized" message which you've posted before is missing in this log, so the timeout is a different issue than in the previous case.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 12:05 pm

I must say that logs don't give me much information what's going beside that there is a timeout
The "ipsec,info,account peer authorized" message which you've posted before is missing in this log, so the timeout is a different issue than in the previous case.
Yeah but as you see can figure it out why. And timeouts are gone now now I have issues with mode and also when settings mode (my-id to fqdn)
[konrad@MikroTik] /ip ipsec policy> ..identity pr
Flags: D - dynamic, X - disabled
0 peer=ike-gcp_casino auth-method=pre-shared-key my-id=fqdn:94.237.xx.xx secret="xxxx" generate-policy=port-override

12:08:39 ipsec -> ike2 reply, exchange: AUTH:1 35.204.xx.xx[4500]
12:08:39 ipsec payload seen: ENC (48 bytes)
12:08:39 ipsec processing payload: ENC
12:08:39 ipsec,debug => iv (size 0x10)
12:08:39 ipsec,debug a0028764 00a8d9d7 669c6e9f e13afa7b
12:08:39 ipsec,debug => plain payload (trimmed) (size 0x8)
12:08:39 ipsec,debug 00000008 00000018
12:08:39 ipsec,debug decrypted
12:08:39 ipsec payload seen: NOTIFY (8 bytes)
12:08:39 ipsec processing payloads: NOTIFY
12:08:39 ipsec notify: AUTHENTICATION_FAILED
12:08:39 ipsec,error got fatal error: AUTHENTICATION_FAILED
Or should I set my new template ?

GCP sends
insertId: "c7lbfvg25fd285"
labels: {…}
logName: "projects/casino-front/logs/cloud.googleapis.com%2Fipsec_events"
receiveTimestamp: "2020-06-29T09:24:10.998457933Z"
resource: {…}
severity: "DEBUG"
textPayload: "generating IKE_AUTH response 1 [ N(AUTH_FAILED) ]"
timestamp: "2020-06-29T09:24:10.972873297Z"
@sindy I have no idea why it stopped working

Ok the issue was with Firewall. I had previously in firewall raw some prerouting entries which I removed and I don't know why they don't add again although I set here
/ip ipsec identity
add generate-policy=port-strict notrack-chain=prerouting peer=ike-gcp_casino secret=xxxx
notrack-chain=prerouting


and how to proceed with BGP ?
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Mon Jun 29, 2020 2:58 pm

and how to proceed with BGP ?
So are you saying that now you have a running tunnel, i.e. there is an installed-sa generated from the template?
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 3:45 pm

and how to proceed with BGP ?
So are you saying that now you have a running tunnel, i.e. there is an installed-sa generated from the template?
Ok to make some clearness what's going on now.
/ip ipsec policy
add action=none dst-address=10.99.6.0/24 src-address=0.0.0.0/0
add action=none dst-address=10.0.0.0/13 src-address=0.0.0.0/0
set 2 proposal=GCP_phase2
add dst-address=169.254.1.2/32 peer=ike-gcp_casino proposal=GCP_phase2 sa-dst-address=35.204.xx.xx sa-src-address=94.237.xx.xx src-address=169.254.1.1/32 tunnel=yes
That's what I added
[konrad@MikroTik] /ip ipsec identity> ..policy pr
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default
 #     PEER                                            TUNNEL SRC-ADDRESS                                                                           DST-ADDRESS                                                                           PROTOCOL   ACTION  LEVEL    PH2-COUNT
 0                                                            0.0.0.0/0                                                                             10.99.6.0/24                                                                          all        none
 1                                                            0.0.0.0/0                                                                             10.0.0.0/13                                                                           all        none
 2 T *                                                        ::/0                                                                                  ::/0                                                                                  all
 3  A  ike-gcp_casino                                  yes    169.254.1.1/32                                                                        169.254.1.2/32                                                                        all        encrypt require          1
 4  DA  ike-gcp_casino                                  yes    0.0.0.0/0                                                                             35.204.xx.xx/32                                                                      all        encrypt unique           1
I had to add BGP like that to establish connections
( Received entries
3 ADb 10.101.0.0/16 169.254.1.2 20
)
but when I want to ping for instance from
10.99.6.2 (which is mikrotik) to a host in GCP 10.101.13.250
I get timeout.
[konrad@MikroTik] /ip ipsec identity> /tool traceroute address=10.101.13.250
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 94.237.xx.xx                     66..   31     0ms   197.6       0   988.7   395.2 host unreachable from 94.237.xx.xx
I think that without ipsec policy it will not go through tunnel like the BGP didn't went through.. and this is something we cannot do as this is the main reason we try to push 0.0.0.0/0 <=> 0.0.0.0/0
And answering finally your questions
Yes tunnel is up
Flags: H - hw-aead, A - AH, E - ESP
0 E spi=0x1D64599 src-address=35.204.xx.xx dst-address=94.237.xx.xx state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=256 auth-key="f36938f75178530fa65600cabde1015f9ca9733e9e7f8158910f0a2fb60cbb46"
enc-key="71a695dd8e04c2117c1f672c1386ab237791ead7e596b7fbff79a6aa26f5b71d" addtime=jun/29/2020 14:36:21 expires-in=1h10m22s add-lifetime=2h24m19s/3h24s current-bytes=189600 current-packets=3160 replay=128

1 E spi=0xD1C79D38 src-address=94.237.xx.xx dst-address=35.204.xx.xx state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=256 auth-key="f38ced110cb246b66eeab648b2cafe709a4f215a4ee46d47dabab3d9f346ea9f"
enc-key="85205ae97a803ec5f1f6a29316d83e533c7009d37e6232d40ca4ca1d456ff26e" add-lifetime=2h24m19s/3h24s replay=128

2 E spi=0x93CFCD7 src-address=35.204.xx.xx dst-address=94.237.xx.xx state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=256 auth-key="0408aac30d5f47d8122e18ce59c51610883c47f9c0205a5324736280a206545c"
enc-key="e25b17c33ecf846ff3ea40bfa5e44d2dce5418731bbbfb111f670b3e2475463f" addtime=jun/29/2020 15:02:52 expires-in=1h36m57s add-lifetime=2h24m22s/3h28s current-bytes=489017 current-packets=5964 replay=128

3 E spi=0x2013652C src-address=94.237.xx.xx dst-address=35.204.xx.xx state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=256 auth-key="bbe1a9483bd32acc69c10453541ed253c8c265305c2bdd064510badb4c4afc10"
enc-key="3e09b65774d4c95ffa8ffdf0e4ed615aedd2512c78b0291916837c53d15ca1a8" addtime=jun/29/2020 15:02:52 expires-in=1h36m57s add-lifetime=2h24m22s/3h28s current-bytes=35991 current-packets=574 replay=128
Last edited by eset on Mon Jun 29, 2020 4:27 pm, edited 2 times in total.
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Mon Jun 29, 2020 4:01 pm

The whole idea was that the policy associated to the peer has to be a 0.0.0.0/0=>0.0.0.0/0 one because the GCP insists on that. You have used a template instead, plus a single policy for just 169.254.1.1=>169.254.1.2, and the peer has accepted that narrow policy, which is quite surprising. So try to add a static policy for 10.101.13.0/24 too, and see what happens. If that works, the GCP does actually not need to have the whole 0.0.0.0/0=>0.0.0.0/0 confirmed from the Mikrotik end, which would make all that exception policy burden redundant (which would be very good).
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 4:30 pm

The whole idea was that the policy associated to the peer has to be a 0.0.0.0/0=>0.0.0.0/0 one because the GCP insists on that. You have used a template instead, plus a single policy for just 169.254.1.1=>169.254.1.2, and the peer has accepted that narrow policy, which is quite surprising. So try to add a static policy for 10.101.13.0/24 too, and see what happens. If that works, the GCP does actually not need to have the whole 0.0.0.0/0=>0.0.0.0/0 confirmed from the Mikrotik end, which would make all that exception policy burden redundant (which would be very good).
If I add static policy 10.101.13.0/24 will end up with something I don't want to have because GCP force 0.0.0.0/0 although they let anything else. But this ends up finally with unstable tunnel which is the whole point of investigating and changing the whole configuration to have stable GCP tunnel. Btw above I showed screen where there wasn't any phase2 for BGP with policy 0.0.0.1 and 128.0.0.0 and ping to 10.101.13.250 worked.
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Mon Jun 29, 2020 4:38 pm

If I add static policy 10.101.13.0/24 will end up with something I don't want to have because GCP force 0.0.0.0/0 although they let anything else. But this ends up finally with unstable tunnel which is the whole point of investigating and changing the whole configuration to have stable GCP tunnel
In that case, use a static policy with tunnel=yes and src-address=0.0.0.0/0 dst-address=0.0.0.0/0 attached to the GCP instead of the template, and remove the static ones linked to that peer.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 4:43 pm

 0                                                            0.0.0.0/0                                                                             10.99.6.0/24                                                                          all        none
 1                                                            0.0.0.0/0                                                                             10.0.0.0/13                                                                           all        none
 2  A  ike-gcp_casino                                  yes    169.254.1.1/32                                                                        169.254.1.2/32                                                                        all        encrypt require          1
 3     ike-gcp_casino                                  yes    10.99.6.0/24                                                                          10.101.0.0/16                                                                         all        encrypt require          0
 4 T *                                                        ::/0                                                                                  ::/0                                                                                  all
 5  DA  ike-gcp_casino                                  yes    0.0.0.0/0                                                                             35.204.160.90/32                                                                      all        encrypt unique           1
Ok I've added static as you see and funny is that this
10.99.6.0/24 <=> 10.101.0.0/16
and no phase2 , no A but ping works. In GCP I didn't received any traffic selector with 10.99.6.0/24 and 10.101.0.0/16 which is good, it isn't visible like 169.254.1.1/24 and 169.254.1.2/24 where you see A and and established - which isn't good from GCP perspective
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 4:50 pm

OK I've set it like this, as you suggested with static (removing template)
[konrad@MikroTik] /ip ipsec identity> ..policy pr
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default
 #     PEER                                            TUNNEL SRC-ADDRESS                                                                           DST-ADDRESS                                                                           PROTOCOL   ACTION  LEVEL    PH2-COUNT
 0 TX*                                                        ::/0                                                                                  ::/0                                                                                  all
 1                                                            0.0.0.0/0                                                                             10.99.6.0/24                                                                          all        none
 2                                                            0.0.0.0/0                                                                             10.0.0.0/13                                                                           all        none
 3     ike-gcp_casino                                  yes    169.254.1.1/32                                                                        169.254.1.2/32                                                                        all        encrypt require          0
 4     ike-gcp_casino                                  yes    10.99.6.0/24                                                                          10.101.0.0/16                                                                         all        encrypt require          0
 5  A  ike-gcp_casino                                  yes    0.0.0.0/0                                                                             0.0.0.0/0                                                                             all        encrypt require          1
and what makes me happy is that BGP network isn't in established state (no 'A')
And google gives me in log something like this (below)
{
 insertId:  "1n3xyb5g29eywcr"   
 labels: {…}   
 logName:  "projects/casino-front/logs/cloud.googleapis.com%2Fipsec_events"   
 receiveTimestamp:  "2020-06-29T13:44:33.150984918Z"   
 resource: {…}   
 severity:  "NOTICE"   
 textPayload:  "CHILD_SA vpn_94.237.10.65{410} established with SPIs cbc70be9_i 0ebfde6c_o and TS 0.0.0.0/0 === 0.0.0.0/0 "   
 timestamp:  "2020-06-29T13:44:33.120739857Z"   
}
And that looks awesome. Ping works.
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Mon Jun 29, 2020 5:16 pm

And that looks awesome. Ping works.
Perfect. So you can remove those two static policies linked to the GCP peer, which show phase 2 count 0 above, and you should be good.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 5:21 pm

btw do I need to have `default` removed? Because right now I don't see if this is required.
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Mon Jun 29, 2020 5:38 pm

You mean those one?
add dst-address=169.254.1.2/32 peer=ike-gcp_casino proposal=GCP_phase2 sa-dst-address=35.204.xx.xx sa-src-address=94.237.10.65 src-address=169.254.1.1/32 tunnel=yes
add dst-address=10.101.0.0/16 peer=ike-gcp_casino proposal=GCP_phase2 sa-dst-address=35.204.xx.xx sa-src-address=94.237.10.65 src-address=10.99.6.0/24 tunnel=yes
Isn't that what you said I need to set up? Because without them ping stops working
I've said earlier today that you should add the second one (10.99.6.0/24=>10.101.0.0/16) as I forgot that if done that way, the tunnel is not stable (I don't remember the beginning of the thread for weeks, sorry). But when the tunnel came up with the 0.0.0.0/0=>0.0.0.0/0 policy as static one, not as a template, those two policies above (169.254.1.1/32=>169.254.1.2/32 and 10.99.6.0/24=>10.101.0.0/16) were inactive, so I can see no reason why they should be present. So if you disable them, and then disconnect and re-connect the tunnel, it should be working with the 0.0.0.0/0=>0.0.0.0/0 alone.

If I remove them, I mean on of them I will lose mikrotik connectivity through SSH and Winbox for example.
I don't see why, or rather I don't understand connectivity from where? It's the excluding policies, with action=none, which should prevent the SSH and Winbox connection from LAN from being kidnapped by the 0.0.0.0/0=>0.0.0.0/0 policy.

btw do I need to have `default` removed? Because right now I don't see if this is required.
You don't, as the default policy is a template, and since you should have generate-policy=no in the identity, hence no template is needed.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 6:11 pm

You mean those one?
add dst-address=169.254.1.2/32 peer=ike-gcp_casino proposal=GCP_phase2 sa-dst-address=35.204.xx.xx sa-src-address=94.237.10.65 src-address=169.254.1.1/32 tunnel=yes
add dst-address=10.101.0.0/16 peer=ike-gcp_casino proposal=GCP_phase2 sa-dst-address=35.204.xx.xx sa-src-address=94.237.10.65 src-address=10.99.6.0/24 tunnel=yes
Isn't that what you said I need to set up? Because without them ping stops working
I've said earlier today that you should add the second one (10.99.6.0/24=>10.101.0.0/16) as I forgot that if done that way, the tunnel is not stable (I don't remember the beginning of the thread for weeks, sorry). But when the tunnel came up with the 0.0.0.0/0=>0.0.0.0/0 policy as static one, not as a template, those two policies above (169.254.1.1/32=>169.254.1.2/32 and 10.99.6.0/24=>10.101.0.0/16) were inactive, so I can see no reason why they should be present. So if you disable them, and then disconnect and re-connect the tunnel, it should be working with the 0.0.0.0/0=>0.0.0.0/0 alone.
If I remove them, I mean on of them I will lose mikrotik connectivity through SSH and Winbox for example.
I don't see why, or rather I don't understand connectivity from where? It's the excluding policies, with action=none, which should prevent the SSH and Winbox connection from LAN from being kidnapped by the 0.0.0.0/0=>0.0.0.0/0 policy.

btw do I need to have `default` removed? Because right now I don't see if this is required.
You don't, as the default policy is a template, and since you should have generate-policy=no in the identity, hence no template is needed.
Yes you're right about that. And what about `default` route? Should it be also removed?
And why this firewall has issue with this.
/ip firewall filter
add action=accept chain=input disabled=yes dst-address=169.254.1.1 dst-port=179 protocol=tcp src-address=169.254.1.2 <= this works and BGP can be established but..
/ip firewall address-list
add address=169.254.1.0/24 list=bgp

add action=accept chain=input comment="accept established connection packets" connection-state=established,related,untracked
add action=accept chain=input disabled=yes dst-address=169.254.1.1 dst-port=179 protocol=tcp src-address=169.254.1.2
add action=drop chain=input comment="drop invalid packets" connection-state=invalid log=yes log-prefix=invalid
add action=jump chain=input comment="Rule for services" jump-target=services

add action=drop chain=services dst-address-list=!bgp dst-port=179 log=yes log-prefix=BGP protocol=tcp src-address-list=!bgp <= doesn't work (?)
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 6:42 pm

it seems that
! in src-address and src-address-list doesn't work I need to set accept instead of drop then it works. Strange.

And is there any way to enable ping on local and gcp side? I mean I can't ping from GCP vpn client the mikrotik seems strange
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 9:03 pm

And I have trouble with this ipsec polices. when enable 0.0.0.0/0 I loose traffic between two mikrotiks. The one which has the tunnel and second in different provider network 10.5.0.0/16. Disable IPsec pings works.
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Mon Jun 29, 2020 9:50 pm

And I have trouble with this ipsec polices. when enable 0.0.0.0/0 I loose traffic between two mikrotiks. The one which has the tunnel and second in different provider network 10.5.0.0/16. Disable IPsec pings works.
10.5.0.0/16 fits into 10.0.0.0/13, hence the action=none dst-address=10.0.0.0/13 policy should prevent the packets from being kidnapped by action=encrypt dst-address=0.0.0.0/0. Let me do some tests here first, I'll be back.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 10:03 pm

And I have trouble with this ipsec polices. when enable 0.0.0.0/0 I loose traffic between two mikrotiks. The one which has the tunnel and second in different provider network 10.5.0.0/16. Disable IPsec pings works.
10.5.0.0/16 fits into 10.0.0.0/13, hence the action=none dst-address=10.0.0.0/13 policy should prevent the packets from being kidnapped by action=encrypt dst-address=0.0.0.0/0. Let me do some tests here first, I'll be back.
Exactly what I thought but I was confused when this didn't worked.

Mikrotik where I've set GCP compatible tunnel (10.6.1.253)
Traceroute to the second mikrotik gives
[konrad@UP-RT-02] /ip firewall nat> /tool traceroute address=10.5.0.120
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1                                  100%    3 timeout
 2                                  100%    3 timeout
 3                                  100%    3 timeout
 4                                  100%    2 timeout
 5                                  100%    2 timeout
From the second mikrotik traceroute to the first one (with tunnel)
[konrad@UP-RT-01] > /tool traceroute address=10.6.1.253
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 10.5.0.1                           0%   24   0.1ms     0.1     0.1     0.2       0
 2                                  100%   24 timeout
 3                                  100%   23 timeout
 4                                  100%   23 timeout
 5                                  100%   23 timeout
 6                                  100%   23 timeout
Interesting is that that second (up-rt-01) has it's gateway visible in the traceroute
Last edited by eset on Mon Jun 29, 2020 10:24 pm, edited 1 time in total.
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Mon Jun 29, 2020 10:23 pm

I'm trying this with 6.45.9 at one end and 6.46.6 on the other, and it works as expected - there is a 0.0.0.0/0<=>0.0.0.0/0 policy between these two Mikrotiks and exceptions from it (action=none) before, and the exceptions work as expected - what is covered by an exception is not kidnapped by the 0.0.0.0/0<=>0.0.0.0/0. Can you show me the current policy list while the tunnel is down? I've specially tested the exceptions at the initiator side (6.46.6).

As for the default route - a default route is not necessary, but some route must exist for any packet to be matched by the IPsec traffic selectors - IPsec traffic selector matching is done after regular routing, and if regular routing has no route for the packet, the packet doesn't reach the policy matching stage. So if you have a dedicated route for the BGP communication, and a dedicated route towards the public IP of the GCP IPsec peer, it should be sufficient, as the BGP will create the remaining routes and the policies will then take care about the rest.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 10:30 pm

I'm trying this with 6.45.9 at one end and 6.46.6 on the other, and it works as expected - there is a 0.0.0.0/0<=>0.0.0.0/0 policy between these two Mikrotiks and exceptions from it (action=none) before, and the exceptions work as expected - what is covered by an exception is not kidnapped by the 0.0.0.0/0<=>0.0.0.0/0. Can you show me the current policy list while the tunnel is down? I've specially tested the exceptions at the initiator side (6.46.6).

As for the default route - a default route is not necessary, but some route must exist for any packet to be matched by the IPsec traffic selectors - IPsec traffic selector matching is done after regular routing, and if regular routing has no route for the packet, the packet doesn't reach the policy matching stage. So if you have a dedicated route for the BGP communication, and a dedicated route towards the public IP of the GCP IPsec peer, it should be sufficient, as the BGP will create the remaining routes and the policies will then take care about the rest.
I've disabled the tunnel
add action=none dst-address=10.0.0.0/13 src-address=0.0.0.0/0
add action=none dst-address=10.99.6.0/24 src-address=0.0.0.0/0
add disabled=yes dst-address=0.0.0.0/0 peer=ike-gcp_casino proposal=GCP_phase2 sa-dst-address=35.204.160.90 sa-src-address=94.237.10.65 src-address=0.0.0.0/0 tunnel=yes
after disabling it traceroute to second mikrotik works
[konrad@UP-RT-02] /ip ipsec policy> /tool traceroute address=10.5.0.120
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 10.6.0.1                           0%    2   0.1ms     0.1     0.1     0.1       0
 2                                  100%    2 timeout
 3                                  100%    2 timeout
 4                                  100%    1 timeout
 5                                  100%    1 timeout
 6                                  100%    1 timeout
and also its gateway appears 10.6.0.1
when I enable IPsec it disappears. Probably that's the reason it doesn't work from both sides.
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Mon Jun 29, 2020 10:50 pm

How is 10.5.0.120 linked with 10.6.0.1? Is it a direct LAN interconnection or some kind of a tunnel (L2TP etc.)?
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 11:01 pm

It's UpCloud Provider.
So if you asked me. They are not behind One NAT
One mikrotik is in Netherland and second is Frankfurt. So basically those are two different DCs.

Communication between them is possible over Local Are Network but 100% sure it's some kind of tunnel between those two regions.
 1  10.5.0.1 (10.5.0.1)  0.103 ms  0.080 ms  0.082 ms
 2  100.68.128.129 (100.68.128.129)  0.233 ms  0.233 ms  0.266 ms
 3  172.21.255.241 (172.21.255.241)  0.400 ms  0.393 ms  0.296 ms
 4  172.21.255.253 (172.21.255.253)  0.261 ms  0.194 ms  0.241 ms
 5  r2-ams1-po1.nl.net.upcloud.com (94.237.0.109)  0.262 ms  0.195 ms  0.245 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  10.6.1.253 (10.6.1.253)  27.484 ms  27.494 ms  27.489 ms
From one of the Linux Servers to the mikrotik traceroute looks like ^ above. Of course this works only when VPN is disabled on the mikrotik
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Mon Jun 29, 2020 11:29 pm

Okay, so there may be some tunnel but the loading and unloading to/from the tunnel is not done by any of the two 'Tiks. Are the routes (10.6.x.y via 100.68.128.129) configured statically or via some dynamic routing protocol? Or is 100.68.128.129 the default gateway (static or DHCP-assigned)?

I am looking for a reason why prevening the 10.0.0.0/13 destination from getting kidnapped by the 0.0.0.0/0=>0.0.0.0/0 policy is not sufficient, and the only things that come to my mind are that
  • the transport packets of the tunnel between 10.5.x.y and 10.6.w.z get kidnapped, but that cannot be the case as there is no tunnel terminated at your Tik so no transport packets to be kidnapped
  • the route to 10.6.x.x is learned dynamically, and the address of the remote peer of the dynamic routing protocol is not protected from being kidnapped
  • you track the availability of the gateway (100.68.128.129) by pinging and if it doesn't respond to pings, the route goes down - in that case, you have to add another action=none policy for dst-address=100.68.128.129/32, because IPsec kidnaps even destinations in connected subnets
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Mon Jun 29, 2020 11:52 pm

Okay, so there may be some tunnel but the loading and unloading to/from the tunnel is not done by any of the two 'Tiks. Are the routes (10.6.x.y via 100.68.128.129) configured statically or via some dynamic routing protocol? Or is 100.68.128.129 the default gateway (static or DHCP-assigned)?

I am looking for a reason why prevening the 10.0.0.0/13 destination from getting kidnapped by the 0.0.0.0/0=>0.0.0.0/0 policy is not sufficient, and the only things that come to my mind are that
  • the transport packets of the tunnel between 10.5.x.y and 10.6.w.z get kidnapped, but that cannot be the case as there is no tunnel terminated at your Tik so no transport packets to be kidnapped
  • the route to 10.6.x.x is learned dynamically, and the address of the remote peer of the dynamic routing protocol is not protected from being kidnapped
  • you track the availability of the gateway (100.68.128.129) by pinging and if it doesn't respond to pings, the route goes down - in that case, you have to add another action=none policy for dst-address=100.68.128.129/32, because IPsec kidnaps even destinations in connected subnets
All routes on mikrotik are set statically . The above example , from Linux server, it's dynamic.
first mikrotik
add distance=1 dst-address=10.0.0.0/13 gateway=10.6.0.1
second mikrotik
add distance=1 dst-address=10.0.0.0/13 gateway=10.5.0.1
So I have no access to 100.68.128.129. It's a network set by UpCloud.
I don't have information on my mikrotik about 100.68.128.129 only 10.5.0.1 and 10.6.0.1 has it and those are gateways for 10.5.0.0/22 and 10.6.0.0/22 subnets.
pinging that 100.68.128.129 gives no route to host
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Mon Jun 29, 2020 11:57 pm

Do you have generate-policy=no in the /ip ipsec identity row? And do you remember the voice password from Sexmission?
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Tue Jun 30, 2020 12:34 am

Do you have generate-policy=no in the /ip ipsec identity row? And do you remember the voice password from Sexmission?
[konrad@UP-RT-02] > /ip ipsec identity pr
Flags: D - dynamic, X - disabled
 0    peer=ike-gcp_casino auth-method=pre-shared-key secret="xxxx" generate-policy=no
I have no generate-policy set. Still no luck even when added that
add action=none dst-address=100.68.128.129/32 src-address=0.0.0.0/0
I believe it's because
that second mikrotik
[konrad@UP-RT-01] /routing bgp network> /tool traceroute address=10.6.1.253 src-address=10.5.0.120 interface=ether2
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 10.5.0.1                           0%   27   0.1ms     0.1     0.1     0.2       0
 2                                  100%   27 timeout
 3                                  100%   27 timeout
 4                                  100%   27 timeout
 5                                  100%   26 timeout
 6                                  100%   26 timeout
Has it's gateway available and my mikrotik doesn't has it.
[konrad@UP-RT-02] > /tool traceroute address=10.5.0.120 interface=ether2
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1                                  100%    2 timeout
 2                                  100%    2 timeout
 3                                  100%    2 timeout
 4                                  100%    1 timeout
 5                                  100%    1 timeout
Sorry do I have what? What Sexmission?
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Tue Jun 30, 2020 1:07 am

And what interesting is that the subnet 10.99.6.0/24 also has connectivity problem.
I've created a temporary server instance in the same local SDN network
root@debian-1cpu-1gb-fi-hel2:~# traceroute 10.99.6.2
traceroute to 10.99.6.2 (10.99.6.2), 30 hops max, 60 byte packets
 1  10.99.6.2 (10.99.6.2)  0.301 ms  0.293 ms  0.282 ms
 

After enabling tunnel on mikrotik (10.6.1.253 with 10.99.6.2 interface also)
[konrad@UP-RT-01] /routing bgp network> /tool traceroute address=10.99.6.3 src-address=10.99.6.2 interface=ether3
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1                                  100%    2 timeout
 2                                  100%    2 timeout
 3                                  100%    2 timeout
 4                                  100%    2 timeout
 5                                  100%    2 timeout

Which is strange because from linux serwer I get normal route
root@debian-1cpu-1gb-fi-hel2:~# traceroute 10.99.6.2
traceroute to 10.99.6.2 (10.99.6.2), 30 hops max, 60 byte packets
 1  10.99.6.2 (10.99.6.2)  0.303 ms  0.284 ms  0.276 ms
Sounds like firewall
Last edited by eset on Tue Jun 30, 2020 1:13 am, edited 1 time in total.
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Tue Jun 30, 2020 1:09 am

I believe it's because that second mikrotik has it's gateway available and my mikrotik doesn't has it.
Nope, the fact that the gateway is shown as "not available" by traceroute while the policy is active is just a symptom, not the cause. When a policy is active, packets which come from IPs matching some policy's dst-address to IPs matching that policy's src-address are not accepted if they don't come via the SA created by that policy. So as none of the addresses of the intermediate hops on the route between 10.5.x.y and 10.6.w.z matches any of the dst-address of the action=none policies, the ICMP "TTL expired" messages coming from them reach the 0.0.0.0/0=>0.0.0.0/0 policy and get dropped because they came in plaintext via ether2, not via the policy's SA. But that should not be the case for 10.5.0.1 which is covered by the 10.0.0.0/13. I went as far as to set exactly the same exception (action=none dst-address=10.0.0.0/13) here, and it does prevent packets to 10.5.0.1 from getting matched by the action=encrypt dst-address=0.0.0.0/0 one. I did the same on the 6.45.9 for 10.6.0.1 with action=none dst-address=10.0.0.0/13, and it works properly as well.

So does a plain ping to 10.6.0.1 from 10.6.a.b break as well when the IPsec tunnel is active? Or only the traceroute?

I'm starting to think that a CHR behaves different than a mipsbe, but I hesitate to believe that.

Sorry do I have what? What Sexmission?
A 1983 sci-fi comedy - https://culture.pl/pl/dzielo/seksmisja- ... -machulski

There is a scene where access to the exit from the underground to the surface is protected by a password, and the guy trying to open it expresses his frustration by shouting a well-known word, which turns out to be the password. That's my feeling about this issue :)
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Tue Jun 30, 2020 1:19 am

[konrad@UP-RT-01] /routing bgp network> /ping 10.5.0.1
  SEQ HOST                                     SIZE TTL TIME  STATUS
    0 10.5.0.1                                   56  64 0ms
    1 10.5.0.1                                   56  64 0ms
    sent=2 received=2 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

[konrad@UP-RT-01] /routing bgp network> /tool traceroute  10.5.0.1
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 10.5.0.1                           0%    2   0.2ms     0.2     0.1     0.2     0.1
This patient (up-rt-01) which is a production router by they way (with NOT adjusted IPsec config yet) has no problem with accessing private lan.

The mikrotik we are talking about up-rt-02 10.6.1.253 with also local SDN 10.99.0.0/24 has the issue
Does it works? No.
[konrad@UP-RT-02] > ping 10.6.0.1
  SEQ HOST                                     SIZE TTL TIME  STATUS
    0 10.6.0.1                                                timeout
    1 10.6.0.1                                                timeout
    sent=2 received=0 packet-loss=100%
[konrad@UP-RT-02] > tool traceroute 10.6.0.1
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1                                  100%    1 timeout
 2                                  100%    1 timeout
 3                                    0%    1     0ms
It looses completely communication with any other hosts around him. It's a shame because right now this settings which are correct with GCP gives me nothing because I can't use that router as gateway to any traffic incoming with destination GCP network
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Tue Jun 30, 2020 1:20 am

A yes, Sex Mission movie. I 'm shocked that you saw it ;)
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Tue Jun 30, 2020 1:33 am

OK, so one last idea before I fall asleep, I've already switched off the PC - once the tunnel gets up, disable and re-enable the action=none dst-address=10.0.0.0/13 policy. I have added it while the tunnel was up, but I also had other action=none policies in place before the tunnel went up, and they are doing their job too (otherwise I'd lose connection to the test Tiks).
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Tue Jun 30, 2020 2:03 am

OK, so one last idea before I fall asleep, I've already switched off the PC - once the tunnel gets up, disable and re-enable the action=none dst-address=10.0.0.0/13 policy. I have added it while the tunnel was up, but I also had other action=none policies in place before the tunnel went up, and they are doing their job too (otherwise I'd lose connection to the test Tiks).
No, nothing. :/
 
User avatar
eset
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 97
Joined: Tue Dec 15, 2015 5:15 pm
Contact:

Re: VPN with GCP

Tue Jun 30, 2020 2:08 am

in /tool torch i see that something is comming from 10.5.0.120 when I start to ping 10.6.1.253
But it only 560ps on RX with IP protocol

I was so close and loosing network connectivity between DCs makes me feel fuckin sad :/

@sindy if you still want to help you can create account in upcloud (they have 100$ free plan) and create two machines one in Finland2 for example and second in Netherland and connect tunnel with gcp on one CHR and see if you loose connectivity between those two CHR
 
sindy
Forum Guru
Forum Guru
Posts: 5383
Joined: Mon Dec 04, 2017 9:19 pm

Re: VPN with GCP

Wed Jul 01, 2020 12:27 am

So I have spawned two 6.45.9 CHRs under Hyper-V on Win10, set up a 0.0.0.0/0 <=> 0.0.0.0/0 IPsec tunnel between them via the DHCP addresses they get from the Hyper-V on their ether1, interconnected ether2 of both using a separate dedicated virtual switch, and gave them addresses 10.5.0.1/30 and 10.5.0.2/30. With enabled action=none src-address=0.0.0.0/0 dst-address=10.0.0.0/13 policies before (above) the action=encrypt src-address=0.0.0.0/0 dst-address=0.0.0.0/0 ones on both, they ping each other's 10.5.0.x address via ether2; when that action=none policy is disabled on both, they ping each other's 10.5.0.x address via the IPsec tunnel.

So I don't understand what else may eventually be wrong about your setup, but the CPU architecture is also not the reason.

[me@HyperV-CHR-1] > ip address print
Flags: X - disabled, I - invalid, D - dynamic
 #   ADDRESS            NETWORK         INTERFACE
 0 D 192.168.204.37/28  192.168.204.32  ether1
 1   10.5.0.1/30        10.5.0.0        ether2
[me@HyperV-CHR-1] > ip ipsec export
# jun/30/2020 21:21:36 by RouterOS 6.45.9
...
/ip ipsec peer
add address=192.168.204.41/32 exchange-mode=ike2 name=chr-2 passive=yes
/ip ipsec identity
add peer=chr-2 secret=a-very-complex-secret
/ip ipsec policy
add action=none dst-address=192.168.204.32/28 src-address=0.0.0.0/0
add action=none dst-address=10.0.0.0/13 src-address=0.0.0.0/0
add dst-address=0.0.0.0/0 peer=chr-2 sa-dst-address=192.168.204.41 sa-src-address=0.0.0.0 src-address=0.0.0.0/0 tunnel=yes

[me@HyperV-CHR-2] > ip address print
Flags: X - disabled, I - invalid, D - dynamic
 #   ADDRESS            NETWORK         INTERFACE
 0 D 192.168.204.41/28  192.168.204.32  ether1
 1   10.5.0.2/30        10.5.0.0        ether2
[me@HyperV-CHR-2] > ip ipsec export
# jun/30/2020 21:22:55 by RouterOS 6.45.9
...
/ip ipsec peer
add address=192.168.204.37/32 exchange-mode=ike2 name=chr-1
/ip ipsec identity
add peer=chr-1 secret=a-very-complex-secret
/ip ipsec policy
add action=none dst-address=192.168.204.32/28 src-address=0.0.0.0/0
add action=none dst-address=10.0.0.0/13 src-address=0.0.0.0/0
add dst-address=0.0.0.0/0 peer=chr-1 sa-dst-address=192.168.204.37 sa-src-address=0.0.0.0 src-address=0.0.0.0/0 tunnel=yes
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.

Who is online

Users browsing this forum: Baidu [Spider], benc1337, Bing [Bot], meazz1 and 113 guests