Google Cloud Platform GCP - VPN - BGP

gargola · Sat Dec 01, 2018 1:50 am

Hi guys.

I'm trying to establish a vpn to GCP network but I have a doubt still can't resolve. I can establish the ipsec tunnel, but I don't have any interface o idea to where to configure the ip to create the bgp sessión. Is the same if I configure the VPN on GCP for static routing, the ipsec works, but I don't where to configure the static routing on my mikrotik to route to the ips on google cloud.

Best regards and thanks in advance.

sri2007 · Sat Dec 01, 2018 5:37 pm

hello! If you have already an IPSec Site-to-Site between Google and your Mikrotik then I think that you'll need to create some special routes using policies and rules to reach the remote server, theoretically I think that it is possible, however can you please post the configs at both sites?

gargola · Sun Dec 02, 2018 12:04 pm

Hi sri2007.

Thank you for your response, here is the configuration:

/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha1 disabled=no enc-algorithms=aes-128-cbc,3des lifetime=30m name=default pfs-group=modp1024

/ip ipsec peer
add address=XX.XX.17.122/32 auth-method=pre-shared-key dh-group=modp1024 disabled=no dpd-interval=30s enc-algorithm=aes-128,3des exchange-mode=ike2 \
    generate-policy=no hash-algorithm=sha1 policy-template-group=default secret=test send-initial-contact=yes
    
/ip ipsec policy
add action=encrypt disabled=no dst-address=10.168.0.0/20 dst-port=any ipsec-protocols=esp level=unique priority=0 proposal=default protocol=all \
    sa-dst-address=XX.XX.17.122 sa-src-address=XX.XX.53.118 src-address=10.0.5.0/29 src-port=any tunnel=yes
    
/ip address
add address=10.0.5.1/29 interface=V2-1 network=10.0.5.0

Actually it seems that I have a one way issue, because my compute engine instance is able to ping my local segment 10.0.5.0/29, but from my Mikrotik I can't ping the cloud instance. I've checked firewall and everything related to it.

Here is the evidence from the cloud instance to my LAN segment. I think is the same issue, my Mikrotik doesn't know how to route to that destination, even so the policy is indicating that any traffic to 10.168.0.0/20 should go through the ipsec tunnel.

xxx@instance-1:~$ ifconfig
ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.168.0.2  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::4001:aff:fea8:2  prefixlen 64  scopeid 0x20<link>
        ether 42:01:0a:a8:00:02  txqueuelen 1000  (Ethernet)
        RX packets 1809  bytes 1041959 (1.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1829  bytes 214386 (214.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 160  bytes 13111 (13.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 160  bytes 13111 (13.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

xxx@instance-1:~$ ping 10.0.5.1
PING 10.0.5.1 (10.0.5.1) 56(84) bytes of data.
64 bytes from 10.0.5.1: icmp_seq=1 ttl=63 time=78.6 ms
64 bytes from 10.0.5.1: icmp_seq=2 ttl=63 time=76.9 ms
64 bytes from 10.0.5.1: icmp_seq=3 ttl=63 time=84.9 ms
^C
--- 10.0.5.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 5ms
rtt min/avg/max/mdev = 76.920/80.126/84.871/3.423 ms

xxx@instance-1:~$ ping 10.168.0.2
PING 10.168.0.2 (10.168.0.2) 56(84) bytes of data.
64 bytes from 10.168.0.2: icmp_seq=1 ttl=64 time=0.015 ms
64 bytes from 10.168.0.2: icmp_seq=2 ttl=64 time=0.043 ms
^C
--- 10.168.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 29ms
rtt min/avg/max/mdev = 0.015/0.029/0.043/0.014 ms

xxx@instance-1:~$ sudo ufw status
Status: inactive
xxx@instance-1:~$

Best regards.

gargola · Sun Dec 02, 2018 7:56 pm

Well, some sleep hours helped, I could figured it out after search a little, I had to add the NAT exception from the src/dst networks. Now I'm able to ping both ways. So the IPSEC tunnel is working.

But at this time is configured with static segments (only 10.0.5.0/29 to 10.168.0.02/0) I have to make it work through BGP to avoid manual configurations. I'm going to test to add a bridge interface with the bgp ip to see if I'm able to establish the session.

gargola · Wed Dec 05, 2018 2:10 am

Finally made it.

To be able to route to the VPN I created a bridge to use it as the interface that has the ip for the BGP session and it turned in to my gateway to the cloud. If you're going to use static routes or policy based VPN is enough with the IPSEC Policies.

Any doubt feel free to contact me.

eset · Wed Jan 16, 2019 11:46 am

Are you using dedicated instance from GCP side to connect with mikrotik or do you use dedicated VPN service from Google?
Two could you share the config ?

gargola · Fri Jul 05, 2019 7:27 pm

Hi Eset.

Sorry for the late response, I didn't received a notification email. I'm using the VPN service from GCP. What config do you need? router side or GCP platform?

errrata · Fri Sep 06, 2019 2:34 am

Hi gargola,

If you don't mind, would be great if you can post here the config from both sides in case someone would like to use a similar scenario.
I am also trying to create a similar Site-To-Site VPN between a mikrotik and a GCP project (using the VPN service), but without any luck, I didn't manage yet to make the two sides communicate

PeterFreeman · Wed Sep 18, 2019 1:06 pm

Hi gargola,

We're having similar issues with MikroTik and GCP VPN with BGP. Although our configuration appears to be correct, we're suffering from packet loss through the tunnel, it's not stable.
If you coudl publish your configuration from both ends we'd like to compare.

Thank you

gargola · Thu Sep 19, 2019 7:37 pm

Hi all.

Just give me the opportunity to have some time from my job and i'll publish the configuration.

PeterFreeman · Fri Sep 20, 2019 11:42 am

Hi gargola,

We used some of your setting for 'dpd' and timeouts and it appears to have solved the problem.
How did you arrive at those settings?
Maybe when we see the full config for both sides all will become clear...
Thank you for your effort and we all look forward to seeing the script when you have time to post.

Many thanks

Pete

gargola · Sat Sep 21, 2019 11:41 pm

Hi Pete

Here is the configuration on the router:

/interface bridge
add name="BGP GCP VPN" protocol-mode=none

/ip address
add address=169.254.0.1/30 comment="BGP GCP" interface="BGP GCP VPN" network=169.254.0.0

/ip ipsec profile
add dh-group=modp1024 enc-algorithm=aes-128 lifetime=10h10m name=profile_1 nat-traversal=no proposal-check=exact
/ip ipsec peer
add address=XX.XXX.XX.221/32 exchange-mode=ike2 name=peer2 profile=profile_1
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc lifetime=3h
/ip ipsec identity
add peer=peer2 secret=same-password-here-than-in-gcp
/ip ipsec policy
add dst-address=169.254.0.2/32 level=unique peer=peer2 sa-dst-address=XX.XXX.XX.221 sa-src-address=0.0.0.0 src-address=169.254.0.1/32 tunnel=yes
add dst-address=10.XX.XX.0/29 level=unique peer=peer2 sa-dst-address=XX.XXX.XX.221 sa-src-address=0.0.0.0 src-address=0.0.0.0/0 tunnel=yes

/routing bgp instance
set default as=64512 client-to-client-reflection=no redistribute-connected=yes redistribute-ospf=yes redistribute-static=yes router-id=XXX.XX.1.1
/routing bgp network
add network=169.254.0.0/30 synchronize=no
/routing bgp peer
add hold-time=infinity in-filter=gcp-in name=peer1 out-filter=gcp-out remote-address=169.254.0.2 remote-as=64513 ttl=default

/routing filter
add action=discard chain=gcp-out prefix=XXX.XXX.XX.64 prefix-length=26
add action=discard chain=gcp-out prefix=169.254.0.0 prefix-length=30

In GCP you have to configure a VPN on the Hybrid connectivity section, refer to attached images.
VPN Tunnel
VPN Gateway
Cloud router and BGP sessión

The information is as follows:
169.254.0.0/30 ips to create the bgp sessión .1 is my router .2 is GCP
10.XX.XX.0/29 is the ip segment I have for my servers in GCP named cloud-servers
XX.XXX.XX.221 GCP public ip address
XXX.XXX.XX.64 is my public ip segment my public ip address is XXX.XXX.XX.118
I have an IPSEC policy to reach 10.XX.XX.0/29 from every segment of my network 0.0.0.0/0
The IPSEC lifetimes are the ones specified in GCP documentation
I'm redistributing the OSPF, static and connected routes to GCP via the BGP
64512 is my ASN and 64513 is GCP ASN

Any doubts just let me know, hope it helps on your configuration.

eset · Mon Dec 02, 2019 12:07 pm

@gargola I have few question.

Do you have any connectivity problems? I wrote to MikroTik support because my tunnel sometimes disconnects and when reconnecting which seems all fine some part of networks defined in policy don't work (Ping doesn't goes through). After disable/enable that policy it starts working again

riizzz · Tue Dec 03, 2019 10:57 am

@gargola
i tried your configuration. Our vpn tunnel is up but Bgp session status not. Do you have another config in firewall filter, mange of nat as well?
also how we test vpn connection? i tried pinging 169.254.0.1 from mikrotik (169.254.0.2) no luck. I also tried creating gce instance using same network not working as well.

gargola · Tue Dec 03, 2019 7:36 pm

Try adding these exceptions to your NAT:

/ip firewall nat
add action=accept chain=srcnat comment="IPSEC NAT Exception" dst-address=169.254.0.6 src-address=169.254.0.5
add action=accept chain=srcnat comment="IPSEC NAT Exception" dst-address=10.0.X.X/29

@gargola
i tried your configuration. Our vpn tunnel is up but Bgp session status not. Do you have another config in firewall filter, mange of nat as well?
also how we test vpn connection? i tried pinging 169.254.0.1 from mikrotik (169.254.0.2) no luck. I also tried creating gce instance using same network not working as well.

gargola · Tue Dec 03, 2019 7:39 pm

@eset yes, sometimes and randomly the ipsec tunnel is connected but no traffic passing through, so the bgp goes down. I have to reset everything manually and it comes back, I got almost fixed it setting the timers a google request them on their manual, but still having some random issues. The easiest way I've found that fix the tunnel is killing the active peer.

@gargola I have few question.

Do you have any connectivity problems? I wrote to MikroTik support because my tunnel sometimes disconnects and when reconnecting which seems all fine some part of networks defined in policy don't work (Ping doesn't goes through). After disable/enable that policy it starts working again

edo89b · Tue Dec 31, 2019 11:24 am

Hi Gargola,

You have been very helpful.

I have little knowledge of bgp dynamic routing, as I usually use the ospf protocol.

I'd ask you if you can help me create an IPSEC configuration between a Mikrotik router and GCP, using two connectivity to create an HA solution.

My problem is the BGP part, which if there are both peer activities the traffic doesn't pass and I can't understand where I'm wrong.

Here is the BGP configuration of my Mikrotik:

/routing bgp instance
set default as=64512 client-to-client-reflection=no out-filter=bgp-out \
redistribute-connected=yes redistribute-ospf=yes redistribute-other-bgp=yes \
redistribute-static=yes router-id=169.254.1.2
/routing bgp aggregate
add include-igp=yes instance=default prefix=172.16.100.0/24
add include-igp=yes instance=default prefix=172.16.0.0/24
add include-igp=yes instance=default prefix=10.171.50.0/24
add include-igp=yes instance=default prefix=10.171.54.0/24
add include-igp=yes instance=default prefix=10.171.55.0/24
add include-igp=yes instance=default prefix=172.16.1.0/24
/routing bgp network
add network=169.254.1.0/30 synchronize=no
add network=169.254.4.0/30 synchronize=no
/routing bgp peer
add hold-time=infinity in-filter=gcp-in-1 name=peer1 out-filter=gcp-out-1 \
remote-address=169.254.1.1 remote-as=64513 ttl=default
add disabled=yes hold-time=infinity in-filter=gcp-in-2 name=peer2 out-filter=\
gcp-out-2 remote-address=169.254.1.5 remote-as=64513 ttl=default

Thanks

farhanfarouqi · Fri Jan 10, 2020 7:05 pm

Hello, Can any one send me the script that how to configure Mikrotik router board 750gr3 with google cdn, also told me the google cdn configuration please I am new in this field

gargola · Tue Feb 11, 2020 5:21 am

@edo89b

I'm neither a BGP expert, but there are several ways to indicate to BGP which path to take, like costs in OSPF, in BGP some of them are:

Weight
Local Preference
Network or Aggregate
Shortest AS_PATH
Lowest origin type
Lowest multi-exit discriminator (MED)

Take a look at them and which one is available in GCP.

@farhanfarouqi
Take a look at my posts, all the documentation is there, just modify as needed for your network.

Best regards.

eset · Fri Feb 14, 2020 9:11 pm

@eset yes, sometimes and randomly the ipsec tunnel is connected but no traffic passing through, so the bgp goes down. I have to reset everything manually and it comes back, I got almost fixed it setting the timers a google request them on their manual, but still having some random issues. The easiest way I've found that fix the tunnel is killing the active peer.

@gargola I have few question.

Do you have any connectivity problems? I wrote to MikroTik support because my tunnel sometimes disconnects and when reconnecting which seems all fine some part of networks defined in policy don't work (Ping doesn't goes through). After disable/enable that policy it starts working again

Unfortunately that's what I do also but that's not a solution for unstable IPsec tunnel. So again I believe MikroTik has serious issue with IPsec. That's not a trick to have stable IPsec connection between two mikrotik devices or virtual instances like CHR rather than having stable connection between mikrotik and other provider e.g GCP or AWS (in the second scenario I don't know if there are the same issues).

MikroTik support have ticket opened for this from 2019 and still no solution. The gave me a incomprehensible suggestion to to disable PFS (when Google Requires that) or maybe I don't understand what they had in mind

Emīls Z.04/12/19 09:40:03
Hello,

Please try the previously suggested workarounds - disabling phase 2 lifetime on MikroTik side and disabling pfs group on MikroTik side. IKEv2 protocol has independent rekeying times so they does not have to match on both sides. Also PFS group can be set to none, there is a reference in the GCP documentation regarding it if you insist on following it despite my recommendations.
"If your VPN gateway requires DH settings for Phase 2, use the same settings you used for Phase 1."

But settings PFS to none will obviously end up with this

So I don't know what they suggest and why like so.

rakateps · Thu Mar 05, 2020 11:08 am

Hi Gargola,

Thank you for this sharing, nice to meet this thread. i have job to create connection from my office (on prem) to gcp, my office use mikrotik router for routing activity. But i dont know how to start to make this job, especially from mikrotik. do u have youtube channel or other for visual tutorial? hope u always healthy

bale · Fri Mar 06, 2020 3:31 pm

Hi Gargola,

Firstly thank you for sharing the experience.

For mikrotik side are you used CLI for config or GUI?
Any tutorial link for mikrotiik newbie user like me?

Thanks & Regards

eset · Wed Jun 03, 2020 1:53 pm

@eset yes, sometimes and randomly the ipsec tunnel is connected but no traffic passing through, so the bgp goes down. I have to reset everything manually and it comes back, I got almost fixed it setting the timers a google request them on their manual, but still having some random issues. The easiest way I've found that fix the tunnel is killing the active peer.

@gargola I have few question.

Do you have any connectivity problems? I wrote to MikroTik support because my tunnel sometimes disconnects and when reconnecting which seems all fine some part of networks defined in policy don't work (Ping doesn't goes through). After disable/enable that policy it starts working again

I have news for that subject
MikroTik + GCP VPN using BGP routing - isn't possible.

By possible I mean, having stable tunnel without any interruption we are facing.
BGP requires from GCP a 0.0.0.0/0 both Remote and Local Traffic Selectors on both sides configured in IPsec policy. But according to MikroTik support it's not possible right now.
I asked official GCP Support about that instability which we are facing and after digging logs the notice that I receive each time

"Warning: Remote traffic selectors narrowed for Child SA: vpn_94.xx.xx.xx. Configured TS: [0.0.0.0/0 ], negotiated TS:[172.16.18.0/24 ]. Please verify configuration on the remote side."

and the reason for that is That I'm using BGP as the route method and they pointed that out and advise to set 0.0.0.0/0 (any) on Peer (mikrotik). And tried but when struggling I wrote to MikroTik support and got

Hello,
RouterOS has policy based IPsec only. You can configure 0.0.0.0/0<->0.0.0.0/0 traffic selector, but you will not be able to route specific traffic over the tunnel, so that really is not an option at this time.

So basically the only way to have stable tunnel between GCP and Mikrotik is to switch into Policy Based routing

djdrastic · Wed Jun 03, 2020 2:38 pm

Hello,
RouterOS has policy based IPsec only. You can configure 0.0.0.0/0<->0.0.0.0/0 traffic selector, but you will not be able to route specific traffic over the tunnel, so that really is not an option at this time.
So basically the only way to have stable tunnel between GCP and Mikrotik is to switch into Policy Based routing

Yeah we've been asking for VTI IPSEC for years now . What many guys have done in days past on traditional non cloud networks is run a GRE or IPIP tunnel and setup IPSEC over the endpoints to be able to run a routing protocol.

Mikrotik is not alone with this , Sophos has finally added VTI to their XG appliances after nearly 5 years of waiting and numerous complaints

The reason why vendors either ignore or take hellishly long to implement this stem from what I understand , most of the linux implementations that a lot of network gear uses only has policy ipsec support and requires some sort of patching or customizing the code to allow virtual tunnel endpoints.

gargola · Wed Jun 03, 2020 6:13 pm

Hi eset, djdrastic.

I'm using policy based:

Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default 
 0 T * group=default src-address=0.0.0.0/0 dst-address=0.0.0.0/0 protocol=all proposal=default template=yes 

 1  A  peer=peer2 tunnel=yes src-address=169.254.0.5/32 src-port=any dst-address=169.254.0.6/32 dst-port=any protocol=all action=encrypt 
       level=unique ipsec-protocols=esp sa-src-address=0.0.0.0 sa-dst-address=35.22X.XX.XX proposal=default ph2-count=1 

 2  A  peer=peer2 tunnel=yes src-address=0.0.0.0/0 src-port=any dst-address=10.0.X.0/29 dst-port=any protocol=all action=encrypt level=unique 
       ipsec-protocols=esp sa-src-address=0.0.0.0 sa-dst-address=35.22X.XX.XX proposal=default ph2-count=1

Policy #1 is to route from the addresses of the bgp session, policy #2 is to route from all my network 0.0.0.0/0 to the /29 that I have in GCP.

I'm getting a very stable tunnel, I mean, yeah I have to reset (kill the connection) but is every 3 or more weeks or sometimes it restore without my intervention

eset · Wed Jun 03, 2020 10:01 pm

Hi eset, djdrastic.

I'm using policy based:
Code: Select all
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default 
 0 T * group=default src-address=0.0.0.0/0 dst-address=0.0.0.0/0 protocol=all proposal=default template=yes 

 1  A  peer=peer2 tunnel=yes src-address=169.254.0.5/32 src-port=any dst-address=169.254.0.6/32 dst-port=any protocol=all action=encrypt 
       level=unique ipsec-protocols=esp sa-src-address=0.0.0.0 sa-dst-address=35.22X.XX.XX proposal=default ph2-count=1 

 2  A  peer=peer2 tunnel=yes src-address=0.0.0.0/0 src-port=any dst-address=10.0.X.0/29 dst-port=any protocol=all action=encrypt level=unique 
       ipsec-protocols=esp sa-src-address=0.0.0.0 sa-dst-address=35.22X.XX.XX proposal=default ph2-count=1 
Policy #1 is to route from the addresses of the bgp session, policy #2 is to route from all my network 0.0.0.0/0 to the /29 that I have in GCP.

I'm getting a very stable tunnel, I mean, yeah I have to reset (kill the connection) but is every 3 or more weeks or sometimes it restore without my intervention

But how you are using bgp In policy based ? You can't do that with gcp when you don't define in gcp that you need to do it. After that you need 0.0.0.0/0 (any) in traffic selector. But this isn't possible from mikrotik side with bgp. That what you have is policy based but configured wrong way. You are using 0.0.0.0/0 here when policy based routing requires specific , defined networks on both sides where 0.0.0.0/0 is not an option. On fact if you are using it like that , according to gcp, you are using basically route based method from gcp not policy. That's why you are still facing issue. Define networks, loose bgp , because that doesn't work either, define static routes and the tunnel will be stable. I was there already, someday using Linux peer with strongswan where had policy based and it never had issue with stability.

Wysłane z mojego ONEPLUS A5000 przy użyciu Tapatalka

agungjies · Fri Jun 12, 2020 9:41 pm

im trying to config another topology,
where i installed CHR mikrotik on GCP,
and i tried to connect VPN from my office to CHR
in my Office, there is no one IP Publik, that why i tried to install CHR and use Static IP by GCP

After I got the Good Connection Using OVPN, Im trying to Connect my-subnet to my CHR too to make connection from Head Office to GCP Internal IP

HO===>OVPN TO=====>CHR ON GCP=====>IPSEC TO====> CLOUD VPN=====>GCP INTERNAL IP

but stil problem on phase 2 authentication, any solution for this case and topology?

log note :
no policy found / generated
got fatal error: AUTHENTICATION_FAILED
killing ike2 SA: IP INTERNAL CHR[500]-IP PUBLIK CLOUD VPN[500]

eset · Sat Jun 13, 2020 12:56 am

why you want to make a IPsec from VM with RouterOS on GCP into GCP Cloud VPN? That's sounds stupid.

Google Cloud Platform GCP - VPN - BGP [SOLVED]

Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP help

Re: Google Cloud Platform GCP - VPN - BGP help

Re: Google Cloud Platform GCP - VPN - BGP help

Re: Google Cloud Platform GCP - VPN - BGP help [SOLVED]

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Re: Google Cloud Platform GCP - VPN - BGP

Who is online