Community discussions

MikroTik App
 
zett93
newbie
Topic Author
Posts: 36
Joined: Mon Feb 10, 2020 3:42 pm

Problem with 2 IPSec IKE2 tunnels to GCloud

Thu Nov 10, 2022 1:49 pm

Hey,

I have a problem as described below:
  • I have two regions in gcloud with two different addresses (192.xx.xx.0/24 and 192.yy.yy.0/24)
  • a tunnel (IKE2) is created for each of these addresses
  • each of the tunnels works on my router, however, when I want to run two at the same time, traffic stops going through one of them
  • in the /ipsec/policy tab all tunnels have the status "established"
  • this is not likely a problem on the gcloud side, as I have the same tunnels set up on fortigate in another location
  • it is also not a problem of RouterOS version (originally 6.47.x, then 6.49.x, now 7.6)
Have you perhaps encountered similar problems? Below I paste the anonymized configuration of the tunnels. IMO it looks like it's losing some static routes, but I can't trace what I'm doing wrong. Can you guys help? :)
# nov/10/2022 12:02:22 by RouterOS 7.6
# software id = QCX6-3PXK
#
# model = RB750Gr3
# serial number = xxxxxxxxxxxx

/ip firewall nat
add action=accept chain=srcnat comment="Accept traffic to Gcloud tunnel" dst-address-list=GCLOUD-LAN \
    src-address-list="LAN NETWORKS" 
add action=accept chain=srcnat comment="Accept traffic from Gcloud tunnel" dst-address-list="LAN NETWORKS" \
    src-address-list=GCLOUD-LAN

/ip ipsec profile
set [ find default=yes ] dh-group=ecp256,modp2048,modp1024 enc-algorithm=\
    aes-256,aes-192,aes-128,3des
add dh-group=modp2048 enc-algorithm=aes-256 hash-algorithm=sha256 name=\
    gcloud_profile

/ip ipsec peer
add address=xx.xx.xx.xx/xx exchange-mode=ike2 local-address=ll.ll.ll.ll/ll \
    name=gcloud_peer_fra profile=gcloud_profile send-initial-contact=no
add address=yy.yy.yy.yy/yy exchange-mode=ike2 local-address=ll.ll.ll.ll/ll \
    name=gcloud_peer profile=gcloud_profile send-initial-contact=no
    
/ip ipsec proposal
add auth-algorithms=sha256 enc-algorithms=aes-256-cbc lifetime=8h name=\
    gcloud_proposal pfs-group=modp2048
    
/ip ipsec identity
add peer=gcloud_peer auth-method=pre-shared-key generate-policy=no #commented
add peer=gcloud_peer_fra auth-method=pre-shared-key generate-policy=no #commented

/ip ipsec policy
add dst-address=192.xx.xx.00/24 peer=\
    gcloud_peer proposal=gcloud_proposal src-address=192.ll.ll.ll/24 tunnel=\
    yes
add dst-address=192.yy.yy.0/24 peer=\
    gcloud_peer_fra proposal=gcloud_proposal src-address=192.ll.ll.ll/24 \
    tunnel=yes
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Problem with 2 IPSec IKE2 tunnels to GCloud

Thu Nov 10, 2022 5:05 pm

Nothing in your configuration seems strange to me, and multiple connections from the same local IP address to multiple distinct remote peers are nothing unusual too.

Is the Mikrotik connected to internet directly or via some firewall (or even NAT) device? I can only imagine a firewall device to have some issues with handling bare ESP - if so, forcing NAT into the path might help.

And just as a blind shot, try setting level=unique for the policies.
 
zett93
newbie
Topic Author
Posts: 36
Joined: Mon Feb 10, 2020 3:42 pm

Re: Problem with 2 IPSec IKE2 tunnels to GCloud

Thu Nov 10, 2022 6:12 pm

Thanks for the reply @sindy

I know it's nothing unusual (on this Mikrotik I have 4 more tunnels to other locations and with them there was no problem, the only difference is exchange-mode=main).

This router has a direct connection to the internet (via PPPoE).

An interesting fact is that if I have both connections raised (gcloud_peer + gcloud_peer_fra), the traffic only goes to the gcloud_peer peer address, while if I stop it, the traffic appears on the gcloud_peer_fra peer. The other tunnels work without any interruption and do not lose any packets.

As for the firewall rules. There is nothing special there (esp and ah have accepts and are at the very top of the stack), in NAT similarly (address lists of these tunnels have accepts in src-nat)

Unfortunately changing level=unique did not bring any change :(

This router has quite a long configuration to paste it in its entirety (a lot of things to hide) - so I don't know if it makes sense to process and paste it all, but please let me know, I'll prepare something

Look at the screens below:
You do not have the required permissions to view the files attached to this post.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Problem with 2 IPSec IKE2 tunnels to GCloud

Thu Nov 10, 2022 6:27 pm

Disable both peers (or identities), then enable them again, to clean up all 4 SAs and let them be recreated. Start pinging both destination networks as you did (it doesn't matter that one of the pings will not work).

Then show me the output of the following (of course you can obfuscate the public IPs):

/ip ipsec active-peers print (only the rows for the relevant peers are interesting)

/ip ipsec installed-sa print where src-address~"ip.of.peer.1|ip.of.peer.2" or dst-address~"ip.of.peer.1|ip.of.peer.2" (again, you can obfuscate the public IPs; no point in obfuscating the keys as they are ephemeral and if you disable the peers/identities again before posting, these values cannot be misused).
 
zett93
newbie
Topic Author
Posts: 36
Joined: Mon Feb 10, 2020 3:42 pm

Re: Problem with 2 IPSec IKE2 tunnels to GCloud

Thu Nov 10, 2022 6:42 pm

It looks like this
/ip ipsec active-peers print
Flags: R - RESPONDER; N - NATT-PEER
Columns: ID, STATE, UPTIME, PH2-TOTAL, REMOTE-ADDRESS
6 R  ip-gloud_peer_fra  established  24s               1  ip-gloud_peer_fra 
7 R  ip-gcloud_peer     established  24s               1  ip-gcloud_peer 


/ip ipsec installed-sa print where....
Flags: S - SEEN-TRAFFIC; H - HW-AEAD; E - ESP
Columns: SPI, STATE, SRC-ADDRESS, DST-ADDRESS, AUTH-ALGORITHM, ENC-ALGORITHM, ENC-KEY-SIZE
#     SPI         STATE   SRC-ADDRESS     DST-ADDRESS     AUTH-ALGORITHM  ENC-ALGORITHM  ENC-KEY-SIZE
0 SHE 0x2034A30   mature  ip-gloud_peer_fra    ip-local_router      sha256          aes-cbc                 256
1 SHE 0xF10FDBF8  mature  ip-local_router      ip-gloud_peer_fra    sha256          aes-cbc                 256
2 SHE 0x41CB194   mature  ip-gcloud_peer       ip-local_router      sha256          aes-cbc                 256
3 SHE 0x60BC48C8  mature  ip-local_router      ip-gcloud_peer       sha256          aes-cbc                 256
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Problem with 2 IPSec IKE2 tunnels to GCloud

Thu Nov 10, 2022 6:54 pm

Sorry, another time, but with /ip/ipsec/installed-sa/print detail ... - I didn't know this was necessary in ROS 7 to see the number of bytes and packets handled by the SA.
 
zett93
newbie
Topic Author
Posts: 36
Joined: Mon Feb 10, 2020 3:42 pm

Re: Problem with 2 IPSec IKE2 tunnels to GCloud

Thu Nov 10, 2022 7:22 pm

Check now :)
Flags: S - seen-traffic; H - hw-aead; A - AH, E - ESP 
 0 SHE spi=0x2034A30 src-address=ip-gloud_peer_fra dst-address=ip-local_router state=mature auth-algorithm=sha256 
       enc-algorithm=aes-cbc enc-key-size=256 auth-key="05144e43ff7ba0e7a5541fea953cb8b7d9732a66ab8138c77f0a2acc2e39c627" 
       enc-key="12c9ca6799bd332d70a5568f5664f008b26c40474824d55c7fc2cd5485da7350" addtime=nov/10/2022 17:35:30 
       expires-in=7h14m42s add-lifetime=6h24m10s/8h13s current-bytes=84 current-packets=1 replay=128 

 1 SHE spi=0xF10FDBF8 src-address=ip-local_router dst-address=ip-gloud_peer_fra state=mature auth-algorithm=sha256 
       enc-algorithm=aes-cbc enc-key-size=256 auth-key="564867967d018d46dd2249d07f7c1af22ea74ddf80a35ee265dae357c8164f54" 
       enc-key="684df450f2ea30cf927fdf318c3f91b9b38be52e370d4cd9fbb2244adc421ddd" addtime=nov/10/2022 17:35:30 
       expires-in=7h14m42s add-lifetime=6h24m10s/8h13s current-bytes=37800 current-packets=450 replay=128 

 2 SHE spi=0x41CB194 src-address=ip-gcloud_peer dst-address=ip-local_router state=mature auth-algorithm=sha256 
       enc-algorithm=aes-cbc enc-key-size=256 auth-key="6865794d34862ecf6127b24dfdb95d6dd6673fd37ba2f6780c820e9f970e43d5" 
       enc-key="0ed13a7633b73c4c69412c4ffd3ebd8043bf8230c20a99582797607e40e84422" addtime=nov/10/2022 17:35:30 
       expires-in=7h14m42s add-lifetime=6h24m10s/8h13s current-bytes=73920 current-packets=880 replay=128 

 3 SHE spi=0x60BC48C8 src-address=ip-local_router dst-address=ip-gcloud_peer state=mature auth-algorithm=sha256 
       enc-algorithm=aes-cbc enc-key-size=256 auth-key="fc5a1d291c74ebf56590ddde9233c83f328bff7a84fb9cca56ce17e3e18e18e4" 
       enc-key="ff21a9fb63cf29c8971e0a4e1feffac5ea2c3e6ef4227acc9287d54a8b7468a1" addtime=nov/10/2022 17:35:30 
       expires-in=7h14m42s add-lifetime=6h24m10s/8h13s current-bytes=37632 current-packets=448 replay=128 
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Problem with 2 IPSec IKE2 tunnels to GCloud

Thu Nov 10, 2022 7:41 pm

That's what I was assuming - the first one, with src-address=ip-gloud_peer_fra, shows only one packet transported, while the two with src-address=ip-local_router show about 450 packets each, whereas the one with src-address=ip-gcloud_peer shows 880 packets (so roughly twice 450) with an average size of 84 bytes.

To double-check, disable and re-enable the peers again, ping only the private subnet server by gcloud peer fra for a minute or so, and then check the same output of /ip/ipsec/installed-sa print detail .... If my assumption is correct, you'll see non-0 packet counts for the SA with src-address=ip-local_router dst-address=ip-gloud_peer_fra (correct) and with src-address=ip-gcloud_peer dst-address=ip-local_router (incorrect).
 
zett93
newbie
Topic Author
Posts: 36
Joined: Mon Feb 10, 2020 3:42 pm

Re: Problem with 2 IPSec IKE2 tunnels to GCloud

Thu Nov 10, 2022 8:31 pm

After reset peers and pinging only gcloud_peer_fra it looks like this.

Ok, did You have any ideas how to resolve that problem? I see something like that first time in my carreer.
Flags: S - seen-traffic; H - hw-aead; A - AH, E - ESP 
12  HE spi=0xD2F455E src-address=ip-gcloud_peer_fra dst-address=ip-local_router state=mature auth-algorithm=sha256 
       enc-algorithm=aes-cbc enc-key-size=256 auth-key="76fd09b74e762b466e045a76418035f0860e873af2959a1e724504b66bf1763a" 
       enc-key="90d0be1e0a3630c714089b57216fe66a92b4d0d12ba610ebdb674ae94c68377f" add-lifetime=6h24m12s/8h16s replay=128 

13  HE spi=0xFFAEFD80 src-address=ip-local_router dst-address=ip-gcloud_peer_fra state=mature auth-algorithm=sha256 
       enc-algorithm=aes-cbc enc-key-size=256 auth-key="62389175bda08cac34571e4b4e2cbfbe0ac509aceec2983ed47898a845536c75" 
       enc-key="c071a500f746ca0f29cc65f594bbaed93bc454e693a4600598422fe0339b6cb8" add-lifetime=6h24m12s/8h16s replay=128 

14  HE spi=0xFC32657 src-address=ip-gcloud_peer_fra dst-address=ip-local_router state=mature auth-algorithm=sha256 
       enc-algorithm=aes-cbc enc-key-size=256 auth-key="6bc87b3ff54074736275add3c3511624e07a9449884e384391d5b009d382666e" 
       enc-key="5ab7ae6bc693206f7623ba1a0cdf598543d6af477586f19d9a0e4d80df189240" add-lifetime=6h24m17s/8h22s replay=128 

15 SHE spi=0x57FBCAC9 src-address=ip-local_router dst-address=ip-gcloud_peer_fra state=mature auth-algorithm=sha256 
       enc-algorithm=aes-cbc enc-key-size=256 auth-key="0985a1e12dc87840ced63c69456a6be9c65b16ba76e8f2363a723caa447520ee" 
       enc-key="6b35ddbbac74142864babf1135844be8907f3e689d3af9911b9945678c7e959e" addtime=nov/10/2022 19:14:20 
       expires-in=7h49m41s add-lifetime=6h24m17s/8h22s current-bytes=53004 current-packets=631 replay=128 

16 SHE spi=0xABB29F4 src-address=ip-gcloud_peer dst-address=ip-local_router state=mature auth-algorithm=sha256 
       enc-algorithm=aes-cbc enc-key-size=256 auth-key="91b6cd02766eebb73fbdffee6c0993db8c34367de0d4ee4b0e20fc64def93e02" 
       enc-key="e3123d8ee5df5632a666e6cc7ddc1455851c597520dc940b2db5eb2ca7273d27" addtime=nov/10/2022 19:14:20 
       expires-in=7h49m43s add-lifetime=6h24m19s/8h24s current-bytes=51912 current-packets=618 replay=128 

17  HE spi=0x150984D0 src-address=ip-local_router dst-address=ip-gcloud_peer state=mature auth-algorithm=sha256 
       enc-algorithm=aes-cbc enc-key-size=256 auth-key="60bcd5a820c0532125828de3a87529bc6dbec5ba908694dca29547d8c9d3c204" 
       enc-key="d52393fb87961c8cfd824640c6fc7cd868d8d99b819a6dd0db3921992a614c76" add-lifetime=6h24m19s/8h24s replay=128 
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Problem with 2 IPSec IKE2 tunnels to GCloud

Thu Nov 10, 2022 9:10 pm

I see something like that first time in my carreer.
Don't worry, so do I. Maybe our careers are just too short so far :)

did You have any ideas how to resolve that problem?
It seems like a bug to me, so I'd first talk to Gcloud support. But you may try to adjust to that bug by configuring both policies with peer=gcloud_peer,gcloud_peer_fra. In theory, both policies will get negotiated with gcloud_peer if both peers are available, and if communication to gcloud_peer eventually gets lost, they will negotiate with ip-gcloud_peer_fra instead (and stay there even if gcloud_peer becomes available again, until gcloud_peer_fra eventually fails).

But I've only tried this method where the remote peers were creating the policies dynamically at their end, so it may fail miserably here.
 
zett93
newbie
Topic Author
Posts: 36
Joined: Mon Feb 10, 2020 3:42 pm

Re: Problem with 2 IPSec IKE2 tunnels to GCloud

Thu Nov 10, 2022 9:45 pm

If both tunnels were in the same region, that could be a solution. On the other hand, one tunnel compiles to Poland, the other to Germany, so perhaps this is the bug....

Especially since I have exactly these two subnets tied up in another location using Fortigate - no problem there....
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Problem with 2 IPSec IKE2 tunnels to GCloud

Thu Nov 10, 2022 11:54 pm

The fact that the two tunnels are intended for two different countries did not prevent Gcloud from sending data from Germany via the tunnel built for Poland. So inside their network something ignores the regional partitioning. That's what I had in mind when saying you have to adjust your side to their buggy behaviour.
 
zett93
newbie
Topic Author
Posts: 36
Joined: Mon Feb 10, 2020 3:42 pm

Re: Problem with 2 IPSec IKE2 tunnels to GCloud

Fri Nov 11, 2022 11:11 am

I changed the rule in google vpn, now the policy includes both subnets (even though the GUI doesn't suggest at all that you can enter a subnet from a different region), threw out the google_fra config completely, added that subnet in the policy using google_peer and now everything works fine. Finally!

Thank you very much for the few messages mentioned, which finally helped to fix my problem :)

Take care!

Who is online

Users browsing this forum: ccrsxx, GoogleOther [Bot], nichky, nickhoulton, onnyloh, outtahere, rolling and 68 guests