IPSEC over GRE - SA installed - but gre interface is down

olof · Tue Oct 27, 2020 7:08 pm

I don't know if this is a configuration error or not, but I am having an issue where I want to setup a ipsec vpn over gre interfaces, that are sourced from loopback interfaces. See attached topology for better understanding. I also attached both side configs.
I am running RouterOS 6.47.6 on RB4011 on both routers.

I expected that if I try to ping other sides loopback address, with source-loopback, it should setup the gre tunnel and I can route over the gre tunnel.
However, the gre interface is never coming up.

[admin@L] > /interface print where name=gre
Flags: D - dynamic, X - disabled, R - running, S - slave 
 #     NAME                                TYPE       ACTUAL-MTU L2MTU  MAX-L2MTU MAC-ADDRESS      
 0     gre                                 gre-tunnel       1300 65535
[admin@L] > /ip route check 172.24.0.2
  status: failed
  
  [admin@L] > /ip ipsec active-peers print  
Flags: R - responder, N - natt-peer 
 #    ID                   STATE              UPTIME          PH2-TOTAL REMOTE-ADDRESS                               DYNAMIC-ADDRESS     
 0 R  1.1.1.2              established        9m5s                    1 1.1.1.2                                     
[admin@L] > /ip ipsec installed-sa print 
Flags: H - hw-aead, A - AH, E - ESP 
 0 HE spi=0x4FB053D src-address=1.1.1.2 dst-address=1.1.1.1 state=mature auth-algorithm=sha1 enc-algorithm=aes-cbc enc-key-size=256 
      auth-key="5291410ef6d7be3fa5197c55ca7fbb0c2a8122bb" enc-key="db58f5eb380e815344ef275c4a4c743c5301c3418b8b5c9175a17f7ab53bd057" 
      add-lifetime=24m4s/30m5s replay=128 

 1 HE spi=0x8C490BB src-address=1.1.1.1 dst-address=1.1.1.2 state=mature auth-algorithm=sha1 enc-algorithm=aes-cbc enc-key-size=256 
      auth-key="a70d7166a1ed4d57dd5371209c143fadc45672b0" enc-key="baa134c797c3d75f09e812730f759f220fd513ebb2d72a470c4dc49b79177e25" 
      add-lifetime=24m4s/30m5s replay=128

See here for this very simple topology

LEFT SIDE CONFIG:
===============

/interface bridge
add name=lo protocol-mode=none
/interface gre
add local-address=10.0.0.1 mtu=1300 name=gre remote-address=10.0.0.2
/ip ipsec peer
add address=1.1.1.2/32 exchange-mode=ike2 name=rb4011
/ip address
add address=1.1.1.1/30 interface=ether1 network=1.1.1.0
add address=192.168.1.1/24 interface=ether5 network=192.168.1.0
add address=10.0.0.1 interface=lo network=10.0.0.1
add address=172.24.0.1/30 interface=gre network=172.24.0.0
/ip ipsec identity
add peer=rb4011 secret=testing1234
/ip ipsec policy
add dst-address=10.0.0.2/32 peer=rb4011 sa-dst-address=1.1.1.2 sa-src-address=1.1.1.1 src-address=10.0.0.1/32 tunnel=yes
/system identity
set name=L

RIGHT SIDE CONFIG:
================

/interface bridge
add name=lo protocol-mode=none
/interface gre
add local-address=10.0.0.2 mtu=1300 name=gre remote-address=10.0.0.1
/ip ipsec peer
add address=1.1.1.1/32 exchange-mode=ike2 name=rb4011
/ip address
add address=1.1.1.2/30 interface=ether1 network=1.1.1.0
add address=192.168.2.1/24 interface=ether5 network=192.168.2.0
add address=10.0.0.2 interface=lo network=10.0.0.2
add address=172.24.0.2/30 interface=gre network=172.24.0.0
/ip ipsec identity
add peer=rb4011 secret=testing1234
/ip ipsec policy
add dst-address=10.0.0.1/32 peer=rb4011 sa-dst-address=1.1.1.1 sa-src-address=1.1.1.2 src-address=10.0.0.2/32 tunnel=yes
/system identity
set name=R

FROM IPSEC LOG:
===============

16:58:19 ipsec,debug 05000000
16:58:19 ipsec initiator selector: 10.0.0.2
16:58:19 ipsec adding payload: TS_I
16:58:19 ipsec,debug => (size 0x18)
16:58:19 ipsec,debug 00000018 01000000 07000010 0000ffff 0a000002 0a000002
16:58:19 ipsec responder selector: 10.0.0.1
16:58:19 ipsec adding payload: TS_R
16:58:19 ipsec,debug => (size 0x18)
16:58:19 ipsec,debug 00000018 01000000 07000010 0000ffff 0a000001 0a000001
16:58:19 ipsec <- ike2 request, exchange: AUTH:1 1.1.1.1[4500] fd9a20f449736340:d654a3db4f90c1b5
16:58:19 ipsec,debug ===== sending 444 bytes from 1.1.1.2[4500] to 1.1.1.1[4500]
16:58:19 ipsec,debug 1 times of 448 bytes message will be sent to 1.1.1.1[4500]
16:58:19 ipsec,debug ===== received 444 bytes from 1.1.1.1[4500] to 1.1.1.2[4500]
16:58:19 ipsec -> ike2 reply, exchange: AUTH:1 1.1.1.1[4500] fd9a20f449736340:d654a3db4f90c1b5
16:58:19 ipsec payload seen: ENC (416 bytes)
16:58:19 ipsec processing payload: ENC
16:58:19 ipsec,debug => iv (size 0x10)
16:58:19 ipsec,debug dc5d00d0 669f2854 69bbc63b fda2e74a
16:58:19 ipsec,debug => plain payload (trimmed) (size 0x84)
16:58:19 ipsec,debug 2700000c 01000000 01010101 2c00001c 02000000 2e1d7e03 720c96bb 2f4e7e08
16:58:19 ipsec,debug a04c4ba7 12ff131f 2d000018 01000000 07000010 0000ffff 0a000002 0a000002
16:58:19 ipsec,debug 21000018 01000000 07000010 0000ffff 0a000001 0a000001 0000002c 00000028
16:58:19 ipsec,debug 01030403 04fb053d 0300000c 0100000c 800e0100 03000008 03000002 00000008
16:58:19 ipsec,debug 05000000
16:58:19 ipsec,debug decrypted
16:58:19 ipsec payload seen: ID_R (12 bytes)
16:58:19 ipsec payload seen: AUTH (28 bytes)
16:58:19 ipsec payload seen: TS_I (24 bytes)
16:58:19 ipsec payload seen: TS_R (24 bytes)
16:58:19 ipsec payload seen: SA (44 bytes)
16:58:19 ipsec processing payloads: NOTIFY (none found)
16:58:19 ipsec ike auth: initiator finish
16:58:19 ipsec processing payload: ID_R
16:58:19 ipsec ID_R (ADDR4): 1.1.1.1
16:58:19 ipsec processing payload: AUTH
16:58:19 ipsec requested auth method: SKEY
16:58:19 ipsec,debug => peer's auth (size 0x14)
16:58:19 ipsec,debug 2e1d7e03 720c96bb 2f4e7e08 a04c4ba7 12ff131f
16:58:19 ipsec,debug => auth nonce (size 0x18)
16:58:19 ipsec,debug 6f42ffb5 0a0445a7 d1a4a3cb aa3fcdb5 99949078 a82f6586
16:58:19 ipsec,debug => SK_p (size 0x14)
16:58:19 ipsec,debug 112804de 2a6b0f89 63111ccf 7ac01e03 c50a57cb
16:58:19 ipsec,debug => idhash (size 0x14)
16:58:19 ipsec,debug 1dc7d6a2 a4f3ecfa 25b4f5f1 e2547caa fa73ff08
16:58:19 ipsec,debug => calculated peer's AUTH (size 0x14)
16:58:19 ipsec,debug 2e1d7e03 720c96bb 2f4e7e08 a04c4ba7 12ff131f
16:58:19 ipsec,info,account peer authorized: 1.1.1.2[4500]-1.1.1.1[4500] spi:fd9a20f449736340:d654a3db4f90c1b5
16:58:19 ipsec processing payloads: NOTIFY (none found)
16:58:19 ipsec peer selected tunnel mode
16:58:19 ipsec processing payload: TS_I
16:58:19 ipsec 10.0.0.2
16:58:19 ipsec processing payload: TS_R
16:58:19 ipsec 10.0.0.1
16:58:19 ipsec my vs peer's selectors:
16:58:19 ipsec 10.0.0.2 vs 10.0.0.2
16:58:19 ipsec 10.0.0.1 vs 10.0.0.1
16:58:19 ipsec processing payload: SA
16:58:19 ipsec IKE Protocol: ESP
16:58:19 ipsec  proposal #1
16:58:19 ipsec   enc: aes256-cbc
16:58:19 ipsec   auth: sha1
16:58:19 ipsec matched proposal:
16:58:19 ipsec  proposal #1
16:58:19 ipsec   enc: aes256-cbc
16:58:19 ipsec   auth: sha1
16:58:19 ipsec,debug => child keymat (size 0x78)
16:58:19 ipsec,debug db58f5eb 380e8153 44ef275c 4a4c743c 5301c341 8b8b5c91 75a17f7a b53bd057
16:58:19 ipsec,debug 5291410e f6d7be3f a5197c55 ca7fbb0c 2a8122bb baa134c7 97c3d75f 09e81273
16:58:19 ipsec,debug 0f759f22 0fd513eb b2d72a47 0c4dc49b 79177e25 a70d7166 a1ed4d57 dd537120
16:58:19 ipsec,debug 9c143fad c45672b0 5a42c6fd 7c6a529e 6316f6cd 87460313
16:58:19 ipsec IPsec-SA established: 1.1.1.1[4500]->1.1.1.2[4500] spi=0x8c490bb
16:58:19 ipsec IPsec-SA established: 1.1.1.2[4500]->1.1.1.1[4500] spi=0x4fb053d

xvo · Tue Oct 27, 2020 10:21 pm

As you are wrapping gre in ipsec you need tunnel=no in ipsec policy.

sindy · Tue Oct 27, 2020 11:17 pm

There are two issues, none of which is obvious.

the one which prevents the setup from working on your table is that in order that a packet could be matched by IPsec policy's traffic selector, it must first be routed using the "normal" routing. It is not important what output interface is chosen as it won't be actually used, but there must be some. The IPsec policy matching is the last thing to be done just before the packet would have been sent out the chosen interface. So you need to add some route towards 10.0.0.2 at left router, and some route towards 10.0.0.1 at right router. You can even use the lo bridge as that route's gateway, it will still work.
in real life deployment with firewall rules, there is another issue, which popped up after some security patch related to GRE. Since then, a GRE packet not belonging to an established connection is labeled as connection-state=invalid, so the default firewall rules which drop connection-state=invalid packets prevents the GRE connections from establishing. Plus there is the specific keepalive method used in GRE, where the active party creates a GRE transport packet with no payload which the passive party would send to the active one, and sends this GRE transport packet as a payload of its own GRE transport packet to the passive party. So the passive party de-capsulates the GRE payload from the received GRE packet and forwards it as any other packet, so the active party receives that GRE transport packet with no payload back. At the passive party, the in-interface for this return packet is the GRE interface, so the firewall rules in filter/forward must allow GRE packets coming in via GRE interfaces to be forwarded. So in both the input and the forward chain of filter in the default firewall rules, protocol=!gre must be added to the "drop invalid" rules.

@xvo's remark would make sense if you used 1.1.1.1 and 1.1.1.2 as GRE's local-address and remote-address; in your setup, you do need the tunnel=yes mode.

One more remark, in my understanding, "voice over IP" means that voice is the payload and IP is the transport. Similarly, IPsec over GRE means to me that you use GRE to transport IPsec as a payload, but your configuration shows the reverse, you use IPsec to transport (and encrypt) GRE as a payload.

xvo · Wed Oct 28, 2020 12:10 am

@xvo's remark would make sense if you used 1.1.1.1 and 1.1.1.2 as GRE's local-address and remote-address; in your setup, you do need the tunnel=yes mode.

Indeed...

olof · Wed Oct 28, 2020 2:01 am

There are two issues, none of which is obvious.
the one which prevents the setup from working on your table is that in order that a packet could be matched by IPsec policy's traffic selector, it must first be routed using the "normal" routing. It is not important what output interface is chosen as it won't be actually used, but there must be some. The IPsec policy matching is the last thing to be done just before the packet would have been sent out the chosen interface. So you need to add some route towards 10.0.0.2 at left router, and some route towards 10.0.0.1 at right router. You can even use the lo bridge as that route's gateway, it will still work.

in real life deployment with firewall rules, there is another issue, which popped up after some security patch related to GRE. Since then, a GRE packet not belonging to an established connection is labeled as connection-state=invalid, so the default firewall rules which drop connection-state=invalid packets prevents the GRE connections from establishing. Plus there is the specific keepalive method used in GRE, where the active party creates a GRE transport packet with no payload which the passive party would send to the active one, and sends this GRE transport packet as a payload of its own GRE transport packet to the passive party. So the passive party de-capsulates the GRE payload from the received GRE packet and forwards it as any other packet, so the active party receives that GRE transport packet with no payload back. At the passive party, the in-interface for this return packet is the GRE interface, so the firewall rules in filter/forward must allow GRE packets coming in via GRE interfaces to be forwarded. So in both the input and the forward chain of filter in the default firewall rules, protocol=!gre must be added to the "drop invalid" rules.
One more remark, in my understanding, "voice over IP" means that voice is the payload and IP is the transport. Similarly, IPsec over GRE means to me that you use GRE to transport IPsec as a payload, but your configuration shows the reverse, you use IPsec to transport (and encrypt) GRE as a payload.

Ok, thanks for your detailed answer.
Regarding issue 1, I did not have any routes in my config besides the directly connected interfaces. So it will be a non-issue once they go live and get a default route.It even worked to have a route to an unreachable IP.

And regarding issue #2, is this a security patch in mikrotik specifically?

I will test more tomorrow. I want to see ipsec performance.
do you think there will be a differnce in performance using "gre over ipsec" or "ipsec over gre"? ipsec over gre can be non-tunneled?

@sindy Is there a prefered way of doing this?
I will use this type of setup to connect a mikrotik to a (vyos) router. The mikrotik might have dynamic IP and might have a private IP. And vyos has static IP. So far I have done the gre over ipsec, and I have used my-id to identify the mikrotik device behind NAT. also will need to have ipsec tunnel working after mikrotik has failed over to a another wan connection. so I guess the ipsec over gre setup will never work for me as you need static public ips for gre to come up

sindy · Wed Oct 28, 2020 8:51 am

So it will be a non-issue once they go live and get a default route. It even worked to have a route to an unreachable IP.

Yes. If the policies are statically configured, a default route is enough. If the policy was generated dynamically at one end, the traffic could leak down the default route in plaintext while no policy would exist while the IPsec "session" would be down for some reason.

regarding issue #2, is this a security patch in mikrotik specifically?

I don't remember the CVE it addresses, it was some mid-6.45 release which has changed the behaviour and started labeling initial packets of GRE connections as invalid ones. I suppose the CVE was a netfilter issue, not a Mikrotik specific one; the patch itself may be a Mikrotik specific one, though.

do you think there will be a differnce in performance using "gre over ipsec" or "ipsec over gre"? ipsec over gre can be non-tunneled?
...
so I guess the ipsec over gre setup will never work for me as you need static public ips for gre to come up

Definitely GRE encrypted using IPsec is the right way to go. It was just the mismatch between the description and the actual configuration that made me comment on it. And yes, bare GRE through NAT only works under specific conditions.

olof · Wed Oct 28, 2020 11:57 am

Definitely GRE encrypted using IPsec is the right way to go. It was just the mismatch between the description and the actual configuration that made me comment on it. And yes, bare GRE through NAT only works under specific conditions.

Ok good to know. Because this also allows me my dynamic setups with failover on a LTE modem.

For future reference, I am posting RB4011 to RB4011 GRE over IPsec performance test results.
I was quite surprised over my results!
I suppose 1414 MTU on gre interface is correct considering I have 1500MTU on physical interface?

Baselining ethernet traffic (no ipsec)

[admin@R] /interface> /tool bandwidth-test address=1.1.1.1 direction=both          
                status: running
              duration: 18s
            tx-current: 977.8Mbps
  tx-10-second-average: 975.2Mbps
      tx-total-average: 974.8Mbps
            rx-current: 975.3Mbps
  rx-10-second-average: 976.9Mbps
      rx-total-average: 972.8Mbps
          lost-packets: 71076
           random-data: no
             direction: both
               tx-size: 1500
               rx-size: 1500
      connection-count: 20
        local-cpu-load: 35%
       remote-cpu-load: 34%

Oneway iperf3 with 4 parallel streams between two laptops, connected to ether5. almost 900Mbps. 9,6% CPU usage by ipsec!

...
[SUM]   0.00-10.00  sec  1.04 GBytes   892 Mbits/sec                  receiver

ethernet                            4.8%
console                               0%
networking                         27.2%
ipsec                               9.6%
management                            0%
unclassified                        8.2%
total                              49.8%

Bandwidth test between the two routers, via the gre over ipsec link. VERY good results in both directions simultaneously!

[admin@R] /interface> /tool bandwidth-test address=172.24.0.1 direction=both 
                status: running
              duration: 35s
            tx-current: 803.8Mbps
  tx-10-second-average: 796.1Mbps
      tx-total-average: 755.4Mbps
            rx-current: 830.0Mbps
  rx-10-second-average: 834.9Mbps
      rx-total-average: 817.4Mbps
          lost-packets: 8301
           random-data: no
             direction: both
               tx-size: 1414
               rx-size: 1414
      connection-count: 20
        local-cpu-load: 80%
       remote-cpu-load: 80%

Various tests with smaller packets in both directions.


[admin@R] /interface> /tool bandwidth-test address=172.24.0.1 protocol=udp local-udp-tx-size=64 remote-udp-tx-size=64 direction=both 
                status: running
              duration: 19s
            tx-current: 41.2Mbps
  tx-10-second-average: 40.8Mbps
      tx-total-average: 36.6Mbps
            rx-current: 41.4Mbps
  rx-10-second-average: 41.9Mbps
      rx-total-average: 38.6Mbps
          lost-packets: 10071
           random-data: no
             direction: both
               tx-size: 64
               rx-size: 64
      connection-count: 20
        local-cpu-load: 76%
       remote-cpu-load: 78%
[admin@R] /interface> /tool bandwidth-test address=172.24.0.1 protocol=udp local-udp-tx-size=500 remote-udp-tx-size=500 direction=both 
                status: running
              duration: 42s
            tx-current: 317.6Mbps
  tx-10-second-average: 316.6Mbps
      tx-total-average: 310.1Mbps
            rx-current: 311.3Mbps
  rx-10-second-average: 310.2Mbps
      rx-total-average: 296.6Mbps
          lost-packets: 8575
           random-data: no
             direction: both
               tx-size: 500
               rx-size: 500
      connection-count: 20
        local-cpu-load: 79%
       remote-cpu-load: 76%
[admin@R] /interface> /tool bandwidth-test address=172.24.0.1 protocol=udp local-udp-tx-size=1000 remote-udp-tx-size=1000 direction=both 
                status: running
              duration: 31s
            tx-current: 577.1Mbps
  tx-10-second-average: 569.8Mbps
      tx-total-average: 549.6Mbps
            rx-current: 566.4Mbps
  rx-10-second-average: 566.8Mbps
      rx-total-average: 543.3Mbps
          lost-packets: 8137
           random-data: no
             direction: both
               tx-size: 1000
               rx-size: 1000
      connection-count: 20
        local-cpu-load: 79%
       remote-cpu-load: 78%

very good!

IPSEC over GRE - SA installed - but gre interface is down [SOLVED]

IPSEC over GRE - SA installed - but gre interface is down

Re: IPSEC over GRE - SA installed - but gre interface is down

Re: IPSEC over GRE - SA installed - but gre interface is down [SOLVED]

Re: IPSEC over GRE - SA installed - but gre interface is down

Re: IPSEC over GRE - SA installed - but gre interface is down

Re: IPSEC over GRE - SA installed - but gre interface is down

Re: IPSEC over GRE - SA installed - but gre interface is down

Who is online