Community discussions

MikroTik App
 
TedRule
just joined
Topic Author
Posts: 15
Joined: Tue Nov 03, 2020 4:41 pm

IPSEC SA DELETE issue with MikroTik <-> Cisco ASA VPN

Wed Apr 14, 2021 4:37 pm

I've been chasing down an intermittent issue which occurs during the renewal of Child SA's on an IPSEC VPN between a MikroTik ( running latest stable v6.48.1 ) and a Cisco ASA running
some v9.1 code.

The problem seems to occur that once the MikroTik issues the DELETE message to the ASA, it may or may not succeed, whereafter it retries, and retries several times, and eventually tears down the Phase1 SA and completely rebuilds the connection with all the Child SAs. At the moment, this seems to break Child SA renewal more than 50% of the time.

The brief snippet of a Syslog of this happening at the MikroTik end shows the "request collision" and the TEMPORARY_FAILURE, whereafter it's all downhill.

Apr 14 13:35:55 MikroTik-LogServer mikrotik-ipsec-info: IPsec-SA established: ASA-endpoint[4500]->MikroTik-endpoint[4500] spi=0x3d0e1b2
Apr 14 13:35:55 MikroTik-LogServer mikrotik-ipsec-info: IPsec-SA established: MikroTik-endpoint[4500]->ASA-endpoint[4500] spi=0x2f1a09ee
Apr 14 13:35:58 MikroTik-LogServer mikrotik-ipsec-info: adding payload: DELETE
Apr 14 13:35:58 MikroTik-LogServer mikrotik-ipsec-info: <- ike2 request, exchange: INFORMATIONAL:3 ASA-endpoint[4500] e446e17750e9e4b6:eb24748934fe2277
Apr 14 13:36:01 MikroTik-LogServer mikrotik-ipsec-info: -> ike2 request, exchange: INFORMATIONAL:65 ASA-endpoint[4500] e446e17750e9e4b6:eb24748934fe2277
Apr 14 13:36:01 MikroTik-LogServer mikrotik-ipsec-info: payload seen: ENC
Apr 14 13:36:01 MikroTik-LogServer mikrotik-ipsec-info: processing payload: ENC
Apr 14 13:36:01 MikroTik-LogServer mikrotik-ipsec-info: request collision
Apr 14 13:36:01 MikroTik-LogServer mikrotik-ipsec-info: reply notify: TEMPORARY_FAILURE
Apr 14 13:36:01 MikroTik-LogServer mikrotik-ipsec-info: adding notify: TEMPORARY_FAILURE
Apr 14 13:36:01 MikroTik-LogServer mikrotik-ipsec-info: <- ike2 reply, exchange: INFORMATIONAL:65 ASA-endpoint[4500] e446e17750e9e4b6:eb24748934fe2277


Meanwhile, amongst a ASA debug trace ( debug crypto ikev2 protocol 2 ) we see this failure to parse the DELETE message.
....
IKEv2-PROTO-2: (16): Received Packet [From MikroTik-endpoint:4500/To ASA-endpoint:4500/VRF i0:f0]
(16): Initiator SPI : E446E17750E9E4B6 - Responder SPI : EB24748934FE2277 Message id: 3
(16): IKEv2 INFORMATIONAL Exchange REQUEST(16):
Payload contents:
IKEv2-PROTO-1: Sum of decrypted payload lengths 12 is not equal to the total length of the decrypted payload 268
(16):
IKEv2-PROTO-1: (16): Failed to decrypt an encrypted payload
IKEv2-PROTO-1: (16): Failed to parse the packet
....

and elsewhere in the retry sequence, we see:

...
KEv2-PROTO-1: Sum of decrypted payload lengths 8 is not equal to the total length of the decrypted payload 264
....
IKEv2-PROTO-1: Sum of decrypted payload lengths 12 is not equal to the total length of the decrypted payload 268
....

What is highly suspicious to my mind is that 268 - 12 == 264 - 8 == 256.

Now the DELETE message is seeming created and sent and misparsed by the ASA BEFORE the ASA issues its next KeepAlive ( msg-id 65 ), and
hence the MikroTik must be issuing a DELETE message which is badly formatted in some manner. It's still possible that the ASA won't
understand the ensuing TEMPORARY_FAILURE message if it doesn't support IKEv2bis, but the rot appears to start with a bug on the MikroTik.
It may even be the some IKEv2 agents fail to correctly check the total length, and hence this bug is never seen on Mik <-> Mik VPNs.

Within the relevant RFC7296, Section 3.1 and 3.2, we see that the IKEv2 Header contains an overall length, whilst each payload contains a payload length field.

As I see it at the moment, the problem would be entirely explained if the IKEv2 Header Length field is "randomly" mis-set to 256 + Payload lengths, but possibly only
when the overall message contains a DELETE payload.

Is this a known issue?

Ted
 
TedRule
just joined
Topic Author
Posts: 15
Joined: Tue Nov 03, 2020 4:41 pm

Re: IPSEC SA DELETE issue with MikroTik <-> Cisco ASA VPN

Thu Apr 15, 2021 3:47 pm

For a little bit more insight, I've turned up the debug wick at both ends of the VPN and I've managed to capture an instance of what I perceive to be the broken DELETE message:

From what I can deduce so far, the unencrypted DELETE payload contains 12 bytes as logged by the MikroTik, i.e.:

................................... 0000000c
03040001 0ad443d2

The overall payload has msg-id 3 and size 0x28 == decimal 40 as shown in the ".... 00000003 00000028" entries in the packet debug.
The encrypted payload has the same msg-id but size 0x12c == decimal 300

Meanwhile, the ASA shows an overall message size of 300 and even a decrypted payload length of 12, but notes the error of a mismatch with the total length
of the decrypted payloads of 268, although I'm not quite sure what fields it's using to come to that conclusion.

Looking at the MikroTik log in more detail, it's apparent that the DELETE message is malparsed by the ASA whenever the overall encrypted payload size exceeds 255,
and as best I can judge, the DELETE message parses Ok on the ASA for all sizes < 256.

The other weird thing which I don't understand, but maybe someone with a better understanding of the underlying encryption protocol can explain, is that the DELETE payload
seems to have an unencrypted size of either 0x24 or 0x28 - which presumably corresponds to a payload body of either 8 or 12 bytes in size. Wierder still is that the size
of the encrypted payload takes on several sizes - so far I've seen 9 different sizes as below with varying frequency. Can anyone explain why the encrypted payload size
varies so much?

We can see from this listing that about 50% of the DELETE messages have an encrypted payload size > 255.

2 0000004c
6 0000005c
2 000000dc
4 000000ec
3 000000fc
1 0000010c
6 0000011c
5 0000012c
3 0000013c


MikroTik ipsec debug packet logging:

Apr 15 11:47:58 MikroTik-LogServer mikrotik-ipsec-info: IPsec-SA established: ASA-endpoint[4500]->MikroTik-endpoint[4500] spi=0x8e17c57
Apr 15 11:47:58 MikroTik-LogServer mikrotik-ipsec-info: IPsec-SA established: MikroTik-endpoint[4500]->ASA-endpoint[4500] spi=0x36120df5
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-info: adding payload: DELETE
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-info: <- ike2 request, exchange: INFORMATIONAL:3 ASA-endpoint[4500] fcc83c561f08240f:2314adbd1c7b7677
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: => outgoing plain packet (size 0x28)
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: fcc83c56 1f08240f 2314adbd 1c7b7677 2a202508 00000003 00000028 0000000c
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 03040001 0ad443d2
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-info: adding payload: ENC
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: fcc83c56 1f08240f 2314adbd 1c7b7677 2e202508 00000003 0000012c 2a000110
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: a1aa514d c9a0aa54 47ad2ff0 02dd6abe dba9d98e 582b76a8 d456d377 441755b4
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 47bc49b2 43069122 2739fa13 8d6d9130 a3938baa 0228a501 ed9c231c 14613c64
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 4972507f 0a0ae567 1c706e45 adf74993 0483d1a2 809cc59b 344417b3 bd57b45b
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 96a8a4d9 f23b3fc1 7c22fcf9 e9012e9e 6a9cfdb5 a7aaf5e5 84f6efd1 a32095dc
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 40c1538f 41d44410 f682f6e4 627196cd c76be7e2 6da4eb02 cd0a8737 ff174a21
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 9b85aee8 cd4237a6 f60df4fc bac4b15a a821c29f 3e6da91e 738f363d 13896d04
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 38855eb7 8d98d3f8 d1bfce66 42a00235 52fed381 bdde56cb f47031ad 950137ec
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: d84c439a 169e0c48 17bffa71 e6ede39d 28bc2ef5 121beb83 a5fe5ab2 ac4de3d7
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 6ef4d55d 9606c660 8c74ea48



debug crypto ikev2 protocol 255:

IKEv2-PROTO-2: (1): Received Packet [From MikroTik-endpoint:4500/To ASA-endpoint:4500/VRF i0:f0]
(1): Initiator SPI : FCC83C561F08240F - Responder SPI : 2314ADBD1C7B7677 Message id: 3
(1): IKEv2 INFORMATIONAL Exchange REQUESTIKEv2-PROTO-3: (1): Next payload: ENCR, version: 2.0 (1): Exchange type: INFORMATIONAL, flags: INITIATOR (1): Message id: 3, length: 300(1):
Payload contents:
IKEv2-PROTO-1: Sum of decrypted payload lengths 12 is not equal to the total length of the decrypted payload 268
(1):
(1): Decrypted packet:(1): Data: 300 bytes
IKEv2-PROTO-1: (1): Failed to decrypt an encrypted payload
IKEv2-PROTO-1: (1): Failed to parse the packet
 
TedRule
just joined
Topic Author
Posts: 15
Joined: Tue Nov 03, 2020 4:41 pm

Re: IPSEC SA DELETE issue with MikroTik <-> Cisco ASA VPN

Fri Apr 16, 2021 7:20 pm

I think I've found a few more clues, having taken apart the DELETE message again.

The mystery deepens, but given the maths below, it suggests that I have a bug on the ASA which is provoked by varying sizes of Padding
being applied by the MikroTik, or possibly some "mispadding" by the MikroTik, but it's very difficult to pin this down.

RFC7296 allows for variable Padding, but even then some of the maths below suggests impossible sized packets.

As before we have the snippets of MikroTik and ASA debugs:

MikroTik ipsec debug packet logging:

Apr 15 11:47:58 MikroTik-LogServer mikrotik-ipsec-info: IPsec-SA established: ASA-endpoint[4500]->MikroTik-endpoint[4500] spi=0x8e17c57
Apr 15 11:47:58 MikroTik-LogServer mikrotik-ipsec-info: IPsec-SA established: MikroTik-endpoint[4500]->ASA-endpoint[4500] spi=0x36120df5
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-info: adding payload: DELETE
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-info: <- ike2 request, exchange: INFORMATIONAL:3 ASA-endpoint[4500] fcc83c561f08240f:2314adbd1c7b7677
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: => outgoing plain packet (size 0x28)
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: fcc83c56 1f08240f 2314adbd 1c7b7677 2a202508 00000003 00000028 0000000c
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 03040001 0ad443d2
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-info: adding payload: ENC
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: fcc83c56 1f08240f 2314adbd 1c7b7677 2e202508 00000003 0000012c 2a000110
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: a1aa514d c9a0aa54 47ad2ff0 02dd6abe dba9d98e 582b76a8 d456d377 441755b4
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 47bc49b2 43069122 2739fa13 8d6d9130 a3938baa 0228a501 ed9c231c 14613c64
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 4972507f 0a0ae567 1c706e45 adf74993 0483d1a2 809cc59b 344417b3 bd57b45b
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 96a8a4d9 f23b3fc1 7c22fcf9 e9012e9e 6a9cfdb5 a7aaf5e5 84f6efd1 a32095dc
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 40c1538f 41d44410 f682f6e4 627196cd c76be7e2 6da4eb02 cd0a8737 ff174a21
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 9b85aee8 cd4237a6 f60df4fc bac4b15a a821c29f 3e6da91e 738f363d 13896d04
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 38855eb7 8d98d3f8 d1bfce66 42a00235 52fed381 bdde56cb f47031ad 950137ec
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: d84c439a 169e0c48 17bffa71 e6ede39d 28bc2ef5 121beb83 a5fe5ab2 ac4de3d7
Apr 15 11:47:59 MikroTik-LogServer mikrotik-ipsec-debug: 6ef4d55d 9606c660 8c74ea48

ASA debug crypto ikev2 protocol 255:

IKEv2-PROTO-2: (1): Received Packet [From MikroTik-endpoint:4500/To ASA-endpoint:4500/VRF i0:f0]
(1): Initiator SPI : FCC83C561F08240F - Responder SPI : 2314ADBD1C7B7677 Message id: 3
(1): IKEv2 INFORMATIONAL Exchange REQUESTIKEv2-PROTO-3: (1): Next payload: ENCR, version: 2.0 (1): Exchange type: INFORMATIONAL, flags: INITIATOR (1): Message id: 3, length: 300(1):
Payload contents:
IKEv2-PROTO-1: Sum of decrypted payload lengths 12 is not equal to the total length of the decrypted payload 268
(1):
(1): Decrypted packet:(1): Data: 300 bytes
IKEv2-PROTO-1: (1): Failed to decrypt an encrypted payload
IKEv2-PROTO-1: (1): Failed to parse the packet


My reading of the error message from the ASA in its debug - given what RFC7296 says about Header/Payload format

....
IKEv2-PROTO-1: Sum of decrypted payload lengths 12 is not equal to the total length of the decrypted payload 268
....

is that the ASA successfully decrypts the Payload length 12 of the DELETE Payload from within the Encrypted Payload,
since it can't possibly know that number without a successful decryption of at least the start of the packet.
However, the ASA then finds that the number 12 doesn't match the size of the Encrypted Payload itself ( 300 - 28 Bytes of IKE Header ) less its own Payload Header,
IV, Padding and Checksum.

That seems to imply that the unencrypted Padding Length in the original packet is broken. It's almost as if 256 Bytes
of extra Padding are added to the Encrypted Payload before it is encrypted, but of course the Padding Length is
limited to a maximum of 255.

RFC7296 Header/Payload formats - for reference - are as below:
                       1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       IKE SA Initiator's SPI                  |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       IKE SA Responder's SPI                  |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Next Payload | MjVer | MnVer | Exchange Type |     Flags     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                          Message ID                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Length                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                        Figure 4: IKE Header Format


                        1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | Next Payload  |C|  RESERVED   |         Payload Length        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | Protocol ID   |   SPI Size    |          Num of SPIs          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~               Security Parameter Index(es) (SPI)              ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                     Figure 17: Delete Payload Format


                        1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | Next Payload  |C|  RESERVED   |         Payload Length        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                     Initialization Vector                     |
   |         (length is block size for encryption algorithm)       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   ~                    Encrypted IKE Payloads                     ~
   +               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |               |             Padding (0-255 octets)            |
   +-+-+-+-+-+-+-+-+                               +-+-+-+-+-+-+-+-+
   |                                               |  Pad Length   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   ~                    Integrity Checksum Data                    ~
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                    Figure 21: Encrypted Payload Format
The MikroTik dumps the non-encrypted DELETE packet in the debug, showing an overall packet length of 0x28

40 (0x28) = IKEHdr(28) + Hdr(4) + Payload (8)

Meanwhile the MikroTik packet dump shows an encrypted packet length of 300, where I'm assuming 8 Bytes for the IV used in AES256 - I think that's correct?

Encrypted DELETE Packet:

300(0x12c) = IKEHdr(28) + Hdr(4) + IV(8?) + Encrypted( Hdr(4) + Payload(8) + Padding + Padding Length(1) ) + Checksum(4)

Hence Encrypted( Hdr(4) + Payload(8) + Padding + Padding Length(1) ) == 300 - 28 - 4 - 8 - 4 == 256

But in this case how can the ASA be deducing that the total decrypted payload length is 268 - or 256 more than the real figure?

We've ALSO seen DELETE messages which fail with a overall size of 316(0x13c).

For 316(0x13c) we have Encrypted( Hdr(4) + Payload(8) + Padding + Padding Length(1) ) == 316 - 28 - 4 - 4 - 8 == 272 == 256 + 16

If we assume - a big assumption I know - that the size of the Encrypted Payload is the same as the un-Encrypted Payload we can estimate how much Padding is applied
in various cases. The various encrypted operations ( CREATE_CHILD_SA / AUTH / DELETE IKE SA / DELETE IPSEC SA ) seem to have the following differing payload sizes:

For 0x170 unencrypted size we see an encrypted size minimum of 0x19c encrypted == Padding of 28 , maximum of 0x1dc == Padding of 92

For 0xa8 unencrypted size we see an encrypted size minimum of 0x11c encrypted == Padding of 100 , maximum of 0x15c == Padding of 164

For 0x24 unencrypted size we see an encrypted size minimum of 0xdc encrypted == Padding of 168 , maximum of 0x11c == Padding of 232

For 0x28 unencrypted size we see an encrypted size minimum of 0x4c encrypted == Padding of 20 , maximum of 0x13c == Padding of 260 which is impossible?

I've also seen that it's all three of the AUTH / DELETE IKE SA / DELETE IPSEC SA operations appear to fail when their encrypted packet size is "too big";
this is NOT just a problem which afflicts the DELETE operations; I simply spotted it there first.

Who is online

Users browsing this forum: Ahrefs [Bot], Amazon [Bot], andreacar, Bing [Bot], f008600, fibracapi, mickeymouse690, sid5632 and 72 guests