Hi everyone, I am trying to establish an IKEv2 IPsec tunnel between a MikroTik RB750Gr3 (RouterOS 7.21.5) and a remote Fritz!Box (FritzOS 8.25). The exact same IKEv2 connection works flawlessly when using strongSwan (swanctl) with the following negotiated proposal: IKE: AES_CBC_256 / HMAC_SHA2_512_256 / PRF_HMAC_SHA2_512 / MODP_2048
However, on RouterOS the tunnel fails during the Phase 1 exchange (SA_INIT). The debug log shows that RouterOS rejects the response from the Fritz!Box with:
"ipsec => invalid payload (size 0x30)"
Here is my current IPsec configuration on RouterOS:
# 1. Profile phase 1 (IKEv2)
/ip ipsec profile add dh-group=modp2048 enc-algorithm=aes-256 hash-algorithm=sha512 prf-algorithm=sha512 name=profile-fritz-strongswan
# 2. Peer (Fritz!Box)
/ip ipsec peer add address=XXXXXX.myfritz.net exchange-mode=ike2 name=peer-fritz-geneva profile=profile-fritz-strongswan
# 3. Mode Config (to request virtual IP virtual 192.168.178.204)
/ip ipsec mode-config add name=modecfg-fritz responder=no use-responder-dns=no
# 4. Proposal Phase 2 (Child SA)
/ip ipsec proposal add auth-algorithms=sha256 enc-algorithms=aes-256-cbc name=proposal-fritz-child pfs-group=none
# 5. Identity ( PSK and ID )
/ip ipsec identity add auth-method=pre-shared-key generate-policy=port-override mode-config=modecfg-fritz my-id=fqdn:MY-ID peer=peer-fritz-geneva remote-id=fqdn:XXXXXX.myfritz.net secret="MY-PRE-SHARED-KEY"
# 6. Policy Template to attach the traffic
/ip ipsec policy add dst-address=0.0.0.0/0 peer=peer-fritz-geneva proposal=proposal-fritz-child src-address=192.168.178.204/32 tunnel=yes
Here is the packet hex payload received from the Fritz!Box (109.205.172.211):
2026-09-03 09:39:20 ipsec processing payload: SA
2026-09-03 09:39:20 ipsec => invalid payload (size 0x30)
2026-09-03 09:39:20 ipsec 22000030 0000002c 01010004 0000000c 0100000c 800e0100 00000008 0300000e
2026-09-03 09:39:20 ipsec 00000008 0400000e 00000008 02000007
Looking at the hex dump, the Fritz!Box sends 4 transforms:
1. ENCR: AES-256-CBC (0100000c 800e0100)
2. INTEG: HMAC-SHA2-512 (0300000e)
3. D-H: MODP-2048 (0400000e)
4. PRF: HMAC-SHA2-256 (02000007)
It seems FritzOS responds with a PRF of SHA-256 while using SHA-512 for integrity, or formats the transform headers in a non-strict manner.
strongSwan accepts this negotiation without issues, but RouterOS considers the SA payload invalid and drops it immediately. Setting `prf-algorithm=sha256` or `sha512` in RouterOS profile does not resolve the issue (it results in "no proposal chosen" or "invalid payload").
Is this a known strictness/parser issue in RouterOS IKEv2 regarding FritzOS SA_INIT payloads? Is there any workaround on RouterOS without modifying the remote Fritz!Box config? Thanks in advance for any insights!
p.s. wireguard can't be used as the fritzbox is already configured with one connection and allows only one wireguard at the time.
full log:
2026-09-03 09:50:29 ipsec ike2 starting for: 109.205.172.211
2026-09-03 09:50:30 ipsec adding payload: SA
2026-09-03 09:50:30 ipsec,debug => (size 0x30)
2026-09-03 09:50:30 ipsec,debug 00000030 0000002c 01010004 0300000c 0100000c 800e0100 03000008 02000007
2026-09-03 09:50:30 ipsec,debug 03000008 0300000e 00000008 0400000e
2026-09-03 09:50:30 ipsec adding payload: KE
2026-09-03 09:50:30 ipsec,debug => (first 0x100 of 0x108)
2026-09-03 09:50:30 ipsec,debug 00000108 000e0000 7608c994 8df72dbb f8f8f217 c89c1d2f 12a8db79 4ab99656
2026-09-03 09:50:30 ipsec,debug 2d766a0d 6e633985 9948bdb5 18768610 54591c75 d9fb0c4b 3dac750f 6dddc18f
2026-09-03 09:50:30 ipsec,debug 5378818a cf6481f7 b34894f9 271cc641 b456874a 2dfbd77e a7483d87 d618553c
2026-09-03 09:50:30 ipsec,debug c453e5d0 7f4a01ac ac0ca7a9 76aad2c5 4e86ce32 50a53643 416c387a a93f8712
2026-09-03 09:50:30 ipsec,debug 35f85278 4ecee887 017cbcfa f5ea4ee8 68978a9c 756f6952 5ec2867a c9c9f6d2
2026-09-03 09:50:30 ipsec,debug c0e8ab06 6444024e 17a535a9 b727c593 da254e23 9bfa5602 89adb660 bddcb656
2026-09-03 09:50:30 ipsec,debug b1c3c6ca 15bfdf93 22dd9769 cdecc1bc 6c87e003 8d242b15 12238a7f 6f97ef78
2026-09-03 09:50:30 ipsec,debug e467d9ba eee0c5fd b8521cc1 daf58c70 cce8c94f 7929fa2b 4a42f3e6 4b76cd8c
2026-09-03 09:50:30 ipsec adding payload: NONCE
2026-09-03 09:50:30 ipsec,debug => (size 0x1c)
2026-09-03 09:50:30 ipsec,debug 0000001c 7a51cc66 ceb56d8f 397b4c87 b3727fc0 a97b322a d1cc30ae
2026-09-03 09:50:30 ipsec adding notify: NAT_DETECTION_SOURCE_IP
2026-09-03 09:50:30 ipsec,debug => (size 0x1c)
2026-09-03 09:50:30 ipsec,debug 0000001c 00004004 92e1545f 24c1f871 cbcc5280 8c987b96 de8ba3f1
2026-09-03 09:50:30 ipsec adding notify: NAT_DETECTION_DESTINATION_IP
2026-09-03 09:50:30 ipsec,debug => (size 0x1c)
2026-09-03 09:50:30 ipsec,debug 0000001c 00004005 0a828c62 36b1376d 19ed71f8 02ff2398 4b92bf48
2026-09-03 09:50:30 ipsec adding notify: IKEV2_FRAGMENTATION_SUPPORTED
2026-09-03 09:50:30 ipsec,debug => (size 0x8)
2026-09-03 09:50:30 ipsec,debug 00000008 0000402e
2026-09-03 09:50:30 ipsec <- ike2 request, exchange: SA_INIT:0 109.205.172.211[4500] b2ef88ccf209431f:0000000000000000
2026-09-03 09:50:30 ipsec,debug ===== sending 432 bytes from 192.168.1.2[4500] to 109.205.172.211[4500]
2026-09-03 09:50:30 ipsec,debug 1 times of 436 bytes message will be sent to 109.205.172.211[4500]
2026-09-03 09:50:30 ipsec,debug,packet b2ef88cc f209431f 00000000 00000000 21202208 00000000 000001b0 22000030
2026-09-03 09:50:30 ipsec,debug,packet 0000002c 01010004 0300000c 0100000c 800e0100 03000008 02000007 03000008
2026-09-03 09:50:30 ipsec,debug,packet 0300000e 00000008 0400000e 28000108 000e0000 7608c994 8df72dbb f8f8f217
2026-09-03 09:50:30 ipsec,debug,packet c89c1d2f 12a8db79 4ab99656 2d766a0d 6e633985 9948bdb5 18768610 54591c75
2026-09-03 09:50:30 ipsec,debug,packet d9fb0c4b 3dac750f 6dddc18f 5378818a cf6481f7 b34894f9 271cc641 b456874a
2026-09-03 09:50:30 ipsec,debug,packet 2dfbd77e a7483d87 d618553c c453e5d0 7f4a01ac ac0ca7a9 76aad2c5 4e86ce32
2026-09-03 09:50:30 ipsec,debug,packet 50a53643 416c387a a93f8712 35f85278 4ecee887 017cbcfa f5ea4ee8 68978a9c
2026-09-03 09:50:30 ipsec,debug,packet 756f6952 5ec2867a c9c9f6d2 c0e8ab06 6444024e 17a535a9 b727c593 da254e23
2026-09-03 09:50:30 ipsec,debug,packet 9bfa5602 89adb660 bddcb656 b1c3c6ca 15bfdf93 22dd9769 cdecc1bc 6c87e003
2026-09-03 09:50:30 ipsec,debug,packet 8d242b15 12238a7f 6f97ef78 e467d9ba eee0c5fd b8521cc1 daf58c70 cce8c94f
2026-09-03 09:50:30 ipsec,debug,packet 7929fa2b 4a42f3e6 4b76cd8c e3015ca2 52f63d09 2900001c 7a51cc66 ceb56d8f
2026-09-03 09:50:30 ipsec,debug,packet 397b4c87 b3727fc0 a97b322a d1cc30ae 2900001c 00004004 92e1545f 24c1f871
2026-09-03 09:50:30 ipsec,debug,packet cbcc5280 8c987b96 de8ba3f1 2900001c 00004005 0a828c62 36b1376d 19ed71f8
2026-09-03 09:50:30 ipsec,debug,packet 02ff2398 4b92bf48 00000008 0000402e
2026-09-03 09:50:30 ipsec,debug ===== received 424 bytes from 109.205.172.211[4500] to 192.168.1.2[4500]
2026-09-03 09:50:30 ipsec,debug,packet b2ef88cc f209431f 06704b8f 7587c31a 21202220 00000000 000001a8 22000030
2026-09-03 09:50:30 ipsec,debug,packet 0000002c 01010004 0000000c 0100000c 800e0100 00000008 0300000e 00000008
2026-09-03 09:50:30 ipsec,debug,packet 0400000e 00000008 02000007 28000108 000e0000 b8694a52 a229c76e dbde7846
2026-09-03 09:50:30 ipsec,debug,packet 1340cf7e 26952336 1f78b5fc f844fc65 e079d2b6 2e93b093 333fd7b4 a9111ae9
2026-09-03 09:50:30 ipsec,debug,packet 5c7c5c79 278e8197 a11fb382 94f565e8 78cfaded 9d9409e3 273f751f 6df28335
2026-09-03 09:50:30 ipsec,debug,packet a67fc6f8 74fc878d 0029e166 e1d0259f e6826246 3d20872e 5115f58b 6f3604b0
2026-09-03 09:50:30 ipsec,debug,packet d20de763 beb3c709 167a129f 0690843c d5efb6cd 4c5c0183 62e25345 22156bae
2026-09-03 09:50:30 ipsec,debug,packet d90805aa ee98004d 0d345c63 dbfd9bb9 4e94e9a3 c171d9c3 a9c31c08 8676a082
2026-09-03 09:50:30 ipsec,debug,packet 859368a4 70b1c70e 46b65e1f 8822af0f 146a93e6 5c1ccecc 05fa70f9 c94a5178
2026-09-03 09:50:30 ipsec,debug,packet 61daf9bd a74a8496 7c5d494f 73546b6f ea1473e6 8ec6aa34 102a9141 252a3c25
2026-09-03 09:50:30 ipsec,debug,packet 277dd0a9 1860937c ea6fc0e1 57590b08 9ffaca87 2900001c b4ed7143 f96e110d
2026-09-03 09:50:30 ipsec,debug,packet 9ed1d78d b8182dca ece3b27a d2c82c34 2900001c 00004004 248fb02a a9a11d87
2026-09-03 09:50:30 ipsec,debug,packet f50f4818 fa951a3c 5e48301c 0000001c 00004005 ca31c3c2 61fda89b 2ab5af86
2026-09-03 09:50:30 ipsec,debug,packet 2f2df95d 40e98690
2026-09-03 09:50:30 ipsec -> ike2 reply, exchange: SA_INIT:0 109.205.172.211[4500] b2ef88ccf209431f:06704b8f7587c31a
2026-09-03 09:50:30 ipsec ike2 initialize recv
2026-09-03 09:50:30 ipsec payload seen: SA (48 bytes)
2026-09-03 09:50:30 ipsec payload seen: KE (264 bytes)
2026-09-03 09:50:30 ipsec payload seen: NONCE (28 bytes)
2026-09-03 09:50:30 ipsec payload seen: NOTIFY (28 bytes)
2026-09-03 09:50:30 ipsec payload seen: NOTIFY (28 bytes)
2026-09-03 09:50:30 ipsec processing payloads: NOTIFY
2026-09-03 09:50:30 ipsec notify: NAT_DETECTION_SOURCE_IP
2026-09-03 09:50:30 ipsec notify: NAT_DETECTION_DESTINATION_IP
2026-09-03 09:50:30 ipsec processing payload: SA
2026-09-03 09:50:30 ipsec => invalid payload (size 0x30)
2026-09-03 09:50:30 ipsec 22000030 0000002c 01010004 0000000c 0100000c 800e0100 00000008 0300000e
2026-09-03 09:50:30 ipsec 00000008 0400000e 00000008 02000007