I've spent several days debugging the problem and it turned out that the previously working proposal or AES-GCM started to fail in phase 2 negotiations
The only way to make the tunnel work again was to revert to AES-CBC ciphers witch is quite unfortunate.
Originally the proposals were configured as:
ipsec logging was reporting
add auth-algorithms=sha256,null enc-algorithms=aes-128-gcm lifetime=20m name=GCM-proposal pfs-group=ec2n185
ipsec invalid encryption algorithm 20.
switching from CGM to CTR resulted in the same error
ipsec pfkey update failed.
switching to CBC immediately established phase 2
also during the debug I've noticed another strange thing in phase 2 in the logs:
my proposals were only for GCM-128, bit somehow both routers were attempting to negotiate 160-bit keys
21:49:03 ipsec,debug (trns_id=AES-GCM-ICV16 encklen=128 authtype=254) 21:49:03 ipsec,debug (trns_id=AES-GCM-ICV16 encklen=160 authtype=254)
Can anybody help with setting up GCM ciphers in 6.41+ software or advise on bug fixes timeframes?