[Wireguard] "Handshake for peer did not complete after 5 seconds" after upgrade

After upgrading one endpoint of the Wireguard tunnel from 7.13.5 to 7.15.3 (7.14.1 was also broken) the tunnel stops working and in the logs the following messages repeats:

  • Handshake for peer did not complete after 5 seconds
  • Sending handshake initiation to peer
    Do I have to change something in my configuration after such an upgrade?