Ok, I have found the issue.
The TCP SYN from the Server is being sent to the Router and the Router forwards it to the VPN-GW in the same Subnet, which then routes the Packet to the VPN Client.
But as they are in the same Subnet, the TCP SYN/ACK is sent directly from the VPN-GW to the Server.
Because of that, the Router can’t track the Connection and drops it as invalid.