... I've tried using certificates but they just don't work....
I've been using a Mikrotik AC^2 client connecting to a Debian server with IPSEC-secured GRE for quite some time.
Recently I switched to certificate based auth (previously, PSK and then RSA keys).
Shared some tips here, hope this may be useful:
Basically, one difficult part is creating the certs (for me at least), another is making sure the certs have the necessary stuff in their subjectAltNames, and finally debugging (troubleshooting) can be quite difficult. For this I suggest watching your server logs as Mikrotik is trying to connect.
If necessary it's also possible to enable IPSEC logging on the Mikrotik side.
FWIW, my current strongSwan config.
local_addrs = 220.127.116.11 # my server public IP
version = 2
proposals = aes128-sha256-ecp256
auth = pubkey
certs = newtun_server_1.pem
auth = pubkey
cacerts = newtun_CA.pem
local_ts = dynamic[gre]
remote_ts = dynamic[gre]
mode = transport
esp_proposals = aes128-sha256-ecp256
There are two certs on the server - the CA and the server's own cert, and also a private key for the server cert.
The client (Mikrotik) has its own cert (issued by the same CA) and also the server cert. Both are set in IP / IPSec / Identity.
The idea is that the server validates the certificate sent by the client against the CA that it has. It's also possible to validate against a specific client certificate (I'm not doing that).
I'm running RouterOS 6.44 beta, it has some IPSec fixes which help stability. Everything is running very well for me (finally) but yes it took some time.
/ip ipsec export
# feb/13/2019 22:52:31 by RouterOS 6.44beta75
# software id = 7BZI-A8N0
# model = RouterBOARD D52G-5HacD2HnD-TC
/ip ipsec peer
add address=18.104.22.168/32 comment=gre-tunnel1 exchange-mode=ike2 local-address=22.214.171.124 name=myipsecserver
/ip ipsec profile
set [ find default=yes ] dh-group=ecp256 enc-algorithm=aes-128 hash-algorithm=sha256 nat-traversal=no
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-128-cbc pfs-group=ecp256
/ip ipsec identity
add auth-method=rsa-signature certificate=newtun_ac2_1.crt_0 peer=myipsecserver remote-certificate=newtun_server_1.crt_0
/ip ipsec policy
add dst-address=126.96.36.199/32 protocol=gre src-address=188.8.131.52/32