After a long while of searching and testing I managed to get Mikrotik <-> Racoon on Linux 2.6.25 working with rsa-sig and certificates. The error messages I got with racoon initially looked like the ones I had with the Cisco routers, so I assume the problem was the same. I will test again with Cisco...