So I just got it working.
I’m not sure whether I didn’t made any other change, but the step between not working and working was to go to IP/IPSec/Policy and enable the default entry I have there.
/ip ipsec policy print
Flags: T - template, X - disabled, D - dynamic, I - inactive, * - default
0 T * group=default src-address=::/0 dst-address=::/0 protocol=all
proposal=default template=yes
With this policy disabled, I could see incoming connections in the log, and I could see even remote peers connected for a moment in IPSec/Remote Peers, but after a moment they disconnected with timeout. After enabling it, it works like a charm.
(On a side note, it was this thread that kicked me to look on the policy.)