Low IPSEC TCP Rate unless I use Torch

Wondering if anyone can give me any pointers. I have an IPSEC VPN to a site and I relay/route certain traffic through it.
When I try to SSH to a server at the site that I am connecting to, the echoback is super delayed. To type out this post would take minutes for example.
I have servers on the internet that I get to (for ACL reasons) by routing through this IPSEC connection and their echoback is nice and snappy.

I discovered tonight during diagnostics however, when I use TORCH on the IPSEC interface, the problem goes away.
Any pointers or ideas where I could start looking?

Are you using fasttrack??

I am, yes

https://wiki.mikrotik.com/wiki/Manual:IP/IPsec#NAT_and_Fasttrack_Bypass, see IP/Firewall/Raw solution.