@anav: It is the setup in http://forum.mikrotik.com/t/routeros-blatantly-ignores-pref-src-can-this-really-be-a-bug/180360/38 so public IP and multiple WANs.
However, upon more and more debugging it turns out the issue is still the DNAT, or, more accurately the missing source address.
It seems that for some packets, a different source address is selected, to the backwards translation on the DNAT fails. In the sniffer output it can be seen that packages are re-transmitted multiple times. I assume this is because subsequent packets never made it to the client.
Hence best to continue in the DNAT/pref-src thread