I wish I had a more detailed post to direct you to, but check out this for now:
http://forum.mikrotik.com/viewtopic.php ... 50#p534818
Likely what you are seeing is an issue with the hardware encryption driver that causes problems with packet delivery. With TCP, this often results in poor single thread performance. You can verify by switching to software encryption.
Oh man you were spot on. Hardware encryption is broken in my case all the way from 6.23.1 up to 6.35.1 on CCR1016 and is pretty much useless one threaded. This means every file copy through tge tunnel.
Thanks for mentioning this.
In talking to Mikrotik, it sounds like the issue with hardware right now is having packets from the same connection processed by different cores causes the encapsulated packets to be sent out of order. This results in the device on the remote end of the tunnel getting duplicate acks, out-of-order packets, etc (tcp assumes packets are lost, tries to compensate, but that takes additional time and bandwidth to be consumed in the process). All that provides poor throughput on a single stream. Of course, that is worse for certain connections (higher latency) and services like smb that aren't built to perform well on error prone connections (especially older versions). Many times you can't control what type of traffic and services will be used on a connection, so it is best to just fix the issue on the device doing the encryption that is introducing these issues. I think one solution they might be considering is locking a connection to a single core. The good news is that the Tile chipset should be able to do several hundred mbps of hardware encryption on a single core, which is better than the non-offloaded, software encryption.
I'm doing multiple software tunnels with load balancing as a bandaid fix until they can release the fix for the hardware encryption driver. This gives me ~150Mbps per stream multiplied by number of tunnels, which is better than I can get with the current poor quality hardware encryption.
I asked for an update on my ticket with Mikrotik and they say they are working on a fix, which is great news. Hopefully that won't take too long.