Forget all the stuff above.
Just setting RX and TX flow control on the CHR interface to auto fixed my issue. (off by default)
Strangely, off or auto is still off when doing a
/interface/ethernet/monitor 0
, but somehow it works. I guess setting to auto changes something else also.