Basically, what is the secret???
I can easily get into an RB200 via the serial port but have yet to manage it in any of my RB500 units.
Try to set baud-rate to 115200.
I think it will solve your problem.
I wish 
It was the first thing I tried (well, second thing)
I had a look at the settings via winbox and they seem fine, though the baud rate seems to be defaulted to auto and not 115200. Have tried to force it to 115200. 9600, 19200 and even 110 via winbox but still cannot connect via the serial port.
It’s very frustrating since it hooks up to an RB230 right away with no fuss what so ever.
I wouldn’t normally be too worried, but I am trying to configure an RB512 and can’t go in via the ethernet after I changed the IP address of it. It won’t even ping the IP (though arp -a shows it to be set as it gives the correct mac address). Anyway, that is by the by as I really want to work out how to connect via the serial port for all future occasions such as this one. I am “practising” serial connections using a spare RB532 but having no luck, though as I said when I use a spare RB230 no problem at all.
Help 
I have had the same problem, even after trying every parameter selection that I could think of. I first noticed the problem on RC4, but it seemed to work fine on older boards running RC2(I don’t know the Bootloader versions that they came installed with). I tried a RB512 today, and I could not get anything but nonsense in the terminal, even though the board would boot just fine. Can you not connect to your RB5x2 by MAC address? I discovered that sometimes Winbox by MAC address doesn’t work properly if you have more than one network interface enabled on the Winbox machine, if that helps any…
Hitek
In a way I’m glad to hear it is not just me!
By the way, I managed to get access to the RB512 using the mac address and winbox (hadn’t realised I could do this!).
Have you checked that all pins on the null-modem cable are wired correctly?
I have 4 null-modem (so they say on the package) cables next to me, all of them are wired differently. All of them work on routerboards 230 but not all of them work with other types of equipment.
Thanks, that sorted it.
I dug out a different cable and it worked first time. 
Wonder what it is that allows the RB230 to work but not the RB512 etc?
the handshake lines ? by default, MT needs hardware handshake I think.
We have loads of 3 wire serial leads, none of them work with MT as default.
Regards
I was a little hasty saying that all works for RB230, only 3 works with RB230.
PC based equipment in general wants to have hardware flow control by default. It’s possible to circumvent this by using software flow control but that means you have to have access in the first place.
Serial lines is one of those things that really have alot of “standards” 
We set our laser equipment firmware, and PC software, to use 3 wires no handshake, so it works with all PCs and any cable so long as it has the 3 wires. We have an error-checked protocol to avoid errors or lost characters, so over-run isn’t a problem.
On MT, you could go into winbox, set the serial port to “no handshake” using command line, I think, then use the serial port with 3 wires only. I think MT then “remembers” this setting.
Otherwise, you need the proper cable …
Regards