Don’t forget certificates. I import a certificate for a new user I add (I don’t use a password on the account, but I do on the certificate). I put that new user in a new group I create that has only SSH access. I remove the SSH privilege from admin group. I then put SSH on a random, high-numbered port as someone else suggested.
That setup doesn’t eliminate the possibility of continued brute-force attacks, but the non-standard port makes it much less likely, and the certificate reduces the chance of success to just about zero. Even if someone could gain access through the ssh-only user account, there’s not much they can do unless they can launch a second session as admin through the ssh connection.
If for some reason a certificate could not be used, I would still make the other changes above, but change the name of the account with ssh privilege to something difficult to guess. That way guessing the username is as difficult as guessing the password.
VPN is a great idea but not possible if you manage multiple routers not owned by the same person or company. Plus, having ssh as a back door is nice if the VPN breaks. That’s a real possibility with firmware updates, but ssh is simple enough that it always seems to work.