Is there not a way that specific router tells its name to that program performing the quiery?
No, because that is not how traceroute works. It prints out the found IP addresses and if it can find a reverse DNS entry for such an IP it will show you the associated FQDN.
So all you need to do is make sure you have a DNS up and running und configured your windows box to use it, you traceroute can as well and put teh required information in your DNS. It does not matter if you do it with bind, Microsoft stuff - any DNS server will do (that's what standards are for).
I understand probably somewhere a system must exist that binds that name with the IP of the router.
Yes, that's what DNS is for. It not only translates domain names to IPs but can also be used the other way around.
Is that done inside the router?
Only if you run some DNS service on your router and your windows box uses it.
Or does the traceroute program (pingplotterpro) have to send a query to a dns server to ask which names goes with what IP?
Bingo!
In this latter scenario, does that mean that I have to tell that dns server all my router's IP's and their names?
Bingo again! We have another lucky winner here..
If I use OpenDNS for all my dns quieries, how do I do that?
Don't know that service. If unsure, ask them. But if your routers use private IPs you'll need an internal DNS server. And if the public IPs of your routers don't belong to you, you'll have to ask your provider to propagate the reverse DNS information in their DNS servers according to your wishes.