It seems my router is not sending an IPv6 DNS server in the router advertisement. And I don’t know how to control which DNS server it advertises. How do I do this? I can’t find documentation for this.
Answering my own question partially. Once I googled the right terms, I found some help. I was thinking somehow I should be able to completely control what information is sent in the router advertisement, which is what I was searching for. This seems to not be the case. The router mostly decides what info is put into the router advertisement.
(I am able to configure the router advertisement on my other routers.)
So anyway, you have to go to /ipv6 nd and set advertise-dns=yes for the interface you want it advertised on.
Then if you have a particular server you want advertised, you go to /ip dns (not /ipv6 dns because it doesn’t exist), and add your ipv6 dns server as a static one.
But the problem with that is the router advertisement combines that static server with whatever ipv6 DNS servers it got from your upstream provider, which I don’t want but I can’t find a way to control that.
Probably because I don’t seem to have direct control over the router advertisement 
This hopefully will be fixed at some point.