But this is not handled by the router, right?
Indeed not. You set things up to forward ports 80 and 443 to the internal machine for all HTTP/HTTPS accesses, and then configure the web server to respond differently depending on what name it was called by for each individual request.
This is ancient (1997) tech, now widely supported. It goes back to HTTP/1.1, being one of the things added relative to the now totally obsolete HTTP/1.0. About the only things doing HTTP/1.0 now are people hand-typing the protocol, tiny embedded systems, and badly outdated code frozen in time for the past two decades.
web proxy...Nginx...redirect based on host name.
Or, just make it your web server, period, full stop. Note the "server_name" directive in its
virtual host configuration docs. You can set up as many of those as you need. My public nginx server responds to half a dozen different names.
If you have to do reverse proxying atop that, fine, but for static hosting, you don't need to go to that extent.