That's totally possible. With 5 publics each pretty straight forward even. What you get is if a router failed, and the rest of the network shouldn't care, only side effect be either increased CPU on the one routers. The main feature, beyond hardware failover, is you can control which router is doing the work simply by changing the VRRP priority. And externally IP ports open to the internet can use the VRRP address for incoming connection, and as long as firewall still works, losing a router be harmless here too. The new connection tracking should make this relatively seamless to the clients.
Now without doing anything on the client VLANs, how well the work overall depending HOW the traffic routers from the gateway address of your LAN/VLANs to the internet. Since those can only have one gateway, if that VLAN gateway router goes out, without VRRP, the entire VLAN loses internet. And assuming the VLAN are bridged to both router. To avoid neeing VRRP-per-client-VLAN, you could use duplicate DHCP servers, for each VLAN, with only difference is the VLAN GW in DHCP uses the "local/active" VRRP's router VLAN IP – using the Up/Down event of VRRP write a small script that enables/disables the DHCP Server to make this happen. Essentially use the WAN VRRP as signal to change DHCP. Obviously VRRP per VLAN be ideal but does get complex as the number of VLANs goes up.
I think this post is closer to what you're trying to do:
viewtopic.php?p=895300&hilit=VRRP+WAN#p895300