InterVLAN routing between multiple routers

Greetings,

I have two routers, Router1 and Router2. Both routers are set up with VLAN 10 and VLAN 20. The routers are connected together with an untagged /30.

I want Router1 VLAN10 to be able to communicate with Router2 VLAN 10. But I never want Router1 VLAN 10 to be able to communicate with Router2 VLAN 20. I can easily block VLAN 10 from communcating with VLAN 20 on the same router. But how would I block VLAN 10 to VLAN 20 on different routers? Once the packets are routed, they lose their VLAN tag. So I think I would need to do this by IP address? With a larger network, maintaining IP lists could get cumbersome, is there an easy way to do this?