Does this setup makes sense?

[tangent]

what happens when vlanX needs to hit VLANY, is this somehow routed between subnets at wirespeed then??

Use some combination of hardware inter-VLAN routing and hardware VLAN filtering. This takes up precious ACL rule space, but 128 rules is enough for a home lab, no problem.