You should migrate from a dual bridge config to a single bridge config and configure the correct VLANs on the correct ports along with PVID, taggged and untagged status per port. Once done, you can enable vlan filtering on the bridge.
The link from the switch would just be for tagged packets and would have both VLANs configured on it. The other ports (1&2, 3&4) would be untagged for their respective matching VLAN on the switch. If you need to assign services for each VLAN - instead of using the bridge you would create two VLAN interfaces assigned to the one bridge and make the assignment to each VLAN interface (ip address, DHCP server, etc)