I don't have a device with more than one switch chip to test my hypothesis. However, I imagine it is possible to bridge all ports and keep using HW to deal with VLANs... by simply using bridge as a dumb switch (VLAN unaware), just the way it used to be done in pre-6.41 (or whatever) times. The only thing it needs to be done is to make
switchX-cpu interfaces members of same VLANs - those that should pass between both switch chips. And one has to revert to the old way of configuring other interfaces made members of same dumb bridge (e.g. wifi interfaces).
Just beware that RB2011 has two different switch chips that, when it comes to VLANs, behave slightly differently (check
switch chip features).
It's hard to judge if the solution of poster of this post is the most/only correct as he didn't post the original (poorly performing) setup ... so we'll never know what was the cause of slow performance.