So what is your traffic speed and what is your CPU usage? It sounds like the bandwidth of your traffic may be overwhelming the CPU, whereas when it is offloaded through the switch chip, it works fine.
My traffic is pretty low, I have set up for one vlan=6 passthrough from ether1 to ether5 using bridges, and it is usually about 15Mbps (IPTV). This is also happening even when traffic is super low, 1Mbps. CPU usage is also low, 1-3% max.