Since 7.23 we have this:
Which means you can do some filtering in the mangle table, where the interface matcher behaves in the way that you want.
Of course, when dropping in mangle you should also use the connection-state=new condition, because usually there is no accept established, related rule at the top of the chains like in the filter table.