It’s not unusual to want not only for packets to reach the server, but also to get some response from it. 
So on routing level, both LANs must be able to reach each other. And what can actually pass will be up to firewall rules.
And even OP’s original config, think about it, if mangle rule sends all traffic to internet, including traffic that has other LAN as destination, then what’s the simplest fix? Tell the rule to do what it does now, except for local traffic, e.g. with dst-address=! (“!” means “not”).