1.) You only have two mangle rules listed there, and you do not list your routing table, hard to tell what is going on without all of the necessary information.
2.) According to your rules you are marking traffic coming INTO the router form the outside and trying to load balance it, that will not work. You need to mark outbound traffic from your LAN. There are 3 possible moving parts when you are dealing with multiple internet connections:
a.) Traffic generated by a user that you want to policy base route/load balance
b.) Traffic that the router is responding to itself
c.) Traffic coming into the LAN from the outside (such as a port forward)
Each one of these requires a set of rules to make them work. Most people just care about part “a” of it, but if you want, for example, to be able to connect to both public IP addresses of the router then you need rules for case “b” to take care of that.
Here is a working example for case a and the limited information provided.
What happens is a new connection comes into the LAN and it hits the mangle prerouting chain. The packet will either match for the dst-address or not, if it does it hits the first rule. If it does not, the second rule matches. It then will get passed down in the firewall rule chain to the mark routing rules. This will then tell the router what internet connection it should use for that specific traffic.
The problem you are running into is the requested traffic is coming into the router, and then the router tries to load balance that traffic and will sometimes send it out the wrong interface.