The reason why it didn't work with the previous mangle rules can be explained, but first, some literatures:
As you already did in your original version of the mangle rules, normally the correct thing to do is to exclude packet destined to the router itself from being affected by mangle mark-routing. The reason is that because in the default routing lookup order (listed under policy-rules) mangle comes before local.
If the destination of a packet is one of the router's IP addresses, normally if you want the router to be able to receive it, it must reach the local chain of the routing lookup. But if you applied mangle mark-routing on the packet, it will be handled by the mangle chain, which by default comes before local. Which means the router will not be able to receive the packets.
That's why normally you see the condition such as dst-address-type=!local added to the mangle rules, like you also did with the mark-connection rules. So that the packet with destination address being an address of the router will be excluded from mark-connection and thus later from mark-routing.
Usually that's enough, however, not when DSTNAT is involved. If you look at the linked flowchart:
Mangle prerouting comes at #4, before DSTNAT, which is at #10. So when the client make a DNS query to 8.8.8.8, the initial UDP53 or TCP53 packet that hits the mangle mark-connection rule still has dst-address=8.8.8.8, which does satisfy dst-address-type=!local because 8.8.8.8 is not one of the router's IP addresses. The connection will be marked. The later mark-routing rule will apply the routing mark on the packet.
Then at #10 DSTNAT redirect changes 8.8.8.8 to the router's IP address on the LAN interface.
At #11 where Routing Lookup happens, the packet is handled by the mangle routing decision chain, but the packet has a destination address of type local.
The mangle chain can only use the routing table pointed at by the routing mark for the lookup, but for a packet destined for the router, those tables are useless, only the local routing decision chain has the ability to correctly handle them, but this chain sits below mangle. As a result, the packet is routed outside of the router, using one of the matching routes of the table (for example a dst-address=0.0.0.0/0 route that sends the packets to the internet), or if no active matching route is found, the packet will be discarded.
The fix is to make sure that those DNS packets are not affected by your mark-connection / mark-routing rules, as you've figured out with: