I tried this but it did not work, and I think I know why. I did run a packet capture to have a better understanding of this.
When a connection is initiated from the internet (SYN), coming into my network via ISP B (backup route for my netowrk), it is correctly routed to the internal host based on the port forwarding rules. The internal host responds back to the connection request (SYN-ACK). Once the SYN-ACK reaches the router, it uses the default primary route (ISP - A) to send out the packets. This results in a successful connection never being formed. In order to form a successful connection, the SYN-ACK should have been sent back to the Internet host from the ISP -B.
Your suggested configuration allows me to mark the incoming packets from ISP B, but the requirement here is to somehow mark the reply packets to the traffic we have already marked. Basically, the need is to preserve/mark connections (similar to how NAT/PAT preserves connections), and mark/re-route the reply traffic to the incoming traffic from ISP-B (not the incoming traffic itself).
I’m very new to creating Mangle rules. I saw 3 fields for mark - Packet mark, Routing mark and Connection mark. Could using Connection mark here be of some help?