Can someone help create mangle rules/PBR for return traffic to internet?
I have ISP A-primary(behind a NAT of another router) and ISP B- backup(PPPoE connection from my router).
I can only use ISP B to reach my home network from the outside(IPSec tunnel, Port forwarding to internal network devices, etc).
Problem is, when I try to access from outside(internet) using the ISP B, the reply traffic going out uses the primary route.
I'd like to mark the incoming traffic from ISP B somehow, and make the return traffic for it use the ISP B route instead of the primary route.
Please let me know if you would like to know more specifics of my setup.

Well yes,

(1) create the necessary table.
/routing table add name=SpecificTraffic fib ( necessary for Ver 7 firmware )

(2) create the mangle rule to identify the traffic,
/ip firewall mangle
add chain=prerouting in-interface=WANX mark=no-mark action=mark-routing new-routing-mark=SpecificTraffic { (same for v6 and v7)}

(3) Create an additional route besides the standard route for WANX.
add dst-address=0.0.0.0/0 gateway=WANX-gateway routing-mark=Specific Traffic (<--- V6 )
add dst-address=0.0.0.0/0 gateway=WANX-gateway table=SpecificTraffic (<--- V7)

I tried this but it did not work, and I think I know why. I did run a packet capture to have a better understanding of this.

When a connection is initiated from the internet (SYN), coming into my network via ISP B (backup route for my netowrk), it is correctly routed to the internal host based on the port forwarding rules. The internal host responds back to the connection request (SYN-ACK). Once the SYN-ACK reaches the router, it uses the default primary route (ISP - A) to send out the packets. This results in a successful connection never being formed. In order to form a successful connection, the SYN-ACK should have been sent back to the Internet host from the ISP -B.

Your suggested configuration allows me to mark the incoming packets from ISP B, but the requirement here is to somehow mark the reply packets to the traffic we have already marked. Basically, the need is to preserve/mark connections (similar to how NAT/PAT preserves connections), and mark/re-route the reply traffic to the incoming traffic from ISP-B (not the incoming traffic itself).

I'm very new to creating Mangle rules. I saw 3 fields for mark - Packet mark, Routing mark and Connection mark. Could using Connection mark here be of some help?

Well yes,

(1) create the necessary table.
/routing table add name=SpecificTraffic fib ( necessary for Ver 7 firmware )

(2) create the mangle rule to identify the traffic,
/ip firewall mangle
add chain=prerouting in-interface=WANX mark=no-mark action=mark-routing new-routing-mark=SpecificTraffic { (same for v6 and v7)}

(3) Create an additional route besides the standard route for WANX.
add dst-address=0.0.0.0/0 gateway=WANX-gateway routing-mark=Specific Traffic (<--- V6 )
add dst-address=0.0.0.0/0 gateway=WANX-gateway table=SpecificTraffic (<--- V7)

Unless you post your latest config its hard to say what is going on.

/export file=anynameyouwish ( minus router serial number and any public WANIP info )

Attaching my config here
I have redacted some of the personal information with ##### in the file, like usernames, local IP address leases (LAN), etc
Also adding some context about the WAN interfaces, there are a total of 3 (you can ignore the 3rd one which is LTE)
Primary internet interface - ExcitelFTTH - Primary - connects to a fiber ONU 192.168.9.1
Secondary internet interface - BSNL (PPPoE), physically connects via the interface called LHG (BSNL was temporarily disabled at the time I collected the config, normally it is not disabled)

Happy to give you more info if you need any.

Thanks

Unless you post your latest config its hard to say what is going on.

/export file=anynameyouwish ( minus router serial number and any public WANIP info )

Assuming: These are the current two routes......
/ip route
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
192.168.9.1 pref-src="" routing-table=main scope=30 suppress-hw-offload=\
no target-scope=10
add comment="For Cloud Update" disabled=no distance=1 dst-address=0.0.0.0/0 \
gateway=BSNL pref-src="" routing-table=only_via_BSNL scope=30 \
suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=BSNL pref-src="" \
routing-table=OutInitConn_BSNL scope=30 suppress-hw-offload=no \
target-scope=10

Suggest that you make WAN2 a higher distance which clearly points as ISP1 being the primary and the ISP2 being the secondary or backup.
The route for your second ISP has to be the normal standard route.
Then you add an additional route for the markings...........

Speaking about mangling
/ip firewall mangle
add action=mark-connection chain=prerouting comment=\
"Connection from WAN (BSNL)" in-interface=BSNL new-connection-mark=\
conn_from_outside_BSNL passthrough=yes

Modify this to.....
add action=mark-connection chain=prerouting comment=\
"Connection from WAN (BSNL)" in-interface=BSNL mark=no=mark new-connection-mark=\
conn_from_outside_BSNL

The routing.... ( should include the standard router for BSNL and the additional table route )
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=BSNL pref-src="" \
table=main scope=30 suppress-hw-offload=no \
target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=BSNL pref-src="" \
routing-table=OutInitConn_BSNL scope=30 suppress-hw-offload=no \
target-scope=10

+++++++++++++++++++++++++++++++++++++

I am not sure what effect the ddns update route has for mycloud, but suggest you disable that route for testing purposes to eliminate it as a factor in getting the connection to BSNL working properly.

I had a hard time understanding your config I would much simplify it with one bridge and vlans ( and not use vlan1 ) etc..... but if its working for you............. no need.

I know the config is a bit messy
I'll try to put here just the snippets of the config relevant to this topic.

First things first, I did take your advice and change the distance of BSNL from 2 to 10, and the LTE connection to 20 to give a more clear idea on the preference of each route.
This is the current config for routes:
You were interested in the extra route I have marked For Cloud Update. It is linked to this topic
viewtopic.php?p=973860#p973860
Instead of adding a new routing table for the mangle rule this time, I used the same routing table I have for the DDNS update

Currently the routing looks like this
All the routes above are in the 'main' routing table except the last one.

Coming to the mangle rule for the inbound traffic from BSNL
I did try using the exact command you mentioned in the last post but that seemed to have a syntax error point where the CLI gave me the syntax error marker is mark=no=mark . I did try changing it to mark=no-mark, didnt help either. Looks like 'mark' is not a valid command in mangle.

Anyway, after all of this was in place, i.e. Mangle rule configured to mark everything that comes in via BSNL, with action to mark routing, and then the routing mark pinting to the only_via_BSNL routing table. I did try reaching an inside host from an outside network. I also had a PCAP running to give me a clear understanding of the traffic flow
Looks like its the exact same thing happening again. I'll try to explain how the TCP handshake went.

----Excitel is WAN1 and BSNL is WAN2----

-SYN is recieved at the BSNL interface from the outside host on the Internet.
-RouterOS runs NAT, and forwards it to the internal host (192.168.1.2)
-SYN-ACK is received from 192.168.1.2 on the internal bridge interface
-NAT changes the source IP to (Public IP) BSNL and destination to the remote(outside host) IP
-Routing forwards the NATed traffic to the MAC address of the Excitel ONU via the ethernet interface that connects to it. ------------This is the step where things go wrong.
-The remote host would eventually receive the SYN-ACK, but from an IP (Public IP of the Excitel network) which it never sent a SYN to.
-Hence an ACK never comes back and the TCP handshake fails.

I still think the reason behind this is the direction of traffic. Our marked traffic for routing is the traffic coming in from BSNL. (Direction - BSNL>Internal host) but the traffic we really want the policy to apply to is the reply traffic (Outbound from the internal host back to the router).
We need to somehow identify/mark the return traffic (maybe using the NAT table if its possible, because its working as expected till the NAT process). The layer 3 forwarding of the traffic is working as expected, its just the actual frame being forwarded at layer 2 where the problem really is.

Hoping that my ideas here help with figuring out something.

Any ideas from someone after my last post?

post the complete config and I will have a look.,

here is the config file
like the last time, I have redacted the personal info and replaced it with #############

Anything new here?
Did my last detailed post help with any ideas?

(1) I find this confusing........
add action=masquerade chain=srcnat comment="NAT to Internet - FTTH" \
out-interface=ExcitelFTTH

I dont think its correct or required.

(2) Nor do I understand this address, you have already it in pppoe....... so do you have three WANS then ????
add address=192.168.9.10/24 comment=Excitel interface=ExcitelFTTH network=\
192.168.9.0

(3) Not sure its a good idea to have your LAN pool and L2TP VPN lan pool in the same subnet but I dont know and its probably fine. Just something I would never do.

(4) Where is 192.168.10.0 located, I dont see it defined anywhere but in the routing??
/ip route
add check-gateway=ping disabled=no distance=1 dst-address=192.168.10.0/24 \
gateway=192.168.6.2 pref-src="" routing-table=main scope=30 \
suppress-hw-offload=no target-scope=10

(5) Why is there a routing for FFTH, this is done through PPPOE.....

(6) This rule says nothing................... purpose?
/routing rule
add action=lookup disabled=no table=main

(7) I would expect a routing rule for this route.........
add comment="For Cloud Update" disabled=no distance=1 dst-address=0.0.0.0/0 \
gateway=BSNL pref-src="" routing-table=only_via_BSNL scope=30 \
suppress-hw-offload=no target-scope=10

something along the lines of:
add routing-mark=only_via_BSNL action=lookup-only-in-table table=VIA-BSNL

You will note I changed the name of the table as i found it very confusing to name the table the same as the routing mark.............
They are two separate functions. One names the table to tell the router which table to look up for routing purposes, the routing mark is used to tag/identify traffic for later use.............

+++++++++++++++++++++++

Overall I dont understand what you are doing with the second WAN and why it appears you are trying to get two WANs out of the pppoe connection.