Community discussions

MikroTik App
 
petterg
Member Candidate
Member Candidate
Topic Author
Posts: 230
Joined: Wed Sep 16, 2009 2:55 pm

Routing table ignoring routing mark

Sun Aug 26, 2012 2:17 am

I followed the wiki at http://wiki.mikrotik.com/wiki/PCC to set up loadbalancing of two internet connections. Because of an ipsec tunnel I added the following:
/ip firewall mangle add chain=prerouting dst-address=172.29.5.0/24 action=accept
/ip firewall mangle add chain=prerouting dst-address=1.2.3.0/24 action=mark-connection new-connection-mark=ISP1_conn
/ip firewall mangle add chain=output dst-address=1.2.3.0/24 action=mark-connection new-connection-mark=ISP1_conn
(1.2.3.0/24 is a subnet including the ipsec remote peer 1.2.3.4. 172.29.5.0/24 is the ipsec remote local subnet)

I suspected it didn't work (ipsec was unstable), so I did some testing. I deleted the two routes to 0.0.0.0/0 without routing-mark and added the following:
/ip firewall filter add action=log chain=output disabled=no dst-address=1.2.3.0/24 log-prefix="ipsec out:"  routing-mark=to_ISP1
/ip firewall filter add action=log chain=forward disabled=no dst-address=1.2.3.0/24 log-prefix="ipsec fwd:"  routing-mark=to_ISP1
Now, when I ping 1.2.3.3 or 1.2.3.4 (or try to establish ipsec from my local side to 1.2.3.4) I see in the log quite frequently that the log display the package will be using ISP2 as out interface, with ISP1 as source address! I think that explains why the ipsec is so unstable. (Further logging show that ipsec packets go out on IPS2 interface, and returning packets come on ISP1 interface.)
And the fact that anything shows up in the log at all proves that the routing-mark is correct.

Then I run the following commands:
/ip route
add dst-address=0.0.0.0/0 gateway=10.111.0.1 distance=5 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.112.0.1 distance=2 check-gateway=ping
In my understanding of routing-mark, these two rules should not make any difference at all, as there are other routing rules with lower distance that matches the routing-mark. But they do make a difference. After these two commands are entered the packets to the ipsec peer is sent on the ISP2 interface 100% of the times, except when establishing the tunnel. (For some reason tunnel is established. But all packets thereafter are sent on the wrong interface.) And ping are always sent on the wrong interface.

But if I add the route
add dst-address=1.2.3.0/24 gateway=10.111.0.1 distance=1 check-gateway=ping
everything works fine with the ipsec. But still most of the packages to the internet are using ISP2.


So, my question. Is it me doing something completely wrong here, or is routing-mark frequently ignored in the routing table?

The real setup where I experienced the issue was on a RB450G with ROS4.11
The issue is reproduced on a RB751G with ROS 5.11, which I used for all testing/debug. To make sure I didn't fail on translating the addresses in the commands from the wiki, I setup a complete test environment using the exact same addresses as the wiki, using two more rb751g as ISP* routers, and even two more to create a "internet" with an remote ipsec peer.
 
petterg
Member Candidate
Member Candidate
Topic Author
Posts: 230
Joined: Wed Sep 16, 2009 2:55 pm

Re: Routing table ignoring routing mark

Sun Aug 26, 2012 5:06 am

A litle bit of additional information:
This logging catches the ipsec packets with wrong out-interface:
/ip firewall mangle 
add action=log chain=postrouting disabled=no dst-address=192.168.9.123 log-prefix=POST: routing-mark=to_ISP1
While these two does not:
/ip firewall nat
add action=log chain=srcnat disabled=yes dst-address=1.2.3.0/24 log-prefix=SRC:
add action=log chain=srcnat disabled=yes dst-address=1.2.3.0/24 log-prefix=SRCmark: routing-mark=to_ISP1
(rules moved to top of list)

SRCNAT is, as far as I know, the last place in the router one can see the packets before they leave. I find it strange that the packets doesn't show up there.
 
petterg
Member Candidate
Member Candidate
Topic Author
Posts: 230
Joined: Wed Sep 16, 2009 2:55 pm

Re: Routing table ignoring routing mark

Tue Aug 28, 2012 2:44 am

Nobody has experience with routing-mark?
 
lelo
just joined
Posts: 9
Joined: Fri Apr 04, 2014 3:20 pm
Location: Poland

Re: Routing table ignoring routing mark

Thu Oct 30, 2014 2:50 pm

Hi petterg,

I am not sure if my issue is related to the same problem, but I had also problem with marking IPsec packets
and I have found reason/solution/bug ??.

I do not want to paste here all my config, so I will try explain just the problem the best way I can.

I have dual WAN config (no BGP) (with additional scripts for WAN fail-over switching, but it does not matter).

I have mangle rules, based on this site:
https://aacable.wordpress.com/2011/07/2 ... t-by-zaib/

just short of few ones:
/ip firewall mangle
add chain=input in-interface=WAN1 action=mark-connection new-connection-mark=WAN1_conn
add chain=input in-interface=WAN2 action=mark-connection new-connection-mark=WAN2_conn
In my case I have chain=prerouting here, so all incomming connections to the router (both router itself and destined for my local subnets) are connection-marked.

I have VPNs connections with remote locations,
(GRE tunnels working over IPsec).
The GRE tunnels work on local IP's (10.255.255.10/32 - central office, 10.255.255.20/32 - remote office).
The IPsec policies are set up only for those IP's so they are encapsulated and encrypted with IPsec.

I have two GRE tunnels to every remote location (one working on our WAN1 and second working on our WAN2).

What I have discovered, that outgoing IPsec packets that are marked to WAN2 are not really outgoing with correct interface.
(they were outgoing throught WAN1 (default)).
As both our ISP's does not filter our source addresses, we can use IP address assigned by ISP1 and packet can succesfully go out using ISP2 interface and vice-versa. So I was not even conscious, that bad thing is happening..

After 2 day hard investigation, log analisys, I have found the reason !!

All decrypted packets that comes from IPsec tunnel (those one 10.255... on which GRE tunnels are working) in firewall acts with incomming interface = physical interface that IPsec packet was received.
That is actually nothing new for me, I knewed it before and this is probably OK.
This way, according to mangle rules from above, the GRE connections (with local/private IPs) are also connection-marked,
and causes that outgoing GRE packet is correctly marked for correct outgoing WAN interface.

But it completely does not matter, as the outgoing GRE packet is catched by IPsec polisy, encrypted, encapsulated, and whole process starts again. http://wiki.mikrotik.com/images/thumb/2 ... _final.png

and.. here the problem beggins.. !
Now, the outgoing IPsec packet should be also marked and routed via the correct wan interface, but this is not working.
Either route-marking rule is not working or the routing mark is ignored during the routing.

and.. now the solution/reason ! This problem with marking/routing of IPsec packets is casued by previously route-marking of packet that was encapsulated into IPsec !!!!!! (GRE packets in my case)

I have completely no idea.. what is the relation/reason of route-marking packets that comes into IPsec tunnel, with marking IPsec packets itself ???

Solution was very simple, I have updated the above rules with destination IP subnets of WAN links,
so now only real packets coming into WAN interfaces destinet to public IP's are connection marked.
(GRE packets are not marked now).

I have made several tests, not only with GRE but with simple ICMP ping over those private IP's used by GRE.
And I can confirm: every packet that is marked for routing but finally encapsulated into IPsec packet, cause that route-marking for this IPsec packet is not working.

Please somebody guys from MikroTik, confirm if this is a bug or I do not know something ?
(checked on ROS 6.18 and 6.20)
 
611
newbie
Posts: 37
Joined: Wed Oct 17, 2018 10:12 am

Re: Routing table ignoring routing mark

Mon May 22, 2023 10:19 am

It's 2023 and ROS v7.9, the same thing, still undocumented.

I've resorted to just resetting the routing mark as the first rule in mangle output chain:
/ip firewall mangle add chain=output action=mark-connection connection-state=new,untracked ipsec-policy=out,ipsec new-connection-mark=no-mark passthrough=yes log-prefix=cm-reset comment="reset connection mark for outgoing packets to be IPsec encrypted as they break IPsec route adjustment (routing marks on encrypted packets stop working)" 
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7041
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Routing table ignoring routing mark

Mon May 22, 2023 12:40 pm

Look at the packet flow diagram and specifically ispec example
https://help.mikrotik.com/docs/display/ ... n+RouterOS

If you want to reroute ipsec packets then you have to do routing marks after the ipsec encapsulation. Ipsec doesn't care about any prior marks or prior routing decisions. It encapsulates and sends packet according to its policies.

If you want to reroute incoming decrypted packets then same thing applies but in reverse order, marking must be done after the decryption process.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Routing table ignoring routing mark

Mon May 22, 2023 2:26 pm

What is the difference between IPsec tunnel mode and IPsec transport mode?

IPsec tunnel mode is used between two dedicated routers, with each router acting as one end of a virtual "tunnel" through a public network. In IPsec tunnel mode, the original IP header containing the final destination of the packet is encrypted, in addition to the packet payload. To tell intermediary routers where to forward the packets, IPsec adds a new IP header. At each end of the tunnel, the routers decrypt the IP headers to deliver the packets to their destinations.

In transport mode, the payload of each packet is encrypted, but the original IP header is not. Intermediary routers are thus able to view the final destination of each packet — unless a separate tunneling protocol (such as GRE) is used.

Does the posting above means that RouterOS is always using a tunnel? In transport mode only the payload is encrypted so normal routing is used.

Source: https://www.cloudflare.com/learning/net ... -is-ipsec/
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7041
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Routing table ignoring routing mark

Mon May 22, 2023 2:38 pm

It is still ipsec encapsulation/decapsulation and after the encap/decap packet is not the same.

Who is online

Users browsing this forum: biomesh, GuJack20, korg, mogiretony and 85 guests