Community discussions

 
User avatar
eworm
Member
Member
Topic Author
Posts: 334
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

blackhole/unreachable with IPSec policies

Sat Jul 13, 2019 12:13 pm

Hello everybody,

with my current VPN provider I use l2tp/IPSec, which works with an interface. I add routing marks, then add a route for these marks to my interface. A second route makes sure no traffic is routed when the interface is down:
/ ip route add distance=1 gateway=l2tp-pia routing-mark=via-vpn
/ ip route add distance=10 routing-mark=via-vpn type=unreachable
Now migrating to NordVPN with just IPSec/IKEv2 verything works fine, including new connection-mark option for mode-config from 6.46beta9. I am missing one detail, though: If the peer is down all traffic is routed to my ISP without VPN. What's the best way to have the functionality of blackhole/unreachable route with IPSec policies?
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
msatter
Forum Guru
Forum Guru
Posts: 1114
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: blackhole/unreachable with IPSec policies

Sat Jul 13, 2019 12:46 pm

I am using a in rules that drops forwarded traffic that has the connection mark and goes out through the WAN. I am using the specific ports now instead of connection mark.

I am using this this for L2TP/IPSEC but maybe this can be used for IKEv2 to.

Sindy made a blackhole by using an extra bridge.

Then:
/ ip route add distance=1 connection-mark=via-vpn type=unreachable
Last edited by msatter on Sat Jul 13, 2019 12:52 pm, edited 1 time in total.
Two RB760iGS (hEX S) in series. One does PPPoE/IKEv2 and the other does the rest of the tasks.
Running:
RouterOS 6.46Beta / Winbox 3.19 / MikroTik APP 1.2.8
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
eworm
Member
Member
Topic Author
Posts: 334
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: blackhole/unreachable with IPSec policies

Sat Jul 13, 2019 12:50 pm

With l2tp this is quite easy as routing goes to different interfaces. With IPSec policies things work different.
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
msatter
Forum Guru
Forum Guru
Posts: 1114
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: blackhole/unreachable with IPSec policies

Sat Jul 13, 2019 12:55 pm

You want to stop traffic what is not caught by IPSEC and so it is normal traffic.
Two RB760iGS (hEX S) in series. One does PPPoE/IKEv2 and the other does the rest of the tasks.
Running:
RouterOS 6.46Beta / Winbox 3.19 / MikroTik APP 1.2.8
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
eworm
Member
Member
Topic Author
Posts: 334
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: blackhole/unreachable with IPSec policies

Sat Jul 13, 2019 1:51 pm

Yes. But I can not decide by out interface as that does not differ with policies.
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
msatter
Forum Guru
Forum Guru
Posts: 1114
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: blackhole/unreachable with IPSec policies

Sat Jul 13, 2019 2:09 pm

IKEv2 is not having it's own interface and encrypted traffic uses WAN. Traffic (not encrypted) marked for IKEv2 has so nothing lost in routing so when detected be indicated as unreachable.
Last edited by msatter on Sat Jul 13, 2019 2:13 pm, edited 1 time in total.
Two RB760iGS (hEX S) in series. One does PPPoE/IKEv2 and the other does the rest of the tasks.
Running:
RouterOS 6.46Beta / Winbox 3.19 / MikroTik APP 1.2.8
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
sindy
Forum Guru
Forum Guru
Posts: 3488
Joined: Mon Dec 04, 2017 9:19 pm

Re: blackhole/unreachable with IPSec policies

Sat Jul 13, 2019 2:10 pm

The extra bridge with no member ports and no IP address assigned is used as a gateway of routes to all destinations to which the traffic should only get via the IPsec SA.
The point is that IPsec policies' traffic selectors choose the packets for interception on their way to the interface, so any route which doesn't let them get out via an interface will also prevent them from being intercepted by the policy.
And although it normally doesn't make much sense to use a point-to-multipoint interface as a gateway, there are cases where it does, so it is a permitted configuration which is likely to stay.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eworm
Member
Member
Topic Author
Posts: 334
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: blackhole/unreachable with IPSec policies

Sun Jul 14, 2019 12:00 am

I added this rule as a workaround... It catches the packets if the dynamic rule by mode-config is not present.
/ip firewall nat add action=src-nat chain=srcnat connection-mark=via-vpn to-addresses=127.0.0.1
However it is kind of blackhole only, there's no way to make the client receive unreachable messages. Also packets are sent to the default gateway, with unroutable address 127.0.0.1. This may bring some undesired side effects.

I could think of double-NAT with a pair of ipip Interfaces... But that makes the configuration a lot more complicated compared with what I have now.

sindy, I did not get the idea of your thoughts. If I route the packets to a bridge with no ports and no addresses I have a blackhole. But there is no way for the packets to be encrypted and sent the right way, no? Can you explain in more detail and possibly with example configuration?
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
sindy
Forum Guru
Forum Guru
Posts: 3488
Joined: Mon Dec 04, 2017 9:19 pm

Re: blackhole/unreachable with IPSec policies

Sun Jul 14, 2019 9:06 am

sindy, I did not get the idea of your thoughts. If I route the packets to a bridge with no ports and no addresses I have a blackhole. But there is no way for the packets to be encrypted and sent the right way, no? Can you explain in more detail and possibly with example configuration?
The order of actions is
  1. use routing to find the outgoing interface
  2. execute the postrouting chain of the firewall (including srcnat)
  3. check a match to IPsec policy and send the packet via the policy's SA if it matches
  4. send the packet out the interface chosen in step 1 if it didn't match any IPsec policy
So the difference between a type=blackhole route and a bridge with no ports is that a blackhole route drops the packet already at step 1, whereas a "blackhole bridge" only drops it if it didn't match any IPsec policy.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
msatter
Forum Guru
Forum Guru
Posts: 1114
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: blackhole/unreachable with IPSec policies

Sun Jul 14, 2019 11:32 am

I added this rule as a workaround... It catches the packets if the dynamic rule by mode-config is not present.
/ip firewall nat add action=src-nat chain=srcnat connection-mark=via-vpn to-addresses=127.0.0.1
However it is kind of blackhole only, there's no way to make the client receive unreachable messages. Also packets are sent to the default gateway, with unroutable address 127.0.0.1. This may bring some undesired side effects.
Mine would be:
/ip firewall filter add action=reject chain=forward connection-mark=via-vpn out-interface-list=WAN reject-with=icmp-host-unreachable
Two RB760iGS (hEX S) in series. One does PPPoE/IKEv2 and the other does the rest of the tasks.
Running:
RouterOS 6.46Beta / Winbox 3.19 / MikroTik APP 1.2.8
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
sindy
Forum Guru
Forum Guru
Posts: 3488
Joined: Mon Dec 04, 2017 9:19 pm

Re: blackhole/unreachable with IPSec policies

Sun Jul 14, 2019 10:23 pm

@msatter, /ip firewall filter rules act before before policy matching, so your rule will drop the packets also if the IPsec policy is active. You also cannot add an ipsec-policy=out,none condition into the /ip firewall filter rule for the purpose, because before the src-nat happens after filter rules have already acted, all transit traffic matches that condition so again everything would be dropped before it could get src-nated and subsequently matched by the policy.

@eworm's solution with an action=src-nat rule, which is shadowed by the dynamically added one while the IKEv2 connection is up, can be improved this way in terms that the packets not intercepted by the ipsec policy won't make it even to the default gateway, but I can see no advantage as compared to my "blackhole bridge" way, and I'm afraid that having /interface bridge filter rules defined has some impact even on operation of bridges to which those rules are unrelated.

However, I could not find any way how to provide the icmp feedback (administratively prohibited, unreachable etc.) except by using the ipip tunnel to self instead of the "blackhole bridge", and using policy routing to send anything that comes in via the "far" end of that tunnel to a marked route with type=unreachable or prohibit. The question is whether it is worth the effort - for me it wasn't, so I didn't test it and thus I may be missing some caveats. Plus when the VPN connection drops for a while and then re-establishes, silent drops don't break the established TCP sessions whereas icmp feedback packets likely would; this was another reason for me not to pay attention to the possibility to send them.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
msatter
Forum Guru
Forum Guru
Posts: 1114
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: blackhole/unreachable with IPSec policies

Sun Jul 14, 2019 10:53 pm

I do not agree because forwarded traffic is only dropped if it on his way to the WAN while it is tagged to go through the IKEv2 connection.

In the NAT itself the first split is made by only NAT traffic to the WAN the is not connection marked.

So if traffic is on it's way through Mangle and marked and the NAT IKEv2 is not already inserted or is removed it will not find NAT to the Wan. The filter rule is there to be absolute sure noting goes out the Wan that not meant to be and so being a back-stop.

If you don't use mode-config (source address/connection marking) traffic can escape because it is not marked or source bound. If possible the omitting the NAT to Wan is an option.
Two RB760iGS (hEX S) in series. One does PPPoE/IKEv2 and the other does the rest of the tasks.
Running:
RouterOS 6.46Beta / Winbox 3.19 / MikroTik APP 1.2.8
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1114
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: blackhole/unreachable with IPSec policies

Sun Jul 14, 2019 11:16 pm

The next step is when it hit the IKEv2 Nat rule and the connection is not there anymore what happens then?
It will head for the WAN which is transfering anything not specific caught by other routing rules.

Outgoing encrypted traffic will fall dead because the receiving IKEv2 server is not there anymore.
Plain traffic that I see is that with other IPSEC connections, that I get traffic not for me but for the previous user of that dynamic IP address.
The VPN provider should block these cross overs, it is leaking traffic.

Statefull filtering will stop that but if you want you can see the content if you accept it. (think peer to peer sharing)
Two RB760iGS (hEX S) in series. One does PPPoE/IKEv2 and the other does the rest of the tasks.
Running:
RouterOS 6.46Beta / Winbox 3.19 / MikroTik APP 1.2.8
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
eworm
Member
Member
Topic Author
Posts: 334
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: blackhole/unreachable with IPSec policies

Mon Jul 15, 2019 12:46 am

The order of actions is
  1. use routing to find the outgoing interface
  2. execute the postrouting chain of the firewall (including srcnat)
  3. check a match to IPsec policy and send the packet via the policy's SA if it matches
  4. send the packet out the interface chosen in step 1 if it didn't match any IPsec policy
So the difference between a type=blackhole route and a bridge with no ports is that a blackhole route drops the packet already at step 1, whereas a "blackhole bridge" only drops it if it didn't match any IPsec policy.
Well, 1. is just routing to the default gateway, no? Then 2. does the src-nat with dynamic address from mode-config, 3. is ipsec encapsulation after firewall. At this point we do not have any way to manipulate the out interface. So how should 4. work? Oder did I misunderstood 1.?
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
sindy
Forum Guru
Forum Guru
Posts: 3488
Joined: Mon Dec 04, 2017 9:19 pm

Re: blackhole/unreachable with IPSec policies

Mon Jul 15, 2019 9:50 am

@msatter, I'll come back to your posts later, first the OP. But maybe the following is an answer to your posts too?

@eworm,
Oder did I misunderstood 1.?
Either you did, or I've misunderstood your goal.

My understanding of your goal is that you want to be sure that those pakets, which should be sent via the VPN, will under no circumstances get to the destination via any other path if the VPN connection fails. The static srcnat rule you've posted above supports that understanding.

Is my understanding correct? I'll only continue explaining if you confirm that.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eworm
Member
Member
Topic Author
Posts: 334
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: blackhole/unreachable with IPSec policies

Mon Jul 15, 2019 10:55 am

@eworm,
Oder did I misunderstood 1.?
Either you did, or I've misunderstood your goal.

My understanding of your goal is that you want to be sure that those pakets, which should be sent via the VPN, will under no circumstances get to the destination via any other path if the VPN connection fails. The static srcnat rule you've posted above supports that understanding.

Is my understanding correct? I'll only continue explaining if you confirm that.
Yes, correctly.
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
sindy
Forum Guru
Forum Guru
Posts: 3488
Joined: Mon Dec 04, 2017 9:19 pm

Re: blackhole/unreachable with IPSec policies  [SOLVED]

Mon Jul 15, 2019 1:05 pm

OK, so I continue.

What happens with a packet if it is not intercepted by an IPsec policy is determined before the match to IPsec policy's traffic selector is attempted. There is no L3 packet processing layer after the IPsec policy matching which could take any action if no policy matches the packet. All routing and all evaluation of routing-mark, connection-mark etc. takes place before IPsec policy matching as this diagram shows.

Vice versa, if the packet is not routed towards any interface because it matches a blackhole, prohibit or unreachable route, it can never be matched by any IPsec policy (nor even src-nated).

In case of IKEv2 with mode-config and the src-nat rule it creates dynamically, there is additionally the src-nat action in step 2 which is only taken if that IKEv2 connection is active.

But already this is a caveat: only the initial packet of each tracked connection is actually treated by the srcnat chain of /ip firewall nat, whereas the src-nat action on the subsequent outbound packets, as well as the "un-src-nat" action on the inbound packets of the same connection, is done by the connection tracking module alone. So all packets, which belong to connections which have been src-nated to the IP assigned by the VPN server by the dynamically added src-nat rule while the VPN was up, will continue to be src-nated by the connection tracking module even after the VPN connection breaks. But as no IPsec policy will matche them while the VPN will be down, they will be sent out the gateway interface chosen by routing, except that with a wrong source IP address. How far such packet can get depends on the anti-spoofing policies applied along the path, and there is unfortunately no guarantee that your ISP will drop them, so they may even make it to the destination - either with the source address being the one you got from the VPN server when the IKEv2 connection came up, or with your public IP if you don't have a public IP on the Mikrotik itself but on ISP's routing modem.

When your client host initiates a new connection while the VPN has already disconnected, the dynamically created src-nat rule is not present in the srcnat chain, so also in the postrouting chain, the initial packet is treated as any other one.

So:
  • to make sure that the packets can be matched by an IPsec policy, they must be routed out via some interface,
  • to make sure that they won't leak if they are not matched by an IPsec policy (because the VPN is down), they must be routed out via an interface which actually sends them nowhere.
These two requirements taken together mean that packets belonging to this category must have their own route in step 1 (with the blackhole bridge as a gateway interface), not the common default route which usually uses the WAN as its gateway.

But again, it is just one way how to do it. Your approach, with the "emergency" src-nat rule changing the source IP of those packets which otherwise the dynamically added src-nat rule would process, works as well, except that if used without additional measures, it only prevents the leaked packets from carrying your WAN IP, which may not be sufficient as said above - if your ISP box does NAT, it is likely not to care about the source IP of the packet you've sent to it, so the leaked packets will be sent to the original destination with your normal public IP. So to make that approach fulfil the real goal, you have to complement it with the bridge filter setup I've linked to above, which is the last place where you can stop packets with undesired src-address from being actually sent. However, there are complications associated to that step:
  • you cannot insert the bridge filter into the path if the WAN interface is an L3 tunnel (such as PPPoE),
  • if also the WAN interface address is dynamically assigned, the bridge filter rule must work with a subnet or more than one may be necessary to let through the traffic which needs to be let through and drop the leaks.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eworm
Member
Member
Topic Author
Posts: 334
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: blackhole/unreachable with IPSec policies

Mon Jul 15, 2019 3:38 pm

Ah, got it! :D :lol:

My false assumption was that I thought...
  • Routing with type=blackhole is the same as routing to an interface with no addresses. Of course it is not.
And even more important that I thought...
  • Routing decision is done earlier in flow for unencrypted packet. It is not, or better: Looks like routing decision is reconsidered after IPSec encapsulation.
So yes, this is the simplest and most straight forward solution. The important configuration bits look like this:
# create a bridge with no ports and no addresses
/interface bridge add name=blackhole

# route the vpn traffic to this interface
/ip route add distance=1 gateway=blackhole routing-mark=via-vpn
Well, that's it, no? Of course you need to add routing-mark (for route above) and connection-mark (for dynamic src-nat from mode-config) in firewall mangle.

BTW, previously I used an unreachable route. After sindy's comments I reconsidered and I am happy with blackhole now. Probably it's a good idea if connections do not break if vpn link is down for just a short moment.

sindy, I owe you a beer (or two or some more... ) Thanks a lot for help and detailed explanation!
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
sindy
Forum Guru
Forum Guru
Posts: 3488
Joined: Mon Dec 04, 2017 9:19 pm

Re: blackhole/unreachable with IPSec policies

Mon Jul 15, 2019 4:14 pm

Looks like routing decision is reconsidered after IPSec encapsulation.
Well... the IPsec transport packet carrying the encrypted original packet is actually a brand new one locally originated by the Mikrotik, so it is not a reconsidered routing decision, it is merely an independent routing decision.

Well, that's it, no? Of course you need to add routing-mark (for route above) and connection-mark (for dynamic src-nat from mode-config) in firewall mangle.
Yes.

Probably it's a good idea if connections do not break if vpn link is down for just a short moment.
Unfortunately it is probably useless in your particular scenario because if I get the behaviour right, you are not guaranteed to get the same IP address after re-connection, no matter how long the disconnection took. In my scenarios where the VPN server is the same one all the time and each user gets the same address all the time it is another story.

I owe you a beer (or two or some more...)
Thanks, I'll let you know once I'll be roaming nearby :) But the probability seems to be low for next few months.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
sindy
Forum Guru
Forum Guru
Posts: 3488
Joined: Mon Dec 04, 2017 9:19 pm

Re: blackhole/unreachable with IPSec policies

Mon Jul 15, 2019 4:23 pm

@msatter, did my detailed post #17 explain what I had in mind when saying that your rule suggested in post #10 will drop the packets regardless whether they would be finally intercepted by an IPsec policy? I still have a feeling that the mutual misunderstanding may come from the fact that you use a different overall concept which I didn't get, and that rule is just a part of it.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
eworm
Member
Member
Topic Author
Posts: 334
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: blackhole/unreachable with IPSec policies

Mon Jul 15, 2019 5:35 pm

@msatter, did my detailed post #17 explain what I had in mind when saying that your rule suggested in post #10 will drop the packets regardless whether they would be finally intercepted by an IPsec policy? I still have a feeling that the mutual misunderstanding may come from the fact that you use a different overall concept which I didn't get, and that rule is just a part of it.
He is using L2TP/IPSEC and has an interface. :wink:
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
sindy
Forum Guru
Forum Guru
Posts: 3488
Joined: Mon Dec 04, 2017 9:19 pm

Re: blackhole/unreachable with IPSec policies

Mon Jul 15, 2019 5:58 pm

He is using L2TP/IPSEC and has an interface. :wink:
Nope, he has a very clever arrangement where one machine provides an IKEv2 connection and another machine chooses which traffic will use the VPN and which will use the direct WAN, which he uses to beat the trouble to do that on a single machine (as you cannot tell the policy what to intercept and what to ignore using any other means than src-address whilst the src-address keeps changing).
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
msatter
Forum Guru
Forum Guru
Posts: 1114
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: blackhole/unreachable with IPSec policies

Mon Jul 15, 2019 7:39 pm

I have to have a good look at it when I have access to Winbox and my RouterOS in front of me. Seeing what is catching the first and tailing packets. So I have to come back on this.

I am using L2TP/IPSEC on the inner box and the outer (GW) is transparant for that. IKEv2 is routed on the inner bock and is then fasttracked. The outer box does the IKEv2 stuff which can't be fasttracked.

The advantages is more speed due to double processing power and lower temperatures in the inner box. I don't have a 4011 which can archive even higher encrypted transfer rates.
Two RB760iGS (hEX S) in series. One does PPPoE/IKEv2 and the other does the rest of the tasks.
Running:
RouterOS 6.46Beta / Winbox 3.19 / MikroTik APP 1.2.8
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)

Who is online

Users browsing this forum: No registered users and 53 guests