IPSec via second ISP don't work in 7.21.3

RouterOS General
1

Hello everyone.
Need some help with settings IPSec via second ISP.
Sorry if i wrote something wrong.

I have 2 offices. First office with mikrotik have 2 ISP, second office have 1 ISP.

In office with mikrotik where 2 ISP, first ISP default, second ISP backup.

I established IPSec connection via first ISP and IPSec works fine.
But when i tried to establish IPSec via second ISP, i got error "no phase 2"

In second office i got log and see this(I changed IP):
X_IP - first ISP IP from mikrotik
Y_IP - second ISP IP for backup from mikrotik
"CONNECTION STATUS: DOWN... peer: X_IP:4500, phase1_id: Y_IP"

What mean, mikrotik still use default ISP, not backup but in phase1_id IP address is second ISP

I read documentation about IPSec and second ISP here:

https://help.mikrotik.com/docs/spaces/ROS/pages/11993097/IPsec#IPsec-Manuallyspecifyinglocal-addressparameterunderPeerconfiguration

I found this section:
'IPsec, as any other service in RouterOS, uses the main routing table regardless of what local-address parameter is used for Peer configuration. It is necessary to apply routing marks to both IKE and IPSec traffic.'

I tried to create mangle rule and add route for marked traffic via second ISP like in documentation.
But mikrotik still use default ISP for establish IPSec not backup ISP.

I tried to check mikrotik mangle log and see this:
X_IP - first ISP IP from mikrotik
Y_IP - second ISP IP for backup from mikrotik
Z_IP - second office IP
ether6 - default ISP port

IPSEC TEST output: in:(unknown 0) out:ether6, connection-state:established,snat proto UDP, Y_IP:4500->Z_IP:4500, NAT (Y_IP:4500->X_IP:49715)->Z_IP:4500, len 272

Mikrotik Model: CCR1009-8G-1S
RouterOS: 7.21.3 (stable)

Can anyone help to understand why this happened and how to say mikrotik to use second ISP and don't NAT to first ISP?

2

well, i had a similar situation with wireguard
idea is same, so maybe this will bring you closer to solution

mangle
you need to mark incomming packets with your DSTPORT
mark incommint connections on ISP2 interface
mark prerouting: packet->conn->routingMark
mark output packets (SRC PORT)->conn->routingMark (this is for router originated traffic)

then in firewall->nat
create few NAT rules
with marked packets to ISP2 if
marked connections to ISP2 if
and last all on outgoing ISP2 interface

ok, it looks overcomplicated but you will learn about mangle, you will see packets/connections and then you could decide what to keep :slight_smile:

hope it will be less strougle than WG

good news is, if this working with WG, it will work with anything :slight_smile:

3

There is not enough information about OP's problem in the post to determine where things are going sideways, but...

WG and IPSec are way different in this regard. Wireguard works by providing a virtual interface that is tapped in-kernel, so packets are absorbed and re-emitted, plus it has the additional complication of source address selection. IPSec on the other hand is a filter that matches certain packets, encapsulates them and reinjects them into the packet flow.

Because the packet flow is so different, what works for one is unlikely to help with the other. As an additional difficulty, IPSec has separate IKE and ESP traffic.

4

What is information i can additionally provide for more understanding?

5

Thanks for reply.
I don't clearly understand which settings need to do on mikrotik.

Can you provide your setting for example?

6

In particular:

  • your setup around having two IPs/ISPs (second routing table? vrf?)
  • public routable IP or CGNAT?
  • your IPSec configuration, both phase 1 and phase 2
  • your current mangle and routing rules setup
  • it's helpful to create fw logs for both IKE and encrypted traffic (as you don't have a valid phase 2 policy, probably there is no enrypted traffic)

It's useful to include both parts of the configuration you consider relevant and the full config. Sanitize/mask what you wish, just make sure it's reversible.

Use code blocks when including code.

7

I have

  1. just simple routing table created for IPSec via second ISP, and added route to routes
    DST-ADDRESS GATEWAY ROUTING-TABLE DISTANCE
    As 0.0.0.0/0 my second ISP IP backup 2

  2. Have 2 public routable IPs, not CGNAT

  3. My IPSec setting work with first ISP and they are simular for my second ISP, just changed only IP in Peer

  4. I created mangle rules like in documentation for my second office IP. I don't use routing rules

chain=output action=mark-connection new-connection-mark=ipsec1 passthrough=yes protocol=udp dst-address=second office's IP dst-port=500,4500

chain=output action=mark-connection new-connection-mark=ipsec1 passthrough=yes protocol=ipsec-esp dst-address=second office's IP

chain=output action=mark-routing new-routing-mark=backup passthrough=no connection-mark=ipsec1 log=yes

  1. When i enabled ipsec log i found only this error: ts_unacceptable, because first ISP send request with information about second ISP in packet.
    "CONNECTION STATUS: DOWN... peer: X_IP:4500, phase1_id: Y_IP"

I can't provide configuration, because have very more information for mask.

8

I ran into the same problem. It’s interesting that it worked on an earlier version of ROS, possibly 7.19.

/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-table=main
add disabled=no dst-address=0.0.0.0/0 gateway=pppoe-out2 routing-table=rtab-1

/ip firewall mangle
add chain=output action=mark-connection new-connection-mark=mark1 passthrough=yes protocol=udp dst-address=1.2.3.4 dst-port=500,4500
add chain=output action=mark-connection new-connection-mark=mark1 passthrough=yes protocol=ipsec-esp dst-address=1.2.3.4
add action=mark-routing chain=prerouting connection-mark=mark1 new-routing-mark=rtab-1

The remote MikroTik (1.2.3.4) sees the incoming connection, but the remote address is set to the address from isp1. Maybe if I specify the ISP2 address in /ip ipsec peer, that will be enough, but my ISP2 has a dynamic address.

9

That's not very helpful.

You will have to set local and remote addresses correctly on both sides as a first step.

@Nikalia You also have to do that if the address is dynamic. Use scripting, either for dhcp or pppoe.

And log IKE traffic at the fw level, so that it's clear which IP and connection is being used.

10

Thank for your information about 7.19
I have another Mikrotik, maybe i will try to downgrade version and check

11

I correctly set addresses on both sides.

Problem now not in my setting.
Problem now in fact that mangle, NAT and route setting fully ignored by ROS 7.21.3 if set them like documentaion saying for work with second ISP.

And i can't understand why

12

Without seeing the actual configuration it is complicated to help, but here's what can cause trouble:

  • "vanilla" IPsec works with policies and traffic selectors. At any time, only a single policy out of multiple ones with the same traffic selector (source and destination addresses and ports) can be active. So if you use vanilla IPsec, you must set the peer item of a single policy to a comma-separated list comprising the primary and standby peer, rather than set two policies, each linked to its own peer
  • on the dual-WAN end, configuring the local-address parameters of the two IPsec peers is not sufficient. The peer set to use the secondary address will listen on that address and use it as a source one for packets it will send, but the packets sent from that address will still use routing table main (whose default gateway is usually reachable through the primary WAN interface) unless you use a routing rule (or a firewall mangle rule assigning a routing mark) to tell the router to use a different routing table for traffic with that source address. So you have to define an additional routing table named e.g. via-ISP2, put a default route whose gateway is reachable via the secondary WAN into that table, and add a routing rule that says src-address=x.x.x.x action=lookup-only-in-table table=via-ISP2.
  • if all this works, only one of the peers will show an active phase 2 at a time; to make phase 2 active on the second one, you have to break phase 1 on the previously working one. Once phase 2 for the policy gets established using the new (previously standby) peer, it will stay there even if phase 1 on the other peer re-establishes; this is by design, so if you want the phase 2 traffic to use the primary WAN whenever it is available, you have to use scripting to do that.

Or you can save yourself the burden with vanilla IPsec and its policies, use two IPsec peers, each with its own policy in transport mode, to encrypt two IPIP tunnels, and use normal routing with OSPF and possibly BFD to choose between the IPIP tunnels. This approach makes the routing simpler and the failover faster, the price to pay is the volume of the BFD traffic.

13

You can ask this question to any popular AI, and it will give the same answer as in the wiki. Since that’s exactly how it used to work — using a simple mangle and dst.address, UDP\4500. I don’t know what has changed in the new firmware.

14

I would say that's a very good approach.

BFD's handshake frequency is easy to tune to match the bandwidth of each link, and the big advantage of OSPF/BFD is the extremely fast rerouting when a link goes down. The BFD handshake is also extremely lightweight, so the traffic will hardly even be noticeable.

15

Not sure why it worked in 7.19, but if you have only the rules you've shown, it must have worked due to some circumstances that occured by chance.

When the router sends a packet that does not belong to an already existing conversation and no source address is explicitly stated for it using some configuration setting, it chooses the source address for it depending on the result of routing. So if the IPsec peer acts as an initiator, no local-address is set for it, and there is no routing rule, the source address is set to the one attached to the WAN through which the gateway of the corresponding route in table main is accessible. If such a packet hits an action=mark-routing rule in mangle in chain output, it gets routed one more time using the table indicated using new-routing-mark, but this second round of routing does not change its source address. So even if the new route uses a different out-interface than the previously chosen one, the packet leaves with the source address chosen initially, unless there is a src-nat (or masquerade, or netmap) rule.

Given that you say that the IP address of the secondary WAN changes dynamically, I suppose you haven't set local-address for the peer?

Also, you assign the new-routing-mark using a rule that matches on the destination address and port; do you connect to two distinct remote peers using your two WANs, or do you have the same setup like the OP and there remote address is the same for both your peers?

16

I have no idea what contracts the mobile ISPs on the north pole offer, but in some real countries, an LTE backup uplink may have quite a restricted data cap, so one only wants to use it for the "real" traffic. Hence my warning about the BFD volume. And yes, the BFD rate can be set very differently for each path.

17

Well, Santa is actually pretty generous when it comes to free traffic and bandwidth up here at the North Pole :smiley:

Yeah, if you're on a limited data plan, you obviously need to be a bit careful. We run 3 pps on our CPEs with LTE and NR/NSA links, which is just about enough to keep CA active all the time. With NR/SA it doesn't really matter, since CA kicks in within just a few ms.