HowTo: Monitor and Recover IPsec-Connections using PowerShell (IPsec Watch-Dog)

RouterOS General
1

I am using ROS7 for establishing an IPsec-Tunnel. The tunnels’ purpose is, to provide access to a special TCP-based service.
From time to time, this IPsec-Tunnel enters some kind of invalid state: The tunnel is established, but there is no communication to the TCP-based service possible. In this state the “active peer” usually shows an empty field for “PH2 Total”.
To recover from this odd state, I just have to remove the active peer - Some seconds later, everything will usually be fine again.

For this reason, I have written a short PowerShell-script, which should be monitoring the IPsec-Tunnel and recover from such an odd state automatically. For doing so, there are some prerequisites:

  1. You need a windows machine to execute the script every 2 minutes
  2. Get the PowerShell-Script “Invoke-MikrotikRestAPI.ps1” from http://forum.mikrotik.com/t/rest-api-powershell-running-a-script-remotely-from-windows/156792/2 and place it in the same directory with this monitoring-script
  3. You need to implement some kind of “kill-switch” to ensure, that TCP-Packets destined for the IPsec-Tunnel will not leak through the Default-GW (see: http://forum.mikrotik.com/t/site-to-site-tunnel-with-one-site-using-dynamic-ipv4/177586/23 )
  4. For each IPsec-Tunnel to be monitored: Define a specific LAN-IP-Address (e.g.: 192.168.88.150)
  5. This IP-Address must be assigned as an >additional< IP-Address to your Windows-Network-Interface-Card
  6. This IP-Address must be “mangle”-d by ROS to assign the appropriate “connection-mark”. Something like this should do the job:
/ip firewall mangle add chain=prerouting src-address=192.168.88.150 action=mark-connection new-connection-mark=MyIPsecMark passthrough=yes comment="Mark IPsec-Traffic"

The PowerShell script will use the default IP-Address of the Windows-PC to check Internet-Connectivity (is 8.8.8.8 reachable?) - If Internet-Connection is okay, the script will try a TCP-Connect by using the secondary LAN-IP-Address (i.e. 192.168.88.150) to each host, you provide with the variable $RemoteHosts .
If none of the RemoteHosts can be connected using the secondary IP-Address, an IPsec-“remove connection” will be executed on ROS using MikroTik’s Rest-API.
In some rare situations the newly upcoming connection will result in an “active peer” with an empty “ID” (usually “ID” will be the host-name of the IPsec-Server). In this case, removing the active peer will not restore TCP-connection via IPsec … I decided to trigger a ROS-Reboot as a final IPsec-recover-solution (only applied, when “remove connection” will not solve the problem).

Let’s have a look at the script:

$VPNs= @{
    "VPN-Server1.example.com" = "192.168.88.150"
    "VPN-Server2.example.com" = "192.168.88.151"
}
$RemoteHosts = @("MyRemoteHost1.example.com","MyRemoteHost2.example.com")
$RemotePort  = 443
$RouterIP    = "192.168.88.1"
$RouterPass  = "MyROS-Password"


function TestConn($source, $dest, $port) {
    $src= [System.Net.IPEndPoint]::new([ipaddress]::Parse($source),0)
    $tc = [System.Net.Sockets.TcpClient]::new($src)

    $ErrorActionPreference = "SilentlyContinue"
    $tc.Connect($dest, $port)
    $ErrorActionPreference = "Continue"

    if ($tc.Connected) {
        Write-Host "Connected to $($dest):$port via $source"
        return $true
    } else {
        Write-Host "Not connected to $($dest):$port via $source"
        return $false
    }
}


if ((Test-NetConnection 8.8.8.8).PingSucceeded) {
    foreach ($VPNserver in $VPNs.Keys) {
        $SrcIP=$VPNs[$VPNserver]
        $i=0;
        $RemoteHosts | ForEach-Object {
            if (-not (TestConn $SrcIP $_ $RemotePort)) {$i++}
        }
        if ($i -eq $RemoteHosts.Count) {
            .\Invoke-MikrotikRestAPI.ps1 -RouterIP $RouterIP -User admin -Password $RouterPass -ExecutionMode Command -Data "/ip ipsec active-peers remove [/ip ipsec active-peers find where id=""$VPNserver""]"
            Write-Output "$(Get-Date -format "yyyy\/MM\/dd HH-mm-ss"): killed connection $VPNserver" >> Log.txt

            Start-Sleep -Seconds 15

            if (-not (TestConn $SrcIP $RemoteHosts[0] $RemotePort)) {
                .\Invoke-MikrotikRestAPI.ps1 -RouterIP $RouterIP -User admin -Password $RouterPass -ExecutionMode Command -Data '/system reboot'
                Write-Output "$(Get-Date -format "yyyy\/MM\/dd HH-mm-ss"): triggered reboot" >> Log.txt
                Exit
            }
        }
    }
} else {
    Write-Output "$(Get-Date -format "yyyy\/MM\/dd HH-mm-ss"): No Internet-Connectivity!" >> Log.txt
}

A Log-File will be placed in the working-directory of this Script - Refer to this Log-File to see all IPsec-Recovering-attempts.

Works fine for me :slight_smile:

Site-to-Site-Tunnel with one site using dynamic IPv4