Community discussions

MikroTik App
 
bp0
newbie
Topic Author
Posts: 31
Joined: Thu May 06, 2021 5:06 pm

If wireguard fails to connect, it won't ever try again

Mon Apr 25, 2022 4:05 am

I have a hAP ac2 at a remote location that connects to a wireguard VPN to be managed.
The remote wireguard config is:
/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1 private-key="..."
/interface wireguard peers
add allowed-address=10.0.99.0/24, endpoint-address=someplace.com endpoint-port=13232 interface=wireguard1 \
    persistent-keepalive=25s public-key="..."
Today the power went out, and when it came back online, the router did not reconnect to the VPN. It seems that while the cable modem was reconnecting, wireguard failed to handshake and then the router just never tried again. I was able to log in another way and get it to come back by just poking a peer config value. (set 0 endpoint-port=13232 didn't change anything but triggered reconnect). I'm wondering if there is a way to tell it to try again some time after failing?
I did some tests with another router and a hotspot and I can recreate the situation. If the link is up, but the internet connection is not ready yet, it fails and seems to never try again.
 
bp0
newbie
Topic Author
Posts: 31
Joined: Thu May 06, 2021 5:06 pm

Re: If wireguard fails to connect, it won't ever try again

Mon Apr 25, 2022 4:35 am

It seems there will only be an attempt if there is traffic for it. I thought that is what persistent-keepalive was for.
I will try to just send traffic through every few minutes.
/system script add name=wg-verify source=":tool ping 10.0.99.1 count=1"
/system scheduler add interval=5m name=wg-verify on-event=wg-verify
 
User avatar
eworm
Forum Guru
Forum Guru
Posts: 1070
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: If wireguard fails to connect, it won't ever try again

Mon Apr 25, 2022 9:40 am

You can use netwatch for that:
/tool/netwatch/add host=10.0.99.1 interval=5m
But I think this issue is something different: If the endpoint address is a name (not ip address) it is resolved just once. If this fails the peer is stuck.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: If wireguard fails to connect, it won't ever try again

Mon Apr 25, 2022 1:31 pm

A startup script could be made to check if domains stated in WG are being resolved. And if so then that specific WG is reactivated. Only reactivate when there is no active connection.

It could be a loop untill all WG are reactivated or at a set time-out. As startup schedule, it will be started only once. The loop should havea let's say 15 seconds delay so the router keeps it's cool.

When IP addresses are used the script skips checking of that specific WG.

If IP also not get connected script could also reacivate those. This would be a different version of the same scipt.

Here a Netwatch version pinging a IP on the other side of the VPN. If no connection the WG is restarted after 5 minutes. This timecould be shortened.

viewtopic.php?p=922247&p922247#p922247
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Mon Apr 25, 2022 2:22 pm

Check out Para 6 user article - viewtopic.php?t=182340
 
bp0
newbie
Topic Author
Posts: 31
Joined: Thu May 06, 2021 5:06 pm

Re: If wireguard fails to connect, it won't ever try again

Mon Apr 25, 2022 4:39 pm

The netwatch tool and toggle peer seems to be the right solution, thanks folks.
 
bp0
newbie
Topic Author
Posts: 31
Joined: Thu May 06, 2021 5:06 pm

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 5:34 pm

The netwatch tool and toggle peer seems to be the right solution, thanks folks.
So it happened again and the wireguard link didn't come back. I suspect its because it lost power twice within 10 minutes and the netwatch down-script only runs on the first change to down. If it was already down, then it will not run again at the next interval if the result is still down even if it might now succeed. I will try to go back to a regular scheduled script that pings and then toggles based on result no matter what the previous state.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 5:36 pm

ISPs dont change their stripes every 10 minutes???
WTF is anybody having their router or modem NOT on an UPS ???

Do you mean this didnt work
>>>>>>>
:delay 25
/interface wireguard peer disable 0
:delay 5
/interface wireguard peer enable 0
:log info "WGPeer toggled"
.................
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 5:44 pm

There is also this script by msatter................... ( in case he updates it further link is also provided )
viewtopic.php?p=930256#p929076
{; # BeginOfScript

# scripted by msatter
# function: bring up stalled WireGuard interfaces after restart of the router

:local timesRetried   15    ; # how many times WireGuard is tried to be restarted
:local loopDelay      10s   ; # (loopDelay * timesRetried ) = total timeout
:local restarted      true  ; # set default to true
:local domainResolved false ; # also checking if the endpoint domain-names could be resolved
:local retried        0     ; # set to starting value 

:while ( $restarted && ($retried < $timesRetried) ) do={
    # loop till all Wireguard interfaces are working or there the maximum retries is reached
    :set $restarted false
    /interface/wireguard
    :foreach wg in=[find disabled=no] do={ 
        :local peer [get $wg name]
        # scripted by Anav looking for domain names. Adapted by msatter.
        :foreach i in=[peers/find interface=$peer endpoint-address~"[a-z]\$"] do={
            :if ( [:resolve [peers/get $i value-name=endpoint-address]] ) do={
                :set $domainResolved true
                :set $lastHandshake [peers/get $i last-handshake]
                :if ( ([:tostr $lastHandshake] = "") || ($lastHandshake > [:totime [peers/get $i persistent-keepalive]]) ) do={
                    disable $peer; :delay 1s; enable $peer ; # restarting the WireGuard connection
                    :set $restarted true
                }; # EndIf
            }; # EndIf
        }; # EndForeach
    }; # EndForeach
    :if ( $restarted ) do={
        :put "Check loop: $retried"
        :set $retried ($retried + 1) 
        :put "Checking loop: $retried"
        :delay $loopDelay ; # waiting time till following check
    }; #EndIf
}; # EndWhile
:if ( !$domainResolved ) do={
    :put "One or more domains could not be resolved, all/some domain based endpoints could not be brought up in the set time of ($timesRetried * $loopDelay)"
} else={
    :if ( $restarted && ($retried > $timesRestied) ) do={:put "Not all WireGuard interfaces could be brought up in the set time of ($timesRetried * $loopDelay)"}
    :if ( !$restarted && ($retried > 0) ) do={:put "No WireGuard interfaces are down, after $retried retries"}
    :if ( $retried = 0 ) do={:put "No WireGuard interfaces had to be restared"}
}; # EndElse

}; #EndOfScript[code]
Last edited by anav on Sat Apr 30, 2022 7:40 pm, edited 1 time in total.
 
bp0
newbie
Topic Author
Posts: 31
Joined: Thu May 06, 2021 5:06 pm

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 5:45 pm

ISPs dont change their stripes every 10 minutes???
WTF is anybody having their router or modem NOT on an UPS ???

Do you mean this didnt work
>>>>>>>
:delay 25
/interface wireguard peer disable 0
:delay 5
/interface wireguard peer enable 0
:log info "WGPeer toggled"
.................
That did work, in that it toggled the interface and logged "WGPeer toggled" in the log, but it still wasn't ready, so the next time netwatch ran and found it was "down", it was already in "down" state and didn't run that code again. This is a somewhat temporary situation, and I definitely need to just fix the power problem, but just wanted to share that the solution given using netwatch has a small flaw.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 6:00 pm

To clarify, if the other end was still down, the netwatch basically stopped resetting the interface because it was still down from the initial failure of power at the other end.......
 
bp0
newbie
Topic Author
Posts: 31
Joined: Thu May 06, 2021 5:06 pm

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 6:25 pm

To clarify, if the other end was still down, the netwatch basically stopped resetting the interface because it was still down from the initial failure of power at the other end.......

I think it happened that the cable modem hadn't connected yet at the first attempt to toggle. Looks like it missed it by about 8 seconds. 22:28:12 is when the modem gets the actual public IP and is fully connected.
 apr/29 22:27:59 system,info wireguard peer entry changed
 apr/29 22:28:04 system,info wireguard peer entry changed
 apr/29 22:28:04 script,info WGPeer toggled
 apr/29 22:28:12 dhcp,critical,error dhcp-client on ether1 lost IP address 192.168.100.10 - received NAK from dhcp server
 
holvoetn
Forum Guru
Forum Guru
Posts: 5321
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 6:34 pm


That did work, in that it toggled the interface and logged "WGPeer toggled" in the log, but it still wasn't ready, so the next time netwatch ran and found it was "down", it was already in "down" state and didn't run that code again. This is a somewhat temporary situation, and I definitely need to just fix the power problem, but just wanted to share that the solution given using netwatch has a small flaw.
Hmmm, that's indeed a possible way to get stuck.
How about this: instead of using netwatch that little piece of script should somehow run under scheduler, every couple of minutes or so.
And netwatch should then enable or disable the scheduler part depending on the outcome.
A 2-layered approach.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 6:39 pm

Sounds good holvoe............ how would you do that.......
Can netwatch enable a schedule (on down) and then disable a schedule (on up).
 
holvoetn
Forum Guru
Forum Guru
Posts: 5321
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 6:43 pm

Why not ?
Enabling/disabling something in scheduler is a command like any other so can be scripted perfectly.

If I got some time tomorrow I'll look into it (unless someone else already stepped in before).
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 6:50 pm

Why not ?
Enabling/disabling something in scheduler is a command like any other so can be scripted perfectly.

If I got some time tomorrow I'll look into it (unless someone else already stepped in before).
Thats not what you said, you said netwatch enabling the schedular name so a script will run and netwatch disabling the scheduler name
Assuming you mean if netwatch is down and then if netwatch is up.



You may be conversant in netwatch, scripts, schedules many like me are not LOL
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 6:59 pm

@Anav (hi!) is better you post the link instead of the script because msatter modify the script, and here remain forever the old version...
 
holvoetn
Forum Guru
Forum Guru
Posts: 5321
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 7:16 pm

Thats not what you said, you said netwatch enabling the schedular name so a script will run and netwatch disabling the scheduler name
Assuming you mean if netwatch is down and then if netwatch is up.

You may be conversant in netwatch, scripts, schedules many like me are not LOL
That is still what I said but you misinterpreted it.
In scheduler, use script to toggle every couple of minutes (or script from msatter with all bells and whistles).

On netwatch down - enable schedule for that script
On Netwatch up - disable schedule for that script

Done.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 7:27 pm

Okay, sorry my bad.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Sat Apr 30, 2022 7:45 pm

@Anav (hi!) is better you post the link instead of the script because msatter modify the script, and here remain forever the old version...
Yes, I linked to the post, but no guarantee it will include certain improvements ;-).

In the meantime i know you can solve this issue in your sleep, probably only two snores and it would be done!!
How to put this into a script>

:delay 25
/interface wireguard peer disable 0
:delay 5
/interface wireguard peer enable 0
:log info "WGPeer toggled"


Then assign a schedule to this script, lets say every 20 seconds it runs.
BUT here is the challenge.
Enable the scheduler (and thus the script) via Netwatch down condition for wireguard.
Disable the scheduler (and thus the script) via Netwatch up condition for wireguard.
 
holvoetn
Forum Guru
Forum Guru
Posts: 5321
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: If wireguard fails to connect, it won't ever try again

Sun May 01, 2022 12:52 pm

Rough POC (Proof of Concept)

Define script to toggle WG peer settings:
/system script
add dont-require-permissions=no name=ToggleWGPeer owner=*** policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon
    source="
    :delay 25
    /interface wireguard peer disable 0
    :delay 5
    /interface wireguard peer enable 0
    :log info \"WGPeer toggled\"
    "

Define schedule for earlier created script:
I used 2 minute period here, adjust as needed for your own purposes.
/system scheduler
add disabled=yes interval=2m name=ScheduleWGToggle
on-event="/system script run ToggleWGPeer"
policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=may/01/2022 start-time=11:34:35

Define Netwatch up/down actions:
Assuming address to be verified "on the other side" is 10.255.255.1 (change as needed for your purposes)
Assumption 2: run every minute (in most cases there is nothing to do... but I prefer not to wait too long when something does go haywire)
/tool netwatch
add down-script="
    log warning \"NETWATCH - WG Down\"
    /system scheduler set ScheduleWGToggle disabled=no"
    host=10.255.255.1
    up-script="/system scheduler set ScheduleWGToggle disabled=yes"
 
Tested on mAPLite lying around here. Manually disabled WG Peer and after some time, the flow starts ...
  • Scheduler gets enabled
  • Script is run
  • WG peer status is toggled
  • Other side is reachable again and scheduler gets disabled.

So I think this might work 8)
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Sun May 01, 2022 4:20 pm

But isnt it a problem if the host is dynamic not static and thus one cannot use netwatch???>>>>
(netwatch will not accept mynetname..... )

Another way perhaps is to keep the schedule LIVE, on all the time, and have it simply check the netwatch state, to determine action.
Lets say run it every 2 minutes......
-->check netwatch status
if up, then do nothing
if down then toggle interfaces. ??

In either case, the weak link is that netwatch cannot handle dyndns name or so it seems (im probably doing something wrong).
Last edited by anav on Sun May 01, 2022 4:34 pm, edited 1 time in total.
 
holvoetn
Forum Guru
Forum Guru
Posts: 5321
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: If wireguard fails to connect, it won't ever try again

Sun May 01, 2022 4:28 pm

But isnt it a problem if the host is dynamic not static and thus one cannot use netwatch???>>>>
(netwatch will not accept mynetname..... )

Aha, but here comes once again a useful case for using IP addresses on the WG interfaces :lol:
If done consistently, there will ALWAYS be an IP address you know which can be looked out for. Always.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Sun May 01, 2022 4:35 pm

Ahh so we are not checking the endpoint address on this one we are simply checking the 'reachable IP address' at the other end.
Got it!!!
 
holvoetn
Forum Guru
Forum Guru
Posts: 5321
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: If wireguard fails to connect, it won't ever try again

Sun May 01, 2022 4:42 pm

Ahh so we are not checking the endpoint address on this one we are simply checking the 'reachable IP address' at the other end.
Got it!!!
Correct.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Sun May 01, 2022 4:52 pm

But what about the idea of running a script every 2 minutes that checks the netwatch state (part1) KINDA reverse thinking........
If the state is down the script part 2 is executed, if the state is up, dont execute part 2.
 
User avatar
Znevna
Forum Guru
Forum Guru
Posts: 1347
Joined: Mon Sep 23, 2019 1:04 pm

Re: If wireguard fails to connect, it won't ever try again

Sun May 01, 2022 4:52 pm

All this seems overcomplicated for no reason.
Since the case that started this topic is about "after reboot" why don't we use some simpler logic, like:
If peer X isn't disabled & has endpoint-address set to hostname/not empty & current-endpoint-address is empty, restart peer X
? Wouldn't this be simpler? 0o
Last edited by Znevna on Sun May 01, 2022 5:15 pm, edited 1 time in total.
 
holvoetn
Forum Guru
Forum Guru
Posts: 5321
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: If wireguard fails to connect, it won't ever try again

Sun May 01, 2022 5:04 pm

Problem here is that when netwatch is in the down state, it will not try again.
If dns resolving is not up before netwatch is fired, you're stuck.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Sun May 01, 2022 6:21 pm

That is why the OP posted, he tried the obvious easiest netwatch example and it fails if the endpoint server has a longer outage then conceived on the netwatch script.]
So one could change the script to 75 seconds perhaps or better as Holve has suggested......

@holve - if concerned about flash utilzation. The netwatch method uses flash 4 times ( scheduler enabled, interface toggled off, interface toggled on, scheduler disabled).
The script centric approach uses flash 2 times (interface toggled off, interface toggled on)

Can you write a script (runs every 2 minutes), that basically has two parts.

Part A - check netwatch status
Part B
(i) if up then do nothing;
(ii) if down then run script to toggle wg interface off, delay 25, then toggle interface on.

etc.........
 
holvoetn
Forum Guru
Forum Guru
Posts: 5321
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: If wireguard fails to connect, it won't ever try again

Sun May 01, 2022 6:24 pm

Only when something needs to be done.
Otherwise nothing happens with flash.
 
reinerotto
Long time Member
Long time Member
Posts: 519
Joined: Thu Dec 04, 2008 2:35 am

Re: If wireguard fails to connect, it won't ever try again

Mon May 02, 2022 11:08 pm

Today the power went out, and when it came back online, the router did not reconnect to the VPN. It seems that while the cable modem was reconnecting, wireguard failed to handshake and then the router just never tried again.
It is not a wireguard problem, but MTs implementation/usage of wireguard.
I have a fleet of openwrt-based routers to maintain, and this behaviour only to be observed in one special situation:
wireguard has problems to start up the link, in case of wrong system time. So, in case the MT-box after boot is far off correct time,
such behaviour can occure.
You need to make shure, that system time (NTP, RTC) is properly set, before starting wireguard.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: If wireguard fails to connect, it won't ever try again

Mon May 02, 2022 11:14 pm

Well what you are describing is a time lag for
a. the NTP to resolve to the correct time?
b. the endpoint address to resolve to the correct IP if its a dynamic dyndns name for endpoint?

Other than that it should simply work.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: If wireguard fails to connect, it won't ever try again

Mon May 02, 2022 11:20 pm

I have not those problems and I have an internal time server and DNS resolving is no problem because the domains used, are in static DNS.

The router is then hitting the ground running ready to fire up the WG connection.

When using domain names for NTP then first the DNS has to work and that is depending again of a working connection. Thanks for mentioning the NTP so I can add that to my script that re-instates DNS entries even before the connection is up.
 
reinerotto
Long time Member
Long time Member
Posts: 519
Joined: Thu Dec 04, 2008 2:35 am

Re: If wireguard fails to connect, it won't ever try again

Mon May 02, 2022 11:28 pm

On openwrt I _MUST_ make shure, that NTP syncs the system time, before wireguard starts up.
Which needs re-arranging the startup-sequence regarding NTP service, and single-shot update of system time via NTP, not just graceful adjustment in small increments, to be on safe side or to handle worst case: firmware-update OTA, having very old initial system time.
Not a big deal on real open source :-)
Regarding dynDNS, I have no experience, cause not used.
However, correct name resolution for WGs startup seems to be obvious.
 
holvoetn
Forum Guru
Forum Guru
Posts: 5321
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: If wireguard fails to connect, it won't ever try again

Mon May 02, 2022 11:36 pm

It is not a wireguard problem, but MTs implementation/usage of wireguard.
...
This.

E.g. on windows, wg client will simply REFUSE to start without proper dns.
Error. As it should be.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: If wireguard fails to connect, it won't ever try again

Tue May 03, 2022 12:29 am

Script to restore resolved domains on restart or crash:

Link: viewtopic.php?t=185647

Who is online

Users browsing this forum: cciprian, FlowerShopGuy, johnson73, loloski, seutertje and 70 guests