How to auto-reboot if remote IP down for 5 minutes

[vitorcsp]

Hi!
Very nice, Congratulations !

[maikel]

Thanks Tom and John,

I can verify this works on a MT RouterBOARD wAP R-2nD with RouterOS v6.43.7
I only commented out the first logging and changed action to “/system reboot”. I scheduled for every two minutes

Cheers,
Maikel

@mistry7

@TomjNorthIdaho

FYI - I am sometime planning to re-write a 2’nd newer version of my netwatch-script-reboot-on-ping-loss program.

Hi Tom,

I like the idea of adding the option for a second IP address to ping against to make double sure failure of the primary IP address isn’t causing a false WAN failure.

I’m using a stripped down (simplified) version of your original script along with Netwatch to restart an attached USB cellular modem when needed. It’s been working really well.

Using the script has greatly reduced false triggers which used to occur whenever my cellular provider refreshes the USB modem’s DHCP lease… usually daily. That refresh temporarily interrupts WAN access for about 30 to 60 seconds, so I have the script reboot after 2 minutes of unsuccessful pings spaced 10 seconds apart.

Here’s the script I’m currently using:

:local continue true;
:local counter 0;
:local maxcounter 12;
:local sleepseconds 10;
:local goodpings 0;
:log error "Script will further test ping to WAN in $sleepseconds seconds and will continue for $maxcounter x $sleepseconds seconds.";
:while ($continue) do={
    :set counter ($counter + 1);
    :delay $sleepseconds;
    :if ([/ping 8.8.8.8 interval=1 count=1] =0) do={
        :log warning "Ping to WAN failed on attempt $counter of $maxcounter - Will try again in $sleepseconds seconds.";
    } else {
        :log warning "Ping to WAN succeeded on attempt $counter of $maxcounter - No Further testing needed and script will exit.";
        :set continue false;
        :set goodpings ($goodpings +1);
        };
    :if ($counter=$maxcounter) do={
        :set continue false;
        }
    }
:if ($goodpings = 0) do={
    :/system routerboard usb power-reset duration=5
    }

Netwatch script executed on Down:

log warning "Netwatch ping to WAN failed.";/system script run WANPing

Netwatch script executed on Up:

/tool e-mail send to=xxxxxxxx@xxxxxxx.net subject="MikroTik Netwatch Ping Failure" body="MikroTik router Netwatch detected a ping to WAN failure and the WANPing script was executed.  Check router log for details.";log warning "Netwatch ping failure notification email sent."

Have a great day,

John

[drdedus]

hello and thank you.
i get a message in log that is not enough permissions.
what permissions are need it just to reboot after 5 minutes?

[gyropilot]

hello and thank you.
i get a message in log that is not enough permissions.
what permissions are need it just to reboot after 5 minutes?

I encountered the same problem and just checked the “Don’t Require Permissions” checkbox on the script setup screen. Now I don’t know if that’s considered safe from a router security standpoint, but it eliminated the problem.

John

[TomjNorthIdaho]

Updated version(s) of my WatchDog auto-reboot scripts I am now using.

FYI - Back when I first posted my script to auto reboot a Mikrotik when the network lost connectivity , it used to work. Since then (years ago and several ROS versions ago) my scripts ran great - but newer versions of ROS have changed some permissions needed for scripts to perform what my scripts were trying to do. The good news is I have changed and modified my scripts and for me , they now function better.

Below are some instructions and the scripts to configure your Mikrotik to use my WatchDog auto-reboot scripts.

1st - On your core router facing the Internet , create a loopback interface (or a bridge or any interface) that has an ip address of 192.0.2.254 / 32
Note , 192.0.2.254 is a safe IP address to use because it is not on the Internet and the entire 192.0.2.0/24 network is intended to be used for in-house lab testing networks. After you have a 192.0.2.254/32 IP address on your network, every device in your network should be able to ping 192.0.2.254

2nd - Create three scripts on all of your other Mikrotik routers using the following:

Script name: "RebootScript"
Script contents;
/system reboot
/system reboot

Check the policy reboot


Script name: "Wlan1-SiteSurvey"
Script contents:
/interface wireless scan wlan1 duration=90 save-file=ManualSiteSurvey-wlan1-90seconds--txt.rtf
Check Polity; ftp , read , policy , password , sensitive , reboot , write , test , sniff , romon

Script name: "Wlan2-SiteSurvey"
Script contents:
/interface wireless scan wlan2 duration=90 save-file=ManualSiteSurvey-wlan1-90seconds--txt.rtf
Check Polity; ftp , read , policy , password , sensitive , reboot , write , test , sniff , romon


3rd , create two netwatch settings;

3A) Netwatch Host (IP address): 192.0.2.254

Interval: 00:05:00

Down action:

:local addresstoping 192.0.2.254
:local continue true
:local counter 0
:local maxcounter 60
:local sleepseconds 5
:local goodpings 0

:log warning "NetWatchVersion2-SiteSurvey - -----> Tom's NetWatchVersion2-SiteSurvey"
:log warning "NetWatchVersion2-SiteSurvey - -----> Ping testing for $maxcounter times every $sleepseconds seconds"

:while ($continue) do={
:set counter ($counter + 1)
:delay $sleepseconds

:if ([/ping $addresstoping interval=1 count=1] =0) do={
:log warning "NetWatchVersion2-SiteSurvey - ----->ping to $addresstoping failed on attempt $counter of $maxcounter -- Will try again in $sleepseconds seconds"
} else {
:log warning "NetWatchVersion2-SiteSurvey - -----> ping success on to $addresstoping attempt $counter of $maxcounter <----- No Further testing needed --- Program will exit -----"
exit
quit
end
:set continue false
:set goodpings ($goodpings +1)
}
:if ($counter=$maxcounter) do={:set continue false;}
}

:log warning "NetWatchVersion2-SiteSurvey - Performing site survey 1-of-2 using wlan1 for 20 seconds"
/interface wireless scan wlan1 duration=20 save-file=SiteSurvey-wlan1-20seconds--txt.rtf
:log warning "NetWatchVersion2-SiteSurvey - Performing site survey 2-of-2 using wlan1 for 90 seconds"
/interface wireless scan wlan1 duration=90 save-file=SiteSurvey-wlan1-90seconds--txt.rtf
:log warning "NetWatchVersion2-SiteSurvey - Site Surveys completed and saved to files"
:delay 10
exit
quit
end

3B) Netwatch Host (IP address): 192.0.2.254

Interval: 00:05:00

Down action:

:local addresstoping 192.0.2.254
:local continue true
:local counter 0
:local maxcounter 120
:local sleepseconds 5
:local goodpings 0

:log error "-NetWatchVersion2-Reboot-NotConnected - ----> Tom's NetWatchVersion2"
:log error "NetWatchVersion2-Reboot-NotConnected - -----> Ping testing for $maxcounter times every $sleepseconds seconds"

:while ($continue) do={
:set counter ($counter + 1)
:delay $sleepseconds

:if ([/ping $addresstoping interval=1 count=1] =0) do={
:log error "NetWatchVersion2-Reboot-NotConnected - ----->ping to $addresstoping failed on attempt $counter of $maxcounter -- Will try again in $sleepseconds seconds"
} else {
:log warning "NetWatchVersion2-Reboot-NotConnected - -----> ping success on to $addresstoping attempt $counter of $maxcounter <----- No Further testing needed --- Program will exit -----"
exit
quit
end
:set continue false
:set goodpings ($goodpings +1)
}
:if ($counter=$maxcounter) do={:set continue false;}
}

:log error "NetWatchVersion2-Reboot-NotConnected - --- executing reboot script in to seconds"
:delay 10
/system script run RebootScript
:delay 10
:log error "NetWatchVersion2-Reboot-NotConnected - --- I should of rebooted by now - will retry a reboot script againt in to seconds"
:delay 10
/system script run RebootScript
:delay 10
exit
quit
end

These scripts and netwatch settings now do the following when a Mikrotik can NOT ping 192.0.2.254

• One netwatch will perform a site-survey and store the results as a file. Using winbox, you can drag the files it creates "SiteSurvey-wlan1-20seconds--txt.rtf" and "SiteSurvey-wlan1-90seconds--txt.rtf" to your windows desktop and double-click on it to open it and see the results of a remote site-survey.

• One netwatch will perform a reboot if the Mikrotik remains unable to ping 192.0.2.254 after a few minutes.

• The reboot function has a long enough period prior to a reboot to allow a remote wireless client Mikrotik to perform a site-survey possibly reconnect.

• The scripts will abort if/when 192.0.2.254 starts to answer pings prior to a reboot

If you wish to test this , try this testing procedure:

• configure your core router to answer a ping to 192.0.2.254
• place the above scripts and netwatch code on a test Mikrotik

FYI - Use these scripts at your own risk and assume I may errors in my code
FYI - If you wish to auto reboot all of your mikrotiks and also have your remote Mikrotiks all perform site-surveys (in my case , hundreds to thousands of customer wireless client Mikrotiks) , simple disable 192.0.2.254 on your core router for about 10 minutes.


Please post any results or your ideas on how to improve these WatchDog auto-site-survey auto-reboot scripts.


North Idaho Tom Jones

[Jotne]

script

Can you please code tag your post?
Makes it easier to read.

Like this:

:local addresstoping 192.0.2.254
:local continue true
:local counter 0
:local maxcounter 60
:local sleepseconds 5
:local goodpings 0
#
:log warning "NetWatchVersion2-SiteSurvey - -----> Tom's NetWatchVersion2-SiteSurvey"
:log warning "NetWatchVersion2-SiteSurvey - -----> Ping testing for $maxcounter times every $sleepseconds seconds"
#
:while ($continue) do={
:set counter ($counter + 1)
:delay $sleepseconds
#
:if ([/ping $addresstoping interval=1 count=1] =0) do={
:log warning "NetWatchVersion2-SiteSurvey - ----->ping to $addresstoping failed on attempt $counter of $maxcounter -- Will try again in $sleepseconds seconds"
} else {
:log warning "NetWatchVersion2-SiteSurvey - -----> ping success on to $addresstoping attempt $counter of $maxcounter <----- No Further testing needed --- Program will exit -----"
exit
quit
end
:set continue false
:set goodpings ($goodpings +1)
}
:if ($counter=$maxcounter) do={:set continue false;}
}
#
:log warning "NetWatchVersion2-SiteSurvey - Performing site survey 1-of-2 using wlan1 for 20 seconds"
/interface wireless scan wlan1 duration=20 save-file=SiteSurvey-wlan1-20seconds--txt.rtf
:log warning "NetWatchVersion2-SiteSurvey - Performing site survey 2-of-2 using wlan1 for 90 seconds"
/interface wireless scan wlan1 duration=90 save-file=SiteSurvey-wlan1-90seconds--txt.rtf
:log warning "NetWatchVersion2-SiteSurvey - Site Surveys completed and saved to files"
:delay 10
exit
quit
end

[Krumpeir]

Try this. Without “script error: no such item (4)” problem:

1st - add Netwatch with Host: 8.8.8.8 and to down place:

:if ([/system resource get uptime]>30) do={
  :log info "Not ping on 8.8.8.8";
  /system script run AutomaticRestart;
}

2nd - add script with name “AutomaticRestart” and source:

:local PingCount 300;
:local stop false;
:while ([/ping 8.8.8.8 count=1 interval=5]=0 && stop=false) do={
  :set PingCount ($PingCount-5);
  #:log info "Reboot after $PingCount s";
 :if ($PingCount<=0) do={
    :set stop true;
    :log info "NOT PING ON 8.8.8.8 - REBOOT!!!";
    /system reboot;
  };
};

:set PingCount (300-$PingCount);
:if (stop=false) do={:log info "Reboot stop after $PingCount/300 s";};

[Ksee]

Good day Guys,

I have a similar request but not really the same,

What i would like to do is add a script for when a route becomes unreachable that the Mikrotik reboots.
What i would like to achieve is im having issues with stale voice connections and my PABX requires a reboot before they work again has anyone solved this or have an idea what to do ?

[indy0077]

@timberwolf

@Saasouh

Dear friend
I am really in need for that script, I have tried it but it seems that it’s not working with me .

This is for you:

======================================================================
Here is the information:

First - I assume your Mikrotik router performing the test is IP address 192.168.61.9

Second - I assume the remote device you want your Mikrotik router to test to is 192.168.61.7

Here are the instructions:

#1 Winbox into your 192.168.61.9 Mikrotik

#2 Go to Tools - Netwatch

#3 If you have anything in there then remove it

#4 Click on the RED “+”

#5 On the Host tab.
Type in “192.168.61.7”
Interval should be “00:01:00”
Timeout should be “1000”

#6 click on the Down tab.
Paste in the entire line below:
log warning “Netwatch missed a ping to 192.168.61.7 - starting 5 minute timeout script” ; /system script run NetWatchBoot-192.168.61.7

Now push the Enter button (you want your curser on the bottom on a blank line)

What you typed in should look like this:

|-----------------------------------------------
|log warning "Netwatch missed a ping to
|192.168.61.7 - starting 5 minute timeout
|“script” ; /system script run NetWatchBoot-
|192.168.61.7
|
|
|------------------------------------------------
Note - You need to make sure you pushed the Enter button after the last line and hit the “APPLY” button.

**** At this point - you have finished part 1 of 2 things you need to do *****

#7 Go to System - Scripts

#8 If you have anything in there then remove it

#9 Click on the RED “+”

#10 In the name box, paste in the line below:

NetWatchBoot-192.168.61.7

#11 The owner should be “admin” and make sure every check box is checked.

#12 In the Source: box, paste in the following 3 lines:[/color]
:if ([/ping 192.0.2.254 interval=5 count=60] =0) do={
/system reboot
}


Now push the Enter button (you want your curser on the bottom on a blank line)

What you typed in should look like this:
|---------------------------------------------
|:if ([/ping 192.0.2.254 interval=5 count=60]
|=0) do={
|/system reboot
|}
|
|
|--------------------------------------------

Note - You need to make sure you pushed the Enter button after the last line and hit the “APPLY” button.

***** At this point - you have finished part 2 of 2 things you need to do - it is running now *****
INFORMATION - INFORMATION - INFORMATION -

The Netwatch you just configured (part 1 of 2) checks every minute to see if it can ping “192.168.61.7”.
If it can ping “192.168.61.7” then the script will exit and auto run another test a minute later.

If the Netwatch script can not ping “192.168.61.7” then it will run the system-script you created (part 2 of 2).
The system-script will try every 60 seconds to ping “192.168.61.7” for five times.
If there is an answer to ping to “192.168.61.7” during this 5-minute period, then the system-script will exit.
If there are no answers to pings to “192.168.61.7” during the 5-minute period, then the Mikrotik router will reboot.

NOTE - NOTE - NOTE
Every time the Netwatch-script fails to ping “192.168.61.7” it will place a blue line in your Mikrotik log.
If you see many blue lines in your log file that look like
Netwatch missed a ping to 192.168.61.7 - starting 5 minute timeout script
Then you should check how good your connections are from your Mikrotik router to the remote 192.168.61.7 device

I hope this helps
Tom Jones - A WISP in North Idaho

Hi Tom,
I’m trying your script, but I get an error in the log like this: “could not run script Reboot-ping-failed: not enough permissions”