Community discussions

MikroTik App
 
Stril
Member Candidate
Member Candidate
Topic Author
Posts: 200
Joined: Fri Nov 12, 2010 7:18 pm

Scheduler stops executing script

Mon Aug 23, 2021 5:37 pm

Hi!

I have a quite strange problem.

I am using a simple scheduler script on a CCR1009:
/system scheduler
add interval=27m name=pull_blacklists on-event=\
    "/system script run get_blocklists" policy=\
    ftp,reboot,read,write,policy,test,password,sniff
    start-date=jun/20/2020 start-time=00:05:15
The script is running fine for some weeks and then it stops suddenly to get executed. "Next run" does not update again in the GUI.

Do you have any idea, why this happens?
If I change the "Start-Time" for one second in that situation, the script continues to run (for some weeks).

Thank you for your help
Stril
 
Stril
Member Candidate
Member Candidate
Topic Author
Posts: 200
Joined: Fri Nov 12, 2010 7:18 pm

Re: Scheduler stops executing script

Fri Sep 17, 2021 10:07 am

...am I the only one, seeing that problem?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Scheduler stops executing script

Fri Sep 17, 2021 10:41 am

This is not diagnosticable, the scheduler is ok,
you must also post the script you try to run...
 
Stril
Member Candidate
Member Candidate
Topic Author
Posts: 200
Joined: Fri Nov 12, 2010 7:18 pm

Re: Scheduler stops executing script

Fri Sep 17, 2021 10:57 am

Hi!

There are multiple scripts on the router and all of them are running fine for some weeks and then, they stop simultaniously.
 
User avatar
SiB
Forum Guru
Forum Guru
Posts: 1888
Joined: Sun Jan 06, 2013 11:19 pm
Location: Poland

Re: Scheduler stops executing script

Fri Sep 17, 2021 11:03 am

In scheduler put line's like:
/log warning "ScriptX Starts"
/system script run get_blocklists
/log warning "ScriptX End"

And similar line in your log and your log-s show you if scripts is realy working or not and at what step it's breake.

Try run manualy your script and check if he success.
 
Stril
Member Candidate
Member Candidate
Topic Author
Posts: 200
Joined: Fri Nov 12, 2010 7:18 pm

Re: Scheduler stops executing script

Fri Sep 17, 2021 11:38 am

Hi!

Scripts are running fine! 100%.
The only problem is, that the scheduler does not try to run them after some thousand successful runs.

This is also visible at "next-run" in scheduler, which is in the past in that case.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Scheduler stops executing script

Fri Sep 17, 2021 11:55 am

@Stril I asked you to post the script here for further analysis,
but your assumption that the executed script is perfect and does not block the scheduler, makes me make this decision:

End of help from my side.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11383
Joined: Thu Mar 03, 2016 10:23 pm

Re: Scheduler stops executing script

Fri Sep 17, 2021 12:25 pm

Scripts are running fine! 100%.
The only problem is, that the scheduler does not try to run them after some thousand successful runs.

This is also visible at "next-run" in scheduler, which is in the past in that case.

Since problem seems to be connected to internal state of your router and normal users have no way of analyzing it, it's useless to pursue this matter over the (user) forum. I guess your next steps should be: 1) create a supout.rif file when scheduler stops executing scripts and 2) open a ticket with support@mikrotik.com.
 
User avatar
SiB
Forum Guru
Forum Guru
Posts: 1888
Joined: Sun Jan 06, 2013 11:19 pm
Location: Poland

Re: Scheduler stops executing script

Fri Sep 17, 2021 12:40 pm

My last suggestion.
Do this log warning at script who have max of running counter - probably it is first to have a problem.

Next is to use a total free https://deadmanssnitch.com/plans a one triger, this give you a URL... and if this URL is not run in x time then this portal give you notification that it not see new request.
https://deadmanssnitch.com I use and similar projects to notify when some process not finish it's job, general they are created to be check a linux CRON jobs but you can use them in many ways.
When you receive that e-mail notify then go to MikroTik, create a supout.rif and sent it directly to MikroTik, we at forum can only confirm that should be work and that config is ok.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Scheduler stops executing script

Fri Sep 17, 2021 1:13 pm

@SiB, next time the scheduler start must check if previous is finished, if not warn user on some way.

pseudocode scheduler
set global varialble randomnameJhdsfg to "endscript" if the variable do not already exist
check global variable randomnameJhdsfg if it is set to "endscript", if it is not, warn user on some way
set global varialble randomnameJhdsfg to "startscript"
run the needed script
set global varialble randomnameJhdsfg to "endscript"
 
User avatar
SiB
Forum Guru
Forum Guru
Posts: 1888
Joined: Sun Jan 06, 2013 11:19 pm
Location: Poland

Re: Scheduler stops executing script

Fri Sep 17, 2021 1:45 pm

rextended write
@SiB, next time the scheduler start must check if previous is finished, if not warn user on some way.
I know that you do all by scripting :).
But, sometimes to not open the open door, I just add at the end of schedulers/script just one line of code like:
/tool fetch url=https://nosnch.in/00abcdef3f
/tool e-mail send to=00abcdef3f@nosnch.in
and it's all. If scripts stops working by update this remote service by url/email, Then this online service send me e-mail (in free account at end of Hour/day/week/month).

Of course OP this thread should add just log warning and watch in logs if scripts run or not... many way can be done to diagnostic this.
Thats why we like MikroTik.. by scripting :)
 
Stril
Member Candidate
Member Candidate
Topic Author
Posts: 200
Joined: Fri Nov 12, 2010 7:18 pm

Re: Scheduler stops executing script

Fri Sep 17, 2021 2:01 pm

Hi!

I just added the log-warning.

Here the main script:
:log info message=("Start Sending Report");
:local ipList value="";
:foreach tmpAddress in=[/ip firewall address-list find where list=HONEYPOT] do={
:set $attackip value=([/ip firewall address-list get $tmpAddress value-name=address]);
:log info message=("$attackip"."Report to AbuseIPDB");
:do {/tool fetch keep-result=no http-method=post url="https://api.abuseipdb.com/api/v2/report" http-data="key=xxxx&categories=14&comment=Portscan&ip=$attackip"} on-error={:log info message="Error for Report of IP $attackip"}
:delay 6000ms;
};
:log info message=("Sending Report End");
One additioinal question:
Does scheduler stop executing ALL jobs, if one job does not finish?
 
Stril
Member Candidate
Member Candidate
Topic Author
Posts: 200
Joined: Fri Nov 12, 2010 7:18 pm

Re: Scheduler stops executing script

Fri Sep 17, 2021 2:05 pm

@Stril I asked you to post the script here for further analysis,
but your assumption that the executed script is perfect and does not block the scheduler, makes me make this decision:
You are right, this was not correct - Sorry for that!
I was thinking, that if all the scheduler jobs stop working simultaneously, the script itself cannot be the reason.

Did you ever see, that scheduler stops executing new jobs, because one script is "blocked" and that it restarts, by re-adjusting the start-time?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Scheduler stops executing script

Fri Sep 17, 2021 2:44 pm

Sometime "fetch" freeze for answer from remote site and lock the script (and the scheduler)
on-error can not catch indefinite waiting....

also

":set $attackip value="

where is defined "attackip"?
and :set must be used without the $

only 6 seconds between fetch notification?
some fetch can sovrappose...

How many IP you report? IF the list is huge can take some hours...
You delete the IP or continuosly you resend again notice for already notified IP?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Scheduler stops executing script

Fri Sep 17, 2021 2:55 pm

Rewrited script, without change logic
:log info "Start Sending Report"
/ip firewall address-list
:foreach tmpAddress in=[find where list="HONEYPOT"] do={
    :local attackip [get $tmpAddress address]
    :log info "BEGIN $attackip Report to AbuseIPDB"
    :do { /tool fetch keep-result=no http-method=post  \
                      http-data="key=xxxx&categories=14&comment=Portscan&ip=$attackip" \
                      url="https://api.abuseipdb.com/api/v2/report"
    } on-error={:log error "Error for Report of IP $attackip"}
# added for debug
    :log info "END $attackip Report to AbuseIPDB"
    :delay 10s
}
:log info "Sending Report End"
 
Stril
Member Candidate
Member Candidate
Topic Author
Posts: 200
Joined: Fri Nov 12, 2010 7:18 pm

Re: Scheduler stops executing script

Fri Sep 17, 2021 3:20 pm

Hi!

THANK YOU! That's a great help.
I changed the script with your input. I have no idea, why it did, what it had to do despite of the wrong parts...

Is there any possibility to to catch a freeze?

Currently, I am sending the list every hour and as I set timeouts to the "HONEYPOT"-list of less than one hour, there should be very few "double-send".
How does the scheduler work, if the script did not complete before the next run?

Is there any possibility to exit a "foreach" after x entries? Is there something like an exit-condition?

Thank you very much!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Scheduler stops executing script

Fri Sep 17, 2021 3:24 pm

fetch can cause a infinite DELAY, not infinite loop,
all is freezed, waiting fetch to finish, is not a cycle than can be autochecked if executed too much time and autoexit...

Try my script, if fail we add asyncronous fetch execution
Something like that, you can see my Snippets, on my signature the link.
Autoclose after 20s of waiting...
    :local jobid [:execute script="/tool fetch ............"]
    :local sec 0
    :while (([:len [/sys script job find where .id=$jobid]] = 1) && ($sec < 20)) do={
        :set sec ($sec + 1)
        :delay 1s
    }
 
Stril
Member Candidate
Member Candidate
Topic Author
Posts: 200
Joined: Fri Nov 12, 2010 7:18 pm

Re: Scheduler stops executing script

Fri Sep 17, 2021 3:26 pm

Thank you! I will try both for the next weeks and report back, if there are any problems.

Thank you for your advice and sorry for my first reaction...
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Scheduler stops executing script

Fri Sep 17, 2021 3:29 pm

No problem
 
ConnyMercier
Forum Veteran
Forum Veteran
Posts: 723
Joined: Tue Dec 17, 2019 1:08 pm

Re: Scheduler stops executing script

Fri Sep 17, 2021 11:39 pm

Quick question for the Guru's
@rextended, @SiB, @mkx

Q1:
Wenn using parameters start-date and start-time
does RouterOS have to recalculate Next run every single time from these parameters or only on startup?

Q2:
Would it make a performace difference to change the schedule start-time to startup ?
 
User avatar
SiB
Forum Guru
Forum Guru
Posts: 1888
Joined: Sun Jan 06, 2013 11:19 pm
Location: Poland

Re: Scheduler stops executing script

Sat Sep 18, 2021 1:49 am

ConnyMercier write
Q1: Wenn using parameters start-date and start-time
does RouterOS have to recalculate Next run every single time from these parameters or only on startup?
When you change start-time then next run will change, be calculated.
When you change interval then next run will change, be calculated.
When you change start-date to past then this not change anything but if you put start-date as feature date then of course it will recalculate and it will be waitng to that date.
Q2: Would it make a performace difference to change the schedule start-time to startup ?
When you use start-time=startup then start-date is ignored. It's just wait for reboot action to run on-event.
This is similar to crontab the @reboot triger.
About "performace difference" I not know what you mean. I use only one startup scheduler at time, never use few with that action. And every time when I use startup action I use at first line the "delay 20s" because I want to ROS finish initialization all his interfaces, connecting vpn's etc.
 
ConnyMercier
Forum Veteran
Forum Veteran
Posts: 723
Joined: Tue Dec 17, 2019 1:08 pm

Re: Scheduler stops executing script

Sat Sep 18, 2021 2:20 am

Thank you @SiB for the quick response

Let me clarify my question a little bit...
Let take a look at the Scheduler from Strill
/system scheduler
add interval=27m name=pull_blacklists on-event=\
    "/system script run get_blocklists" policy=\
    ftp,reboot,read,write,policy,test,password,sniff
    start-date=jun/20/2020 start-time=00:05:15
If i would restart my Mikrotik-Router with this scheduler active now (18.09.2021 01:16:22)
RouterOS will need on startup to calculate in increments of 27minutes starting jun/20/2020 00:05:15 to know when the "next run" takes place, Right ?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Scheduler stops executing script

Sat Sep 18, 2021 12:04 pm

When you change a particular value of the start time (i.e. xx:xx:xx, not "startup", the scheduler calculates the subsequent actual startup times from the new value and the "interval" value. So you can set a start time deep in the past, and repeated runs will continue even after reboots. E.g. to schedule a run for every Sunday at 3 AM, set start-date to 05-Sep-2021, start-time to 03:00:00, and interval to 1w.

Who is online

Users browsing this forum: Google [Bot], ofatieiev and 52 guests