[PROPOSAL] Event driven scripting

RouterOS General
1

While working with scripting in ROS we have to revert to scheduler have a script being executed in a interval or on startup. This leaves time before a script can detect a change and make the its own changes in the setting or give a warning to the user.

My proposal is to give each path/option a unique identifier and sent that on any edit/create/remove/disable to an new script collection. The new script collection could looks like schedule, but it is only search for a matching script with the unique number as identifier.

O Noooooo, not that also!!! Changes made by ROS like with Netwatch, in paths that are read-only should be able to trigger a script. Think of /user/active keeping track of users being logged in.

The “event-list” should receive the following data:

  • unique number for finding and start the appropriate script, if present
  • path (example: path=/interface/wireguard/peers/)
  • .id, the line being effected (example: id=*5)
  • type of change (add/delete/edit/(disable/update))

The unique identifier is available in every menu suited for this and if already every option has an unique identifier then make this available to the user to be copied.

The “event-list” should have two fields, the unique identifier and a settable time-out. The timeout is there to have an delay before the script is being executed. This because often a user edits a change or edit shortly after confirming. So the script execution can be delay, if an edit is made then the second “trigger” will reset the timeout and the clock starts to run again.

I hope that the unique identifier is already present in ROS and that can be implemented without much effort. I advise to do it graduate release of the identifiers.

ANY thought, suggestions or complaints feel free to post those here and I am only interested in having Mikrotik a good input for this so that it is also attractive for them to implement it.

Update: as unique identifier you can also think of the converting the path to an hash. There is a short version of paths so it can be made fitting.
Also indicate in each menu, I am using WinBox, that an event-list entry is active for this menu. An apply or OK will trigger the event-list.

2

I agree whole hardly with the concept. Config be a lot cleaner if these various “events” can be assigned a script. It’s not easily to look at a config and go through /sys/script & /sys/schedule to actually get a sense of what’s going WITHOUT reading each script while a script being tied to some path/command etc, you start with some context on script.

I’d say for sure the on-XXX= scheme should be more places! My example be /interface/detect-internet … it theoretically does something useful, but there is no way (without scheduler polling) to hook/alter it’s operation.

Now things like “/ip address print follow as-value” should allow a do={} action to “monitor” a value… BUT it there is no do={} on print so CANNOT – unlike the API which has a “listen”. But that be at least one “event driven” way of listening for changes from script, with minimal changes.

And, perhaps similarly if all “set” or “add” supported on-change={} attribute in config (perhaps others on-XXXX?) … since that be more tied to the eventual “id” – just without a register step.

On your larger concept…
I’ve thought it be nice to “register” functions and listeners to “paths”… e.g. ideally update the data shown in /console/inspect to extend the “schema”. And have that “schema extension scripts” as part of config…so you could essentially have a wrapper layer to do config using higher functions than the add/set primitives.

3

I agree that there should be many more places in RouterOS where an event would trigger a script, but I don’t understand why you propose all those implementation details.
As Amm0 also writes, there already are several event driven scripts, and the way they are tied to the respective events is quite clean and clear.
In some cases the availability (and documentation) of parameters available to the script, depending on the actual instance that started them, should be improved.

Also I think a valuable addition would be a system logging action “script” that runs a script whenever log events are sent to it.
Of course the script receives the log topics and message as a parameter.
That in itself would already enable a lot of event-driven scripting.

4

I did not want to fall back on existing routines because that will complicated things. If you want to have existing routines trigger scripts then a extra unique number has to be present that for example logging from a firewall rule. This already beyond the current proposal I made. I want to keep is simple only a limited parameters are sent back.

What is not guaranteed is that the situation expected by the script is not the situation anymore of moment of the trigger being was sent out. The time-out will catch most of the swift changes but it is not 100%. If you want 100% then there should a kind of transaction number. This could be the an timestamp that is sent to the script and the mutation timestamp stays is attached to the .id . If the timestamps don’t match then the script reacts on that by stopping further processing and to back to waiting state.

Also the .id can be different or not existing anymore then the timestamp compare, will stop processing. No overruling or ignoring possible. There may be places where this would be possible but for first lets keep it simple as possible.

This also makes is possible to know the last changes on information and sort on it in a firewall. So you can see which rule has been changed and in which time order.

I just was thinking how to have all those transactions being processed. When a trigger is arriving first checked first if the there is a matching script available. The check will also retrieve the time-out time and set that the just stacked item. The record will then have two timestamps, one the transaction timestamp and a timestamp increased with the time-out.

The stack is constantly processed till it is empty. Record that where the timestamp + time-out is not yet expired, goes back on the top of the stack again.

5

@Amm0:

I don’t know how Mirkrotik has implemented follow on print, and that could be an testing environment before taking the big leap before going system wide.

Then you could create a script that is executed when the same has is added to the pint command line featuring following an extra parameter “on-change-script”. There is no hash needed to be stated, because the path is used for the hash. However you have to know the naming of the script to be created.
This could be the parameter for showing the hash for the script name “shown-script-hash”

You can run a command from the root path so ROS will have to take the correct path from the command line to produce the correct hash and the print command has to contain following.

I just used /system history for the first time and you can see a lot and it will take a while look at all the options. But here I can go back and find changes.
What I like to implement with using the timestamp is only the last change time on one specific line.
That could be extended to history so that you get an list of the know changes on that line. But that is a different proposal/request.

ROS contains a lot of stuff but also a lot is not integrated with each other and are separate blocks.

6

I liked the implementation idea by pe1chl, it’s llike doing a tail -f on the log. Pass each and every line to a string/topic matcher and you got an event trigger.

7

I guess I’m not seeing need for the “event-list” table, if the commands or paths themselves support events specific to the command.

Taking your /user/active one… why not something more like
/user/active on-add={} on-remove={} on-update=()

Or just /user… where on-XXX= could be the path itself (or prefixed “listen” command to regularize it)…
/user listen on-disable={} on-enable={} on-XXX= …

If I get your suggestion, you’re thinking something like this:
/events/map path=/user/active script={:put “$id $parent-id $path $change”; :if ($change = “…”) do={…} }

@msatter, maybe some “pseudo-script” help explain what your thinking…

8

+1 for this.

9

Agree. And, “tail -f” == “print follow” … but there is no pipe | in RouterOS…

So, print supporting a “do=” go a long way, IMO…

/log print follow do={ 
     :put "$.dead $.id  $.nextid  $buffer $message $time $topics"
} where topics~container
10

Yes, but there is no need for such a special construct. There aleady are system logging actions of 5 types: disk, echo, email, memory, remote.
These actions take the logging events and do something: write to disk, print on console, send as email, put in memory (for Log window), send to remote syslog server.
When a new action type “script” is added, it can work in the same way: each time a message goes to that action, a script is called with the log message.
The script can then further look what to do (ignore it or do some processing).
And the user can define log rules for whatever message topic they want to handle in a script, and point it to the script action.
There can also be more than one script action, each with a different script, and with one or more rules pointing to it (just like there can be more than one disk or remote action).
It all fits very neatly in the existing system.

Even more powerful would be when the “Rules” would be extended with a “Regexp” so you can pre-filter messages before sending them to the action.
That could also be useful in other cases, e.g. when sending messages to email where you want to limit the spam. Sometimes the topics are too broad.

Finally, a good improvement of the logging mechanism would be to assign a unique log ID to every message that can possibly be logged in the system, and include that as a topic with each message. So you can define rules that match only for a single message.

All this has been discussed before, but nothing was ever changed. Maybe I should copy/paste this into a support ticket?

11

You likely should. They do eventually process them (e.g. I put one in for /tool/snmp-get missing “as-value”, took 2 years, but showed up a couple months ago and issue was marked fixed)

Yes, certainly “/log add action=script …” be clearer & log provides a pretty good source of events (see the all Splunk scripts)

Not quite @msatter’s original suggestion of a generalized approach to events… Both have some merit, just different. I’m fine with baby steps in any direction here.

12

I have submitted feature request SUP-124741.

13

I’ll add that the new MQTT subscribe in 7.11 – which screams for some events like on-recv= doesn’t have it… You’d think MT would have include some event handler for, well, an MQTT “event”. Instead, doc show “print” e.g. so polling is needed… but no examples of even that: https://help.mikrotik.com/docs/display/ROS/MQTT#MQTT-Subscribe

Maybe a wait for something here.

14 



{
:local killscript [:execute {/user print follow-only where [:foreach item in=[/user/active find] do={
   :if ([/user find name=[/user/active get $item]->"name" disabled] ) do={
     :do {/user/active request-logout $item} on-error={}}} ]}];
:delay 10s;
:do { /system script job remove $killscript } on-error={}
}

This will follow any changes for 10 seconds in /user and if so it will check if any user not active anymore and look if they have any local sessions open. The disabled user will be disconnect locally.

You have to limit the period the script is running in the background otherwise it is not terminated and keeps running till you kill it or your router crashes. :wink:

Source: https://help.mikrotik.com/docs/display/ROS/Scripting#Scripting-Commands and then the sample for :execute

15

And watching the log and being able to script on a change:

# activate reading the .id from global
:global eventPathLogTopicContainer

# remove previous running event scripts (:execute)
:do { /system script job remove $eventPathLogTopicContainer } on-error={:log error "Script .id eventPathLogTopicContainer not found"}

:global eventPathLogTopicContainer [:execute {
          /user print follow-only where topics~"container"[ :put "container scripting over here" ] } ]

Stop the script from running endlessly:

/system script job remove $eventPathLogTopicContainer
16

Interesting… So, if I understand your code “follow-only” already executes code for each item found – just no do={} is required & works today…

Now I’m not sure the :execute is even needed (for log etc) in my quick test at CLI…

/log print follow-only as-value where topics~".*" [:put "---- $[:tostr $topics] - $message"]

so it does provide the items from the log into the [ code ] provided at end, like $topics** and $message.

Although I still can’t why explain this works based on any reading of the docs… e.g. you have no operator, and somehow there is an implicit pipe & xargs equivalents …


** $topics is an array, why there is a $[:tostr $topics] in example – otherwise you get the output N times (N = number of topics). I know the “no array in string rule”, but took me a minute to remember myself & even that weird logic is documented…

17

The execute is there to wrap the print and code. You can’t press CTRL-C when it is still following and running in a script.

I moved the termination of execute to the top, so that any new start, will first terminate any existing same running event script.

Now you can set in scheduler that the event script starts on boot and a second script running at midnight for 24 hours and repeat that daily.

18

Yeah the :execute makes sense. More that I just didn’t know that syntax on print even existed :wink:

That why there should be more events :wink:

19

You can now write any event catcher you want. You can use the path directly if available, the log or history. Maybe even more and I think MT is even using this themselves, in the background.

This still very granular and no event timestamps control avaiable and no .id to know which setting has exactly been changed.

20

And your still left with parsing message= with log :wink: – now @jotne seems to extract a lot even though the log format isn’t regularized (or complete)…
NOW, since print works on anything so your right… other applications here…

While your scheduler trick is nifty … part of the underlying issue is “config readability” IMO – every script you add, make it hard to just look at a config and figure out what’s going on since a lot of logic gets buried in script/schedule.

I’m still a fan of more potential stuff like “/user on-disable={ remove session code } on-remove={ code… }” or “/container [find …] set on-output={ code to parse container output }” etc etc… Or something like your original “event registry” scheme…