Community discussions

MikroTik App
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

[PROPOSAL] Event driven scripting

Wed Aug 09, 2023 11:47 am

While working with scripting in ROS we have to revert to scheduler have a script being executed in a interval or on startup. This leaves time before a script can detect a change and make the its own changes in the setting or give a warning to the user.

My proposal is to give each path/option a unique identifier and sent that on any edit/create/remove/disable to an new script collection. The new script collection could looks like schedule, but it is only search for a matching script with the unique number as identifier.

O Noooooo, not that also!!! Changes made by ROS like with Netwatch, in paths that are read-only should be able to trigger a script. Think of /user/active keeping track of users being logged in.

The "event-list" should receive the following data:
- unique number for finding and start the appropriate script, if present
- path (example: path=/interface/wireguard/peers/)
- .id, the line being effected (example: id=*5)
- type of change (add/delete/edit/(disable/update))

The unique identifier is available in every menu suited for this and if already every option has an unique identifier then make this available to the user to be copied.

The "event-list" should have two fields, the unique identifier and a settable time-out. The timeout is there to have an delay before the script is being executed. This because often a user edits a change or edit shortly after confirming. So the script execution can be delay, if an edit is made then the second "trigger" will reset the timeout and the clock starts to run again.

I hope that the unique identifier is already present in ROS and that can be implemented without much effort. I advise to do it graduate release of the identifiers.

ANY thought, suggestions or complaints feel free to post those here and I am only interested in having Mikrotik a good input for this so that it is also attractive for them to implement it.

Update: as unique identifier you can also think of the converting the path to an hash. There is a short version of paths so it can be made fitting.
Also indicate in each menu, I am using WinBox, that an event-list entry is active for this menu. An apply or OK will trigger the event-list.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4240
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: [PROPOSAL] Event driven scripting

Wed Aug 09, 2023 6:08 pm

I agree whole hardly with the concept. Config be a lot cleaner if these various "events" can be assigned a script. It's not easily to look at a config and go through /sys/script & /sys/schedule to actually get a sense of what's going WITHOUT reading each script while a script being tied to some path/command etc, you start with some context on script.

I'd say for sure the on-XXX= scheme should be more places! My example be /interface/detect-internet ... it theoretically does something useful, but there is no way (without scheduler polling) to hook/alter it's operation.

Now things like "/ip address print follow as-value" should allow a do={} action to "monitor" a value... BUT it there is no do={} on print so CANNOT – unlike the API which has a "listen". But that be at least one "event driven" way of listening for changes from script, with minimal changes.

And, perhaps similarly if all "set" or "add" supported on-change={} attribute in config (perhaps others on-XXXX?) ... since that be more tied to the eventual "id" – just without a register step.

On your larger concept...
I've thought it be nice to "register" functions and listeners to "paths"... e.g. ideally update the data shown in /console/inspect to extend the "schema". And have that "schema extension scripts" as part of config...so you could essentially have a wrapper layer to do config using higher functions than the add/set primitives.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10516
Joined: Mon Jun 08, 2015 12:09 pm

Re: [PROPOSAL] Event driven scripting

Thu Aug 10, 2023 7:53 pm

I agree that there should be many more places in RouterOS where an event would trigger a script, but I don't understand why you propose all those implementation details.
As Amm0 also writes, there already are several event driven scripts, and the way they are tied to the respective events is quite clean and clear.
In some cases the availability (and documentation) of parameters available to the script, depending on the actual instance that started them, should be improved.

Also I think a valuable addition would be a system logging action "script" that runs a script whenever log events are sent to it.
Of course the script receives the log topics and message as a parameter.
That in itself would already enable a lot of event-driven scripting.
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: [PROPOSAL] Event driven scripting

Thu Aug 10, 2023 8:43 pm

I did not want to fall back on existing routines because that will complicated things. If you want to have existing routines trigger scripts then a extra unique number has to be present that for example logging from a firewall rule. This already beyond the current proposal I made. I want to keep is simple only a limited parameters are sent back.

What is not guaranteed is that the situation expected by the script is not the situation anymore of moment of the trigger being was sent out. The time-out will catch most of the swift changes but it is not 100%. If you want 100% then there should a kind of transaction number. This could be the an timestamp that is sent to the script and the mutation timestamp stays is attached to the .id . If the timestamps don't match then the script reacts on that by stopping further processing and to back to waiting state.

Also the .id can be different or not existing anymore then the timestamp compare, will stop processing. No overruling or ignoring possible. There may be places where this would be possible but for first lets keep it simple as possible.

This also makes is possible to know the last changes on information and sort on it in a firewall. So you can see which rule has been changed and in which time order.

I just was thinking how to have all those transactions being processed. When a trigger is arriving first checked first if the there is a matching script available. The check will also retrieve the time-out time and set that the just stacked item. The record will then have two timestamps, one the transaction timestamp and a timestamp increased with the time-out.

The stack is constantly processed till it is empty. Record that where the timestamp + time-out is not yet expired, goes back on the top of the stack again.
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: [PROPOSAL] Event driven scripting

Thu Aug 10, 2023 9:07 pm

@Amm0:

I don't know how Mirkrotik has implemented follow on print, and that could be an testing environment before taking the big leap before going system wide.

Then you could create a script that is executed when the same has is added to the pint command line featuring following an extra parameter "on-change-script". There is no hash needed to be stated, because the path is used for the hash. However you have to know the naming of the script to be created.
This could be the parameter for showing the hash for the script name "shown-script-hash"

You can run a command from the root path so ROS will have to take the correct path from the command line to produce the correct hash and the print command has to contain following.

I just used /system history for the first time and you can see a lot and it will take a while look at all the options. But here I can go back and find changes.
What I like to implement with using the timestamp is only the last change time on one specific line.
That could be extended to history so that you get an list of the know changes on that line. But that is a different proposal/request.

ROS contains a lot of stuff but also a lot is not integrated with each other and are separate blocks.
Last edited by msatter on Thu Aug 10, 2023 9:30 pm, edited 1 time in total.
 
4lphanumeric
newbie
Posts: 32
Joined: Wed Jan 16, 2019 1:00 pm

Re: [PROPOSAL] Event driven scripting

Thu Aug 10, 2023 9:16 pm

I liked the implementation idea by pe1chl, it's llike doing a tail -f on the log. Pass each and every line to a string/topic matcher and you got an event trigger.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4240
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: [PROPOSAL] Event driven scripting

Thu Aug 10, 2023 9:19 pm

I guess I'm not seeing need for the "event-list" table, if the commands or paths themselves support events specific to the command.

Taking your /user/active one... why not something more like
/user/active on-add={} on-remove={} on-update=()

Or just /user... where on-XXX= could be the path itself (or prefixed "listen" command to regularize it)...
/user listen on-disable={} on-enable={} on-XXX= ...

If I get your suggestion, you're thinking something like this:
/events/map path=/user/active script={:put "$id $parent-id $path $change"; :if ($change = "...") do={...} }

@msatter, maybe some "pseudo-script" help explain what your thinking...
Last edited by Amm0 on Thu Aug 10, 2023 9:48 pm, edited 2 times in total.
 
syadnom
Forum Veteran
Forum Veteran
Posts: 816
Joined: Thu Jan 27, 2011 7:29 am

Re: [PROPOSAL] Event driven scripting

Thu Aug 10, 2023 9:28 pm

+1 for this.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4240
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: [PROPOSAL] Event driven scripting

Thu Aug 10, 2023 9:58 pm

I liked the implementation idea by pe1chl, it's llike doing a tail -f on the log.
Agree. And, "tail -f" == "print follow" ... but there is no pipe | in RouterOS...

So, print supporting a "do=" go a long way, IMO...
/log print follow do={ 
     :put "$.dead $.id  $.nextid  $buffer $message $time $topics"
} where topics~container 
 
pe1chl
Forum Guru
Forum Guru
Posts: 10516
Joined: Mon Jun 08, 2015 12:09 pm

Re: [PROPOSAL] Event driven scripting

Thu Aug 10, 2023 10:39 pm

Yes, but there is no need for such a special construct. There aleady are system logging actions of 5 types: disk, echo, email, memory, remote.
These actions take the logging events and do something: write to disk, print on console, send as email, put in memory (for Log window), send to remote syslog server.
When a new action type "script" is added, it can work in the same way: each time a message goes to that action, a script is called with the log message.
The script can then further look what to do (ignore it or do some processing).
And the user can define log rules for whatever message topic they want to handle in a script, and point it to the script action.
There can also be more than one script action, each with a different script, and with one or more rules pointing to it (just like there can be more than one disk or remote action).
It all fits very neatly in the existing system.

Even more powerful would be when the "Rules" would be extended with a "Regexp" so you can pre-filter messages before sending them to the action.
That could also be useful in other cases, e.g. when sending messages to email where you want to limit the spam. Sometimes the topics are too broad.

Finally, a good improvement of the logging mechanism would be to assign a unique log ID to every message that can possibly be logged in the system, and include that as a topic with each message. So you can define rules that match only for a single message.

All this has been discussed before, but nothing was ever changed. Maybe I should copy/paste this into a support ticket?
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4240
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: [PROPOSAL] Event driven scripting

Thu Aug 10, 2023 11:20 pm

You likely should. They do eventually process them (e.g. I put one in for /tool/snmp-get missing "as-value", took 2 years, but showed up a couple months ago and issue was marked fixed)

Yes, certainly "/log add action=script ..." be clearer & log provides a pretty good source of events (see the all Splunk scripts)

Not quite @msatter's original suggestion of a generalized approach to events... Both have some merit, just different. I'm fine with baby steps in any direction here.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10516
Joined: Mon Jun 08, 2015 12:09 pm

Re: [PROPOSAL] Event driven scripting

Fri Aug 11, 2023 12:17 am

I have submitted feature request SUP-124741.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4240
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: [PROPOSAL] Event driven scripting

Fri Aug 11, 2023 7:49 pm

I'll add that the new MQTT subscribe in 7.11 – which screams for some events like on-recv= doesn't have it... You'd think MT would have include some event handler for, well, an MQTT "event". Instead, doc show "print" e.g. so polling is needed... but no examples of even that: https://help.mikrotik.com/docs/display/ ... -Subscribe

Maybe a wait for something here.
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 12:41 am

I liked the implementation idea by pe1chl, it's llike doing a tail -f on the log.
Agree. And, "tail -f" == "print follow" ... but there is no pipe | in RouterOS...

So, print supporting a "do=" go a long way, IMO...
/log print follow do={ 
     :put "$.dead $.id  $.nextid  $buffer $message $time $topics"
} where topics~container 

{
:local killscript [:execute {/user print follow-only where [:foreach item in=[/user/active find] do={
   :if ([/user find name=[/user/active get $item]->"name" disabled] ) do={
     :do {/user/active request-logout $item} on-error={}}} ]}];
:delay 10s;
:do { /system script job remove $killscript } on-error={}
}
This will follow any changes for 10 seconds in /user and if so it will check if any user not active anymore and look if they have any local sessions open. The disabled user will be disconnect locally.

You have to limit the period the script is running in the background otherwise it is not terminated and keeps running till you kill it or your router crashes. ;-)

Source: https://help.mikrotik.com/docs/display/ ... g-Commands and then the sample for :execute
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 2:22 pm

And watching the log and being able to script on a change:
# activate reading the .id from global
:global eventPathLogTopicContainer

# remove previous running event scripts (:execute)
:do { /system script job remove $eventPathLogTopicContainer } on-error={:log error "Script .id eventPathLogTopicContainer not found"}

:global eventPathLogTopicContainer [:execute {
          /user print follow-only where topics~"container"[ :put "container scripting over here" ] } ]
Stop the script from running endlessly:
/system script job remove $eventPathLogTopicContainer
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4240
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 3:40 pm

Interesting... So, if I understand your code "follow-only" already executes code for each item found – just no do={} is required & works today...

Now I'm not sure the :execute is even needed (for log etc) in my quick test at CLI...
/log print follow-only as-value where topics~".*" [:put "---- $[:tostr $topics] - $message"] 
so it does provide the items from the log into the [ code ] provided at end, like $topics** and $message.

Although I still can't why explain this works based on any reading of the docs... e.g. you have no operator, and somehow there is an implicit pipe & xargs equivalents ...


** $topics is an array, why there is a $[:tostr $topics] in example – otherwise you get the output N times (N = number of topics). I know the "no array in string rule", but took me a minute to remember myself & even that weird logic is documented...
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 4:21 pm

The execute is there to wrap the print and code. You can't press CTRL-C when it is still following and running in a script.

I moved the termination of execute to the top, so that any new start, will first terminate any existing same running event script.

Now you can set in scheduler that the event script starts on boot and a second script running at midnight for 24 hours and repeat that daily.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4240
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 4:51 pm

Yeah the :execute makes sense. More that I just didn't know that syntax on print even existed ;)
You can't press CTRL-C when it is still following and running in a script.
That why there should be more events ;)
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 5:37 pm

You can now write any event catcher you want. You can use the path directly if available, the log or history. Maybe even more and I think MT is even using this themselves, in the background.

This still very granular and no event timestamps control avaiable and no .id to know which setting has exactly been changed.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4240
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 5:59 pm

This still very granular and no event timestamps control avaiable and no .id to know which setting has exactly been changed.
And your still left with parsing message= with log ;) – now @jotne seems to extract a lot even though the log format isn't regularized (or complete)...
NOW, since print works on anything so your right... other applications here...

While your scheduler trick is nifty ... part of the underlying issue is "config readability" IMO – every script you add, make it hard to just look at a config and figure out what's going on since a lot of logic gets buried in script/schedule.

I'm still a fan of more potential stuff like "/user on-disable={ remove session code } on-remove={ code... }" or "/container [find ...] set on-output={ code to parse container output }" etc etc.... Or something like your original "event registry" scheme...
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 6:17 pm

A lot of users wish to do something, when a rule is hit in the firewall. That is now possible because you can log it. If you can log it then you can create an event trigger for that.
Create a rule and activate logging on the rule and set log-prefix to lets say "IamHit". The you scan the log for that text in the log message and voila, there you have your event trigger. Think of port knocking and only activate the next rule when the previous knock was correct.

You have to structure you events and that is why I have proposed using the path in ROS. The name for the process .id of the execute, is the event + path so you can match that with the name in schedule. You can log inside the event script and so track when it is used/(re)started/stopped.

And I don't say in any way that this the solution, but this is something that we can now work with. My original proposal is allowing much more control over what exactly is happening and react precise to that.

ps. I can parse the log-line but that breaks :execute being run in the background and I loose the process .id I will think about parsing the line somehow. You have the trigger and then you can also have a second script being started to retrieve the matching log line. But then, are you retrieving the correct line because an other event could have just happened. That is where time stamping comes in handy.

update: however :execute can write to file
 :execute {} file=eventUser.txt
and between {} sits the script. The file contains the text that print sees. I did not test with :log.

A bit tricky to generate the file with log because it is overwritten, but I got some lines from logging in and out:

18:00:48 system,info,account user test logged out from 192.x.x.x. via winbox
18:00:48 system,info,account user test logged out from 192.x.x.x via local

The size of the file indicates (>2) if there something in there.

The rest I leave to others, I have shown the building blocks and how you can see what and how stored in the file.

Have fun ;-)
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4240
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 7:21 pm

And I don't say in any way that this the solution, but this is something that we can now work with. My original proposal is allowing much more control over what exactly is happening and react precise to that.
Fair enough – didn't mean to hijack your thread/idea. I think your "event-list" is a good concept...

Essentially you're suggesting RouterOS act like some MQ publisher of path+attributes, with RSC scripts being subscribers. I suppose another way to skin the cat is RouterOS could be MQTT publisher, with path+attributes (e.g. it's config/counters/etc) as "topics" to subscribe. Since RouterOS already supports subscribe (although still needs a script handler...), a new "RouterOS MQTT Publisher", provide routers events a local script BUT, also, remotely via remote MQTT subscribers to the RouterOS publisher...

Whether it actually needs to use actual MQTT protocol, IDK... my point is more there already RouterOS syntax for MQTT publish/subscribe that could be "borrowed" for your event-list...
 
User avatar
Kentzo
Long time Member
Long time Member
Posts: 618
Joined: Mon Jan 27, 2014 3:35 pm
Location: California

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 9:30 pm

The described model heavily reminds me of frontend web development. Except RouterOS is nowhere near having debugging and introspection capabilities of a modern browser. And its programming language is very ill suited for anything beyond trivial actions.

Perhaps when you hit a use cases like that it’s time boot up a linux based NOS: highly introspectable, debuggable and with an arsenal of mature programming languages.

Or start with a container + API.

My area is CPE and I use scripts exclusively as workarounds that could be replaced by proper feature implementation by Mikrotik.
 
syadnom
Forum Veteran
Forum Veteran
Posts: 816
Joined: Thu Jan 27, 2011 7:29 am

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 10:26 pm

Perhaps when you hit a use cases like that it’s time boot up a linux based NOS: highly introspectable, debuggable and with an arsenal of mature programming languages.
You're asking too much, for the most part we don't need this and if we did, having event driven scripts would allow us to also do this.

For instance, I want to know if a port drops from 1000 to 100. I can poll though through a script routinely, but it would by much nicer to just have that logged change trigger a script. ie, this is something that happened to a port, and it was a speed change. make a script subscribed to 'ethernet ports' and 'speed change' and then only trigger that script when matched.

Or a BGP peer. goes down, that's BGP, peer-status, down and triggers a script that uses fetch to push that to some other 'more capable' script.

I might do that do push to my QoE appliance so it can update topology for the downed primary peer or something like that.
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 10:26 pm

@Amm0 I am open to any suggestions and that also forced me to look deeper in the CLI and then saw an example that I did not understand before for a long time. Am I the first one that discovered the potential of this bit of code that MT just left there to be discovered?

To me, MQTT is only polling and when you describe you still have to initiate a pol to retrieve data. I like a clean approach for ROS and that it pushes data on change, and not when a request is done.

Looking at API it has the listen command and there you need also to terminate, when you don't want to receive updates anymore:

listen command is available where console print command is available, but it does not have expected effect everywhere (i.e. may not work)
- "!re" sentences are generated as something changes in a particular item list
- when an item is deleted or disappears in any other way, the '!re' sentence includes the value '=.dead=yes'
- This command does not terminate. To terminate it, use /cancel command.

https://help.mikrotik.com/docs/display/ ... ive/listen

This compares close to what is done with :execute{} and stop the :execute by terminating the running event script. I don't get any direct information of the change itself other than being written to a file.

So for CLI, there could be improvements taken from the API. What about an interpreter giving CLI direct access to API. There is already a cross over and you will find proplist under print enabling to only show a specific value.

https://help.mikrotik.com/docs/display/ ... alCommands

Under /user the F1-key shows:

Proplist ::= ValueName[,Proplist]
ValueName ::= address | comment | disabled | expired | group | ...
disabled -- Defines whether item is ignored or used
group -- Group management
address -- Network address part of addresses user is allowed to use
comment -- Short description of the item
name -- User name
password -- User password

/log

Proplist ::= ValueName[,Proplist]
ValueName ::= buffer | message | time | topics
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 11:42 pm

For instance, I want to know if a port drops from 1000 to 100. I can poll though through a script routinely, but it would by much nicer to just have that logged change trigger a script. ie, this is something that happened to a port, and it was a speed change. make a script subscribed to 'ethernet ports' and 'speed change' and then only trigger that script when matched.
Monitor can see all interface speeds however I use that.
@/interface ethernet print proplist=name,speed         
Flags: X - DISABLED, R - RUNNING; S - SLAVE
Columns: NAME, SPEED
 #    NAME          SPEED
 0 X  ether1             
 1 R  ether2             
 2 X  ether3             
 3 X  ether4             
 4 X  ether5             
 5 RS ether6             
 6  S ether7             
 7  S ether8             
 8 RS ether9             
 9 RS ether10            
10 R  sfp-sfpplus1  1Gbps
I can only see the speed of the SFP, only that one can be used for events through print follow-only.
 
User avatar
Kentzo
Long time Member
Long time Member
Posts: 618
Joined: Mon Jan 27, 2014 3:35 pm
Location: California

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 11:49 pm

having event driven scripts would allow us to also do this.
It's for sure flexible, but raises a lot of question. E.g. given a series of related events, what is the proper order? If your action on an event leads to it being re-triggered, then what? If you have multiple actions on an event, how to determine execution order?

And are absolutely no debugging facilities, moreover a mistake (broken configuration or an indirect but indefinite CPU-consuming loop) will leave system inaccessible. And there won't be any post-mortem facilities either such as core dump.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4240
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: [PROPOSAL] Event driven scripting

Sat Aug 12, 2023 11:55 pm

Am I the first one that discovered the potential of this bit of code that MT just left there to be discovered?
The print follow [ code ] was new to me at least. So shoving into an :execute+scheduler to have a "long poll" certainly novel.


https://help.mikrotik.com/docs/display/ ... ive/listen
[...]
So for CLI, there could be improvements taken from the API. What about an interpreter giving CLI direct access to API. There is already a cross over and you will find proplist under print enabling to only show a specific value.
Also similar to what happens in winbox, you open a window and the stats update dynamically. AFAIK, it does similar to the "API listen" (and when you close window, tells stops the "listen"). But why I mention it, is it's something RouterOS seem to already do (e.g. push values on change)...

And I think your print follow [ code] scheme actually does that ;)

To me, MQTT is only polling and when you describe you still have to initiate a pol to retrieve data. I like a clean approach for ROS and that it pushes data on change, and not when a request is done.
More example... Likely off-topic. But "RouterOS config/stats as a MQTT Publisher" seems like a different topic – only related since allow "events" between multiple routers.

The described model heavily reminds me of frontend web development.
Copying a successful model isn't a necessarily a bad approach. But, yes, if you squint your eyes, the "config paths" start look like HTML5 XPath, so does an on-change= in RouterOS config line seem that bad?

Part of the problem today is scripts are more procedural than needed. Some more "event handler", however, allow more excel-style coding where it's needed in context, and that avoid some need for monolithic scripts in scheduler/script. Obviously scripting could use other improvements too (e.g. JSON) :)

Perhaps when you hit a use cases like that it’s time boot up a linux based NOS: highly introspectable, debuggable and with an arsenal of mature programming languages.
You're asking too much, for the most part we don't need this and if we did, having event driven scripts would allow us to also do this.
Agree, just [a] "listen" ;)

@syadnom, a good example be multi-path wireless. Let say linking setting the PCC values based on some wireless statistics. Totally possible today, but now some important routing logic is kinda hidden in a schedule script – contributing to @Kentzo's points. If you could "bind" a variable to another via script (e.g. via some kinda listen), the link between the firewall mangle and scripts that's dynamic/event-driven set based on another RouterOS attribute be useful...
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: [PROPOSAL] Event driven scripting

Sun Aug 13, 2023 12:23 am

having event driven scripts would allow us to also do this.
It's for sure flexible, but raises a lot of question. E.g. given a series of related events, what is the proper order? If your action on an event leads to it being re-triggered, then what? If you have multiple actions on an event, how to determine execution order?

And are absolutely no debugging facilities, moreover a mistake (broken configuration or an indirect but indefinite CPU-consuming loop) will leave system inaccessible. And there won't be any post-mortem facilities either such as core dump.
Sequencing and outdated events are subscribed in my postings and the stack is self cleaning.

Same events arriving in the timeout period pushed the already waiting even out and resets the timeout period. Secondly if there is near miss between the events the timestamp from the first event is then different than the timestamp of the last change. Result the first event is ignored because the timestamps do not match. The next event has the correct timestamp and is then accepted. The original timestamp is on the item itself.

Order is FIFO and if the timeout for starting the script has not expired then the event goes back to the top of the stack.
In the end, only one event goes to execution of the script, however the timestamp has to match. In case of a storm the timestamps do not match so no script is executed.

The stack grows then fast but the arrival of the same events pushes out any earlier events. A form of storm control could be needed.
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: [PROPOSAL] Event driven scripting

Sun Aug 13, 2023 12:29 am

Perhaps when you hit a use cases like that it’s time boot up a linux based NOS: highly introspectable, debuggable and with an arsenal of mature programming languages.
You're asking too much, for the most part we don't need this and if we did, having event driven scripts would allow us to also do this.

For instance, I want to know if a port drops from 1000 to 100. I can poll though through a script routinely, but it would by much nicer to just have that logged change trigger a script. ie, this is something that happened to a port, and it was a speed change. make a script subscribed to 'ethernet ports' and 'speed change' and then only trigger that script when matched.
I can't test this one short term. This should log when the interface changes it's speed.
/interface ethernet monitor [find where "ether2" !"1Gbps"] do={:log warning "Ether2 changed it's speed"}
Yes it is strange filter, inverting the speed present. Brain hurts from it. ;-)
 
User avatar
Kentzo
Long time Member
Long time Member
Posts: 618
Joined: Mon Jan 27, 2014 3:35 pm
Location: California

Re: [PROPOSAL] Event driven scripting

Sun Aug 13, 2023 1:54 am

Copying a successful model isn't a necessarily a bad approach.
I did not mean it in derogatory sense. In turn it’s a good example of issues can and of programming models that can be used to tackle them. Just think about all the available tooling, is it feasible to expect something similar from Mikrotik given its closed source nature?
 
msatter
Forum Guru
Forum Guru
Topic Author
Posts: 2939
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: [PROPOSAL] Event driven scripting

Sun Aug 13, 2023 11:41 am

I can vow that atleast I have no prior knowledgement of any programing models. ;-) Just adapting on my way.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4240
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: [PROPOSAL] Event driven scripting

Sun Aug 13, 2023 6:30 pm

I'm guess I'm not looking for a programing environment here...just a better way to expressing existing config than actually long/fragile/complex scripts. And 100% agree some scripts should indeed get replaced with built-in things – I can search "My posts" to find dozens ;). But not all scripts are necessarily as a hack around lack of support of something...

And RouterOS does know when stuff changes & in some places some built-in functionality that invoke some custom script be a feature, not a workaround... And think it's removing some need for programming constructs like loops and polling with some kinda "events", actually simplify scripting, not make it more complex...

e.g. RouterOS config acted more like Excel (or Lotus 1-2-3), which billions seem to have learned. Like if you have a spreadsheet cell with some formula "=B3*C5", that cell will automatically change if either B3 or C5 changes. No loops, no polling, just an expression.

If something like this was possible, that's more of what I'd be looking for from "events" – a config item bound/linked/mapped/etc to an attribute and/or script & saved with the config so it just run on a change (no scheduling/polling):
/system note set note=[/system identity listen name]
where if the system name was changed, the system note would update & the "listen formula" was stored in the config, not replaced.
I'd be fine with @msatter's event-list ... still the same "if this changes, do that"
 
syadnom
Forum Veteran
Forum Veteran
Posts: 816
Joined: Thu Jan 27, 2011 7:29 am

Re: [PROPOSAL] Event driven scripting

Sun Aug 13, 2023 8:22 pm

Again, I caution requesting the moon and stars here or getting too wrapped up in details.

There are events that happen that we want to trigger an action on. Asking mikrotik to also tie in specific triggers across the system is a really big ask vs triggering a script.

It's up to the operator to not over complicate their scripts that would cause logic loops.

events should have an event id that can be logged. That event ID can easily be used in the script triggering to only allow one execution per event. push to logs "event id duplicate in script xyz"
 
pe1chl
Forum Guru
Forum Guru
Posts: 10516
Joined: Mon Jun 08, 2015 12:09 pm

Re: [PROPOSAL] Event driven scripting

Tue Aug 22, 2023 2:39 pm

As I wrote above I have made a ticket with a suggestion for improvement of the logging, and it was put on list for possible implementation.
(with the usual disclaimer that it is unknown in what version it would become available)
 
osamahfarhan
just joined
Posts: 10
Joined: Sat Mar 14, 2020 2:35 am
Contact:

Re: [PROPOSAL] Event driven scripting

Tue May 28, 2024 12:14 am


Now things like "/ip address print follow as-value" should allow a do={} action to "monitor" a value... BUT it there is no do={} on print so CANNOT – unlike the API which has a "listen". But that be at least one "event driven" way of listening for changes from script, with minimal changes.

And, perhaps similarly if all "set" or "add" supported on-change={} attribute in config (perhaps others on-XXXX?) ... since that be more tied to the eventual "id" – just without a register step.

On your larger concept...
I've thought it be nice to "register" functions and listeners to "paths"... e.g. ideally update the data shown in /console/inspect to extend the "schema". And have that "schema extension scripts" as part of config...so you could essentially have a wrapper layer to do config using higher functions than the add/set primitives.

look at this
viewtopic.php?t=207973

/ip hotspot user print follow-only where [$EventHandler Name=$name];

Who is online

Users browsing this forum: No registered users and 31 guests