[PROPOSAL] Event driven scripting

[msatter]

A lot of users wish to do something, when a rule is hit in the firewall. That is now possible because you can log it. If you can log it then you can create an event trigger for that.
Create a rule and activate logging on the rule and set log-prefix to lets say “IamHit”. The you scan the log for that text in the log message and voila, there you have your event trigger. Think of port knocking and only activate the next rule when the previous knock was correct.

You have to structure you events and that is why I have proposed using the path in ROS. The name for the process .id of the execute, is the event + path so you can match that with the name in schedule. You can log inside the event script and so track when it is used/(re)started/stopped.

And I don’t say in any way that this the solution, but this is something that we can now work with. My original proposal is allowing much more control over what exactly is happening and react precise to that.

ps. I can parse the log-line but that breaks :execute being run in the background and I loose the process .id I will think about parsing the line somehow. You have the trigger and then you can also have a second script being started to retrieve the matching log line. But then, are you retrieving the correct line because an other event could have just happened. That is where time stamping comes in handy.

update: however :execute can write to file

 :execute {} file=eventUser.txt

and between {} sits the script. The file contains the text that print sees. I did not test with :log.

A bit tricky to generate the file with log because it is overwritten, but I got some lines from logging in and out:

18:00:48 system,info,account user test logged out from 192.x.x.x. via winbox
18:00:48 system,info,account user test logged out from 192.x.x.x via local

The size of the file indicates (>2) if there something in there.

The rest I leave to others, I have shown the building blocks and how you can see what and how stored in the file.

Have fun

[Amm0]

And I don’t say in any way that this the solution, but this is something that we can now work with. My original proposal is allowing much more control over what exactly is happening and react precise to that.

Fair enough – didn’t mean to hijack your thread/idea. I think your “event-list” is a good concept…

Essentially you’re suggesting RouterOS act like some MQ publisher of path+attributes, with RSC scripts being subscribers. I suppose another way to skin the cat is RouterOS could be MQTT publisher, with path+attributes (e.g. it’s config/counters/etc) as “topics” to subscribe. Since RouterOS already supports subscribe (although still needs a script handler…), a new “RouterOS MQTT Publisher”, provide routers events a local script BUT, also, remotely via remote MQTT subscribers to the RouterOS publisher…

Whether it actually needs to use actual MQTT protocol, IDK… my point is more there already RouterOS syntax for MQTT publish/subscribe that could be “borrowed” for your event-list…

[Kentzo]

The described model heavily reminds me of frontend web development. Except RouterOS is nowhere near having debugging and introspection capabilities of a modern browser. And its programming language is very ill suited for anything beyond trivial actions.

Perhaps when you hit a use cases like that it’s time boot up a linux based NOS: highly introspectable, debuggable and with an arsenal of mature programming languages.

Or start with a container + API.

My area is CPE and I use scripts exclusively as workarounds that could be replaced by proper feature implementation by Mikrotik.

[syadnom]

Perhaps when you hit a use cases like that it’s time boot up a linux based NOS: highly introspectable, debuggable and with an arsenal of mature programming languages.

You’re asking too much, for the most part we don’t need this and if we did, having event driven scripts would allow us to also do this.

For instance, I want to know if a port drops from 1000 to 100. I can poll though through a script routinely, but it would by much nicer to just have that logged change trigger a script. ie, this is something that happened to a port, and it was a speed change. make a script subscribed to ‘ethernet ports’ and ‘speed change’ and then only trigger that script when matched.

Or a BGP peer. goes down, that’s BGP, peer-status, down and triggers a script that uses fetch to push that to some other ‘more capable’ script.

I might do that do push to my QoE appliance so it can update topology for the downed primary peer or something like that.

[msatter]

@Amm0 I am open to any suggestions and that also forced me to look deeper in the CLI and then saw an example that I did not understand before for a long time. Am I the first one that discovered the potential of this bit of code that MT just left there to be discovered?

To me, MQTT is only polling and when you describe you still have to initiate a pol to retrieve data. I like a clean approach for ROS and that it pushes data on change, and not when a request is done.

Looking at API it has the listen command and there you need also to terminate, when you don’t want to receive updates anymore:

listen command is available where console print command is available, but it does not have expected effect everywhere (i.e. may not work)

• “!re” sentences are generated as something changes in a particular item list
• when an item is deleted or disappears in any other way, the ‘!re’ sentence includes the value ‘=.dead=yes’
• This command does not terminate. To terminate it, use /cancel command.

https://help.mikrotik.com/docs/display/ROS/API#API-/user/active/listen

This compares close to what is done with :execute{} and stop the :execute by terminating the running event script. I don’t get any direct information of the change itself other than being written to a file.

So for CLI, there could be improvements taken from the API. What about an interpreter giving CLI direct access to API. There is already a cross over and you will find proplist under print enabling to only show a specific value.

https://help.mikrotik.com/docs/display/ROS/Command+Line+Interface#CommandLineInterface-GeneralCommands

Under /user the F1-key shows:

Proplist ::= ValueName[,Proplist]
ValueName ::= address | comment | disabled | expired | group | …
disabled – Defines whether item is ignored or used
group – Group management
address – Network address part of addresses user is allowed to use
comment – Short description of the item
name – User name
password – User password

/log

Proplist ::= ValueName[,Proplist]
ValueName ::= buffer | message | time | topics

[msatter]

For instance, I want to know if a port drops from 1000 to 100. I can poll though through a script routinely, but it would by much nicer to just have that logged change trigger a script. ie, this is something that happened to a port, and it was a speed change. make a script subscribed to ‘ethernet ports’ and ‘speed change’ and then only trigger that script when matched.

Monitor can see all interface speeds however I use that.

@/interface ethernet print proplist=name,speed         
Flags: X - DISABLED, R - RUNNING; S - SLAVE
Columns: NAME, SPEED
 #    NAME          SPEED
 0 X  ether1             
 1 R  ether2             
 2 X  ether3             
 3 X  ether4             
 4 X  ether5             
 5 RS ether6             
 6  S ether7             
 7  S ether8             
 8 RS ether9             
 9 RS ether10            
10 R  sfp-sfpplus1  1Gbps

I can only see the speed of the SFP, only that one can be used for events through print follow-only.

[Kentzo]

having event driven scripts would allow us to also do this.

It’s for sure flexible, but raises a lot of question. E.g. given a series of related events, what is the proper order? If your action on an event leads to it being re-triggered, then what? If you have multiple actions on an event, how to determine execution order?

And are absolutely no debugging facilities, moreover a mistake (broken configuration or an indirect but indefinite CPU-consuming loop) will leave system inaccessible. And there won’t be any post-mortem facilities either such as core dump.

[Amm0]

Am I the first one that discovered the potential of this bit of code that MT just left there to be discovered?

The print follow [ code ] was new to me at least. So shoving into an :execute+scheduler to have a “long poll” certainly novel.

https://help.mikrotik.com/docs/display/ROS/API#API-/user/active/listen
[…]
So for CLI, there could be improvements taken from the API. What about an interpreter giving CLI direct access to API. There is already a cross over and you will find proplist under print enabling to only show a specific value.

Also similar to what happens in winbox, you open a window and the stats update dynamically. AFAIK, it does similar to the “API listen” (and when you close window, tells stops the “listen”). But why I mention it, is it’s something RouterOS seem to already do (e.g. push values on change)…

And I think your print follow [ code] scheme actually does that

To me, MQTT is only polling and when you describe you still have to initiate a pol to retrieve data. I like a clean approach for ROS and that it pushes data on change, and not when a request is done.

More example… Likely off-topic. But “RouterOS config/stats as a MQTT Publisher” seems like a different topic – only related since allow “events” between multiple routers.

@Kentzo

The described model heavily reminds me of frontend web development.

Copying a successful model isn’t a necessarily a bad approach. But, yes, if you squint your eyes, the “config paths” start look like HTML5 XPath, so does an on-change= in RouterOS config line seem that bad?

Part of the problem today is scripts are more procedural than needed. Some more “event handler”, however, allow more excel-style coding where it’s needed in context, and that avoid some need for monolithic scripts in scheduler/script. Obviously scripting could use other improvements too (e.g. JSON)

@syadnom

Perhaps when you hit a use cases like that it’s time boot up a linux based NOS: highly introspectable, debuggable and with an arsenal of mature programming languages.

You’re asking too much, for the most part we don’t need this and if we did, having event driven scripts would allow us to also do this.

Agree, just [a] “listen”

@syadnom, a good example be multi-path wireless. Let say linking setting the PCC values based on some wireless statistics. Totally possible today, but now some important routing logic is kinda hidden in a schedule script – contributing to @Kentzo’s points. If you could “bind” a variable to another via script (e.g. via some kinda listen), the link between the firewall mangle and scripts that’s dynamic/event-driven set based on another RouterOS attribute be useful…

[msatter]

having event driven scripts would allow us to also do this.

It’s for sure flexible, but raises a lot of question. E.g. given a series of related events, what is the proper order? If your action on an event leads to it being re-triggered, then what? If you have multiple actions on an event, how to determine execution order?

And are absolutely no debugging facilities, moreover a mistake (broken configuration or an indirect but indefinite CPU-consuming loop) will leave system inaccessible. And there won’t be any post-mortem facilities either such as core dump.

Sequencing and outdated events are subscribed in my postings and the stack is self cleaning.

Same events arriving in the timeout period pushed the already waiting even out and resets the timeout period. Secondly if there is near miss between the events the timestamp from the first event is then different than the timestamp of the last change. Result the first event is ignored because the timestamps do not match. The next event has the correct timestamp and is then accepted. The original timestamp is on the item itself.

Order is FIFO and if the timeout for starting the script has not expired then the event goes back to the top of the stack.
In the end, only one event goes to execution of the script, however the timestamp has to match. In case of a storm the timestamps do not match so no script is executed.

The stack grows then fast but the arrival of the same events pushes out any earlier events. A form of storm control could be needed.

[msatter]

Perhaps when you hit a use cases like that it’s time boot up a linux based NOS: highly introspectable, debuggable and with an arsenal of mature programming languages.

You’re asking too much, for the most part we don’t need this and if we did, having event driven scripts would allow us to also do this.

For instance, I want to know if a port drops from 1000 to 100. I can poll though through a script routinely, but it would by much nicer to just have that logged change trigger a script. ie, this is something that happened to a port, and it was a speed change. make a script subscribed to ‘ethernet ports’ and ‘speed change’ and then only trigger that script when matched.

I can’t test this one short term. This should log when the interface changes it’s speed.

/interface ethernet monitor [find where "ether2" !"1Gbps"] do={:log warning "Ether2 changed it's speed"}

Yes it is strange filter, inverting the speed present. Brain hurts from it.

[Kentzo]

Copying a successful model isn’t a necessarily a bad approach.

I did not mean it in derogatory sense. In turn it’s a good example of issues can and of programming models that can be used to tackle them. Just think about all the available tooling, is it feasible to expect something similar from Mikrotik given its closed source nature?

[msatter]

I can vow that atleast I have no prior knowledgement of any programing models. Just adapting on my way.

[Amm0]

I’m guess I’m not looking for a programing environment here…just a better way to expressing existing config than actually long/fragile/complex scripts. And 100% agree some scripts should indeed get replaced with built-in things – I can search “My posts” to find dozens . But not all scripts are necessarily as a hack around lack of support of something…

And RouterOS does know when stuff changes & in some places some built-in functionality that invoke some custom script be a feature, not a workaround… And think it’s removing some need for programming constructs like loops and polling with some kinda “events”, actually simplify scripting, not make it more complex…

e.g. RouterOS config acted more like Excel (or Lotus 1-2-3), which billions seem to have learned. Like if you have a spreadsheet cell with some formula “=B3*C5”, that cell will automatically change if either B3 or C5 changes. No loops, no polling, just an expression.

If something like this was possible, that’s more of what I’d be looking for from “events” – a config item bound/linked/mapped/etc to an attribute and/or script & saved with the config so it just run on a change (no scheduling/polling):
/system note set note=[/system identity listen name]
where if the system name was changed, the system note would update & the “listen formula” was stored in the config, not replaced.
I’d be fine with @msatter’s event-list … still the same “if this changes, do that”

[syadnom]

Again, I caution requesting the moon and stars here or getting too wrapped up in details.

There are events that happen that we want to trigger an action on. Asking mikrotik to also tie in specific triggers across the system is a really big ask vs triggering a script.

It’s up to the operator to not over complicate their scripts that would cause logic loops.

events should have an event id that can be logged. That event ID can easily be used in the script triggering to only allow one execution per event. push to logs “event id duplicate in script xyz”

[pe1chl]

As I wrote above I have made a ticket with a suggestion for improvement of the logging, and it was put on list for possible implementation.
(with the usual disclaimer that it is unknown in what version it would become available)

[osamahfarhan]

Now things like “/ip address print follow as-value” should allow a do={} action to “monitor” a value… BUT it there is no do={} on print so CANNOT – unlike the API which has a “listen”. But that be at least one “event driven” way of listening for changes from script, with minimal changes.

And, perhaps similarly if all “set” or “add” supported on-change={} attribute in config (perhaps others on-XXXX?) … since that be more tied to the eventual “id” – just without a register step.

On your larger concept…
I’ve thought it be nice to “register” functions and listeners to “paths”… e.g. ideally update the data shown in /console/inspect to extend the “schema”. And have that “schema extension scripts” as part of config…so you could essentially have a wrapper layer to do config using higher functions than the add/set primitives.

look at this
http://forum.mikrotik.com/t/mikrotik-events-script-new-abroach/176282/1

/ip hotspot user print follow-only where [$EventHandler Name=$name];