Persistent Environment Variables

BrianHiggins · Wed Dec 09, 2020 6:47 pm

other than to load them with a script every reboot, is there any way to get an environment variable to persist between multiple reboots?

msatter · Thu Dec 10, 2020 1:03 am

No, this was discussed in the 7.1beta3 thread a few days ago.

liuyao · Sat Jan 30, 2021 11:09 am

by baidu translate ：
I have a method to define a schedule to read environment variables to the array every minute. In the definition of the second boot from the start of the schedule. Use the acquired array to write to the second schedule at any time. Maybe the idea is to make it by yourself

There is only one command to demonstrate ：
:put [/system script environment print as-value ]

Jotne · Sat Jan 30, 2021 1:12 pm

OP did not want a script or using scheduling.

I agree that there should be a way to set persistent variables that survives reboot.
You can add a fw rule it stays trough a reboot, so should a persistent variable do as well.

millenium7 · Mon Feb 08, 2021 6:17 am

For the moment you can kinda-sorta get away with it by storing variables as comments. The layer7-protocol area is not a bad one because it is practically unused these days so it doesn't clutter up the interface if you put a bunch of variables in there
Comments will not store arrays correctly, or rather the 'toarray' command to convert a string back to an array doesn't work because arrays store with semi-colon separates. Whereas the 'toarray' command uses commas to separate them (why mikrotik why?)

But this method works just fine for simpler things. I use it for incrementing a counter in a few scripts. One of them being a template that runs at a later time, where say a router is told to change some config and then reboot. Well the first command is to increment an integer in the comment of that scheduled task, it's first read and if the counter is higher than say 1 then something has gone wrong, commands havn't executed properly, script has failed, it failed to self-delete (otherwise its just going to reboot every single day at 3am forever) etc so it'll then send an email to support desk as well as a remote syslog message to alert that somethings not right and requires manual intervention

rextended · Sat Oct 02, 2021 1:30 pm

search tag # rextended save and restore global variables on reboot

This is for save and restore variables with keeping the right data type on restore:

Scheduled script do that every x minutes, on purpose store only variables, but do not store codes or functions
because till now I do not have find one way to do that.
code = from ":parse" function, like :global c [:parse "/interface print"]
function = created like :global f do={ /interface print }, but is reported as array from :typeof

Only simple arrays are exported, do not work for array of array and more complex data structure.

/system script environment
:foreach item in=[find] do={
    :local vname  [get $item name]
    :local vvalue [get $item value]
    :if ($vvalue~"^\\*") do={:set vvalue "ID$vvalue"}
    :if ($vvalue~"^(\\(code\\)|;\?\\(eva\?l )") do={:set vvalue "(code)"}
    /ip firewall layer7
    remove [find where name=$vname]
    add name=$vname comment="$vvalue"
    :delay 10ms
    :execute "/ip firewall layer7 set [find where name=$vname] regexp=[:typeof \$$vname]"
    :if ($vvalue="(code)") do={:delay 10ms ; set [find where name=$vname] regexp="code"}
}

Store variable name... on name, variable type on regexp, variable contents on comment.

On reboot this script is scheduled at startup, this restore also variables that do not contain readable data, like binary data, lower numbers and special characters:

/ip firewall layer7
:foreach item in=[find where regexp~"^(array|bool|code|id|ip|ip-prefix|ip6|ip6-prefix|lookup|nil|nothing|num|str|time)\$"] do={
    :local vname  [get $item name]
    :local vvalue [get $item comment]
    :local vtype  [get $item regexp]
    /system script environment
    remove [find where name=$vname]
    :if ($vtype~"^(array|ip|ip6|num|str|time)\$") do={
        :execute ":global $vname [:to$vtype [/ip firewall layer7 get [find where name=$vname] comment]]"
    } else={
        :if ($vtype~"^(bool|id|ip-prefix|ip6-prefix|lookup|nil|nothing)\$") do={
            :if ($vtype="bool")         do={:execute ":global $vname [:tobool $vvalue]"}
            :if ($vtype="id")           do={:execute ":global $vname [:toid $[:pick $vvalue [:find $vvalue "*" -1] [:len $vvalue]]]"}
            :if (($vtype="ip-prefix") or \
                 ($vtype="ip6-prefix")) do={:execute ":global $vname [[:parse \":return $vvalue\"]]"}
            :if ($vtype="lookup")       do={:execute ":global $vname \"\$$vname\""}
            :if ($vtype="nil")          do={:execute ":global $vname"}
            :if ($vtype="nothing")      do={:execute ":global $vname [:nothing]"}
        } else={
            # vtype="code"
            :log error "Unknow variable >$vname< of type >$vtype<"
            :execute ":global $vname [/ip firewall layer7 get [find where name=$vname] comment]"
        }
    }
    :delay 10ms
}

True Level 7 rules for Firewall are not "touched" unless they regexp is exactly a single word as the names of reserved variable types.

Someone know the full list of variable types???

standard:
array
bool
id
ip
ip6
num
str
time

broken, but restored with an hack:
ip-prefix
ip6-prefix

not recoverable:
code
function (reported as array)

"undefined" types:
nil
nothing
lookup ($0 on functions)

The script is not fixed for unknow data types discovered from me on 2023:
viewtopic.php?p=1014223#p1014223
apireq (api request)
exclamation
cmd
iterator
backreference

EDIT: added fix for new discovered "op" on RouterOS, thanks to @tabraham report

2025 EDIT: added missing types discovered on 2023

Larsa · Sat Oct 02, 2021 8:39 pm

Any word regarding persistent variables in v7 yet?

rextended · Sun Oct 03, 2021 3:27 am

Sorry, but I think this is out of interest from MikroTik until the release of RouterOS 7.7.7 (stable)

Larsa · Sun Oct 03, 2021 9:38 am

Rumors say 7.9 that also will include the ability to catch well documented error numbers and messages.

rextended · Mon May 09, 2022 1:11 pm

updated for better support ID, code and function variables,
tested on both v6 and v7

eldoncito2019 · Tue May 10, 2022 7:23 pm

Your script works very well friend REX, the only thing is that how could it be removed from the LOG so that it does not register every time the variables are saved in the LAYER 7

EL DONCITO.

rextended · Tue May 10, 2022 8:22 pm

you can't without disabling all logs.

eldoncito2019 · Tue May 10, 2022 10:54 pm

ok.

EL DONCITO.

diamuxin · Mon May 16, 2022 6:26 pm

Very useful this script, thanks!

BR.

rextended · Mon May 16, 2022 6:53 pm

Very thanks!

See also on my signature for other scripts

I do not have catalogued all script I have done... one day...

Sertik · Tue May 31, 2022 10:46 am

Thanks to the author for the very beautifully written scripts for saving and restoring global variables.

There are comments:
In the save script, you need to remove the save "(function)", since it will not be possible to restore the function code.
Only variables should be saved and restored.

When restoring, restoring boolean variables via :execute does not work for me,
all boolean variables are restored as nil (have no values).

I had to insert such a check and restore via :parse

 :if ($vtype = "bool") do={
[[:parse ":global $vname [:to$vtype $[/ip firewall layer7 get [find where name=$vname] regexp]]"]]}

In this form, everything works.

You also need to remove the line from the recovery script

 :if ($vvalue = "(function)") do={:set vtype "function" }

since the functions are not saved and they do not need to be restored, they will have to be redefined again.

rextended · Tue May 31, 2022 3:22 pm

When restoring, restoring boolean variables via :execute does not work for me,
all boolean variables are restored as nil (have no values).

I had to insert such a check and restore via :parse
Code: Select all
 :if ($vtype = "bool") do={
[[:parse ":global $vname [:to$vtype $[/ip firewall layer7 get [find where name=$vname] regexp]]"]]}

Yes, a bug, but simply solved with

:if ($vtype="bool") do={:execute ":global $vname [:tobool $vvalue]"}

Thanks for the info, I do some restyling and also fix boolean.

About "(function)" (and CODE!)
it was put there for some reasons:
1) manage all the types as possible,
2) in the future maybe they can be managed directly, and it is already ready
3) is ready for add on script the search for file in (flash/)functions/vname.rsc or (flash/)codes/vname.rsc and load it at startup, if the file exist...
On this way also codes and functions are restored...
But is not intended to store inside the file the code "(eval /interface print)" because also if restored, can be restored only as string,
actually there is no way to convert directly "(eval /interface print)" to code or function.

Sertik · Tue May 31, 2022 4:37 pm

It's all clear. But, if you insert the saving of a function variable as, for example, (function), then during the subsequent restoration, the code of active functions in the environment will be corrupted and they will need to be redefined. If you do not save the functions, then you do not need to restore them, then the code of the active environment functions will be saved and they will be operational after restoring global variables of other types.

rextended · Tue May 31, 2022 5:38 pm

You forget something: is for restore variables on reboot, not at any arbitrary time...
On reboot no previous variables or functions or codes exists...

And see point 3) on previous post...

Sertik · Wed Jun 01, 2022 10:56 am

I have not forgotten, your script is also convenient for saving and restoring variables not only during reboot, but also at any time!

Sertik · Wed Jun 01, 2022 10:59 am

I made one function out of it that allows, depending on the value of the passed argument (save, restore, erase and print), to save, restore variables, as well as delete them from the Layer7 storage or print a list of storage variables to the terminal and log

Sertik · Wed Jun 01, 2022 11:02 am

I'll run around a little and put it here

Sertik · Wed Jun 01, 2022 12:29 pm

Here's what it looks like:

#
# The function of saving/restoring global environment variables v 01/06/2022
# $1 parameter can take values:
# "save" - saving global variables in /ip firewall layer7
# "recover" - restoring global variables from layer7
# "erase" - cleaning layer7 only ! from global variables
# "print" - printing to the log and terminal of the list of variables located at the time of printing in Layer7
#

:global FuncGlobal do={
:if ([:typeof $0]="lookup") do={
:local count 0
:if (($1="save") or ($1="recover") or ($1="erase") or ($1="print")) do={
:if ($1="save") do={
/system script environment
:foreach item in=[find] do={
    :local vname [get $item name]
    :local vvalue [get $item value]
    /ip firewall layer7
    remove [find where name=$vname]
    :if (([:typeof [:find [:tostr $vvalue] "(eval " -1]] = "nil") and ([:typeof [:find [:tostr $vvalue] "(evl " -1]] = "nil")) do={
        :if ([:find $vvalue "*" -1] = 0) do={:set vvalue "ID$vvalue"}
        add name=$vname regex="$vvalue"
        :set count ($count+1)
          } 
        :delay 10ms
        :execute "/ip firewall layer7 set [find where name=$vname] comment=[:typeof \$$vname]"
}
:log warning ("Function $0 has saved $count variables")
:return $count}

:if ($1="recover") do={
/ip firewall layer7
:foreach item in=[find where comment~"^(array|bool|id|ip|ip-prefix|ip6|ip6-prefix|num|str|time|code|nothing|nil)\$"] do={
    :set count ($count+1)
    :local vname [get $item name]
    :local vvalue [get $item regexp]
    :local vtype [get $item comment]
#    :if ($vvalue = "(function)") do={:set vtype "function" }
    /system script environment
    remove [find where name=$vname]
    :if ($vtype~"^(array|bool|id|ip|ip6|num|str|time)\$") do={
        :if ($vtype = "id") do={
            :set vvalue [:pick $vvalue [:find $vvalue "*" -1] [:len $vvalue]]
            :execute ":global $vname [:to$vtype $vvalue]"}
        :if ($vtype = "bool") do={
           [[:parse ":global $vname [:to$vtype $[/ip firewall layer7 get [find where name=$vname] regexp]]"]]
# :if ($vtype="bool") do={:execute ":global $vname [:tobool $vvalue]"}
        } else={
            :execute ":global $vname [:to$vtype [/ip firewall layer7 get [find where name=$vname] regexp]]"
        }
    } else={
        :if ($vtype~"^(ip-prefix|ip6-prefix|nothing|nil)\$") do={
            :if ($vtype~"^(ip-prefix|ip6-prefix)\$") do={:execute ":global $vname [[:parse \":return $vvalue\"]]"}
            :if ($vtype="nothing") do={:execute ":global $vname [:nothing]"}
            :if ($vtype="nil") do={:execute ":global $vname"}
        } else={
            :log error "Unknow variable type >$vtype<"
            :execute ":global $vname [/ip firewall layer7 get [find where name=$vname] regexp]"
        }
    }
    :delay 10ms;
}
:log warning ("Function $0 restored $count variables")
:return $count}

:if ($1="erase") do={
/ip firewall layer7
:foreach item in=[find where comment~"^(array|bool|id|ip|ip-prefix|ip6|ip6-prefix|num|str|time|code|nothing|nil)\$"] do={remove $item; :set count ($count+1)
}
:log warning ("The repository has been cleared of $count global variables")
:return $count} 

:if ($1="print") do={
/ip firewall layer7
:foreach item in=[find where comment~"^(array|bool|id|ip|ip-prefix|ip6|ip6-prefix|num|str|time|code|nothing|nil)\$"] do={
    :set count ($count+1)
    :local vname [get $item name];
    :local vvalue [get $item regexp];
    :local vtype [get $item comment];
 :put ("$vname "."$vtype "."$vvalue");
 :log info ("$vname "."[ $vtype ] "."$vvalue");
}
:log warning ("In Layer7 repository has $count global variables")
:put ("In Layer7 repository has $count global variables")
:return $count}

} else={
:if ([:typeof $1]="nothing") do={:log error ("The function $0 parameter is not set"); :return ("Error $0")}}        
 :log error ("The function $0 parameter is not valid"); :return ("Error $0");
 }
}

# usage examples:
 
# :log info [$FuncGlobal save]
# :log info [$FuncGlobal recover]
# :log info [$FuncGlobal erase]
# :log info [$FuncGlobal print]

# examples of erroneous usage:

# :log info [$FuncGlobal]
# :log info [$FuncGlobal anovertext]

The function can be called to save variables with the command [$FuncGlobal save] with the desired frequency. Recovery from layer7 is possible at any time, including at the start of the router. To clear leyer7 of global variables, use [$FuncGlobal erase]. It is planned to finalize with the introduction of restoring only part of the variables (for example, with a certain prefix of names)

diamuxin · Wed Jun 01, 2022 2:54 pm

Very interesting, thanks!

BR.

rextended · Wed Jun 01, 2022 10:53 pm

@Sertik,
Thanks, as usual, for pointing out the original author of the script...

Really kind...

eldoncito2019 · Mon Aug 08, 2022 9:17 pm

and if i want to save an ip variable in ip firewall address list and then restore it back on reboot?

rextended · Mon Aug 08, 2022 9:24 pm

Don't start with the absurd questions, like the other topic. ( viewtopic.php?t=187812 )
If you save something statically in the firewall's ip address list, it is automatically restored on reboot, without you needing to do anything.

Open your topic, if you don't understand, without going offtopic here.

diamuxin · Tue May 23, 2023 5:55 pm

Hi rextended, when I run your first script to save the global variables I get an error:

ROS Version 7.9.1
[admin@MikroTik] > sys scr run save-global-variables
failure: bad regexp: unmatched ()

ROS Version 7.10beta5
[admin@MikroTik] > sys scr run save-global-variables
failure: bad regexp: too many ()

Could you check it? thanks.

BR.

rextended · Wed May 24, 2023 10:38 am

Since this method use RegEx inside layer 7 to save variables, check if on some variables are present too many ( ) and let me know if the problem is one variable or one function/code...

add
:put $vname
after
:local vname [get $item name]
for print what variable is the problem and check the variable/function/code content...

...actually perhaps it is better that I change the point where to save the variables, otherwise if there are too many parentheses inside a textual variable it gives an error as if it were a function.

EDIT: switched regex/comment functionality, now (except for functions, codes and nested arrays) can work.

rextended · Wed May 24, 2023 11:08 am

...

You try the "new version" if do same error?

diamuxin · Wed May 24, 2023 11:57 am

You try the "new version" if do same error?

Hi Rex,

Sorry for the delay, I just tested both scripts and they are now working correctly, both in v7.9.1 and v10beta5.

I am also happy because I see that it is now possible to save/restore the content of a function. Congratulations!

BR.

rextended · Wed May 24, 2023 12:05 pm

[…] it is now possible to save/restore the content of a function. Congratulations!

EEEEEEEHHHHHHHHHHHHH???????????

diamuxin · Wed May 24, 2023 12:08 pm

[…] it is now possible to save/restore the content of a function. Congratulations!
EEEEEEEHHHHHHHHHHHHH???????????

I am going to capture screenshots, just a second

--------

Ready:

a) Backup to Layer7

b) Restore to Environment

Am I right?

rextended · Wed May 24, 2023 12:09 pm

Check twice.... after reboot....

If the function is
:global f do={ /interface print }

It's normal this:
:put [$f]
(function)

Is just a text "(function)" or another script set again the function correctly...

diamuxin · Wed May 24, 2023 12:20 pm

Check twice.... after reboot....

If the function is
:global f do={ /interface print }

It's normal this:
:put [$f]
(function)

Is just a text "(function)" or another script set again the function correctly...

look at the screenshots

..

rextended · Wed May 24, 2023 12:33 pm

Yes, but the function work?

Is easy copy & paste the "eval / evl" code, but is restored just as string...

on the code is present:
:if ($vvalue~"\\(eval|evl) ") do={:set vvalue "(function)"}

is missing one (another) ( before eval, now the script is fixed

diamuxin · Wed May 24, 2023 1:08 pm

Yes, but the function work?

Is easy copy & paste the "eval / evl" code, but is restored just as string...

on the code is present:
:if ($vvalue~"\\(eval|evl) ") do={:set vvalue "(function)"}

is missing one (another) ( before eval, now the script is fixed

You are right, some functions are not restored properly. With the last modification you have made now save the functions as "(function)" in the comments.

Everything is ok, thanks.

rextended · Wed May 24, 2023 1:17 pm

maybe in the future they will allow to save the functions, in the end it is only text that is stored with a 1 bit instead of 0....

Sertik · Mon Jun 05, 2023 3:01 pm

If the Mikrotik developers allow saving the function code in Layer7 and restoring it from there ... this can lead to both good (for example, creating copyright protection) and dangerous consequences (for example, creating virus scripts)

rextended · Mon Jun 05, 2023 3:09 pm

If the Mikrotik developers allow saving the function code in Layer7 and restoring it from there ... this can lead to both good (for example, creating copyright protection) and dangerous consequences (for example, creating virus scripts)

Completely nonsense: if a "virus" has access to the routerboard, it doesn't give a damn about "reading" functions from Layer7,
it simply puts the function it likes into memory as it already does now...

And speaking of copyrighted materials, I would never trust to insert any script of which I don't see the unencrypted source,
there is already trust in what MikroTik writes in its sources, without letting us see them...

Sertik · Mon Jun 05, 2023 3:23 pm

The future will show nonsense or not nonsense.

flydvorkin · Sat Jun 17, 2023 1:46 am

I found another lost data type:

:put [:typeof (>[])]

Return:
op

this data type behaves like code

rextended · Sat Jun 17, 2023 2:22 am

I found another lost data type:
Code: Select all
:put [:typeof (>[])]
Return:
op

this data type behaves like code

????????????????????????????????????????????????????????????

Thanks........

Amm0 · Sat Jun 17, 2023 2:44 am

I found another lost data type:

Fascinating...

:global optype (>[:global z "blah"])                   
$optype                             
:put $z
# "blah"

Even can be a function with params using :do...

:global optype (>[:do {:put "$1"}])

:put [:typeof $optype]             
# op

$optype "hmm"                      
# hmm

:put $optype
# (evl (evl /docommand=;(evl (evl /putmessage=$1))))

Or with (>{}) it's an empty array...

:global shortThanToArray (>{})
:put "$shortThanToArray $[:typeof $shortThanToArray] $[:len $shortThanToArray]" 
 # array 0

And according /console/inspect, it valid syntax like "(>"... e.g.
/console/inspect input="(" request=completion

Columns: TYPE, COMPLETION, STYLE, OFFSET, PREFERENCE, SHOW
TYPE COMPLETION STYLE OFFSET PREFERENCE SHOW
completion none 2 80 no
completion ( syntax-meta 2 75 no
completion $ syntax-meta 2 75 no
completion [ syntax-meta 2 75 no
completion { syntax-meta 2 75 no
completion " syntax-meta 2 75 no
completion ! syntax-meta 2 75 no
completion - syntax-meta 2 75 no
completion ~ syntax-meta 2 75 no
completion > syntax-meta 2 75 no
completion syntax-meta 2 75 no
completion <value> none 2 -1 no

flydvorkin · Fri Jun 23, 2023 1:39 am

Also i found that the 2 this variants to print "code":
1 - directly printing by "put":

:put $varWithCodeDataType
or
:put [ :parse "local var" ]
or
:put ( $function->1 )

2 - print global function parsed code value from /system script environment

:put [ /system script environment get [ find name="globalFunction" ] value ]

both this variants printing text of parsed code is not completely.
For comparison use 3-rd method:

/environment print

this printing ALL global variables, but if find the needed text of the code in the printout - will see difference with 1st and 2nd methods:

For example, let source:
local var
then parse them to code-data-type and printing with all methods.

1st and 2nd methods are returned:

(evl /localname=$var)

but 3rd method (store code to global variable and print) are returned:

{(evl [/local{name=$var}])}

It seems 3rd method are more preferred to analyse parsed code.

Unfortunately, the "/environment print" command are not have any arguments. But it possible redirect console output with :execute to file (or directly in variable with ros 7.8+ by "as-string" argument), and extract needed part.
I wrote a usefull function that does all this work automatically

rextended · Mon Jun 26, 2023 12:59 am

op..."only pointer" ?

Probably is only for point data inside complex array without everytime specify full path...

Experimental code

:global arraytest {"a";"b";"c";{"q";"z"}}
:put $arraytest

:global pointer (>($arraytest->1))
:put $pointer
:put [$pointer]

:set ($arraytest->1) "x" 
:put $arraytest
:put $pointer
:put [$pointer]

:set [$pointer] "W"
:set pointer "J"
:put $arraytest
:put $pointer
:put [$pointer]

:global pointer (>($arraytest->3))
:put $pointer
:put [$pointer]
:put ([$pointer]->1)

:set ([$pointer]->1) "RR"
:put $pointer
:put [$pointer]
:put $arraytest

BrianHiggins · Mon Jul 31, 2023 6:55 pm

I found another lost data type:
Code: Select all
:put [:typeof (>[])]
Return:
op

this data type behaves like code

Anyway to use this data type to write a script that uses mac-telnet to log into another device by MAC address and execute a command?

Something I've been struggling with since there seems to be no way to pass credentials into the mac-telnet command once you launch it. (and since someone will suggest it, no I cannot SSH into the device, this need exists in situations where there is no IP address defined on a device, it is ONLY accessible via mac-telnet)

tabraham · Tue Aug 29, 2023 11:06 am

Hi Rex. The code seams to break in ROS 7.11 stable. for the store script

:if ($vvalue~"^($code$|;\\(eva\?l )") do={:set $vvalue "(code)"};

The break is mainly in the regex part. could you please double check? Thanks

rextended · Tue Aug 29, 2023 12:18 pm

Hi Rex. The code seams to break in ROS 7.11 stable. for the store script

:if ($vvalue~"^($code$|;\\(eva\?l )") do={:set $vvalue "(code)"};
The break is mainly in the regex part. could you please double check? Thanks

This is not the original line, for example, someone added $ in front of vvalue on :set, and also add ";" at the end, etc....

I took the opportunity to add the control for the export of "op" (\? in front of ; on the RegEx).

tabraham · Wed Aug 30, 2023 10:07 am

I see. Then could you please confirm that the code above is the final version written by you works for arrays too?

rextended · Wed Aug 30, 2023 10:37 am

Only simple arrays are exported, do not work for array of array and more complex data structure.

fischerdouglas · Fri Jan 03, 2025 2:26 pm

@rextended this is a great script!
It will probably help-me to solve the problem I'm trying to solve.
Thanks!

But... (There is always a but, ha-ha.)
To be sincere, I do not like very much the idea of subverting the use of a field of a specific resource like was done with layer7 firewall rules.
I Understand that it was what the best that could be done whit the resources that were available at that time...
But I feel that Layer7 Firewall Rules could be removed sometime, and then this script would be orphaned.

Last interaction on this thread was Wed Aug 30, 2023 4:37 am
But 4 months later the Json serialize and deserialized resources were added:

What's new in 7.13 (2023-Dec-14 09:24):
*) console - added ":serialize" and ":deserialize" commands for converting values to/from JSON;

with the first documentation of for those resources being published in Scripting - RouterOS - MikroTik Documentation - v. 54 dez. 07, 2023 13:21

I tend to believe that Json support will not be removed... Several "new" features have resources based on it like REST-API, MQTT/IOT/GPIO.

Considering that, I was wondering to use that Json structure to persist the environment variables.

The second part of my ideia was to use /system/script as the persistence method for a json file.
Why /system/script?
In my opinion saving it on a script is the easiest way to persist a text-based file on an editable way and contained in the backup.rsc and not on another extra file.

I will start to think of a way to do that.

Please alert me if I'm trying to reinvent the wheel...
Any help is welcome.

rextended · Sat Jan 04, 2025 12:23 am

I also thought about using json, but...
Put in a table:
on the right the variable types that RouterOS supports (array,bool,id,ip,ip6,num,str,time,ip-prefix,ip6-prefix,nil,nothing),
(ignoring unexportable types code, function, lookup, op, apireq, exclamation, cmd, iterator, backreference)

on the left the variable types that json supports (object, array, number, string, boolean (true;false), null)

and in the center (see other topics in the forum) the data types that serialize and deserialize currently cannot understand or convert on both way...
viewtopic.php?p=1099484#p1099484

So, in conclusion, layer7 is certainly not going away anytime soon,
but hopefully by the time it is removed, it will be possible to use serialize and deserialize using the text field of a script...

A possible solution is to convert everything into a base64 array that contains variable name=type/value triplets, which can be saved as strings in json.
To save more complex arrays, they should be disassembled into easy-to-save sub-arrays, and then reassembled everything together correctly.
I have done already something on 2023:
viewtopic.php?p=1022321#p1022321

fischerdouglas · Sat Jan 04, 2025 5:45 pm

I also thought about using json, but...
...
but hopefully by the time it is removed, it will be possible to use serialize and deserialize using the text field of a script...

Yep... I guess you are right.
I don't like the idea of using layer 7 as a DB, but it is what is available for now.

Maybe in the future MikroTik Guys Accept better the idea of dealing with persisten variables.
Just adding on what already exists on /system/script/environment/ the possibilities:

"D"ynamic entries (what we already know)
Persistent entries

To me, sounds "not-that-hard" for MT-Guys inserting that on RouterOS/CLI/Winbox.
Follows the same concept of all the RouterOS.

palii · Sun Jan 12, 2025 11:22 am

Great job with the variable type handling!

Since I don't use many persistent variables, I was never a fan of using scheduled scripts for them. A couple years ago, I wrote a light and easy function that reads/writes to layer7 protocols directly. It only uses strings by default, but can be adapted.

:global persist do={

    :local varName $1
    :local varValue $2
    :local varID [/ip firewall layer7-protocol find name="$varName"]

    :if ([:typeof $varValue] = "nothing") do={
        :if ($varID != "") do={
            :set $varValue [/ip firewall layer7-protocol get $varID value-name=regexp]
        }
    } else={
        :if ($varID = "") do={
            /ip firewall layer7-protocol add name="$varName" regexp="$varValue"
        } else={
            /ip firewall layer7-protocol set $varID regexp="$varValue"
        }
    }

    return $varValue

}

Usage examples: (name = variable name, value = variable value)

$persist name = Read persistent variable
$persist name value = Write persistent variable
:global name [$persist name] = Read persistent variable and write it to a global variable
:global name [$persist name value] = Write data into a global variable and into a persistent variable at the same time

Notes:

The function uses layer7-protocols to store persistent variables.
Read/write directly to layer7-protocols without scheduled scripts.
All values will be strings.
The function always returns a value.

rextended · Tue Jan 28, 2025 1:15 pm

The function still have problems after 6 years...

Refitted to avoid errors.
Also added the ability to delete the "variable" if delete=yes is specified as parameter.

reworked code

:global l7var do={
    /ip firewall layer7-protocol
    :local varName       [:tostr  $1]
    :local varNewValue   [:tostr  $2]
    :local valuePresent ([:typeof $2] != "nothing")

    :if ($varName = "") do={ :return [:nothing] }

    :local varID [find where name=$varName]

    :if ($delete = "yes") do={ remove $varID ; :return [:nothing] }

    :if ($valuePresent) do={ 
        :if ([:len $varID] = 0) do={
            add name=$varName regexp=$varNewValue
            :set varID [find where name=$varName]
        } else={
            set $varID regexp=$varNewValue
        }
    }

    :if ([:len $varID] != 0) do={ :return [get $varID regexp] }

    :return [:nothing]
}

Amm0 · Tue Jan 28, 2025 5:15 pm

I also thought about using json, but...

[...]

So, in conclusion, layer7 is certainly not going away anytime soon,
but hopefully by the time it is removed, it will be possible to use serialize and deserialize using the text field of a script...

A possible solution is to convert everything into a base64 array
[...]

FWIW, I adapted a test script for serialize/deserialize, to test using JSON to base64 to L7 FW regex storage (and back). I wasn't trying to replace anything here, more just trying the above approach to see if it even works, since I had a complex array already. And, critically, this will only work in 7.16+ (or maybe even needs 7.17).

You can see below how JSON serialization largely preserves types. The only oddity is "0.0" become 0.000000 in JSON, but plus side is "1.1" becomes 1.100000 in JSON (a float). And there is an option in :serialize to prevent that if desired.

{
    # an array with most types, including a JSON "float" 
    :local arr {
        "num"=0;
        "str"="mystr";
        "emptystr"="";
        "ip4"=1.1;
        "ip6"=1::1;
        "prefix4"=1.0.0.1/31;
        "prefix6"=1::1/69;
        "float"="1.1";
        "time"=1s;
        "now"=[:timestamp];
        "null"=[:nothing];
        "list"=(1,2,"three");
        "listlist"={(1,2,3);("a","b","c")}
        "dict"={"a"=1;"b"="z"};
        "dictlist"={{"m"="M"};{"z"="Z"}};
        "dictdict"={"b"={"one"=1;"two"=2};"w"={"1"="one";"2"="two"}};    
        "optype"=(>[:put "echo"]);
        "bignum"=[:tonsec [:timestamp]];
        "bigneg"=(0-[:tonsec [:timestamp]]);
    }
    # helpers for test
    :local prettyprint do={:put [:serialize to=json options=json.pretty $1]}
    :local addtypes do={:local rv $1; :foreach n,a in=$1 do={ :set ($rv->"$n-type") [:typeof $a] }; :return $rv }

    :put "\r\narray BEFORE serialization"
    $prettyprint [$addtypes $arr]

    :put "\r\nconvert to JSON"
    :local json [:serialize to=json $arr]

    :put "\r\nconvert to base64 for storage as RouterOS string"
    :local base64out [:convert to=base64 $json]
    $prettyprint $base64out

    :put "\r\nsave to base64 JSON as unused L7 FW rule"
    :local storename "example-base64-json-to-save-vars"
    :local storeid [/ip/firewall/layer7-protocol/find name=$storename] 
    :if ([:len $storeid]=1) do={
    /ip/firewall/layer7-protocol set $storeid regexp=$base64out 
    } else={
    /ip/firewall/layer7-protocol add name=$storename regexp=$base64out  
    }

    :put "\r\nPRETEND you reboot and come back...so wait 3 seconds"
    :delay 3s

    :put "\r\nsave to base64 JSON as unused L7 FW rule"
    :local base64in [/ip/firewall/layer7-protocol get [find name=$storename] regexp]

    :put "\r\nif base64, restore it to JSON"
    :local newjson [:convert from=base64 $base64in]

    :put "\r\nfinally get the array back, with types perserved by :deserialize JSON" 
    :local arr2 [:deserialize from=json $newjson]

    :put "\r\narray AFTER deserialization"
    $prettyprint [$addtypes $arr2]

    :put "\r\nBONUS: simulate using RESTORED array variables using 'activate-in-context' and 'op' types"
    ((>[:put "now: $now  listlen: $[:len $list] bignum: $bignum  prefix6: $prefix6"]) <%% $arr2)
    :put "... even though its array, you can use them as normal variables without the -> if you use the <%% to unwrap them"
}

array BEFORE serialization
{
"bigneg": -1738076593889845466,
"bigneg-type": "num",
"bignum": 1738076593889955906,
"bignum-type": "num",
"dict": {
"a": 1,
"b": "z"
},
"dict-type": "array",
"dictdict": {
"b": {
"one": 1,
"two": 2
},
"w": {
"1": "one",
"2": "two"
}
},
"dictdict-type": "array",
"dictlist": [
{
"m": "M"
},
{
"z": "Z"
}
],
"dictlist-type": "array",
"emptystr": "",
"emptystr-type": "str",
"float": 1.100000,
"float-type": "str",
"ip4": "1.0.0.1",
"ip4-type": "ip",
"ip6": "1::1",
"ip6-type": "ip6",
"list": [
1,
2,
"three"
],
"list-type": "array",
"listlist": [
[
1,
2,
3
],
[
"a",
"b",
"c"
]
],
"listlist-type": "array",
"now": "2025-01-28 15:03:13",
"now-type": "time",
"null": null,
"null-type": "nil",
"num": 0,
"num-type": "num",
"optype": "(op)",
"optype-type": "op",
"prefix4": "1.0.0.1/31",
"prefix4-type": "ip-prefix",
"prefix6": "1::/69",
"prefix6-type": "ip6-prefix",
"str": "mystr",
"str-type": "str",
"time": "1970-01-01 00:00:01",
"time-type": "time"
}

convert to JSON

convert to base64 for storage as RouterOS string
"eyJiaWduZWciOi0xNzM4MDc2NTkzODg5ODQ1NDY2LCJiaWdudW0iOjE3MzgwNzY1OTM4ODk5NTU5MDYs
ImRpY3QiOnsiYSI6MSwiYiI6InoifSwiZGljdGRpY3QiOnsiYiI6eyJvbmUiOjEsInR3byI6Mn0sInciO
nsiMSI6Im9uZSIsIjIiOiJ0d28ifX0sImRpY3RsaXN0IjpbeyJtIjoiTSJ9LHsieiI6IloifV0sImVtcH
R5c3RyIjoiIiwiZmxvYXQiOjEuMTAwMDAwLCJpcDQiOiIxLjAuMC4xIiwiaXA2IjoiMTo6MSIsImxpc3Q
iOlsxLDIsInRocmVlIl0sImxpc3RsaXN0IjpbWzEsMiwzXSxbImEiLCJiIiwiYyJdXSwibm93IjoiMjAy
NS0wMS0yOCAxNTowMzoxMyIsIm51bGwiOm51bGwsIm51bSI6MCwib3B0eXBlIjoiKG9wKSIsInByZWZpe
DQiOiIxLjAuMC4xLzMxIiwicHJlZml4NiI6IjE6Oi82OSIsInN0ciI6Im15c3RyIiwidGltZSI6IjE5Nz
AtMDEtMDEgMDA6MDA6MDEifQ=="

save to base64 JSON as unused L7 FW rule

PRETEND you reboot and come back...so wait 5 seconds

save to base64 JSON as unused L7 FW rule

if base64, restore it to JSON

finally get the array back, with types perserved by :deserialize JSON

array AFTER deserialization
{
"bigneg": -1738076593889845466,
"bigneg-type": "num",
"bignum": 1738076593889955906,
"bignum-type": "num",
"dict": {
"a": 1,
"b": "z"
},
"dict-type": "array",
"dictdict": {
"b": {
"one": 1,
"two": 2
},
"w": {
"1": "one",
"2": "two"
}
},
"dictdict-type": "array",
"dictlist": [
{
"m": "M"
},
{
"z": "Z"
}
],
"dictlist-type": "array",
"emptystr": "",
"emptystr-type": "str",
"float": 1.100000,
"float-type": "str",
"ip4": "1.0.0.1",
"ip4-type": "ip",
"ip6": "1::1",
"ip6-type": "ip6",
"list": [
1,
2,
"three"
],
"list-type": "array",
"listlist": [
[
1,
2,
3
],
[
"a",
"b",
"c"
]
],
"listlist-type": "array",
"now": "2025-01-28 15:03:13",
"now-type": "time",
"null": null,
"null-type": "nil",
"num": 0,
"num-type": "num",
"optype": "(op)",
"optype-type": "str",
"prefix4": "1.0.0.1/31",
"prefix4-type": "ip-prefix",
"prefix6": "1::/69",
"prefix6-type": "ip6-prefix",
"str": "mystr",
"str-type": "str",
"time": "1970-01-01 00:00:01",
"time-type": "time"
}

BONUS: simulate using array variables in function using the 'activate-in-context'
with 'op' type
now: 2873w5d15:03:13 listlen: 3 bignum: 1738076593889955906 prefix6: 1::/69
... even though its array, you can use them as normal variables without the -> if
you use the <%% to unwrap them

Also, the combo of [:deserialize [:serialize]] works on any type - so input does NOT have to be array – you use that to do something like get an ip-prefix from a string, without an array:

(>[:put [:typeof [:deserialize from=json [:serialize to=json $1]]]]) <%% "1.1.1.1/0"
# ip-prefix

rextended · Tue Jan 28, 2025 7:49 pm

Some tests on vars that do not have sub-arrays...

test variable types code

{
:global globalVars [:toarray ""]
:put $globalVars

:set ($globalVars->"testArray")          [:toarray ""]
:set ($globalVars->"testArray"->"value") [:toarray "a,b"]
:set ($globalVars->"testArray"->"type")  [:typeof ($globalVars->"testArray"->"value")]

:set ($globalVars->"testBool")           [:toarray ""]
:set ($globalVars->"testBool"->"value")  true
:set ($globalVars->"testBool"->"type")   [:typeof ($globalVars->"testBool"->"value")]

:set ($globalVars->"testCode")           [:toarray ""]
:set ($globalVars->"testCode"->"value")  [:parse ";"]
:set ($globalVars->"testCode"->"type")   [:typeof ($globalVars->"testCode"->"value")]

:set ($globalVars->"testID")             [:toarray ""]
:set ($globalVars->"testID"->"value")    [:toid *BEEF6]
:set ($globalVars->"testID"->"type")     [:typeof ($globalVars->"testID"->"value")]

:set ($globalVars->"testIP4")            [:toarray ""]
:set ($globalVars->"testIP4"->"value")   127.0.0.1
:set ($globalVars->"testIP4"->"type")    [:typeof ($globalVars->"testIP4"->"value")]

:set ($globalVars->"testIP4px")          [:toarray ""]
:set ($globalVars->"testIP4px"->"value") 192.168.0.0/24
:set ($globalVars->"testIP4px"->"type")  [:typeof ($globalVars->"testIP4px"->"value")]

:set ($globalVars->"testIP6")            [:toarray ""]
:set ($globalVars->"testIP6"->"value")   127:0:0::1
:set ($globalVars->"testIP6"->"type")    [:typeof ($globalVars->"testIP6"->"value")]

:set ($globalVars->"testIP6px")          [:toarray ""]
:set ($globalVars->"testIP6px"->"value") 192:168:0::0/64
:set ($globalVars->"testIP6px"->"type")  [:typeof ($globalVars->"testIP6px"->"value")]

:global lookupGen do={:return $0}
:set ($globalVars->"testLook")           [:toarray ""]
:set ($globalVars->"testLook"->"value")  [$lookupGen]
:set ($globalVars->"testLook"->"type")   [:typeof ($globalVars->"testLook"->"value")]

:set ($globalVars->"testNil")            [:toarray ""]
:set ($globalVars->"testNil"->"value")   []
:set ($globalVars->"testNil"->"type")    [:typeof ($globalVars->"testNil"->"value")]

:set ($globalVars->"testNth")            [:toarray ""]
:set ($globalVars->"testNth"->"value")   ; # undefined on purpose for nothing
:set ($globalVars->"testNth"->"type")    [:typeof ($globalVars->"testNth"->"value")]

:set ($globalVars->"testNum")            [:toarray ""]
:set ($globalVars->"testNum"->"value")   -256
:set ($globalVars->"testNum"->"type")    [:typeof ($globalVars->"testNum"->"value")]

:set ($globalVars->"testOP")             [:toarray ""]
:set ($globalVars->"testOP"->"value")    (>[])
:set ($globalVars->"testOP"->"type")     [:typeof ($globalVars->"testOP"->"value")]

:set ($globalVars->"testStr")            [:toarray ""]
:set ($globalVars->"testStr"->"value")   [:convert to=base64 "1.1"]
:set ($globalVars->"testStr"->"base64")  true
:set ($globalVars->"testStr"->"type")    [:typeof ($globalVars->"testStr"->"value")]

:set ($globalVars->"testTime")           [:toarray ""]
:set ($globalVars->"testTime"->"value")  1w1d1h1m1s
:set ($globalVars->"testTime"->"type")   [:typeof ($globalVars->"testTime"->"value")]

:foreach name,content in=$globalVars do={
    :if (($content->"type") = "array") do={
        :put ">$name< = >$($content->"type")< >$[:tostr ($content->"value")]< "
    } else={
        :if (($content->"base64") = true) do={
            :put ">$name< = >$($content->"type")< >$[:convert from=base64 ($content->"value")]< "
        } else={
            :put ">$name< = >$($content->"type")< >$($content->"value")< "
        }
    }
}

:local json [:serialize to=json options=json.pretty $globalVars]

# debug
# :put $json

:put "\r\nAfter array -> json -> array conversion:\r\n"

:global newArr [:deserialize from=json options=json.no-string-conversion $json]

:foreach name,content in=$newArr do={
    :if (($content->"type") = "array") do={
        :put ">$name< = >$($content->"type")< >$[:tostr ($content->"value")]< IMPORTED TYPE: >$[:typeof ($content->"value")]<"
    } else={
        :if (($content->"base64") = true) do={
            :put ">$name< = >$($content->"type")< >$[:convert from=base64 ($content->"value")]< IMPORTED TYPE: >$[:typeof ($content->"value")]<"
        } else={
            :put ">$name< = >$($content->"type")< >$($content->"value")< IMPORTED TYPE: >$[:typeof ($content->"value")]<"
        }
    }
}

:put "\r\nNOTES\r\ncode, function, lookup and op are not exportable. Function is also reported as array."
:put "Strings are exported and imported on base64 for prevent 7.16.2 and less errors with implicit conversions to json,"
:put "on importing strings already exist options=json.no-string-conversion that prevents implicit conversions from json"
:put "MISSING: apireq, backreference, cmd, exclamation, iterator\r\n"
}

fischerdouglas · Wed Jan 29, 2025 1:53 pm

@rextended
I see some good moves from MikroTik in this 7.18 beta and also on some previous version.

":serialize" and ":deserialize"
":jobname"
JSON support on scripting
Exposing on GUI "Script Environment"
CEF format logging
Back To Home
Cloud Flie-Share

Most of then, to me, seams to be part of a bigger thing.
Could it be related to the MikroTik Devices Controller ?

Would this be a suitable time to reinforce the request about a persistent variable manager?

Amm0 · Wed Jan 29, 2025 5:30 pm

We've seen over the course of the V7 various changes to permissions and policy. Like, well, ":global" is truely global anymore, etc.... So exactly how to expose "persistent variables" in existing policy model is where some thorny issue may come up, since the underlying user/policy/group is pretty limited today.

Given one of the more common use cases for "persistent variables" be to store some username/password for some /tool/fetch call... You'd likely also want to mark "persistent variables" as "sensitive" somehow, and have OS treat accordingly, including redacting them in logs. Supporting saving/restoring/"adding" to /sys/env/print is something they avoided for a decade, possibly since it also mean you can run the low-level "(code)" type, which is also stored there, without going through the script compiler first & compiler enforces some rules on what can be generated.

My guess is that they need to modernize the user policy/AAA stuff before we'll see this – since today it's bandaid-on-bandaid today there.

Amm0 · Wed Jan 29, 2025 5:35 pm

Could it be related to the MikroTik Devices Controller ?

My guess is that webfig717+/winbox4 being "done" is the gating item there, I'd imagine they'll re-use those UI components, but those aren't done given the Winbox4 beta thread

.