Persistent Environment Variables

Sertik · June 5, 2023, 12:23pm

The future will show nonsense or not nonsense.

flydvorkin · June 16, 2023, 10:46pm

I found another lost data type:

:put [:typeof (>[])]

Return:
op

this data type behaves like code

rextended · June 16, 2023, 11:22pm

???

Thanks…

Amm0 · June 16, 2023, 11:44pm

Fascinating…

:global optype (>[:global z "blah"])                   
$optype                             
:put $z
# "blah"

Even can be a function with params using :do…

:global optype (>[:do {:put "$1"}])

:put [:typeof $optype]             
# op

$optype "hmm"                      
# hmm

:put $optype
# (evl (evl /docommand=;(evl (evl /putmessage=$1))))

Or with (>{}) it’s an empty array…

:global shortThanToArray (>{})
:put "$shortThanToArray $[:typeof $shortThanToArray] $[:len $shortThanToArray]" 
 # array 0

And according /console/inspect, it valid syntax like “(>”… e.g.
/console/inspect input=“(” request=completion

Columns: TYPE, COMPLETION, STYLE, OFFSET, PREFERENCE, SHOW
TYPE COMPLETION STYLE OFFSET PREFERENCE SHOW
completion none 2 80 no
completion ( syntax-meta 2 75 no
completion $ syntax-meta 2 75 no
completion [ syntax-meta 2 75 no
completion { syntax-meta 2 75 no
completion " syntax-meta 2 75 no
completion ! syntax-meta 2 75 no
completion - syntax-meta 2 75 no
completion ~ syntax-meta 2 75 no
completion > syntax-meta 2 75 no
completion syntax-meta 2 75 no
completion none 2 -1 no

flydvorkin · June 22, 2023, 10:39pm

Also i found that the 2 this variants to print “code”:
1 - directly printing by “put”:

:put $varWithCodeDataType
or
:put [ :parse "local var" ]
or
:put ( $function->1 )

2 - print global function parsed code value from /system script environment

:put [ /system script environment get [ find name="globalFunction" ] value ]

both this variants printing text of parsed code is not completely.
For comparison use 3-rd method:

/environment print

this printing ALL global variables, but if find the needed text of the code in the printout - will see difference with 1st and 2nd methods:

For example, let source:
local var
then parse them to code-data-type and printing with all methods.

1st and 2nd methods are returned:

(evl /localname=$var)

but 3rd method (store code to global variable and print) are returned:

{(evl [/local{name=$var}])}

It seems 3rd method are more preferred to analyse parsed code.

Unfortunately, the “/environment print” command are not have any arguments. But it possible redirect console output with :execute to file (or directly in variable with ros 7.8+ by “as-string” argument), and extract needed part.
I wrote a usefull function that does all this work automatically

rextended · June 25, 2023, 9:59pm

op…“only pointer” ?

Probably is only for point data inside complex array without everytime specify full path…
:global arraytest {“a”;“b”;“c”;{“q”;“z”}}
:put $arraytest

:global pointer (>($arraytest->1))
:put $pointer
:put [$pointer]

:set ($arraytest->1) “x”
:put $arraytest
:put $pointer
:put [$pointer]

:set [$pointer] “W”
:set pointer “J”
:put $arraytest
:put $pointer
:put [$pointer]

:global pointer (>($arraytest->3))
:put $pointer
:put [$pointer]
:put ([$pointer]->1)

:set ([$pointer]->1) “RR”
:put $pointer
:put [$pointer]
:put $arraytest

BrianHiggins · July 31, 2023, 3:55pm

Anyway to use this data type to write a script that uses mac-telnet to log into another device by MAC address and execute a command? Something I’ve been struggling with since there seems to be no way to pass credentials into the mac-telnet command once you launch it. (and since someone will suggest it, no I cannot SSH into the device, this need exists in situations where there is no IP address defined on a device, it is ONLY accessible via mac-telnet)

tabraham · August 29, 2023, 8:06am

Hi Rex. The code seams to break in ROS 7.11 stable. for the store script

:if ($vvalue~“^((code)|;\(eva?l )”) do={:set $vvalue “(code)”};

The break is mainly in the regex part. could you please double check? Thanks

rextended · August 29, 2023, 9:18am

This is not the original line, for example, someone added $ in front of vvalue on :set, and also add “;” at the end, etc…

I took the opportunity to add the control for the export of “op” (? in front of ; on the RegEx).

tabraham · August 30, 2023, 7:07am

I see. Then could you please confirm that the code above is the final version written by you works for arrays too?

rextended · August 30, 2023, 7:37am

fischerdouglas · January 3, 2025, 12:26pm

@rextended this is a great script!
It will probably help-me to solve the problem I’m trying to solve.
Thanks!

But… (There is always a but, ha-ha.)
To be sincere, I do not like very much the idea of subverting the use of a field of a specific resource like was done with layer7 firewall rules.
I Understand that it was what the best that could be done whit the resources that were available at that time…
But I feel that Layer7 Firewall Rules could be removed sometime, and then this script would be orphaned.

Last interaction on this thread was Wed Aug 30, 2023 4:37 am
But 4 months later the Json serialize and deserialized resources were added:

with the first documentation of for those resources being published in Scripting - RouterOS - MikroTik Documentation - v. 54 dez. 07, 2023 13:21

I tend to believe that Json support will not be removed… Several “new” features have resources based on it like REST-API, MQTT/IOT/GPIO.

Considering that, I was wondering to use that Json structure to persist the environment variables.

The second part of my ideia was to use /system/script as the persistence method for a json file.
Why /system/script?
In my opinion saving it on a script is the easiest way to persist a text-based file on an editable way and contained in the backup.rsc and not on another extra file.

I will start to think of a way to do that.

Please alert me if I’m trying to reinvent the wheel…
Any help is welcome.

rextended · January 3, 2025, 10:23pm

I also thought about using json, but…
Put in a table:
on the right the variable types that RouterOS supports (array,bool,id,ip,ip6,num,str,time,ip-prefix,ip6-prefix,nil,nothing),
(ignoring unexportable types code, function, lookup, op, apireq, exclamation, cmd, iterator, backreference)

on the left the variable types that json supports (object, array, number, string, boolean (true;false), null)

and in the center (see other topics in the forum) the data types that serialize and deserialize currently cannot understand or convert on both way…
http://forum.mikrotik.com/t/bug-report-incorrect-conversion-of-numeric-strings-to-json-in-routeros/178959/2

So, in conclusion, layer7 is certainly not going away anytime soon,
but hopefully by the time it is removed, it will be possible to use serialize and deserialize using the text field of a script…

A possible solution is to convert everything into a base64 array that contains variable name=type/value triplets, which can be saved as strings in json.
To save more complex arrays, they should be disassembled into easy-to-save sub-arrays, and then reassembled everything together correctly.
I have done already something on 2023:
http://forum.mikrotik.com/t/saving-array-to-file/169176/6

fischerdouglas · January 4, 2025, 3:45pm

Yep… I guess you are right.
I don’t like the idea of using layer 7 as a DB, but it is what is available for now.

Maybe in the future MikroTik Guys Accept better the idea of dealing with persisten variables.
Just adding on what already exists on /system/script/environment/ the possibilities:

"D"ynamic entries (what we already know)
Persistent entries

To me, sounds “not-that-hard” for MT-Guys inserting that on RouterOS/CLI/Winbox.
Follows the same concept of all the RouterOS.

palii · January 12, 2025, 9:22am

Great job with the variable type handling!

Since I don’t use many persistent variables, I was never a fan of using scheduled scripts for them. A couple years ago, I wrote a light and easy function that reads/writes to layer7 protocols directly. It only uses strings by default, but can be adapted.

:global persist do={

    :local varName $1
    :local varValue $2
    :local varID [/ip firewall layer7-protocol find name="$varName"]

    :if ([:typeof $varValue] = "nothing") do={
        :if ($varID != "") do={
            :set $varValue [/ip firewall layer7-protocol get $varID value-name=regexp]
        }
    } else={
        :if ($varID = "") do={
            /ip firewall layer7-protocol add name="$varName" regexp="$varValue"
        } else={
            /ip firewall layer7-protocol set $varID regexp="$varValue"
        }
    }

    return $varValue

}

Usage examples: (name = variable name, value = variable value)

$persist name = Read persistent variable
$persist name value = Write persistent variable
:global name [$persist name] = Read persistent variable and write it to a global variable
:global name [$persist name value] = Write data into a global variable and into a persistent variable at the same time

Notes:

The function uses layer7-protocols to store persistent variables.
Read/write directly to layer7-protocols without scheduled scripts.
All values will be strings.
The function always returns a value.

rextended · January 28, 2025, 11:15am

The function still have problems after 6 years…

Refitted to avoid errors.
Also added the ability to delete the “variable” if delete=yes is specified as parameter.
:global l7var do={
/ip firewall layer7-protocol
:local varName [:tostr $1]
:local varNewValue [:tostr $2]
:local valuePresent ([:typeof $2] != “nothing”)

:if ($varName = "") do={ :return [:nothing] }

:local varID [find where name=$varName]

:if ($delete = "yes") do={ remove $varID ; :return [:nothing] }

:if ($valuePresent) do={ 
    :if ([:len $varID] = 0) do={
        add name=$varName regexp=$varNewValue
        :set varID [find where name=$varName]
    } else={
        set $varID regexp=$varNewValue
    }
}

:if ([:len $varID] != 0) do={ :return [get $varID regexp] }

:return [:nothing]

}

Amm0 · January 28, 2025, 3:15pm

FWIW, I adapted a test script for serialize/deserialize, to test using JSON to base64 to L7 FW regex storage (and back). I wasn’t trying to replace anything here, more just trying the above approach to see if it even works, since I had a complex array already. And, critically, this will only work in 7.16+ (or maybe even needs 7.17).

You can see below how JSON serialization largely preserves types. The only oddity is “0.0” become 0.000000 in JSON, but plus side is “1.1” becomes 1.100000 in JSON (a float). And there is an option in :serialize to prevent that if desired.

{
    # an array with most types, including a JSON "float" 
    :local arr {
        "num"=0;
        "str"="mystr";
        "emptystr"="";
        "ip4"=1.1;
        "ip6"=1::1;
        "prefix4"=1.0.0.1/31;
        "prefix6"=1::1/69;
        "float"="1.1";
        "time"=1s;
        "now"=[:timestamp];
        "null"=[:nothing];
        "list"=(1,2,"three");
        "listlist"={(1,2,3);("a","b","c")}
        "dict"={"a"=1;"b"="z"};
        "dictlist"={{"m"="M"};{"z"="Z"}};
        "dictdict"={"b"={"one"=1;"two"=2};"w"={"1"="one";"2"="two"}};    
        "optype"=(>[:put "echo"]);
        "bignum"=[:tonsec [:timestamp]];
        "bigneg"=(0-[:tonsec [:timestamp]]);
    }
    # helpers for test
    :local prettyprint do={:put [:serialize to=json options=json.pretty $1]}
    :local addtypes do={:local rv $1; :foreach n,a in=$1 do={ :set ($rv->"$n-type") [:typeof $a] }; :return $rv }

    :put "\r\narray BEFORE serialization"
    $prettyprint [$addtypes $arr]

    :put "\r\nconvert to JSON"
    :local json [:serialize to=json $arr]

    :put "\r\nconvert to base64 for storage as RouterOS string"
    :local base64out [:convert to=base64 $json]
    $prettyprint $base64out

    :put "\r\nsave to base64 JSON as unused L7 FW rule"
    :local storename "example-base64-json-to-save-vars"
    :local storeid [/ip/firewall/layer7-protocol/find name=$storename] 
    :if ([:len $storeid]=1) do={
    /ip/firewall/layer7-protocol set $storeid regexp=$base64out 
    } else={
    /ip/firewall/layer7-protocol add name=$storename regexp=$base64out  
    }

    :put "\r\nPRETEND you reboot and come back...so wait 3 seconds"
    :delay 3s

    :put "\r\nsave to base64 JSON as unused L7 FW rule"
    :local base64in [/ip/firewall/layer7-protocol get [find name=$storename] regexp]

    :put "\r\nif base64, restore it to JSON"
    :local newjson [:convert from=base64 $base64in]

    :put "\r\nfinally get the array back, with types perserved by :deserialize JSON" 
    :local arr2 [:deserialize from=json $newjson]

    :put "\r\narray AFTER deserialization"
    $prettyprint [$addtypes $arr2]

    :put "\r\nBONUS: simulate using RESTORED array variables using 'activate-in-context' and 'op' types"
    ((>[:put "now: $now  listlen: $[:len $list] bignum: $bignum  prefix6: $prefix6"]) <%% $arr2)
    :put "... even though its array, you can use them as normal variables without the -> if you use the <%% to unwrap them"
}

array BEFORE serialization
{
“bigneg”: -1738076593889845466,
“bigneg-type”: “num”,
“bignum”: 1738076593889955906,
“bignum-type”: “num”,
“dict”: {
“a”: 1,
“b”: “z”
},
“dict-type”: “array”,
“dictdict”: {
“b”: {
“one”: 1,
“two”: 2
},
“w”: {
“1”: “one”,
“2”: “two”
}
},
“dictdict-type”: “array”,
“dictlist”: [
{
“m”: “M”
},
{
“z”: “Z”
}
],
“dictlist-type”: “array”,
“emptystr”: “”,
“emptystr-type”: “str”,
“float”: 1.100000,
“float-type”: “str”,
“ip4”: “1.0.0.1”,
“ip4-type”: “ip”,
“ip6”: “1::1”,
“ip6-type”: “ip6”,
“list”: [
1,
2,
“three”
],
“list-type”: “array”,
“listlist”: [
[
1,
2,
3
],
[
“a”,
“b”,
“c”
]
],
“listlist-type”: “array”,
“now”: “2025-01-28 15:03:13”,
“now-type”: “time”,
“null”: null,
“null-type”: “nil”,
“num”: 0,
“num-type”: “num”,
“optype”: “(op)”,
“optype-type”: “op”,
“prefix4”: “1.0.0.1/31”,
“prefix4-type”: “ip-prefix”,
“prefix6”: “1::/69”,
“prefix6-type”: “ip6-prefix”,
“str”: “mystr”,
“str-type”: “str”,
“time”: “1970-01-01 00:00:01”,
“time-type”: “time”
}

convert to JSON

convert to base64 for storage as RouterOS string
“eyJiaWduZWciOi0xNzM4MDc2NTkzODg5ODQ1NDY2LCJiaWdudW0iOjE3MzgwNzY1OTM4ODk5NTU5MDYs
ImRpY3QiOnsiYSI6MSwiYiI6InoifSwiZGljdGRpY3QiOnsiYiI6eyJvbmUiOjEsInR3byI6Mn0sInciO
nsiMSI6Im9uZSIsIjIiOiJ0d28ifX0sImRpY3RsaXN0IjpbeyJtIjoiTSJ9LHsieiI6IloifV0sImVtcH
R5c3RyIjoiIiwiZmxvYXQiOjEuMTAwMDAwLCJpcDQiOiIxLjAuMC4xIiwiaXA2IjoiMTo6MSIsImxpc3Q
iOlsxLDIsInRocmVlIl0sImxpc3RsaXN0IjpbWzEsMiwzXSxbImEiLCJiIiwiYyJdXSwibm93IjoiMjAy
NS0wMS0yOCAxNTowMzoxMyIsIm51bGwiOm51bGwsIm51bSI6MCwib3B0eXBlIjoiKG9wKSIsInByZWZpe
DQiOiIxLjAuMC4xLzMxIiwicHJlZml4NiI6IjE6Oi82OSIsInN0ciI6Im15c3RyIiwidGltZSI6IjE5Nz
AtMDEtMDEgMDA6MDA6MDEifQ==”

save to base64 JSON as unused L7 FW rule

PRETEND you reboot and come back…so wait 5 seconds

save to base64 JSON as unused L7 FW rule

if base64, restore it to JSON

finally get the array back, with types perserved by :deserialize JSON

array AFTER deserialization
{
“bigneg”: -1738076593889845466,
“bigneg-type”: “num”,
“bignum”: 1738076593889955906,
“bignum-type”: “num”,
“dict”: {
“a”: 1,
“b”: “z”
},
“dict-type”: “array”,
“dictdict”: {
“b”: {
“one”: 1,
“two”: 2
},
“w”: {
“1”: “one”,
“2”: “two”
}
},
“dictdict-type”: “array”,
“dictlist”: [
{
“m”: “M”
},
{
“z”: “Z”
}
],
“dictlist-type”: “array”,
“emptystr”: “”,
“emptystr-type”: “str”,
“float”: 1.100000,
“float-type”: “str”,
“ip4”: “1.0.0.1”,
“ip4-type”: “ip”,
“ip6”: “1::1”,
“ip6-type”: “ip6”,
“list”: [
1,
2,
“three”
],
“list-type”: “array”,
“listlist”: [
[
1,
2,
3
],
[
“a”,
“b”,
“c”
]
],
“listlist-type”: “array”,
“now”: “2025-01-28 15:03:13”,
“now-type”: “time”,
“null”: null,
“null-type”: “nil”,
“num”: 0,
“num-type”: “num”,
“optype”: “(op)”,
“optype-type”: “str”,
“prefix4”: “1.0.0.1/31”,
“prefix4-type”: “ip-prefix”,
“prefix6”: “1::/69”,
“prefix6-type”: “ip6-prefix”,
“str”: “mystr”,
“str-type”: “str”,
“time”: “1970-01-01 00:00:01”,
“time-type”: “time”
}

BONUS: simulate using array variables in function using the ‘activate-in-context’
with ‘op’ type
now: 2873w5d15:03:13 listlen: 3 bignum: 1738076593889955906 prefix6: 1::/69
… even though its array, you can use them as normal variables without the → if
you use the <%% to unwrap them

Also, the combo of [:deserialize [:serialize]] works on any type - so input does NOT have to be array – you use that to do something like get an ip-prefix from a string, without an array:

(>[:put [:typeof [:deserialize from=json [:serialize to=json $1]]]]) <%% "1.1.1.1/0"
# ip-prefix

rextended · January 28, 2025, 5:49pm

Some tests on vars that do not have sub-arrays…
{
:global globalVars [:toarray “”]
:put $globalVars

:set ($globalVars->“testArray”) [:toarray “”]
:set ($globalVars->“testArray”->“value”) [:toarray “a,b”]
:set ($globalVars->“testArray”->“type”) [:typeof ($globalVars->“testArray”->“value”)]

:set ($globalVars->“testBool”) [:toarray “”]
:set ($globalVars->“testBool”->“value”) true
:set ($globalVars->“testBool”->“type”) [:typeof ($globalVars->“testBool”->“value”)]

:set ($globalVars->“testCode”) [:toarray “”]
:set ($globalVars->“testCode”->“value”) [:parse “;”]
:set ($globalVars->“testCode”->“type”) [:typeof ($globalVars->“testCode”->“value”)]

:set ($globalVars->“testID”) [:toarray “”]
:set ($globalVars->“testID”->“value”) [:toid *BEEF6]
:set ($globalVars->“testID”->“type”) [:typeof ($globalVars->“testID”->“value”)]

:set ($globalVars->“testIP4”) [:toarray “”]
:set ($globalVars->“testIP4”->“value”) 127.0.0.1
:set ($globalVars->“testIP4”->“type”) [:typeof ($globalVars->“testIP4”->“value”)]

:set ($globalVars->“testIP4px”) [:toarray “”]
:set ($globalVars->“testIP4px”->“value”) 192.168.0.0/24
:set ($globalVars->“testIP4px”->“type”) [:typeof ($globalVars->“testIP4px”->“value”)]

:set ($globalVars->“testIP6”) [:toarray “”]
:set ($globalVars->“testIP6”->“value”) 127:0:0::1
:set ($globalVars->“testIP6”->“type”) [:typeof ($globalVars->“testIP6”->“value”)]

:set ($globalVars->“testIP6px”) [:toarray “”]
:set ($globalVars->“testIP6px”->“value”) 192:168:0::0/64
:set ($globalVars->“testIP6px”->“type”) [:typeof ($globalVars->“testIP6px”->“value”)]

:global lookupGen do={:return $0}
:set ($globalVars->“testLook”) [:toarray “”]
:set ($globalVars->“testLook”->“value”) [$lookupGen]
:set ($globalVars->“testLook”->“type”) [:typeof ($globalVars->“testLook”->“value”)]

:set ($globalVars->“testNil”) [:toarray “”]
:set ($globalVars->“testNil”->“value”)
:set ($globalVars->“testNil”->“type”) [:typeof ($globalVars->“testNil”->“value”)]

:set ($globalVars->“testNth”) [:toarray “”]
:set ($globalVars->“testNth”->“value”) ; # undefined on purpose for nothing
:set ($globalVars->“testNth”->“type”) [:typeof ($globalVars->“testNth”->“value”)]

:set ($globalVars->“testNum”) [:toarray “”]
:set ($globalVars->“testNum”->“value”) -256
:set ($globalVars->“testNum”->“type”) [:typeof ($globalVars->“testNum”->“value”)]

:set ($globalVars->“testOP”) [:toarray “”]
:set ($globalVars->“testOP”->“value”) (>)
:set ($globalVars->“testOP”->“type”) [:typeof ($globalVars->“testOP”->“value”)]

:set ($globalVars->“testStr”) [:toarray “”]
:set ($globalVars->“testStr”->“value”) [:convert to=base64 “1.1”]
:set ($globalVars->“testStr”->“base64”) true
:set ($globalVars->“testStr”->“type”) [:typeof ($globalVars->“testStr”->“value”)]

:set ($globalVars->“testTime”) [:toarray “”]
:set ($globalVars->“testTime”->“value”) 1w1d1h1m1s
:set ($globalVars->“testTime”->“type”) [:typeof ($globalVars->“testTime”->“value”)]

:foreach name,content in=$globalVars do={
:if (($content->“type”) = “array”) do={
:put ">$name< = >$($content->“type”)< >$[:tostr ($content->“value”)]< "
} else={
:if (($content->“base64”) = true) do={
:put ">$name< = >$($content->“type”)< >$[:convert from=base64 ($content->“value”)]< "
} else={
:put ">$name< = >$($content->“type”)< >$($content->“value”)< "
}
}
}

:local json [:serialize to=json options=json.pretty $globalVars]

debug

:put $json

:put “\r\nAfter array → json → array conversion:\r\n”

:global newArr [:deserialize from=json options=json.no-string-conversion $json]

:foreach name,content in=$newArr do={
:if (($content->“type”) = “array”) do={
:put “>$name< = >$($content->“type”)< >$[:tostr ($content->“value”)]< IMPORTED TYPE: >$[:typeof ($content->“value”)]<”
} else={
:if (($content->“base64”) = true) do={
:put “>$name< = >$($content->“type”)< >$[:convert from=base64 ($content->“value”)]< IMPORTED TYPE: >$[:typeof ($content->“value”)]<”
} else={
:put “>$name< = >$($content->“type”)< >$($content->“value”)< IMPORTED TYPE: >$[:typeof ($content->“value”)]<”
}
}
}

:put “\r\nNOTES\r\ncode, function, lookup and op are not exportable. Function is also reported as array.”
:put “Strings are exported and imported on base64 for prevent 7.16.2 and less errors with implicit conversions to json,”
:put “on importing strings already exist options=json.no-string-conversion that prevents implicit conversions from json”
:put “MISSING: apireq, backreference, cmd, exclamation, iterator\r\n”
}

fischerdouglas · January 29, 2025, 11:53am

@rextended
I see some good moves from MikroTik in this 7.18 beta and also on some previous version.

“:serialize” and “:deserialize”
“:jobname”
JSON support on scripting
Exposing on GUI “Script Environment”
CEF format logging
Back To Home
Cloud Flie-Share

Most of then, to me, seams to be part of a bigger thing.
Could it be related to the MikroTik Devices Controller ?

Would this be a suitable time to reinforce the request about a persistent variable manager?

Amm0 · January 29, 2025, 3:30pm

We’ve seen over the course of the V7 various changes to permissions and policy. Like, well, “:global” is truely global anymore, etc… So exactly how to expose “persistent variables” in existing policy model is where some thorny issue may come up, since the underlying user/policy/group is pretty limited today.

Given one of the more common use cases for “persistent variables” be to store some username/password for some /tool/fetch call… You’d likely also want to mark “persistent variables” as “sensitive” somehow, and have OS treat accordingly, including redacting them in logs. Supporting saving/restoring/“adding” to /sys/env/print is something they avoided for a decade, possibly since it also mean you can run the low-level “(code)” type, which is also stored there, without going through the script compiler first & compiler enforces some rules on what can be generated.

My guess is that they need to modernize the user policy/AAA stuff before we’ll see this – since today it’s bandaid-on-bandaid today there.