Saving array to file

Hi everyone,

I’ve been breaking my teeth on how to best save an array with key/value pairs and retrieve/update them. I am using rosv7.11 and tried many combinations but somehow i can’t seem to retrieve the data and access individual key/value pairs.

The datafile is saved as dataUsage and contains the following {month=8;total=631}

:local currentDate [/system clock get date];
:local currentMonth [:tonum [:pick $currentDate 5 7]];
:local dataFile "dataUsage.txt";

:if ([/file find name=$dataFile] = "") do={
    # datausage.txt file does not exist so create with base values
    /file add name=$dataFile contents="{month=$currentMonth;total=0}";
} else {
    :local fileContent [/file get $dataFile contents];
    :log info [:typeof $fileContent]; # returns string
    :local fileData [:toarray $fileContent];
    :log info [:typeof $fileData]; # returns array
    
    :local fileMonth [:tonum ($fileData->"month")];
    :log info ($fileData->"month"); # returns blank
    :log info ($fileData->"total"); # returns blank

   :if ($fileMonth=$currentMonth) do={
       :log info "Same month";
       # ... processing to be added
       
    } else { 
       :log info "Different month $fileMonth/$currentMonth "; # $fileMonth is blank
       :set ($fileData->"month") $currentMonth;
       :set ($fileData->"total") 0;
    }

     /file remove $dataFile;
     /file add name=$dataFile contents="{month=$($fileData->"month");total=$($fileData->"total")}";
}

Somehow I guess that either my initial saved array/string is not being correctly converted back to an array after reading from file or it is viewed as a single value but I can’t find where my mistake is.

Any insight would be much appreciated or if there is a better way to save and restore a few key/value pairs so that they persist at reboot that would be great.

Thanks
Michaël

where are the quotes between the month name?

:global x {month=Aug;test=0}
:put ($x->“month”)
:global x {month=“Aug”;test=0}
:put ($x->“month”)

That’s true, need quotes. But I think it the serializing an array to disk is the issue… I’m not sure this logic works:

:global before {field1="mystr";field2=2} 
/file add name=diskarray contents=$before
:global after [:toarray [/file get diskarray contents]]

Also using this does not work either: “/file add name=diskarray contents=[:tostr $before]”


Both using the following tests:

# tests...
{
:put "File contains: $[/file get diskarray contents ]"
:put "Array length $[:len $before] should be 2..."
:put "After writing and reading to disk..."
:put "...it's length is $[:len $after] (typeof: $[:typeof $after]) containing:"
:put $after
:put "or \t$[:tostr $after]\tafter :tostr"
:put "with field1 being: $($after->"field1")"
:put "with index 0 being: $($after->0)"
/file remove diskarray
}



File contains: field1=mystr;field2=2
Array length 2 should be 2…
After writing and reading to disk…
…it’s length is 1 (typeof: array) containing:
field1=mystr;field2=2
or field1=mystr;field2=2 after :tostr
with field1 being:
with index 0 being: field1=mystr;field2=2

Thank you both.

The month would be a number (8 for august) but that isn’t so much the problem.

The tests are exactly what I am experiencing as well …
If the serializing of the array to disk is not working with this logic, then how can/should this be done ?

Thanks
Michaël

The arrays must be exported as a set of instructions,
and imported back executing / parsing the instruction set…
example of file content can be:

:set ($arrayname->“column-name”) “value”;:set ($arrayname->“column-name2”) “value2”;:set ($arrayname->“column-name2”) “value2”…

but a presence of " or ; inside the value broken the script…

Is a very complex problem.

You need to save the array to a json, and import back a json in the array…
http://forum.mikrotik.com/t/iterate-over-all-elements-of-an-array-of-unknown-dimension/163033/1

Another thing to keep in mind is the fact that working with “file content” you are limited to an array of size of 4kB (see the other topic)

If the output is put inside one .rsc, can be imported later for recreate the array on another machine…
:global exportarray do={ :global exportarray
:local path “$2”
:local ret “”
/system script environment
:foreach j in=[find] do={
:if ([get $j value] = $1) do={:set path “$$[get $j name]”}
}
:foreach x,y in=$1 do={
:local typex [:typeof $x]
:local typey [:typeof $y]
:local lpath $path
:if ($typex = “str”) do={:set lpath “$path->"$x"”} else={:set lpath “$path->$x”}
:if ([:typeof $y] = “array”) do={
:if ([:len $y] > 0) do={
:set ret “$ret:set ($lpath) [:toarray ""]\r\n”
:set ret “$ret$[$exportarray $y $lpath]\r\n”
} else={
:set ret “$ret:set ($lpath) [:toarray ""]\r\n”
}
} else={
:if ($typey = “apireq” ) do={:set $y “"$y"”}
:if ($typey = “backreference”) do={:set $y “"$y"”}
:if ($typey = “bool” ) do={:set $y “$y” }
:if ($typey = “cmd” ) do={:set $y “"$y"”}
:if ($typey = “code” ) do={:set $y “"$y"”}
:if ($typey = “exclamation” ) do={:set $y “"$y"”}
:if ($typey = “id” ) do={:set $y “[:toid $y]” }
:if ($typey = “ip” ) do={:set $y “$y” }
:if ($typey = “ip6” ) do={:set $y “$y” }
:if ($typey = “ip6-prefix” ) do={:set $y “$y” }
:if ($typey = “ip-prefix” ) do={:set $y “$y” }
:if ($typey = “iterator” ) do={:set $y “"$y"”}
:if ($typey = “lookup” ) do={:set $y “"$y"”}
:if ($typey = “nil” ) do={:set $y “” }
:if ($typey = “nothing” ) do={:set $y [:nothing] }
:if ($typey = “num” ) do={:set $y “$y” }
:if ($typey = “op” ) do={:set $y “"$y"”}
:if ($typey = “str” ) do={:set $y “"$y"”}
:if ($typey = “time” ) do={:set $y “$y” }
:set ret “$ret:set ($lpath) $y\r\n”
}
}
:return [:pick $ret 0 ([:len $ret] - 2)]
}

:global test {month=“Aug”;test=0}
:put “:global test [:toarray ""]\r\n$[$exportarray $test]”
:global test [:toarray “”]
:set ($test->“month”) “Aug”
:set ($test->“test”) 0
EDIT 1-2-3-4: Fixed typos and added array declaration on file (:global test [:toarray “”]), remove duplicated [:toarray “”]., removed useless \r\n at the end of the cycle

Thank you rextended,
very much appreciated! I will check it today and let you know how it went
Grazie mille
Michaël

Only some condition on this preliminary version:
The array must be global!!!


For this array:
http://forum.mikrotik.com/t/iterate-over-all-elements-of-an-array-of-unknown-dimension/163033/1

the result is:
:global ArrayIN [:toarray “”]
:set ($ArrayIN->“alertList”) [:toarray “”]
:set ($ArrayIN->“alertList”->0) “arrayvalue1”
:set ($ArrayIN->“alertList”->1) “arrayvalue2”
:set ($ArrayIN->“audioSettings”) [:toarray “”]
:set ($ArrayIN->“audioSettings”->“DoP”) [:toarray “”]
:set ($ArrayIN->“audioSettings”->“DoP”->“title”) “DoP playback”
:set ($ArrayIN->“audioSettings”->“DoP”->“type”) “boolean”
:set ($ArrayIN->“audioSettings”->“DoP”->“value”) false
:set ($ArrayIN->“audioSettings”->“autoPlay”) [:toarray “”]
:set ($ArrayIN->“audioSettings”->“autoPlay”->“title”) “AutoPlay after boot”
:set ($ArrayIN->“audioSettings”->“autoPlay”->“type”) “boolean”
:set ($ArrayIN->“audioSettings”->“autoPlay”->“value”) false
:set ($ArrayIN->“audioSettings”->“soundCard”) [:toarray “”]
:set ($ArrayIN->“audioSettings”->“soundCard”->“data”) [:toarray “”]
:set ($ArrayIN->“audioSettings”->“soundCard”->“data”->0) [:toarray “”]
:set ($ArrayIN->“audioSettings”->“soundCard”->“data”->0->“id”) 0
:set ($ArrayIN->“audioSettings”->“soundCard”->“data”->0->“name”) “Default”
:set ($ArrayIN->“audioSettings”->“soundCard”->“title”) “Sound Card”
:set ($ArrayIN->“audioSettings”->“soundCard”->“type”) “spinner”
:set ($ArrayIN->“audioSettings”->“soundCard”->“value”) 0
:set ($ArrayIN->“audioSettings”->“soundType”) [:toarray “”]
:set ($ArrayIN->“audioSettings”->“soundType”->“data”) [:toarray “”]
:set ($ArrayIN->“audioSettings”->“soundType”->“data”->0) [:toarray “”]
:set ($ArrayIN->“audioSettings”->“soundType”->“data”->0->“id”) 0
:set ($ArrayIN->“audioSettings”->“soundType”->“data”->0->“name”) “Mono Differential”
:set ($ArrayIN->“audioSettings”->“soundType”->“data”->1) [:toarray “”]
:set ($ArrayIN->“audioSettings”->“soundType”->“data”->1->“id”) 1
:set ($ArrayIN->“audioSettings”->“soundType”->“data”->1->“name”) “Stereo”
:set ($ArrayIN->“audioSettings”->“soundType”->“title”) “Sound Type”
:set ($ArrayIN->“audioSettings”->“soundType”->“type”) “spinner”
:set ($ArrayIN->“audioSettings”->“soundType”->“value”) 1
:set ($ArrayIN->“audioSettings”->“title”) “Audio settings”
:set ($ArrayIN->“broadcastModeSettings”) [:toarray “”]
:set ($ArrayIN->“broadcastModeSettings”->“broadcastModeOptions”) [:toarray “”]
:set ($ArrayIN->“broadcastModeSettings”->“broadcastModeOptions”->“downloadURL”) “no URL provided”
:set ($ArrayIN->“broadcastModeSettings”->“broadcastModeOptions”->“streamURL”) “no URL provided”
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”) [:toarray “”]
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“data”) [:toarray “”]
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“data”->0) [:toarray “”]
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“data”->0->“id”) 0
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“data”->0->“name”) “Default”
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“data”->1) [:toarray “”]
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“data”->1->“id”) 1
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“data”->1->“name”) “Stream URL”
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“data”->2) [:toarray “”]
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“data”->2->“id”) 2
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“data”->2->“name”) “Download URL”
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“title”) “Broadcast Type”
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“type”) “spinner”
:set ($ArrayIN->“broadcastModeSettings”->“broadcastType”->“value”) 0
:set ($ArrayIN->“broadcastModeSettings”->“enabled”) false
:set ($ArrayIN->“broadcastModeSettings”->“title”) “Broadcast mode settings”
:set ($ArrayIN->“deviceInfo”) [:toarray “”]
:set ($ArrayIN->“deviceInfo”->“autoUpgrade”) true
:set ($ArrayIN->“deviceInfo”->“autoUpgradeInstall”) true
:set ($ArrayIN->“deviceInfo”->“cpuTemp”) “48 C”
:set ($ArrayIN->“deviceInfo”->“version”) “2.16.37”
:set ($ArrayIN->“multiroomSettings”) [:toarray “”]
:set ($ArrayIN->“multiroomSettings”->“masterMode”) false
:set ($ArrayIN->“multiroomSettings”->“slaveList”) [:toarray “”]
:set ($ArrayIN->“multiroomSettings”->“slaveList”->0) “arrayvalue1”
:set ($ArrayIN->“multiroomSettings”->“slaveList”->1) “arrayvalue2”
:set ($ArrayIN->“networkSettings”) [:toarray “”]
:set ($ArrayIN->“networkSettings”->“connections”) [:toarray “”]
:set ($ArrayIN->“networkSettings”->“connections”->“LAN”) [:toarray “”]
:set ($ArrayIN->“networkSettings”->“connections”->“LAN”->“addresses”) “192.168.0.101/24”
:set ($ArrayIN->“networkSettings”->“connections”->“LAN”->“dns”) “192.168.0.1”
:set ($ArrayIN->“networkSettings”->“connections”->“LAN”->“gateway”) “192.168.0.1”
:set ($ArrayIN->“networkSettings”->“connections”->“LAN”->“method”) “auto”
:set ($ArrayIN->“networkSettings”->“connections”->“LAN”->“state”) “connected”
:set ($ArrayIN->“networkSettings”->“connections”->“WLAN”) [:toarray “”]
:set ($ArrayIN->“networkSettings”->“connections”->“WLAN”->“state”) “disconnected”
:set ($ArrayIN->“networkSettings”->“interfaces”) [:toarray “”]
:set ($ArrayIN->“networkSettings”->“interfaces”->“LAN”) [:toarray “”]
:set ($ArrayIN->“networkSettings”->“interfaces”->“LAN”->“title”) “Ethernet”
:set ($ArrayIN->“networkSettings”->“interfaces”->“LAN”->“type”) “boolean”
:set ($ArrayIN->“networkSettings”->“interfaces”->“LAN”->“value”) true
:set ($ArrayIN->“networkSettings”->“interfaces”->“WLAN”) [:toarray “”]
:set ($ArrayIN->“networkSettings”->“interfaces”->“WLAN”->“title”) “Wi-Fi”
:set ($ArrayIN->“networkSettings”->“interfaces”->“WLAN”->“type”) “boolean”
:set ($ArrayIN->“networkSettings”->“interfaces”->“WLAN”->“value”) false
:set ($ArrayIN->“networkSettings”->“title”) “Network settings”
:set ($ArrayIN->“networkSettings”->“wifiList”) [:toarray “”]
:set ($ArrayIN->“networkSettings”->“wifiList”->0) “arrayvalue1”
:set ($ArrayIN->“networkSettings”->“wifiList”->1) “arrayvalue2”
If elaborated from someone that have time, the same script can produce something like this directly:
:global ArrayIN {“alertList”=“arrayvalue1”,“arrayvalue2”;[…];“title”=“Network settings”;“wifiList”=“arrayvalue1”,“arrayvalue2”};}

I guess you were not wrong :wink:

Good to know Mikrotik isn’t going to put you out of business with new [:convert] anytime soon.

The timing with which they came up with :convert, and the similarity of what :convert does compared to the scripts I’ve made, is staggering… :wink:

For sure if they had copied my scripts, they would have made fewer mistakes… :laughing:

One tip here, is there are really two kinds of arrays, “indexed lists” ($arr->0) and “key-value maps” ($arr->“key”) - even though both are :typeof == array. The key-value map ones you cannot get back from a string (without @rextended’s parsing code above)… but the list kind you can.

May not work here, but worth mentioning since if you can structure what you read/write to the more basic list array type, that does generally* work:

:put [:typeof [:toarray [:tostr ("zero",123,"with space")]]]
# array

*forgot to mention: except if string in list contain semi-colons

And if one of the :convert options was array to JSON … that change this answer dramatically.

In the meantime I scoured rextended’s extensive code repo and found this that also works perfectly for me
Reading file data

	:local filecontent [/file get $dataFile contents];
    	:global tempconv;
    	[:parse ":global tempconv $filecontent"]
    	:local fileData $tempconv;
	:log info ($fileData->"month");
    	:log info ($fileData->"total");

Saving data to file

     /file remove $dataFile;
     /file add name=$dataFile contents="{month=$($fileData->"month");total=$($fileData->"total")}";

Thanks all
Michaël

Hi,
I noticed that when I convert an array to a file each item in the array is separated by a “;”. So I wrote a script to find all the semicolons and then get all in between to create a new operational array.

I hope this helps you.

# Find a file in the Files List and turn it to operational Array

# Variables
:global ItemsArray;
:local DataFromFile;
:local Deviders;
:local SecondDevider;

# Find if the file exists
:local count 0;
:local L [/file print as-value];
:foreach f in=$L do={
    :if (($f->"name")="DataFile.txt") do={
        :set count ($count + 1);
    }
} 
# If the file exists then update it. If it doesn't exist then create it.
#:if ($count=0) do={ /file add name=DataFile.txt contents="";}
:if ($count=1) do={
    # Find all the ; that devide all the items in the file
    :set DataFromFile (";".[/file get DataFile.txt contents].";");
    :set Deviders;
    :for i from=0 to=([:len $DataFromFile] -1) do={
        :local Char [:pick $DataFromFile $i ];
        :if ($Char=";") do={
            :set Deviders ($Deviders,  $i);
        }
    }
    # Make a new array from the items found in the file
    :local z 1; # Used to calculate the SecondDevider
    :set ItemsArray;
    :foreach FirstDevider in=$Deviders do={
        :set SecondDevider ($Deviders->$z);
        :local Item [:pick $DataFromFile ($FirstDevider+1)  $SecondDevider];
        :set ItemsArray ($ItemsArray, $Item);
        :set z ($z+1); 
    }
}
:put $ItemsArray;