Saving array to file

MichaelDC · Tue Aug 29, 2023 5:50 pm

Hi everyone,

I've been breaking my teeth on how to best save an array with key/value pairs and retrieve/update them. I am using rosv7.11 and tried many combinations but somehow i can't seem to retrieve the data and access individual key/value pairs.

The datafile is saved as dataUsage and contains the following {month=8;total=631}

:local currentDate [/system clock get date];
:local currentMonth [:tonum [:pick $currentDate 5 7]];
:local dataFile "dataUsage.txt";

:if ([/file find name=$dataFile] = "") do={
    # datausage.txt file does not exist so create with base values
    /file add name=$dataFile contents="{month=$currentMonth;total=0}";
} else {
    :local fileContent [/file get $dataFile contents];
    :log info [:typeof $fileContent]; # returns string
    :local fileData [:toarray $fileContent];
    :log info [:typeof $fileData]; # returns array
    
    :local fileMonth [:tonum ($fileData->"month")];
    :log info ($fileData->"month"); # returns blank
    :log info ($fileData->"total"); # returns blank

   :if ($fileMonth=$currentMonth) do={
       :log info "Same month";
       # ... processing to be added
       
    } else { 
       :log info "Different month $fileMonth/$currentMonth "; # $fileMonth is blank
       :set ($fileData->"month") $currentMonth;
       :set ($fileData->"total") 0;
    }

     /file remove $dataFile;
     /file add name=$dataFile contents="{month=$($fileData->"month");total=$($fileData->"total")}";
}

Somehow I guess that either my initial saved array/string is not being correctly converted back to an array after reading from file or it is viewed as a single value but I can't find where my mistake is.

Any insight would be much appreciated or if there is a better way to save and restore a few key/value pairs so that they persist at reboot that would be great.

Thanks
Michaël

rextended · Tue Aug 29, 2023 9:48 pm

where are the quotes between the month name?

example code

:global x {month=Aug;test=0}
:put ($x->"month")
:global x {month="Aug";test=0}
:put ($x->"month")

Amm0 · Wed Aug 30, 2023 12:15 am

That's true, need quotes. But I think it the serializing an array to disk is the issue... I'm not sure this logic works:

:global before {field1="mystr";field2=2} 
/file add name=diskarray contents=$before
:global after [:toarray [/file get diskarray contents]]

Also using this does not work either: "/file add name=diskarray contents=[:tostr $before]"

Both using the following tests:

# tests...
{
:put "File contains: $[/file get diskarray contents ]"
:put "Array length $[:len $before] should be 2..."
:put "After writing and reading to disk..."
:put "...it's length is $[:len $after] (typeof: $[:typeof $after]) containing:"
:put $after
:put "or \t$[:tostr $after]\tafter :tostr"
:put "with field1 being: $($after->"field1")"
:put "with index 0 being: $($after->0)"
/file remove diskarray
}

File contains: field1=mystr;field2=2
Array length 2 should be 2...
After writing and reading to disk...
...it's length is 1 (typeof: array) containing:
field1=mystr;field2=2
or field1=mystr;field2=2 after :tostr
with field1 being:
with index 0 being: field1=mystr;field2=2

MichaelDC · Wed Aug 30, 2023 8:45 am

Thank you both.

The month would be a number (8 for august) but that isn't so much the problem.

The tests are exactly what I am experiencing as well ...
If the serializing of the array to disk is not working with this logic, then how can/should this be done ?

Thanks
Michaël

rextended · Wed Aug 30, 2023 10:45 am

The arrays must be exported as a set of instructions,
and imported back executing / parsing the instruction set...
example of file content can be:

:set ($arrayname->"column-name") "value";:set ($arrayname->"column-name2") "value2";:set ($arrayname->"column-name2") "value2"....

but a presence of " or ; inside the value broken the script....

Is a very complex problem.

You need to save the array to a json, and import back a json in the array...
viewtopic.php?t=191832

Another thing to keep in mind is the fact that working with "file content" you are limited to an array of size of 4kB (see the other topic)

rextended · Wed Aug 30, 2023 3:06 pm

If the output is put inside one .rsc, can be imported later for recreate the array on another machine...

preliminary version of export array code

:global exportarray do={ :global exportarray
                          :local path "$2"
                          :local ret   ""
                          /system script environment
                          :foreach j in=[find] do={
                              :if ([get $j value] = $1) do={:set path "\$$[get $j name]"}
                          }
                          :foreach x,y in=$1 do={
                              :local typex [:typeof $x]
                              :local typey [:typeof $y]
                              :local lpath $path
                              :if ($typex = "str") do={:set lpath "$path->\"$x\""} else={:set lpath "$path->$x"}
                              :if ([:typeof $y] = "array") do={
                                  :if ([:len $y] > 0) do={
                                      :set ret "$ret:set ($lpath) [:toarray \"\"]\r\n"
                                      :set ret "$ret$[$exportarray $y $lpath]\r\n"
                                  } else={
                                      :set ret "$ret:set ($lpath) [:toarray \"\"]\r\n"
                                  }
                              } else={
                                  :if ($typey = "apireq"       ) do={:set $y      "\"$y\""}
                                  :if ($typey = "backreference") do={:set $y      "\"$y\""}
                                  :if ($typey = "bool"         ) do={:set $y        "$y"  }
                                  :if ($typey = "cmd"          ) do={:set $y      "\"$y\""}
                                  :if ($typey = "code"         ) do={:set $y      "\"$y\""}
                                  :if ($typey = "exclamation"  ) do={:set $y      "\"$y\""}
                                  :if ($typey = "id"           ) do={:set $y "[:toid $y]" } 
                                  :if ($typey = "ip"           ) do={:set $y        "$y"  }
                                  :if ($typey = "ip6"          ) do={:set $y        "$y"  }
                                  :if ($typey = "ip6-prefix"   ) do={:set $y        "$y"  }
                                  :if ($typey = "ip-prefix"    ) do={:set $y        "$y"  }
                                  :if ($typey = "iterator"     ) do={:set $y      "\"$y\""}
                                  :if ($typey = "lookup"       ) do={:set $y      "\"$y\""}
                                  :if ($typey = "nil"          ) do={:set $y ""           }
                                  :if ($typey = "nothing"      ) do={:set $y [:nothing]   }
                                  :if ($typey = "num"          ) do={:set $y        "$y"  }
                                  :if ($typey = "op"           ) do={:set $y      "\"$y\""}
                                  :if ($typey = "str"          ) do={:set $y      "\"$y\""}
                                  :if ($typey = "time"         ) do={:set $y        "$y"  }
                                  :set ret "$ret:set ($lpath) $y\r\n"
                              }
                          }
                          :return [:pick $ret 0 ([:len $ret] - 2)]
                        }

:global test {month="Aug";test=0}
:put ":global test [:toarray \"\"]\r\n$[$exportarray $test]"

example of file obtained code

:global test [:toarray ""]
:set ($test->"month") "Aug"
:set ($test->"test") 0

EDIT 1-2-3-4: Fixed typos and added array declaration on file (:global test [:toarray ""]), remove duplicated [:toarray ""]., removed useless \r\n at the end of the cycle

MichaelDC · Wed Aug 30, 2023 3:09 pm

Thank you rextended,
very much appreciated! I will check it today and let you know how it went
Grazie mille
Michaël

rextended · Wed Aug 30, 2023 3:12 pm

Only some condition on this preliminary version:
The array must be global!!!

For this array:
viewtopic.php?t=191832#p973295

the result is:

export example code

:global ArrayIN [:toarray ""]
:set ($ArrayIN->"alertList") [:toarray ""]
:set ($ArrayIN->"alertList"->0) "arrayvalue1"
:set ($ArrayIN->"alertList"->1) "arrayvalue2"
:set ($ArrayIN->"audioSettings") [:toarray ""]
:set ($ArrayIN->"audioSettings"->"DoP") [:toarray ""]
:set ($ArrayIN->"audioSettings"->"DoP"->"title") "DoP playback"
:set ($ArrayIN->"audioSettings"->"DoP"->"type") "boolean"
:set ($ArrayIN->"audioSettings"->"DoP"->"value") false
:set ($ArrayIN->"audioSettings"->"autoPlay") [:toarray ""]
:set ($ArrayIN->"audioSettings"->"autoPlay"->"title") "AutoPlay after boot"
:set ($ArrayIN->"audioSettings"->"autoPlay"->"type") "boolean"
:set ($ArrayIN->"audioSettings"->"autoPlay"->"value") false
:set ($ArrayIN->"audioSettings"->"soundCard") [:toarray ""]
:set ($ArrayIN->"audioSettings"->"soundCard"->"data") [:toarray ""]
:set ($ArrayIN->"audioSettings"->"soundCard"->"data"->0) [:toarray ""]
:set ($ArrayIN->"audioSettings"->"soundCard"->"data"->0->"id") 0
:set ($ArrayIN->"audioSettings"->"soundCard"->"data"->0->"name") "Default"
:set ($ArrayIN->"audioSettings"->"soundCard"->"title") "Sound Card"
:set ($ArrayIN->"audioSettings"->"soundCard"->"type") "spinner"
:set ($ArrayIN->"audioSettings"->"soundCard"->"value") 0
:set ($ArrayIN->"audioSettings"->"soundType") [:toarray ""]
:set ($ArrayIN->"audioSettings"->"soundType"->"data") [:toarray ""]
:set ($ArrayIN->"audioSettings"->"soundType"->"data"->0) [:toarray ""]
:set ($ArrayIN->"audioSettings"->"soundType"->"data"->0->"id") 0
:set ($ArrayIN->"audioSettings"->"soundType"->"data"->0->"name") "Mono Differential"
:set ($ArrayIN->"audioSettings"->"soundType"->"data"->1) [:toarray ""]
:set ($ArrayIN->"audioSettings"->"soundType"->"data"->1->"id") 1
:set ($ArrayIN->"audioSettings"->"soundType"->"data"->1->"name") "Stereo"
:set ($ArrayIN->"audioSettings"->"soundType"->"title") "Sound Type"
:set ($ArrayIN->"audioSettings"->"soundType"->"type") "spinner"
:set ($ArrayIN->"audioSettings"->"soundType"->"value") 1
:set ($ArrayIN->"audioSettings"->"title") "Audio settings"
:set ($ArrayIN->"broadcastModeSettings") [:toarray ""]
:set ($ArrayIN->"broadcastModeSettings"->"broadcastModeOptions") [:toarray ""]
:set ($ArrayIN->"broadcastModeSettings"->"broadcastModeOptions"->"downloadURL") "no URL provided"
:set ($ArrayIN->"broadcastModeSettings"->"broadcastModeOptions"->"streamURL") "no URL provided"
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType") [:toarray ""]
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"data") [:toarray ""]
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"data"->0) [:toarray ""]
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"data"->0->"id") 0
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"data"->0->"name") "Default"
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"data"->1) [:toarray ""]
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"data"->1->"id") 1
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"data"->1->"name") "Stream URL"
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"data"->2) [:toarray ""]
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"data"->2->"id") 2
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"data"->2->"name") "Download URL"
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"title") "Broadcast Type"
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"type") "spinner"
:set ($ArrayIN->"broadcastModeSettings"->"broadcastType"->"value") 0
:set ($ArrayIN->"broadcastModeSettings"->"enabled") false
:set ($ArrayIN->"broadcastModeSettings"->"title") "Broadcast mode settings"
:set ($ArrayIN->"deviceInfo") [:toarray ""]
:set ($ArrayIN->"deviceInfo"->"autoUpgrade") true
:set ($ArrayIN->"deviceInfo"->"autoUpgradeInstall") true
:set ($ArrayIN->"deviceInfo"->"cpuTemp") "48 C"
:set ($ArrayIN->"deviceInfo"->"version") "2.16.37"
:set ($ArrayIN->"multiroomSettings") [:toarray ""]
:set ($ArrayIN->"multiroomSettings"->"masterMode") false
:set ($ArrayIN->"multiroomSettings"->"slaveList") [:toarray ""]
:set ($ArrayIN->"multiroomSettings"->"slaveList"->0) "arrayvalue1"
:set ($ArrayIN->"multiroomSettings"->"slaveList"->1) "arrayvalue2"
:set ($ArrayIN->"networkSettings") [:toarray ""]
:set ($ArrayIN->"networkSettings"->"connections") [:toarray ""]
:set ($ArrayIN->"networkSettings"->"connections"->"LAN") [:toarray ""]
:set ($ArrayIN->"networkSettings"->"connections"->"LAN"->"addresses") "192.168.0.101/24"
:set ($ArrayIN->"networkSettings"->"connections"->"LAN"->"dns") "192.168.0.1"
:set ($ArrayIN->"networkSettings"->"connections"->"LAN"->"gateway") "192.168.0.1"
:set ($ArrayIN->"networkSettings"->"connections"->"LAN"->"method") "auto"
:set ($ArrayIN->"networkSettings"->"connections"->"LAN"->"state") "connected"
:set ($ArrayIN->"networkSettings"->"connections"->"WLAN") [:toarray ""]
:set ($ArrayIN->"networkSettings"->"connections"->"WLAN"->"state") "disconnected"
:set ($ArrayIN->"networkSettings"->"interfaces") [:toarray ""]
:set ($ArrayIN->"networkSettings"->"interfaces"->"LAN") [:toarray ""]
:set ($ArrayIN->"networkSettings"->"interfaces"->"LAN"->"title") "Ethernet"
:set ($ArrayIN->"networkSettings"->"interfaces"->"LAN"->"type") "boolean"
:set ($ArrayIN->"networkSettings"->"interfaces"->"LAN"->"value") true
:set ($ArrayIN->"networkSettings"->"interfaces"->"WLAN") [:toarray ""]
:set ($ArrayIN->"networkSettings"->"interfaces"->"WLAN"->"title") "Wi-Fi"
:set ($ArrayIN->"networkSettings"->"interfaces"->"WLAN"->"type") "boolean"
:set ($ArrayIN->"networkSettings"->"interfaces"->"WLAN"->"value") false
:set ($ArrayIN->"networkSettings"->"title") "Network settings"
:set ($ArrayIN->"networkSettings"->"wifiList") [:toarray ""]
:set ($ArrayIN->"networkSettings"->"wifiList"->0) "arrayvalue1"
:set ($ArrayIN->"networkSettings"->"wifiList"->1) "arrayvalue2"

If elaborated from someone that have time, the same script can produce something like this directly:

example code

:global ArrayIN {"alertList"="arrayvalue1","arrayvalue2";[…];"title"="Network settings";"wifiList"="arrayvalue1","arrayvalue2"};}

Amm0 · Wed Aug 30, 2023 3:33 pm

Is a very complex problem.

I guess you were not wrong

Good to know Mikrotik isn't going to put you out of business with new [:convert] anytime soon.

rextended · Wed Aug 30, 2023 3:58 pm

Good to know Mikrotik isn't going to put you out of business with new [:convert] anytime soon.

The timing with which they came up with :convert, and the similarity of what :convert does compared to the scripts I've made, is staggering...

For sure if they had copied my scripts, they would have made fewer mistakes...

Amm0 · Wed Aug 30, 2023 4:04 pm

One tip here, is there are really two kinds of arrays, "indexed lists" ($arr->0) and "key-value maps" ($arr->"key") - even though both are :typeof == array. The key-value map ones you cannot get back from a string (without @rextended's parsing code above)... but the list kind you can.

May not work here, but worth mentioning since if you can structure what you read/write to the more basic list array type, that does generally* work:

:put [:typeof [:toarray [:tostr ("zero",123,"with space")]]]
# array

*forgot to mention: except if string in list contain semi-colons

Amm0 · Wed Aug 30, 2023 4:12 pm

Good to know Mikrotik isn't going to put you out of business with new [:convert] anytime soon.
The timing with which they came up with :convert, and the similarity of what :convert does compared to the scripts I've made, is staggering...

And if one of the :convert options was array to JSON ... that change this answer dramatically.

MichaelDC · Wed Aug 30, 2023 7:49 pm

In the meantime I scoured rextended's extensive code repo and found this that also works perfectly for me
Reading file data

	:local filecontent [/file get $dataFile contents];
    	:global tempconv;
    	[:parse ":global tempconv $filecontent"]
    	:local fileData $tempconv;
	:log info ($fileData->"month");
    	:log info ($fileData->"total");

Saving data to file

     /file remove $dataFile;
     /file add name=$dataFile contents="{month=$($fileData->"month");total=$($fileData->"total")}";

Thanks all
Michaël

phcooper · Sun Oct 01, 2023 4:29 pm

Hi,
I noticed that when I convert an array to a file each item in the array is separated by a ";". So I wrote a script to find all the semicolons and then get all in between to create a new operational array.

I hope this helps you.

# Find a file in the Files List and turn it to operational Array

# Variables
:global ItemsArray;
:local DataFromFile;
:local Deviders;
:local SecondDevider;

# Find if the file exists
:local count 0;
:local L [/file print as-value];
:foreach f in=$L do={
    :if (($f->"name")="DataFile.txt") do={
        :set count ($count + 1);
    }
} 
# If the file exists then update it. If it doesn't exist then create it.
#:if ($count=0) do={ /file add name=DataFile.txt contents="";}
:if ($count=1) do={
    # Find all the ; that devide all the items in the file
    :set DataFromFile (";".[/file get DataFile.txt contents].";");
    :set Deviders;
    :for i from=0 to=([:len $DataFromFile] -1) do={
        :local Char [:pick $DataFromFile $i ];
        :if ($Char=";") do={
            :set Deviders ($Deviders,  $i);
        }
    }
    # Make a new array from the items found in the file
    :local z 1; # Used to calculate the SecondDevider
    :set ItemsArray;
    :foreach FirstDevider in=$Deviders do={
        :set SecondDevider ($Deviders->$z);
        :local Item [:pick $DataFromFile ($FirstDevider+1)  $SecondDevider];
        :set ItemsArray ($ItemsArray, $Item);
        :set z ($z+1); 
    }
}
:put $ItemsArray;