I have been looking closer at all the new scripting improvements that MikroTik recently made. I found a lot of very useful new functionality and wanted to share some of my findings.
One thing that was never clear to me was the maximum size of a variable. Part of the issue is that usually when dealing with really large variables, I’m also dealing with reading and writing to large files, and that part has its own limitations.
There were a number of threads around this that have been useful but not sure how much of this information is valid anymore.
http://forum.mikrotik.com/t/max-size-of-variables-still-at-4096-anwser-is-no/168007/3
http://forum.mikrotik.com/t/the-maximum-size-of-a-read-written-file/167507/1
http://forum.mikrotik.com/t/max-size-of-variables-still-at-4096-anwser-is-no/168007/1
http://forum.mikrotik.com/t/how-to-download-only-one-piece-of-file-at-a-time-with-tool-fetch-and-put-it-inside-a-variable/151020/1
Even the ROS documentation has confusing information sometimes. Example:
In this example, the data is uploaded as a file. Important note, since variable data comes from a file, a file can only be in size up to 4KB. This is a limitation of RouterOS variables.
https://help.mikrotik.com/docs/spaces/ROS/pages/8978514/Fetch
The above implies the variable maximum size is only 4KB. It is not. Maybe that’s the limit with the fetch command, but the variables in general can get much larger (see below my test with 7MB).
Regarding operations with files, this page in the documentation is very helpful:
It is possible to retrieve content and edit files up to 60KB in size. For accessing contents of larger files, please refer to section Get File Contents.
https://help.mikrotik.com/docs/spaces/ROS/pages/2555971/Files
Technically, it’s not 100% correct because 60KB = 61440 bytes. In my tests the max is 61439. If the file is larger, the resulting variable is 0 bytes, no errors, no crashing.
# Testing the limit for /file/get.
# Use 61439 minus 2 characters for quotes added by JSON
{
:local var
:for i from=1 to=61437 do={ :set $var ($var . "a") }
:put [:len $var]
# 61437
:serialize $var to=json file-name=test
:local newvar [/file/get test contents]
:put [:len $newvar]
# 61439
}
####
{
:local var
:for i from=1 to=61438 do={ :set $var ($var . "a") }
:put [:len $var]
# 61438
:serialize $var to=json file-name=test
:local newvar [/file/get test contents]
:put [:len $newvar]
# 0
:put [/file/get test size]
# 61440
}
There is /file/read command which I haven’t seen before, don’t know if it’s new. With this command there is virtually no limit on reading large files, I suppose other than RAM. One important note here is that /file/read as is doesn’t actually read anything into a variable. But when combined with “as-value”, it does exactly what I want.
Here is simple code to read a very large file (7MB in my tests) into a variable using chunks. The nice thing about the “chunk-size” option is that it’s smart enough to stop reading at the end of the file. So I don’t even need to calculate the remainder and read it separately outside of the loop (although it’s easy too).
Note, the chunk-size maximum is 32768 (not 60KB), but it doesn’t matter, we will just use more chunks.
Another note, this reading loop will take some time as it is CPU-intensive for larger files.
{
:local fname servers.txt
:local fsize [/file/get $fname size]
:local max 32768
# Get the number of chunks minus one since the first offset is 0.
:local chunks (($fsize / $max) - 1)
# If there is a remainder, add another chunk.
: if ($fsize > ($max * $chunks)) do={ :set $chunks ($chunks + 1) }
:put $chunks
# 222
:local var
:for i from=0 to=$chunks do={
# Start each read from the next chunk
:local offset ($i * $max)
:local varchunk [/file/read file=$fname offset=$offset chunk-size=$max as-value]
:set $var ($var . ($varchunk->"data"))
}
:put [:len $var]
# 7293564
}
This was all about reading file. How about writing this huge variable back into a file? There is the known old trick with
:execute { :put $var } file=output
But there are several downsides with this approach:
- You have to put the code producing the $var inside :execute block. Or use a global variable, not the best approach for such large variables (but it can be cleared out in the code right away).
- The file extension is always .txt. Of course, with /file/set name= (I believe also a new thing) one can rename it, but it’s an extra step.
- This method always puts “\r\n” line ending at the end of the file. Usually not a big deal but could be important in some cases.
I think a much better and easier approach to write a large variable to a file is with :serialize. It works only with arrays, but nothing prevents us from using a single-value index-based array.
Below is the full code to read a large file in chunks into a variable, then save it to another file. The resulting file is 100% identical to the original one without any further manipulations. I tested by downloading it to a PC and using a binary comparison tool.
{
:local fname servers.txt
:local fsize [/file/get $fname size]
:local max 32768
# Get the number of chunks minus one since the first offset is 0.
:local chunks (($fsize / $max) - 1)
# If there is a remainder, add another chunk.
: if ($fsize > ($max * $chunks)) do={ :set $chunks ($chunks + 1) }
:put $chunks
# 222
:local var {""}
:for i from=0 to=$chunks do={
# Start each read from the next chunk
:local offset ($i * $max)
:local filechunk [/file/read file=$fname offset=$offset chunk-size=$max as-value]
:set ($var->0) (($var->0) . ($filechunk->"data"))
}
:put [:len ($var->0)]
# 7293564
:serialize $var to=dsv delimiter=";" file-name=output.txt
}
I hope this will be useful for the community.