While waiting for MikroTik to implement the ability to read a file larger than 64KB (64512B) as a value for a variable...
I found a way to download only one piece of file at a time with /tool fetch and put it inside a variable
For example this file is 942KB, too big to put it on one variable to be processed to one script that add to one address-list the IP contained.
https://www.iwik.org/ipcountry/US.cidr
but I can read it like 15 separate files of 64KB each (except the last, obviously) for put inside a variable 64512B byte at a time.
I us the http Range Download method for request, from compatible HTTP(s) site, only a piece of the file, like resumed stopped downloads...
http-header-field="Range: bytes=0-64511"
If I check first the file size with standard fetch, I have how much time I must repeat the fetch for read the file parts inside the variable...
If last specified size exceed the file dimension, do not matter, the website send till the end and not more.
https://httpwg.org/specs/rfc7233.html#r ... -specifier
Not all sites support Range Download, but near all have it..
With this method, instead to save all splited parts to disk, and after read one-by-one,
I can put current part on variable for use inside scripting, and on this way scroll the reading inside the file.
ATTENTION:
Now I download the files, it is for demonstration,
but in reality there is no need to download anything
and in the final import script that I do it simply reads only a piece at once on one variable...
like
Code: Select all
{
:local test ([fetch url="https://www.iwik.org/ipcountry/US.cidr" http-header-field="Range: bytes=0-64511" as-value output=user]->"data")
:put $test
}
I make an "auto count parts" and cycle:
Code: Select all
{
:local url "https://www.iwik.org/ipcountry/US.cidr"
:local filesize ([/tool fetch url=$url as-value output=none]->"downloaded")
# 64512 is the max size of RouterOS text variables.
# To insert the incomplete end of the previous file at the beginning of the next file, reduce the size of each piece accordingly.
:local maxsize 64512
:local start 0
:local end ($maxsize - 1)
:local partnumber ($filesize / ($maxsize / 1024))
:local reminder ($filesize % ($maxsize / 1024))
:if ($reminder > 0) do={ :set partnumber ($partnumber + 1) }
:for x from=1 to=$partnumber step=1 do={
/tool fetch url=$url http-header-field="Range: bytes=$start-$end" keep-result=yes dst-path="/part$x.txt"
:set start ($start + $maxsize)
:set end ($end + $maxsize)
}
}
Obviously some line are splitted on two between two parts like
\n69.100.54.120/19\n\
splitted on two, first read end with:
..............\n69.100.5
second read start with:
4.120/19\n\..............
The solution is simple, instead of read exactly the max 64KB (64512B) make part slightly less like 63K (63488B) to have room on variable to put previous "reminder" on top,
like if "reminder" is "69.100.5" and file part is "4.120/19\n\.............." with
:set varcontent "$reminder$filepart"
is obtained inside varcontent "69.100.54.120/19\n\.............."
and the IP is readable correctly.