Community discussions

MikroTik App
 
User avatar
rextended
Forum Guru
Forum Guru
Topic Author
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

How to download only one piece of file at a time with /tool fetch and put it inside a variable  [SOLVED]

Wed Aug 11, 2021 11:55 pm

search tag # rextended load split large file


While waiting for MikroTik to implement the ability to read a file larger than 64KB (64512B) as a value for a variable...

I found a way to download only one piece of file at a time with /tool fetch and put it inside a variable

For example this file is 942KB, too big to put it on one variable to be processed to one script that add to one address-list the IP contained.
https://www.iwik.org/ipcountry/US.cidr

but I can read it like 15 separate files of 64KB each (except the last, obviously) for put inside a variable 64512B byte at a time.

I us the http Range Download method for request, from compatible HTTP(s) site, only a piece of the file, like resumed stopped downloads...

http-header-field="Range: bytes=0-64511"

If I check first the file size with standard fetch, I have how much time I must repeat the fetch for read the file parts inside the variable...

If last specified size exceed the file dimension, do not matter, the website send till the end and not more.
https://httpwg.org/specs/rfc7233.html#r ... -specifier

Not all sites support Range Download, but near all have it..

With this method, instead to save all splited parts to disk, and after read one-by-one,
I can put current part on variable for use inside scripting, and on this way scroll the reading inside the file.

ATTENTION:
Now I download the files, it is for demonstration,
but in reality there is no need to download anything
and in the final import script that I do it simply reads only a piece at once on one variable...


like
{
:local test ([fetch url="https://www.iwik.org/ipcountry/US.cidr" http-header-field="Range: bytes=0-64511" as-value output=user]->"data")
:put $test
}

I make an "auto count parts" and cycle:
{
    :local url "https://www.iwik.org/ipcountry/US.cidr"
    :local filesize ([/tool fetch url=$url as-value output=none]->"downloaded")
# 64512 is the max size of RouterOS text variables.
# To insert the incomplete end of the previous file at the beginning of the next file, reduce the size of each piece accordingly.
    :local maxsize 64512
    :local start 0
    :local end ($maxsize - 1)
    :local partnumber ($filesize / ($maxsize / 1024))
    :local reminder ($filesize % ($maxsize / 1024))
    :if ($reminder > 0) do={ :set partnumber ($partnumber + 1) }
    :for x from=1 to=$partnumber step=1 do={
         /tool fetch url=$url http-header-field="Range: bytes=$start-$end" keep-result=yes dst-path="/part$x.txt"
         :set start ($start + $maxsize)
         :set end ($end + $maxsize)
    }
}


Obviously some line are splitted on two between two parts like
\n69.100.54.120/19\n\
splitted on two, first read end with:
..............\n69.100.5
second read start with:
4.120/19\n\..............

The solution is simple, instead of read exactly the max 64KB (64512B) make part slightly less like 63K (63488B) to have room on variable to put previous "reminder" on top,
like if "reminder" is "69.100.5" and file part is "4.120/19\n\.............." with
:set varcontent "$reminder$filepart"
is obtained inside varcontent "69.100.54.120/19\n\.............."
and the IP is readable correctly.
Last edited by rextended on Sat Aug 28, 2021 5:34 pm, edited 22 times in total.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: I... I... I... found a way to download only one piece of file at a time with /tool fetch ...

Thu Aug 12, 2021 12:03 am

8) 8) 8)
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: I... I... I... found a way to download only one piece of file at a time with /tool fetch ...

Thu Aug 12, 2021 12:14 am

Hmm, that looks indeed like a very ingenious approach to the problem to have the webserver return what you need and deal with the limitation.
But you cannot guarantee the content returned by the http-server is "cut" at the correct position ? Perhaps some of the file contain "half" a piece of string. That line is lost when you later on
process each of the chuncks and process them in a iteration.
Offcourse "error marging" is extremely low so perhaps 99.99999% good approach :D

Good thinking I would say!
 
User avatar
rextended
Forum Guru
Forum Guru
Topic Author
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: I... I... I... found a way to download only one piece of file at a time with /tool fetch ...

Thu Aug 12, 2021 12:19 am

I have already think about that,

last split "\n" reminder is put at the start before next load, is why I split at 63K and not at 64K limit ;)
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: I... I... I... found a way to download only one piece of file at a time with /tool fetch ...

Thu Aug 12, 2021 12:20 am

So to continue on this approach a script should be made where you can

- set a variable on the TLD's in scope for processing (eg. RU,CN,...)
- have an iterative loop fetch these on a similar approach, saving them like RU_part1 .... RU_partX alongside others CN_part1 .... CN_partX etc.
- have an iterative loop process each of them per TLD, so again iterate through the different TLD_partX files until all are processed and added to the appropriate ACL
- cleanup
- hope your Mikrotik doesn't crash due to memory shortage ;-)
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: I... I... I... found a way to download only one piece of file at a time with /tool fetch ...

Thu Aug 12, 2021 12:21 am

I have already think about that,

last split "\n" reminder is put at the start before next load, is why I split at 63K and not at 64K limit ;)
Clever thinking & anticipating for sure!
The only thing indeed is the server-side support. I guess most will honor that request.

With curl you can easily check if this would work on a specific server

$ curl -i -X HEAD --header "Range: bytes=50-100" http://www.example.com/style.css
HTTP/1.1 206 Partial Content
Date: Wed, 01 Mar 2017 13:58:06 GMT
Server: Apache
Last-Modified: Thu, 03 Nov 2016 15:14:44 GMT
ETag: "a002df-160c-54067aa74702f"
Accept-Ranges: bytes
Content-Length: 51
Vary: Accept-Encoding
Content-Range: bytes 50-100/5644
Content-Type: text/css

...on a server that does not support it, you'll get a OK 200

$ curl -i -X HEAD --header "Range: bytes=50-100" http://example.com/image.png
HTTP/1.1 200 OK
Date: Wed, 01 Mar 2017 12:51:47 GMT
Server: Apache
Content-Length: 42760
Last-Modified: Tue, 28 Feb 2017 15:39:32 GMT
Content-Type: image/png

Don't think you can do this with the fetch-tool and parse if you get a "Accept-Ranges" , so the script could report "Not supported on this server" or something.
 
User avatar
vecernik87
Forum Veteran
Forum Veteran
Posts: 882
Joined: Fri Nov 10, 2017 8:19 am

Re: I... I... I... found a way to download only one piece of file at a time with /tool fetch ...

Thu Aug 12, 2021 4:03 am

Image
Nice. +2 kudos and +5 internetz to you!
 
User avatar
rextended
Forum Guru
Forum Guru
Topic Author
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: How to download only one piece of file at a time with /tool fetch and put it inside a variable

Thu Aug 12, 2021 2:27 pm

Very Thanks!
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19099
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: How to download only one piece of file at a time with /tool fetch and put it inside a variable

Thu Aug 12, 2021 4:59 pm

Very nice 5 course dinner, however I am just making cookies, so I do not see the utility :-)
A practical application set of examples would shine some light on the subject, probably none that would apply for the homeowner but ya never know.
 
User avatar
rextended
Forum Guru
Forum Guru
Topic Author
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: How to download only one piece of file at a time with /tool fetch and put it inside a variable

Thu Aug 12, 2021 5:14 pm

One Example to rule them all:
ad block, country block, malware block...

This can be applied to DNS names, HotSpot WalledGarden, Proxy rules, IP or IP-prefix addresses on address-list, etc.

The limits are moved from size per file to memory size, volatile or not, on RouterBOARD.
Volatile or not because the IP or IP/prefix on address-lsit can also be dynamic, not exported with /export.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: How to download only one piece of file at a time with /tool fetch and put it inside a variable

Sat Aug 21, 2021 12:04 am

@rextended, did you find the time or interest to glue some of the existing scripts together ? To have the ability to use your routines for partial downloads, combined with some generic list-loader ?
 
User avatar
rextended
Forum Guru
Forum Guru
Topic Author
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: How to download only one piece of file at a time with /tool fetch and put it inside a variable

Sat Aug 21, 2021 1:26 am

Yes, Is exactly what I "invented" this method for.

This topic and this:
viewtopic.php?f=9&t=166293#p872049

Are joined together on a project I have to load external (black)list and import on MikroTik, no matter if is a list of DNS, IP or IP/prefix
 
profinet
just joined
Posts: 5
Joined: Mon Apr 23, 2018 1:17 pm

Re: How to download only one piece of file at a time with /tool fetch and put it inside a variable

Sun Aug 22, 2021 12:01 am

Hi,
it's possible somehow connect these two script's? I have a trusted source of "bad" ip address, but file has 350kb.
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
:local array [find dynamic list=blacklist]
:foreach value in=$array do={:set array (array,[get $value address])}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:local ip ([:pick $data 0 [:find $data $newsline]].$cidr)
:do {add list=blacklist address=$ip comment=$description timeout=1d} on-error={
:do {set ($array->([:find $array $ip]-[:len $array]/2)) timeout=1d} on-error={}
}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=https://project.turris.cz/greylist-data/greylist-latest.csv newsline=","
and this?
{
    :local url "https://project.turris.cz/greylist-data/greylist-latest.csv"
    :local filesize ([/tool fetch url=$url keep-result=no as-value]->"total")
    :local maxsize 64512
    :local start 0
    :local end ($maxsize - 1)
    :local partnumber ($filesize / ($maxsize / 1024))
    :local reminder ($filesize % ($maxsize / 1024))
    :if ($reminder > 0) do={ :set partnumber ($partnumber + 1) }
    :for x from=1 to=$partnumber step=1 do={
         /tool fetch url=$url http-header-field="Range: bytes=$start-$end" keep-result=yes dst-path="/part$x.txt"
         :set start ($start + $maxsize)
         :set end ($end + $maxsize)
    }
}
Thank you.
 
User avatar
rextended
Forum Guru
Forum Guru
Topic Author
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: How to download only one piece of file at a time with /tool fetch and put it inside a variable

Sun Aug 22, 2021 2:14 pm

As I write on previous post, just wait i make the full script
 
profinet
just joined
Posts: 5
Joined: Mon Apr 23, 2018 1:17 pm

Re: How to download only one piece of file at a time with /tool fetch and put it inside a variable

Sun Aug 22, 2021 2:22 pm

@rextended Thank you for your work.
Now i'm trying edited script from @msatter viewtopic.php?f=9&t=152632#p873943
Seems been work, but i'm not able added delimiter option for next sources like spamhouse, abuse.ch and next. Maybe now will work blacklist from abuseipdb.com.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: How to download only one piece of file at a time with /tool fetch and put it inside a variable

Sun Aug 22, 2021 8:10 pm

@rextended Thank you for your work.
Now i'm trying edited script from @msatter viewtopic.php?f=9&t=152632#p873943
Seems been work, but i'm not able added delimiter option for next sources like spamhouse, abuse.ch and next. Maybe now will work blacklist from abuseipdb.com.
The script does have a delimiter, just its called variable newsline.
I changed the "," towards "\n" (=newline) and I can for example fetch the country IP-blocks from https://www.iwik.org/ipcountry/

I've tested with the largest, the "US" cidr list which has 61250 entries. Sure this took a very long time to process (as predicted) but it is not that important as these lists are rather "static"
The script in its current form I think does not allow you to provide multiple lines of sources and you must make 1 script for each source. It seems to miss a while/do to accomodate that iteration.
 
hkusulja
Frequent Visitor
Frequent Visitor
Posts: 75
Joined: Fri Apr 13, 2012 1:14 am

Re: How to download only one piece of file at a time with /tool fetch and put it inside a variable

Thu Mar 10, 2022 6:33 pm

Hello,
If somebody here knows, how can I use /tool fetch , to upload file (not bigger than 60 KB) from routeros to remote server via HTTP method?
viewtopic.php?p=915562#p915562
 
User avatar
rextended
Forum Guru
Forum Guru
Topic Author
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: How to download only one piece of file at a time with /tool fetch and put it inside a variable

Wed Apr 20, 2022 12:26 pm

Hello,
If somebody here knows, how can I use /tool fetch , to upload file (not bigger than 60 KB) from routeros to remote server via HTTP method?
viewtopic.php?p=915562#p915562

For that, continue here: viewtopic.php?t=181942#p927815
 
User avatar
rextended
Forum Guru
Forum Guru
Topic Author
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: How to download only one piece of file at a time with /tool fetch and put it inside a variable

Thu Jul 13, 2023 1:27 am

Who is online

Users browsing this forum: alexantao and 28 guests