Is 8MB in a variable from a txt file is possible?

Kataius · Thu Feb 29, 2024 8:34 pm

Hi everyone, I would like to make a script to run once a week that updates address list on Mikrotik.
coming from this topic:

viewtopic.php?p=1051864

I found a very useful script for creating variables.
I'm trying to import a file.txt of 8 MB, only it cuts the list to more or less 3 MB.
Was I wrong or is it not possible?

Thank you.

my script is:

:global thefile ""
{
    :local url        http://public-dns.info/nameservers-all.txt ;
    :local filesize   ([/tool fetch url=$url as-value output=none]->"downloaded")
    :local maxsize    64512 ; # is the maximum supported readable size of a block from a file
    :local start      0
    :local end        ($maxsize - 1)
    :local partnumber ($filesize / ($maxsize / 1024))
    :local reminder   ($filesize % ($maxsize / 1024))
    :if ($reminder > 0) do={ :set partnumber ($partnumber + 1) }
    :for x from=1 to=$partnumber step=1 do={
         :set thefile ($thefile . ([/tool fetch url=$url http-header-field="Range: bytes=$start-$end" as-value output=user]->"data"))
         :set start   ($start + $maxsize)
         :set end     ($end   + $maxsize)
    }
}

#:log info "debug_thefile=$thefile"
#/file remove [find where name="check.txt"];
#:execute ":put \$thefile" file=check.txt;

:global content value=$thefile;
:local contentLen value=[:len $content];
:local lineEnd value=0;
:local line value="";
:local lastEnd value=0;
:local addressListName;
:set addressListName "DNS-DOH";

:if ($thefile != null) do={
  :log info "There are some New DNS"
  /ip firewall address-list remove [/ip firewall address-list find list=$addressListName]
  :do {
      :set lineEnd [:find $content "\n" $lastEnd ] ;
      :set line [:pick $content $lastEnd $lineEnd] ;
      :set lastEnd ( $lineEnd + 1 ) ;
      :local entry [:pick $line 0 $lineEnd ]
      :if ($entry~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
            :if ( [:len $entry ] > 0 ) do={
#:log info "debug_entry=$entry"
                /ip firewall address-list add list=$addressListName address=$entry;
            }
      } 
    } while=($lineEnd < $contentLen);
  } else={
  :log info "There no DNS in list"
}

optio · Fri Mar 01, 2024 12:52 pm

From ROS version 7.13 there is /file/read command (currently undocumented) which you can use to read file chunked with arguments chunk-size and offset. First, fetch command will need to download whole file on to FS and not be used to return response data as value to variable.
This can be useful for parsing text by some delimiter, to read file chunked and hold characters buffer variable until delimiter is reached (assuming that text between delimiters will not reach variable size limit). In your case it can be done since new line is delimiter and lines are not long, but this read feature it still not enough for parsing large meta data files like json (depends on structure, some can be parsed like large arrays of smaller items).

rextended · Mon Mar 04, 2024 1:43 pm

From ROS version 7.13 there is /file/read command (currently undocumented) which you can use to read file chunked with arguments chunk-size and offset. First, fetch command will need to download whole file on to FS and not be used to return response data as value to variable.
This can be useful for parsing text by some delimiter, to read file chunked and hold characters buffer variable until delimiter is reached (assuming that text between delimiters will not reach variable size limit). In your case it can be done since new line is delimiter and lines are not long, but this read feature it still not enough for parsing large meta data files like json (depends on structure, some can be parsed like large arrays of smaller items).

On 7.13.5 do not work, it display it only on terminal, not create any variable or is of any use...

But if on future is fixed, like adding as-value or similar...

actually not working example code

:global readfile do={
    :local url        $1
    :local thefile    ""
    :local filesize   [/file get $1 size]
    :local maxsize    32768 ; # is the actual maximum supported readable size of a block from a file
    :local start      0
    :local end        ($maxsize - 1)
    :local partnumber ($filesize / ($maxsize / 1024))
    :local reminder   ($filesize % ($maxsize / 1024))
    :if ($reminder > 0) do={ :set partnumber ($partnumber + 1) }
    :for x from=1 to=$partnumber step=1 do={
         :set thefile ($thefile . [/file read file=$1 chunk-size=$maxsize offset=$start as-value])
         :set start   ($start + $maxsize)
         :set end     ($end   + $maxsize)
    }
    :return $thefile
}

:local test [$readfile test.txt]

optio · Mon Mar 04, 2024 7:05 pm

I didn't tested it, strange it is not initially implemented to return value as default, there is no much practical use without value. Btw., regarding your function, are you sure this will work on large files and $thefile will not reach variable size limit or maybe device memory limit, since it appends to it bytes from all chunks? I think for parsing purposes bytes buffer variable will need to hold only bytes between delimiters (new line in this case) not all file bytes to avoid memory limits and adding parsed IP into address list should be performed inside chunk iteration when delimiter is reached.

Edit:
@Kataius idk. why I this didn't come to my mind initially, since web server supports chunked download, but this is for ROS 7.14 only (read below why):
You can append to bytes buffer fetched chunk bytes and parse buffer by new line delimiter and also match to IP address as validation (you already have that in script) and add that IP into address list, then remove parsed bytes from buffer - this is to avoid buffer var. size limit when bytes are appended in another iteration (lines are not that long), this all needs to be performed inside fetch chunk loop. First you will need to perform HTTP HEAD method request on URL and read "content-length" header to get full size in bytes which you need to calculate number of chunks, but unfortunately HTTP HEAD method for fetch is available from ROS 7.14 so you will need to run that ROS version if you want to achieve this.
Tip: To properly handle possible fetch connection/server errors while populating address list in chunks loop, better use temporary list and when fetch performed successfully for all chunks then remove all items from actually used address list and items in temp address list put into actual address list. If something failed during chunked fetch just remove temp address list for cleanup and finish operation.

rextended · Mon Mar 04, 2024 9:18 pm

The var size limit is the memory on the device.
I take it for granted that one imports 8 megs of address list into appropriate devices, not into little toys like the hAP lite...

viewtopic.php?p=1012752&hilit=readfile#p1012752

optio · Mon Mar 04, 2024 9:42 pm

Ah ok, just current free memory is limit... but also proposed solution for parsing chunks in prev. post which could work on any device without worrying about free memory, for this case doesn't matter, but for parsing 100Mb file maybe it will and it can be implemented in ROS 7.14 without waiting as-value for /file/read in future version

Kataius · Mon Mar 04, 2024 10:30 pm

The var size limit is the memory on the device.
I take it for granted that one imports 8 megs of address list into appropriate devices, not into little toys like the hAP lite...

viewtopic.php?p=1012752&hilit=readfile#p1012752

I have a RB5009UG+S+IN not a little haP lite

optio · Mon Mar 04, 2024 10:46 pm

Maybe there is some limit even if you have free memory for much more bytes...
Try with my approach mentioned in previous post if you skilled with scripting to implement it, actually you don't need HEAD method first to get file bytes, it is more optimal but it works only on ROS 7.14+, as I see you are downloading full file before chunked fetch to read file size so it is possible to do it with that.

rextended · Tue Mar 05, 2024 9:02 am

I doubt anyone in their right mind imports a 100MB list of IPs to block... at that point is faster to block everything and implement a white list...
If you have to work on files, a container is better, or better yet, an external server...

Kataius · Fri Mar 08, 2024 12:08 am

Hi Here,

I worked on two scripts, where I copied rextended's script (please don't hate me for this).
The logic is:

1 fetch the file in X file splits,

2 imports one by one and then deletes them all.

The problem is that the first script blocks the entire execution of the second script because it gives me 1364.1KiB split and the second script doesn't read this big file.
Is there a possibility to split into smaller files and what is it?

Thank you

FETCH and SPLIT

:local orname;
:set orname "DOH";  
:local url "http://public-dns.info/nameservers-all.txt"
:local filesize ([/tool fetch url=$url as-value output=none]->"downloaded")
# 64512 is the max size of RouterOS text variables.
# To insert the incomplete end of the previous file at the beginning of the next file, reduce the size of each piece accordingly.
:local maxsize 64512
:local start 0
:local end ($maxsize - 1)
:local partnumber ($filesize / ($maxsize / 1024))
:local reminder ($filesize % ($maxsize / 1024))
:if ($reminder > 0) do={ :set partnumber ($partnumber + 1) }
    :for x from=1 to=$partnumber step=1 do={
         /tool fetch url=$url http-header-field="Range: bytes=$start-$end" keep-result=yes dst-path="/$orname$x.txt"
         :set start ($start + $maxsize)
         :set end ($end + $maxsize)
    }

IMPORT AND DELETE

:local part 1
:local input ""
:local done false
:local lineEnd value=0;
:local line value="";
:local lastEnd value=0;
:local addressListName;
:set addressListName "TRY";
:local orname;
:set orname "DOH";


/ip firewall address-list remove [/ip firewall address-list find list=$addressListName]
/file
:while (!$done) do={
  :local nameFile ("$orname".$part.".txt")
  :if ([find where name=$nameFile]) do={
    :set input ($input.[get $nameFile contents])
    :local content value=$input;
    :local contentLen [:len $input];
  :do {
      :set lineEnd [:find $input "\n" $lastEnd ];
              :if ([:len $lineEnd] = 0) do={
            :set lineEnd $contentLen
        }
     :set line [:pick $input $lastEnd $lineEnd];
        :set lastEnd ($lineEnd +1);
      :local entry [:pick $line 0 $lineEnd ];
              : if ($line !="") do={
                :if ($entry~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
            :if ( [:len $entry ] > 0 ) do={
                /ip firewall address-list add list=$addressListName address=$entry;
                  :log info "IMPORT"
                  :delay 2
              }
           }
      } 
    } while=($lineEnd < $contentLen);
    /file remove ("$orname".$part.".txt")
    :set part ($part + 1);
  } else={
    :set done true
  }
}

Kataius · Mon Mar 11, 2024 11:51 pm

Try with one up?

Kataius · Sat Mar 30, 2024 1:28 am

An other up?

optio · Sat Mar 30, 2024 6:53 pm

I suspect there is a bug on http-header-field argument for fetch command, maybe it is not sent correctly (I didn't check this over local web server to analyse), download data is always 64512 bytes regardles which range is set in header when output=user as-value:

:put [:len ([/tool fetch url="https://public-dns.info/nameservers-all.txt" http-header-field="Range: bytes=0-1" output=user as-value]->"data")]
64512

when output=file it downloads whole file:

/tool fetch url="https://public-dns.info/nameservers-all.txt" http-header-field="Range: bytes=0-1" output=file
:delay 3
:put [get "nameservers-all.txt" size]
1396869

This indicates that Range header is not sent correctly.

Using curl is ok:

curl -s -H 'Range: bytes=0-1' https://public-dns.info/nameservers-all.txt | wc -c
2

msatter · Sat Mar 30, 2024 10:57 pm

@optio nothing wrong there, Data is a variable and so 64512 bytes is the max.

I did not succeed in reading chunks from files so I am doing other things in the meantime till it get fixed......some day.

optio · Sat Mar 30, 2024 11:49 pm

@optio nothing wrong there, Data is a variable and so 64512 bytes is the max.

It shouldn't be since range header is set. 64512 is because full data is fetched and limited to that and that's wrong (it should be 2 as from curl).

msatter · Sun Mar 31, 2024 4:03 am

Read it like this:

put [:len ...... ->"data")]

Curl wc is word count not bytes.

Update: ran the line in Terminal and RouterOS always read 64512 bytes and only the first number of 0-1 is used. "downloaded=63;duration=00:00:01;status=finished" and 63 * 1024 = 64512 bytes

RouterOS is short cutting here thinking the data is going to be put in a variable and the max size is 64512 bytes so why not stuff all it in there and the user can then use all, or a part.

optio · Sun Mar 31, 2024 12:08 pm

Read it like this:
Curl wc is word count not bytes.

Maybe you missed -c argument...

$ wc --help
Usage: wc [OPTION]... [FILE]...
  or:  wc [OPTION]... --files0-from=F
Print newline, word, and byte counts for each FILE, and a total line if
more than one FILE is specified.  A word is a non-zero-length sequence of
printable characters delimited by white space.
...
  -c, --bytes            print the byte counts
...

Update: ran the line in Terminal and RouterOS always read 64512 bytes and only the first number of 0-1 is used. "downloaded=63;duration=00:00:01;status=finished" and 63 * 1024 = 64512 bytes

RouterOS is short cutting here thinking the data is going to be put in a variable and the max size is 64512 bytes so why not stuff all it in there and the user can then use all, or a part.

It's simple, download using fetch and curl with same Range header:

:put ([/tool fetch url="https://public-dns.info/nameservers-all.txt" http-header-field="Range: bytes=0-1" output=user as-value]->"data")
2607:5300:203:1797::53
...

64512 is just correct count of wrong data, data should contain only first 2 bytes (characters 2 and 6 in this case) because of Range header and http server will return only these if correctly header is sent, like when using curl:

curl -s -H 'Range: bytes=0-1' https://public-dns.info/nameservers-all.txt
26

jaclaz · Sun Mar 31, 2024 12:32 pm

@optio
Excuse me, but from the results of the experiments you posted it could be that the:
"Range: bytes=0-1"
is read/interpreted as (in pseudocode):
"Range: from_offset=0 length=64512"
(because any length<64512 is "rounded up" to 64512)
or
it is simply ignored (for whatever reasons) and it will always return the first 64512 bytes no matter the values you put in range.

Cannot the first 2 bytes be extracted from the retrieved 64512 in a second operation?

It seems to me that what needs to be checked/tested is whether the "from_offset=0" part is respected/used.
I.e., does (in your command/setup):
"Range: bytes=1-1"
return 64512 bytes of which the first one is 07?

optio · Sun Mar 31, 2024 1:10 pm

This is server side operation for Range header (server returns bytes depending on Range header value), see https://developer.mozilla.org/en-US/doc ... ders/Range
That's is my whole point, yes you can extract from these returned bytes some bytes, but you cannot achieve chunked download of all bytes.
If you download all bytes to file with fetch, then you have issue mentioned here where you cannot use /file/read to read all bytes by chunks into variable, if Range header for fetch is working it will be possible then.

msatter · Sun Mar 31, 2024 1:17 pm

@optio I indeed missed the -c

Then in the end. I now think that it the webserver does not send the number of bytes request because it might not be supporting chunking and so sent all data in one go. Because the data variable only can fit 64512 bytes is that what you get returned by RouterOS. The rest of the file is ignored and my earlier conclusion that it was a kind of sliding window was also wrong.

If chunking is supported on the other end then RouterOS should be working correct.

By the way it is a possible way to test if a webserver is supporting chunking in RouterOS scripting. If you get a len: of 64512 or > then the requested number of bytes +1, then there is no support for chunking available.

CHECK:

:put ([/tool fetch url="https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv" http-header-field="Ran
ge: bytes=0-80" output=user as-value]->"data")   
# For the terms of use see https://view.sentinel.turris.cz/greylist-data/LICENSE.

:put [:len ([/tool fetch url="https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv" http-header-fiel
d="Range: bytes=0-80" output=user as-value]->"data")] 
81

optio · Sun Mar 31, 2024 1:29 pm

Yes, that's strange, I just checked with local http server and indeed it returns ok:

curl -s -H 'Range: bytes=0-1' http://192.168.100.21/info.php
<?

:put ([/tool fetch url="http://192.168.100.21/info.php" http-header-field="Range: bytes=0-1" output=user as-value]->"data")                                            
<?

Apache forensic module log:

+45708:66093837:0|GET /info.php HTTP/1.1|Host:192.168.100.21|User-Agent:curl/8.7.1|Accept:*/*|Range:bytes=0-1
+45709:66093877:0|GET /info.php HTTP/1.1|Accept-Encoding:deflate, gzip|Host:192.168.100.21|Range:bytes=0-1|User-Agent:MikroTik

But it still confusing why with curl is working on server for https://public-dns.info/nameservers-all.txt url, as I see difference is only that fecth sends Accept-Encoding header, but this is it seems server side issue.

Edit:
And there is workaround

:put ([/tool fetch url="https://public-dns.info/nameservers-all.txt" http-header-field="Range: bytes=0-1, Accept-Encoding:" output=user as-value]->"data")  
26

It seems default Accept-Encoding header from fetch is creating issue on server side.

msatter · Sun Mar 31, 2024 1:44 pm

# :put [:len ([/tool fetch url="https://public-dns.info/nameservers-all.txt" http-header-field="Range: bytes=0-1, a
ccept-encoding:" output=user as-value]->"data")]
2
# :put [:len ([/tool fetch url="https://public-dns.info/nameservers-all.txt" http-header-field="Range: bytes=0-1, a
ccept-encoding:compress" output=user as-value]->"data")]
2
# :put [:len ([/tool fetch url="https://public-dns.info/nameservers-all.txt" http-header-field="Range: bytes=0-1, a
ccept-encoding:gzip" output=user as-value]->"data")]        
64512
# :put [:len ([/tool fetch url="https://public-dns.info/nameservers-all.txt" http-header-field="Range: bytes=0-1, a
ccept-encoding:compress gzip" output=user as-value]->"data")]  
64512

# :put [:len ([/tool fetch url="https://public-dns.info/nameservers-all.txt" http-header-field="Range: bytes=0-1, a
ccept-encoding:gzip|deflate" output=user as-value]->"data")] 
2

No gzip support activated on the server?

optio · Sun Mar 31, 2024 2:22 pm

No gzip support activated on the server?

Range on server side is not supported when gzip encoding is set, all gzipped data is retrieved then. This is some issue on Caddy http server, public-dns.info is using Caddy as I see from response headers.

curl -s https://public-dns.info/nameservers-all.txt | wc -c
1396869
curl -s -H 'Range: bytes=0-1' https://public-dns.info/nameservers-all.txt | wc -c
2
curl -s -H 'Accept-Encoding: gzip' https://public-dns.info/nameservers-all.txt | wc -c
517520
curl -s -H 'Accept-Encoding: gzip' -H 'Range: bytes=0-1' https://public-dns.info/nameservers-all.txt | wc -c
517520

msatter · Sun Mar 31, 2024 2:24 pm

Hi Here,

I worked on two scripts, where I copied rextended's script (please don't hate me for this).
The logic is:

1 fetch the file in X file splits,

2 imports one by one and then deletes them all.

The problem is that the first script blocks the entire execution of the second script because it gives me 1364.1KiB split and the second script doesn't read this big file.
Is there a possibility to split into smaller files and what is it?

Thank you

Seems to work now in 7.15Beta4

# /file> read offset=6 chunk-size=42 file=asci.inc 
  data: $ASCI " !\" \$ &'()*+,-./0123456789:;<=>\?

# /file> read offset=0 chunk-size=42 file=asci.inc 
  data:  :set $ASCI " !\" \$ &'()*+,-./0123456789:

optio · Sun Mar 31, 2024 2:54 pm

@Kataius as you can see issue is troubleshooted, override Accept-Encoding header in fetch so that gzip encoding will not be set in it and it should work (it's a server issue).
P.S. use https url scheme since for http server respond with redirect to https, fetch will perform 1 less request.

msatter · Sun Mar 31, 2024 3:49 pm

And reading to file and import from file:

# Turris Import by Blacklister
# 20210823 new version that directly downloads from the external server
# 20240331 rewritten to fetch the whole file and write it to a local file and then import it
{
# import config - delay for slow routers
#:delay 1m
/ip firewall address-list
:local update do={
 :put "Starting import of address-list: $listname"
 :local filesize ([/tool fetch url=$url src-address=10.10.10.10 keep-result=no as-value]->"total")
 :local start 0
 :local maxsize 32767;		# requested chunk size
 :local end ($maxsize - 1);	# because start is zero the maxsize has to be reduced by one
 :local partnumber	($filesize / ($maxsize / 1024)); # how many chunk are maxsize
 :local remainder	($filesize % ($maxsize / 1024)); # the last partly chunk 
 :if ($remainder > 0) do={ :set partnumber ($partnumber + 1) }; # total number of chunks
  :put "Deleting all Dynamic enties in address-list: $listname"
  :if (heirule != null) do={:put "Using as extra filtering: $heirule"}
  :if ($heirule = null) do={:set $heirule "."}
  
 # remove the current list completely
 :do { /ip firewall address-list remove [find where list=$listname dynamic]} on-error={};
 # fetching WHOLE FILE from the webserver
 /tool fetch url=$url dst-path="/$listname.txt"
 :for x from=1 to=$partnumber step=1 do={ :put "Reading Part: $x $start - $end"
   :local data ([:file read offset=$start chunk-size=$maxsize file="$listname.txt" as-value]->"data")
   # Only remove the first line only if you are not at the start of list
   :if ($start > 0) do={:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]}
     :while ([:len $data]!=0) do={ :put $data
       :local line [:pick $data 0 [:find $data "\n"]]; # create only once and checked twice as local variable
       :if ($line~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}" && $line~heirule) do={    
        :do {add list=$listname address=[:pick $data 0 [:find $data $delimiter]] comment=$description timeout=$timeout} on-error={}; # on error avoids any panics        
       }; # if IP address && extra filter if present
      :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]; # removes the just added IP from the data array
      # Cut of the end of the chunks by removing the last lines...very dirty but it works
      :if ([:len $data] < 256) do={:set data [:toarray ""]}    
     }; # while

  :set start (($start-512) + $maxsize); # shifts the subquential starts back with 512
  :set end (($end-512) + $maxsize); # shift the subquential ends back by 512 to keep the 
 }; #do for x
 :put "Completed importing $listname."
}; # do
$update url=https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv listname=turris delimiter=, timeout=8d heirule=http
}

Starting import of address-list: turris
Deleting all Dynamic enties in address-list: turris
Using as extra filtering: http
      status: finished
  downloaded: 258KiBz pause]
       total: 258KiB
    duration: 1s

Reading Part: 1 0 - 32766
Reading Part: 2 32255 - 65021
Reading Part: 3 64510 - 97276
Reading Part: 4 96765 - 129531
Reading Part: 5 129020 - 161786
Reading Part: 6 161275 - 194041
Reading Part: 7 193530 - 226296
Reading Part: 8 225785 - 258551
Reading Part: 9 258040 - 290806
Completed importing turris.

optio · Sun Mar 31, 2024 4:04 pm

Yes, file chunk read to variable works on ROS 7.14:

:local data ([/file read offset=0 chunk-size=2 file=nameservers-all.txt as-value]->"data")
:put $data
26

It depends which ROS version @Kataius is using, more optimal is to download whole file and parse its data by chunks than using multiple requests for chunks.

MTNick · Sun Mar 31, 2024 5:15 pm

Greetings msatter. I tried the script above but it fails & returns the error below:
Download from view.sentinel.turris.cz FAILED: could not bind: Address not available

Same with testing spamhaus:
Download from www.spamhaus.org FAILED: could not bind: Address not available

Address resolves & is alive:
ping www.spamhaus.org
PING www.spamhaus.org.cdn.cloudflare.net (104.16.198.238): 56 data bytes
64 bytes from 104.16.198.238: icmp_seq=0 ttl=56 time=16.672 ms
64 bytes from 104.16.198.238: icmp_seq=1 ttl=56 time=14.764 ms
64 bytes from 104.16.198.238: icmp_seq=2 ttl=56 time=14.971 ms

And reading to file and import from file:

# Turris Import by Blacklister
# 20210823 new version that directly downloads from the external server
# 20240331 rewritten to fetch the whole file and write it to a local file and then import it
{
# import config - delay for slow routers
#:delay 1m
/ip firewall address-list
:local update do={
 :put "Starting import of address-list: $listname"
 :local filesize ([/tool fetch url=$url src-address=10.10.10.10 keep-result=no as-value]->"total")
 :local start 0
 :local maxsize 32767;		# requested chunk size
 :local end ($maxsize - 1);	# because start is zero the maxsize has to be reduced by one
 :local partnumber	($filesize / ($maxsize / 1024)); # how many chunk are maxsize
 :local remainder	($filesize % ($maxsize / 1024)); # the last partly chunk 
 :if ($remainder > 0) do={ :set partnumber ($partnumber + 1) }; # total number of chunks
  :put "Deleting all Dynamic enties in address-list: $listname"
  :if (heirule != null) do={:put "Using as extra filtering: $heirule"}
  :if ($heirule = null) do={:set $heirule "."}
  
 # remove the current list completely
 :do { /ip firewall address-list remove [find where list=$listname dynamic]} on-error={};
 # fetching WHOLE FILE from the webserver
 /tool fetch url=$url dst-path="/$listname.txt"
 :for x from=1 to=$partnumber step=1 do={ :put "Reading Part: $x $start - $end"
   :local data ([:file read offset=$start chunk-size=$maxsize file="$listname.txt" as-value]->"data")
   # Only remove the first line only if you are not at the start of list
   :if ($start > 0) do={:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]}
     :while ([:len $data]!=0) do={ :put $data
       :local line [:pick $data 0 [:find $data "\n"]]; # create only once and checked twice as local variable
       :if ($line~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}" && $line~heirule) do={    
        :do {add list=$listname address=[:pick $data 0 [:find $data $delimiter]] comment=$description timeout=$timeout} on-error={}; # on error avoids any panics        
       }; # if IP address && extra filter if present
      :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]; # removes the just added IP from the data array
      # Cut of the end of the chunks by removing the last lines...very dirty but it works
      :if ([:len $data] < 256) do={:set data [:toarray ""]}    
     }; # while

  :set start (($start-512) + $maxsize); # shifts the subquential starts back with 512
  :set end (($end-512) + $maxsize); # shift the subquential ends back by 512 to keep the 
 }; #do for x
 :put "Completed importing $listname."
}; # do
$update url=https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv listname=turris delimiter=, timeout=8d heirule=http
}

Starting import of address-list: turris
Deleting all Dynamic enties in address-list: turris
Using as extra filtering: http
      status: finished
  downloaded: 258KiBz pause]
       total: 258KiB
    duration: 1s

Reading Part: 1 0 - 32766
Reading Part: 2 32255 - 65021
Reading Part: 3 64510 - 97276
Reading Part: 4 96765 - 129531
Reading Part: 5 129020 - 161786
Reading Part: 6 161275 - 194041
Reading Part: 7 193530 - 226296
Reading Part: 8 225785 - 258551
Reading Part: 9 258040 - 290806
Completed importing turris.

optio · Sun Mar 31, 2024 5:19 pm

maybe src-address argument for fetch is causing the problem?

P.S.
I think

:local filesize ([/tool fetch url=$url src-address=10.10.10.10 keep-result=no as-value]->"total")

is not needed, you can get file size from downloaded file, it will avoid double fetching of same content.

:local filesize [/file get "$listname.txt" size]

MTNick · Sun Mar 31, 2024 5:22 pm

Greetings msatter. It works now. The issue was a source address in the script. Removed it. But logging only logs the download a s finished, nothing else. See screenshot below.

RB5009
ROS 7.14.2

Removed src-address from script. Now works
:local filesize ([/tool fetch url=$url src-address=10.10.10.10 keep-result=no as-value]->"total")

Screen Shot 2024-03-31 at 10.20.52 AM.png

MTNick · Sun Mar 31, 2024 5:24 pm

Greeting optio. Yep, that's what it was. Caught it after I posted. Stupid me didn't notice it until a second look lol. Thank you for the quick catch & help

optio · Sun Mar 31, 2024 5:28 pm

I edited above comment meanwhile, this request can be avoided. Order of commands needs to be adapted to get filesize after file is fetched and set other variables and put rest of the code below it which depends on filesize.

MTNick · Sun Mar 31, 2024 5:40 pm

Greetings optio. I changed the line as you suggested. It downloaded it once, as you indicated it will do & avoid another download for file size. But logging, is non existent. Only tell me that download is finished. It also seems to keep the file on the router. It doesn't delete it. Is that intentional?

For logging, I took the below out of msatter's previous script & the logging started to work. It's missing throughout the script.
:if ($nolog = null) do={:log warning "Starting import of address-list: $listname"}

After comparing the 2 scripts, here's what I came up with & logging works good:

# Turris Import by Blacklister
# 20210823 new version that directly downloads from the external server
# 20240331 rewritten to fetch the whole file and write it to a local file and then import it
{
# import config - delay for slow routers
#:delay 1m
/ip firewall address-list
:local update do={
 :put "Starting import of address-list: $listname"
 :log warning "Starting import of address-list: $listname"
 :local filesize ([/tool fetch url=$url keep-result=no as-value]->"total")
 :local start 0
 :local maxsize 32767;		# requested chunk size
 :local end ($maxsize - 1);	# because start is zero the maxsize has to be reduced by one
 :local partnumber	($filesize / ($maxsize / 1024)); # how many chunk are maxsize
 :local remainder	($filesize % ($maxsize / 1024)); # the last partly chunk 
 :if ($remainder > 0) do={ :set partnumber ($partnumber + 1) }; # total number of chunks
  :put "Deleting all Dynamic enties in address-list: $listname"
  :log warning ("Conditional deleting all".$dynamic." entries in address-list: $listname")
  :if (heirule != null) do={:put "Using as extra filtering: $heirule"}
  :if ($heirule = null) do={:set $heirule "."}
  
# remove the current list completely
 :do { /ip firewall address-list remove [find where list=$listname dynamic]} on-error={};
 # fetching WHOLE FILE from the webserver
 /tool fetch url=$url dst-path="/$listname.txt"
 :for x from=1 to=$partnumber step=1 do={ [:put "Reading Part: $x $start - $end"] [:log warning "Reading Part: $x $start - $end"]
   :local data ([:file read offset=$start chunk-size=$maxsize file="$listname.txt" as-value]->"data")
   # Only remove the first line only if you are not at the start of list
   :if ($start > 0) do={:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]}
     :while ([:len $data]!=0) do={ :put $data
       :local line [:pick $data 0 [:find $data "\n"]]; # create only once and checked twice as local variable
       :if ($line~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}" && $line~heirule) do={    
        :do {add list=$listname address=[:pick $data 0 [:find $data $delimiter]] comment=$description timeout=$timeout} on-error={}; # on error avoids any panics        
       }; # if IP address && extra filter if present
      :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]; # removes the just added IP from the data array
      # Cut of the end of the chunks by removing the last lines...very dirty but it works
      :if ([:len $data] < 256) do={:set data [:toarray ""]}    
     }; # while

  :set start (($start-512) + $maxsize); # shifts the subquential starts back with 512
  :set end (($end-512) + $maxsize); # shift the subquential ends back by 512 to keep the 
 }; #do for x
 :put "Completed importing $listname."
 :log warning "Completed importing $listname."
}; # do
$update url=https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv listname=turris delimiter=, timeout=8d heirule=http
}

Screen Shot 2024-03-31 at 11.36.56 AM.png

optio · Sun Mar 31, 2024 6:40 pm

Script needed some adjustments for single fetch, try with this:

# Turris Import by Blacklister
# 20210823 new version that directly downloads from the external server
# 20240331 rewritten to fetch the whole file and write it to a local file and then import it
{
# import config - delay for slow routers
#:delay 1m
/ip firewall address-list
:local update do={
 :put "Starting import of address-list: $listname"
 /tool fetch url=$url dst-path="/$listname.txt" as-value
 # delay to wait file flush after fetch
 :delay 1
 :local filesize [/file get "$listname.txt" size]
 :local start 0
 :local maxsize 32767;		# requested chunk size
 :local partnumber	($filesize / $maxsize); # how many chunk are maxsize
 :local remainder	($filesize % $maxsize); # the last partly chunk
 :if ($remainder > 0) do={ :set partnumber ($partnumber + 1) }; # total number of chunks
 :put "Deleting all Dynamic enties in address-list: $listname"
 :if (heirule != null) do={:put "Using as extra filtering: $heirule"}
 :if ($heirule = null) do={:set $heirule "."}

 # remove the current list completely
 :do {remove [find where list=$listname dynamic]} on-error={};

 :for x from=1 to=$partnumber step=1 do={
   :local data ([:file read offset=$start chunk-size=$maxsize file="$listname.txt" as-value]->"data")
   # Only remove the first line only if you are not at the start of list
   :if ($start > 0) do={:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]}
   :while ([:len $data]!=0) do={
     :local line [:pick $data 0 [:find $data "\n"]]; # create only once and checked twice as local variable
     :if ($line~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}" && $line~heirule) do={
       :local addr [:pick $data 0 [:find $data $delimiter]]
       :put "Adding address: $addr"
       :do {add list=$listname address=$addr comment=$description timeout=$timeout} on-error={}; # on error avoids any panics
     }; # if IP address && extra filter if present
     :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]; # removes the just added IP from the data array
   }; # while

   :set start ($start + $maxsize)
 }; #do for x
 /file remove "$listname.txt"
 :put "Completed importing $listname."
}; # do
$update url=https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv listname=turris delimiter=, timeout=8d heirule=http
}

MTNick · Sun Mar 31, 2024 7:01 pm

Greetings optio. The only thing the logging is missing is the total count of the imported IP addresses per list. How can I add the total count imported? The script will have more lists. I have quite a few lists actually, totaling around 80k addresses

optio · Sun Mar 31, 2024 7:30 pm

In update function after partnumbers :for loop use:

:local listCount [:len [find list=$listname dynamic]]

And log listCount variable or use as return value of function and use it after function call...

Examples:

...
:local update do={
...
 :local listCount [:len [find list=$listname dynamic]]
 :put "Completed importing $listname, added addresses count: $listCount"
}; # do
...

or

...
:local update do={
...
 :put "Completed importing $listname."
 :return [:len [find list=$listname dynamic]]
}; # do
:local turrisCount [$update url=https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv listname=turris delimiter=, timeout=8d heirule=http]
...
# Use turrisCount var for log, sum with other lists count for overall count log, etc...

msatter · Sun Mar 31, 2024 8:24 pm

Sorry for still having the src-address-10.10.10.10 in the code, I thought I had removed it. It was more a proof-of-concept than actual production code.

Removing the file after download should be optional and the file could then be reused on reboot, even before the connection comes up.

optio · Sun Mar 31, 2024 8:58 pm

Removing the file after download should be optional and the file could then be reused on reboot, even before the connection comes up.

Depends on usage, in this case if in current script

/file remove "$listname.txt"

is just removed it will not work anyway, update function needs to be modified to have argumet for skip fetching (or error ignored with do {} on-error={}) because fetch will fail if there is no connection, also file should not be downloaded in root on some devices with 16MB flash because it will not exist after reboot and if file exists check needs to be added before parsing to avoid other errors if script is executed when file is not downloaded (or deleted) and there is no internet connection

Improvements can be made to make script more robust.

MTNick · Sun Mar 31, 2024 10:26 pm

Greetings optio. I added the code you provided to the "delete addresses" & "importing addresses. Works great. Very happy here. Thank you so much for helping me out!

@msatter. No need to apologize. It's my fault for not looking through the script thoroughly. What you & rextended have done with these scripts in the last fews years is amazing. A huge Thank You for the amazing work & progress!

Edit: My original gripe, spamhaus still won't work. Deletes the list but then fails. tried several delimiters. No luck

Edited script & logging screenshot is below:

# Turris Import by Blacklister
# 20210823 new version that directly downloads from the external server
# 20240331 rewritten to fetch the whole file and write it to a local file and then import it
{
# import config - delay for slow routers
#:delay 1m
:log warning "Blocker script is running..."
/ip firewall address-list
:local update do={
 :put "Starting import of address-list: $listname"
 :log warning "Starting import of address-list: $listname"
 :local filesize ([/tool fetch url=$url keep-result=no as-value]->"total")
 :local start 0
 :local maxsize 32767;		# requested chunk size
 :local end ($maxsize - 1);	# because start is zero the maxsize has to be reduced by one
 :local partnumber	($filesize / ($maxsize / 1024)); # how many chunk are maxsize
 :local remainder	($filesize % ($maxsize / 1024)); # the last partly chunk 
 :if ($remainder > 0) do={ :set partnumber ($partnumber + 1) }; # total number of chunks
  :put "Deleting all Dynamic entries in address-list: $listname"
  :log warning "Deleting all Dynamic entries in address-list: $listname"
  :local listCount [:len [find list=$listname dynamic]]
  :put "Completed deleting $listname, added addresses count: $listCount"
  :log warning "Completed deleting $listname, deleted addresses count: $listCount"
  :if (heirule != null) do={:put "Using as extra filtering: $heirule"}
  :if ($heirule = null) do={:set $heirule "."}
  
 # remove the current list completely
 :do { /ip firewall address-list remove [find where list=$listname dynamic]} on-error={};
 # fetching WHOLE FILE from the webserver
 /tool fetch url=$url dst-path="/$listname.txt"
 :for x from=1 to=$partnumber step=1 do={ [:put "Reading Part: $x $start - $end"] [:log warning "Reading Part: $x $start - $end"]
   :local data ([:file read offset=$start chunk-size=$maxsize file="$listname.txt" as-value]->"data")
   # Only remove the first line only if you are not at the start of list
   :if ($start > 0) do={:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]}
     :while ([:len $data]!=0) do={ :put $data
       :local line [:pick $data 0 [:find $data "\n"]]; # create only once and checked twice as local variable
       :if ($line~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}" && $line~heirule) do={    
        :do {add list=$listname address=[:pick $data 0 [:find $data $delimiter]] comment=$description timeout=$timeout} on-error={}; # on error avoids any panics        
       }; # if IP address && extra filter if present
      :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]; # removes the just added IP from the data array
      # Cut of the end of the chunks by removing the last lines...very dirty but it works
      :if ([:len $data] < 256) do={:set data [:toarray ""]}    
     }; # while

  :set start (($start-512) + $maxsize); # shifts the subquential starts back with 512
  :set end (($end-512) + $maxsize); # shift the subquential ends back by 512 to keep the 
 }; #do for x
 :put "Completed importing $listname."
 :local listCount [:len [find list=$listname dynamic]]
 :put "Completed importing $listname, added addresses count: $listCount"
 :log warning "Completed importing $listname, added addresses count: $listCount"
}; # do
$update url=https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv listname=turris delimiter=, timeout=8d heirule=http
}
:log warning "Blocker script is COMPLETE"

Screen Shot 2024-03-31 at 3.17.16 PM.png

optio · Sun Mar 31, 2024 11:48 pm

Did you check that script is actually working correctly? Try to find all matching lines from downloaded file with regex ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}.*http.*$ and you will see that 2935 lines are matching for import and from your log count is 2934, as I analized last matching line is skipped - 87.236.176.70,"ftp,http,smtp,telnet". Also it is adding duplicates, but that is not an issue for address list becauses addresses are unique per list, just is not optimized.

msatter · Mon Apr 01, 2024 1:04 am

@optio you really need to cut the chunks because you don't know if you get chunk-cuts in the middle of the IP address. So I re-introduced the cut-back of 512 bytes and early ends of 256 bytes in $data. To read the complete list I have added that the last chunk does not end early, when 256 bytes are left over.

# Turris Import by Blacklister and edited by Optio
# 20210823 new version that directly downloads from the external server
# 20240331 rewritten to fetch the whole file and write it to a local file and then import it
# 20240401 avoiding perfect storm by reducing chunkSize when calculation the remainder
{
# import config - delay for slow routers
#:delay 1m
/ip firewall address-list
:local update do={
 :put "Starting import of address-list: $listname"
 /tool fetch url=$url dst-path="/$listname.txt" as-value
 # delay to wait file flush after fetch
 :delay 1
 :local filesize [/file get "$listname.txt" size]
 :local start 0
 :local chunkSize 32767;		# requested chunk size
 :local partnumber	($filesize / $chunkSize); # how many chunk are chunkSize
 :local remainder	($filesize % ($chunkSize-512)); # the last partly chunk and use reduced chunkSize
 :if ($remainder > 0) do={ :set partnumber ($partnumber + 1) }; # total number of chunks
 :put "Deleting all Dynamic enties in address-list: $listname"
 :if (heirule != null) do={:put "Using as extra filtering: $heirule"}
 :if ($heirule = null) do={:set $heirule "."}

 # remove the current list completely
 :do {remove [find where list=$listname dynamic]} on-error={};

 :for x from=1 to=$partnumber step=1 do={
   :local data ([:file read offset=$start chunk-size=$chunkSize file="$listname.txt" as-value]->"data")
   # Only remove the first line only if you are not at the start of list
   :if ($start > 0) do={:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]}
   :while ([:len $data]!=0) do={
     :local line [:pick $data 0 [:find $data "\n"]]; # create only once and checked twice as local variable
     :if ($line~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}" && $line~heirule) do={
       :local addr [:pick $data 0 [:find $data $delimiter]]
       #:put "Adding address: $addr"
       :do {add list=$listname address=$addr comment=$description timeout=$timeout} on-error={}; # on error avoids any panics
     }; # if IP address && extra filter if present
     :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]; # removes the just added IP from the data array
     # Cut of the end of the chunks by removing the last lines...very dirty but it works
     :if (([:len $data] < 256) && (x < $partnumber)) do={:set data [:toarray ""]}   
   }; # while

   #:set start ($start + $chunkSize)
   :set start (($start-512) + $chunkSize); # shifts the subquential starts back with 512
   
 }; #do for x
 /file remove "$listname.txt"
 :put "Completed importing $listname."
}; # do
$update url=https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv listname=turris delimiter=, timeout=8d heirule=http
$update url=https://www.spamhaus.org/drop/drop.txt delimiter=("\_") listname=z-blocklist-SpamHaus timeout=2d
$update url=https://www.spamhaus.org/drop/edrop.txt delimiter=("\_") listname=z-blocklist-SpamHaus-edrop timeout=2d
}

@MTNick from the spamhaus topic you need the delimiter=("\_")
If you have the link for me then I could have a look at it?

MTNick · Mon Apr 01, 2024 1:38 am

@MTNick from the spamhaus topic you need the delimiter=("\_")
If you have the link for me then I could have a look at it?

Greetings msatter. I have tried that delimiter, several of them, including not listing one. Neither work. Below are the spamhaus links:
$update url="https://' . "www.spamhaus.org/drop/drop.txt" delimiter=("\_") listname=z-blocklist-SpamHaus timeout=2d
$update url="https://" . "www.spamhaus.org/drop/edrop.txt" delimiter=("\_") listname=z-blocklist-SpamHaus-edrop timeout=2d

**** added " . " to the links for forum formatting purposes ****

MTNick · Mon Apr 01, 2024 1:56 am

Did you check that script is actually working correctly? Try to find all matching lines from downloaded file with regex ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}.*http.*$ and you will see that 2935 lines are matching for import and from your log count is 2934, as I analized last matching line is skipped - 87.236.176.70,"ftp,http,smtp,telnet". Also it is adding duplicates, but that is not an issue for address list becauses addresses are unique per list, just is not optimized.

@optio You were right. I assumed it was working right. Said it deleted 2934 & entered 2934. Running the new script from msatter above, added more addresses. So, it wasn't downloading all of them as you said. See the difference in previous download vs new script download. Deleted 3834 & added 4007:

Screen Shot 2024-03-31 at 6.53.26 PM.png

optio · Mon Apr 01, 2024 2:26 am

@optio you really need to cut the chunks because you don't know if you get chunk-cuts in the middle of the IP address. So I re-introduced the cut-back of 512 bytes and early ends of 256 bytes in $data. To read the complete list I have added that the last chunk does not end early, when 256 bytes are left over.

It can be done with additional data buffer variable which holds bytes from previevious chunk which are left after last new line character, new data chunk needs to be apended on that data buffer, parsed by lines and matching by regex (single, since in script two are used on same line, regex is expensive for cpu, better to use single string pattern than combining multiple with &&) and then bytes removed up to (and along with) last new line character. No need to seek offset backward and multiple parsing of same data will be avoided.

msatter · Mon Apr 01, 2024 12:50 pm

@MTNick from the spamhaus topic you need the delimiter=("\_")
If you have the link for me then I could have a look at it?
Greetings msatter. I have tried that delimiter, several of them, including not listing one. Neither work. Below are the spamhaus links:
$update url="https://' . "www.spamhaus.org/drop/drop.txt" delimiter=("\_") listname=z-blocklist-SpamHaus timeout=2d
$update url="https://" . "www.spamhaus.org/drop/edrop.txt" delimiter=("\_") listname=z-blocklist-SpamHaus-edrop timeout=2d

**** added " . " to the links for forum formatting purposes ****

#$update url=https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv listname=turris delimiter=, timeout=8d heirule=http
$update url=https://www.spamhaus.org/drop/drop.txt delimiter=("\_") listname=z-blocklist-SpamHaus timeout=2d
$update url=https://www.spamhaus.org/drop/edrop.txt delimiter=("\_") listname=z-blocklist-SpamHaus-edrop timeout=2d

I don't find any problems importing SpamHaus:
z-blocklist-SpamHaus -> 1008 entries
z-blocklist-SpamHaus-edrop -> 363 entries

msatter · Mon Apr 01, 2024 1:03 pm

It can be done with additional data buffer variable which holds bytes from previevious chunk which are left after last new line character, new data chunk needs to be apended on that data buffer, parsed by lines and matching by regex (single, since in script two are used on same line, regex is expensive for cpu, better to use single string pattern than combining multiple with &&) and then bytes removed up to (and along with) last new line character. No need to seek offset backward and multiple parsing of same data will be avoided.

I still like the quick-and-dirty method I introduced a few years ago. It is simple and with the omitting of the last early-end it also reads the last line for sure.

Then going back 512 bytes could create a problem with a file that is using all the chunks fully.

 :local chunkSize 32767;		# requested chunk size
 :local partnumber	($filesize / $chunkSize); # how many chunk are chunkSize
 :local remainder	($filesize % ($chunkSize - 512); # the last partly chunk
 :if ($remainder > 0) do={ :set partnumber ($partnumber + 1) }; # total number of chunks

Each subsequent chunk slides back 512 bytes, then also reduce the chunkSize with 512 for calculating the remainder. This should omit the risk of a perfect storm happening.

MTNick · Mon Apr 01, 2024 6:25 pm

Greetings msatter. I have tried that delimiter, several of them, including not listing one. Neither work. Below are the spamhaus links:
$update url="https://' . "www.spamhaus.org/drop/drop.txt" delimiter=("\_") listname=z-blocklist-SpamHaus timeout=2d
$update url="https://" . "www.spamhaus.org/drop/edrop.txt" delimiter=("\_") listname=z-blocklist-SpamHaus-edrop timeout=2d

**** added " . " to the links for forum formatting purposes ****
Code: Select all
#$update url=https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv listname=turris delimiter=, timeout=8d heirule=http
$update url=https://www.spamhaus.org/drop/drop.txt delimiter=("\_") listname=z-blocklist-SpamHaus timeout=2d
$update url=https://www.spamhaus.org/drop/edrop.txt delimiter=("\_") listname=z-blocklist-SpamHaus-edrop timeout=2d
I don't find any problems importing SpamHaus:
z-blocklist-SpamHaus -> 1008 entries
z-blocklist-SpamHaus-edrop -> 363 entries

Greetings msatter. Spamhaus is working great! Thank you!

However, Sentinel (turris) is not importing all of the addresses. If you look at the csv file, there are 10100 addresses. 4007 were deleted & only 2869 were imported. See screenshot below

Screenshot 2024-04-01 at 11.20.48 AM.png

msatter · Mon Apr 01, 2024 6:56 pm

@MTNick that is correct, because of the heirule=http. Only importing lines containing "http" in the word(s) after each line.

If you want to import all, then omit the heirule. This way you can create different address-lists from the same source file.

MTNick · Mon Apr 01, 2024 7:13 pm

@MTNick that is correct, because of the heirule=http. Only importing lines containing "http" in the word(s) after each line.

If you want to import all, then omit the heirule. This way you can create different address-lists from the same source file.

msatter, Thank you for the explanation of the heirule. Appreciate that. Everything is working as expected. Thank you so much!! Also a Thank You to optio as well. Appreciate the assistance from both of you!

I've added the logging to the script, with a few additions. The code for logging was provided by optio. These are label with ":log warning" added mostly right below the ":put" lines

# Turris Import by Blacklister and edited by Optio
# 20210823 new version that directly downloads from the external server
# 20240331 rewritten to fetch the whole file and write it to a local file and then import it
# 20240401 avoiding perfect storm by reducing chunkSize when calculation the remainder
{
# import config - delay for slow routers
:delay 1m
:log warning "IP-Blocker script is running..."
/ip firewall address-list
:local update do={
 :put "Starting import of address-list: $listname"
 :log warning "Starting import of address-list: $listname"
 /tool fetch url=$url dst-path="/$listname.txt" as-value
 # delay to wait file flush after fetch
 :delay 1
 :local filesize [/file get "$listname.txt" size]
 :local start 0
 :local chunkSize 32767;		# requested chunk size
 :local partnumber	($filesize / $chunkSize); # how many chunk are chunkSize
 :local remainder	($filesize % ($chunkSize-512)); # the last partly chunk and use reduced chunkSize
 :if ($remainder > 0) do={ :set partnumber ($partnumber + 1) }; # total number of chunks
 :put "Deleting all Dynamic enties in address-list: $listname"
 :log warning "Deleting all Dynamic entries in address-list: $listname"
 :local listCount [:len [find list=$listname dynamic]]
 :put "Completed deleting $listname, added addresses count: $listCount"
 :log warning "Completed deleting $listname, deleted addresses count: $listCount"
 :if (heirule != null) do={:put "Using as extra filtering: $heirule"}
 :if ($heirule = null) do={:set $heirule "."}

 # remove the current list completely
 :do {remove [find where list=$listname dynamic]} on-error={};

 :for x from=1 to=$partnumber step=1 do={
   :local data ([:file read offset=$start chunk-size=$chunkSize file="$listname.txt" as-value]->"data")
   # Only remove the first line only if you are not at the start of list
   :if ($start > 0) do={:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]}
   :while ([:len $data]!=0) do={
     :local line [:pick $data 0 [:find $data "\n"]]; # create only once and checked twice as local variable
     :if ($line~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}" && $line~heirule) do={
       :local addr [:pick $data 0 [:find $data $delimiter]]
       #:put "Adding address: $addr"
       :do {add list=$listname address=$addr comment=$description timeout=$timeout} on-error={}; # on error avoids any panics
     }; # if IP address && extra filter if present
     :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]; # removes the just added IP from the data array
     # Cut of the end of the chunks by removing the last lines...very dirty but it works
     :if (([:len $data] < 256) && (x < $partnumber)) do={:set data [:toarray ""]}   
   }; # while

   #:set start ($start + $chunkSize)
   :set start (($start-512) + $chunkSize); # shifts the subquential starts back with 512
   
 }; #do for x
 /file remove "$listname.txt"
 :put "Completed importing $listname."
 :local listCount [:len [find list=$listname dynamic]]
 :put "Completed importing $listname, added addresses count: $listCount"
 :log warning "Completed importing $listname, added addresses count: $listCount"
 :put "Completed delete of downloaded file $listname"
 :log warning "Completed delete of downloaded file $listname"
}; # do
$update url=https://iplists.firehol.org/files/firehol_level2.netset delimiter=("\n") listname=z-blocklist-FireHOL-L2 timeout=3d
$update url=https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv listname=z-blocklist-Sentinel delimiter=, timeout=3d heirule=http
$update url=https://www.spamhaus.org/drop/drop.txt delimiter=("\_") listname=z-blocklist-SpamHaus timeout=3d
$update url=https://www.spamhaus.org/drop/edrop.txt delimiter=("\_") listname=z-blocklist-SpamHaus-edrop timeout=3d
}
:log warning message="IP-Blocker script is COMPLETE"

msatter · Tue Apr 02, 2024 12:13 am

@MTNick optimized logging, and also the displayed text when running in terminal.

Updated: shortend the time a list is unavailable on import by generating a temporary list and then after remove of the old one replacing it with temporary one containing the updated list. Advantage is that the old list is still active during reading the file and while it is being imported.

# Turris Import by Blacklister and edited by Optio
# 20210823 new version that directly downloads from the external server
# 20240331 rewritten to fetch the whole file and write it to a local file and then import it
# 20240401 avoiding perfect storm by reducing chunkSize when calculation the remainder
# 20240402 adding importing new address to temporary list and swap them out with the active list avoiding the list being not active for a short time as possible
# also save and display a count of static addresses present in a address-list
{
# import config - delay for slow routers
#:delay 1m
:log warning "IP-Blocker script started"
/ip firewall address-list
:local update do={
 
 :if (heirule != null) do={:set $filtering ", filtering on: $heirule"}
 :put "Start importing address-list: $listname$filtering"
 :log warning "Start importing address-list: $listname$filtering"
 
 /tool fetch url=$url dst-path="/$listname.txt" as-value
 # delay to wait file flush after fetch
 :delay 1
 :local filesize [/file get "$listname.txt" size]
 :local start 0
 :local chunkSize 32767;		# requested chunk size
 :local partnumber	($filesize / $chunkSize); # how many chunk are chunkSize
 :local remainder	($filesize % ($chunkSize-512)); # the last partly chunk and use reduced chunkSize
 :if ($remainder > 0) do={ :set partnumber ($partnumber + 1) }; # total number of chunks
 
 :local listCount [:len [find list=$listname dynamic]]
 
 :put "Deleting $listCount entries (dynamic) from address-list: $listname"
 :log warning "Deleting $listCount entries (dynamic) from address-list: $listname"

 :if ($heirule = null) do={:set $heirule "."}

 # remove the current dynamic entries completely
 #:do {remove [find where list=$listname dynamic]} on-error={};
 
 :set $listnameTemp ($listname."temp")
 
 :for x from=1 to=$partnumber step=1 do={
   :local data ([:file read offset=$start chunk-size=$chunkSize file="$listname.txt" as-value]->"data")
   # Only remove the first line only if you are not at the start of list
   :if ($start > 0) do={:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]}
   :while ([:len $data]!=0) do={
     :local line [:pick $data 0 [:find $data "\n"]]; # create only once and checked twice as local variable
     :if ($line~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}" && $line~heirule) do={
       :local addr [:pick $data 0 [:find $data $delimiter]]
       :do {add list=$listnameTemp address=$addr comment=$description timeout=$timeout} on-error={}; # on error avoids any panics
     }; # if IP address && extra filter if present
     :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]; # removes the just added IP from the data array
     # Cut of the end of the chunks by removing the last lines...very dirty but it works
     :if (([:len $data] < 256) && (x < $partnumber)) do={:set data [:toarray ""]}   
   }; # while

   #:set start ($start + $chunkSize)
   :set start (($start-512) + $chunkSize); # shifts the subquential starts back with 512
 }; #do for x
 
  /file remove "$listname.txt"
  :put "Deleted downloaded file: $listname.txt"
  :log warning "Deleted downloaded file: $listname.txt"
 
 # Swap out temp list and active list, shorten the time the list is empty
 :do {set list=$listnameTemp [find list=$listname !dynamic]}; # backup any fixed IP addresses to the temporary list
 :do {remove [find list=$listname]} on-error={}; # empty the complete list
 :do {set list=$listname [find list=$listnameTemp]} on-error={
 							:put "Import failed: while swapping out the the old list with the temperorary list: $listname";
 							:log error "Import failed: while swapping out the the old list with the temperorary list: $listname"
 							}
 
 :set $staticCount ""
 :if ([:len [find list=$listname !dynamic]] > 0) do={:set $staticCount "of which $[:len [find list=$listname !dynamic]] are static addresses"}
 
 :if ([:len [find list=$listnameTemp]] < 1) do={
 	:local listCount [:len [find list=$listname]]
 
 	:put "Completed updating address-list $listname with $listCount addresses $staticCount"
 	:log warning "Completed updating address-list $listname with $listCount addresses $staticCount"
 }
 
}; # do
$update url=https://iplists.firehol.org/files/firehol_level2.netset delimiter=("\n") listname=z-blocklist-FireHOL-L2 timeout=3d
$update url=https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv listname=z-blocklist-Sentinel delimiter=, timeout=8d heirule=http
$update url=https://www.spamhaus.org/drop/drop.txt delimiter=("\_") listname=z-blocklist-SpamHaus timeout=3d
$update url=https://www.spamhaus.org/drop/edrop.txt delimiter=("\_") listname=z-blocklist-SpamHaus-edrop timeout=3d

:log warning message="IP-Blocker script COMPLETED running"
}

MTNick · Tue Apr 02, 2024 12:57 am

@msatter Perfect! The above script is awesome. It's smooth & every step is logged with entry counts as well. Once again, I can't thank you enough. Thank you for this update & cleaning up the dirt I added to your script

I know nothing about scripting. But, I'll attempt anything with some direction.

Quick question, was the way I entered it wrong? Would/could it of caused issues? Just asking for reference & knowledge

msatter · Tue Apr 02, 2024 1:49 am

I have made an other update to the import now being aware of any static entries added by user to the active list. Those will be saved and a count of them being displayed on import.

It was not "dirt" but more streamlining the text to be more informative to an user.

You are not using the [] option available in this forum to display code and you added "" to the URL in the $update config line and that interfered with accepting the config line.

MTNick · Tue Apr 02, 2024 3:31 pm

Greeting msatter. Thank you. I removed all "" from the URL for all of the lists. I had no idea that this caused issues. Happy to report that everything is working great!

jvanhambelgium · Thu Apr 25, 2024 9:53 am

Perhaps interesting for those collecting various sources to feed the scripts.

https://docs.paloaltonetworks.com/resou ... ng-service

Palo Alto also provides for free various curated lists like for M365,Azure,GCP,Zoom etc,etc

Imports work just fine with the current script.

Larsa · Thu Apr 25, 2024 12:30 pm

I believe that https://iplists.firehol.org has the most comprehensive collection of IP address lists, statistics, and clickable maps indicating where the crooks are located. Palo Alto is one of many contributors.

actually not working example code

Who is online