Community discussions

MikroTik App
 
texmeshtexas
Member Candidate
Member Candidate
Topic Author
Posts: 151
Joined: Sat Oct 11, 2008 11:17 pm

Download list from https similar to team-cymru

Sun May 01, 2022 4:17 am

I found this script and haven been using it successfully to download bogon lists.
However, I'm trying to use the same script to download a different list from Talos that is offered in same exact format but there is something about it that the fetch command does not like.
the list is located here https://www.talosintelligence.com/docum ... -blacklist
anyone know why fetch is happy with https://www.team-cymru.org/Services/Bog ... s-ipv4.txt but not https://www.talosintelligence.com/docum ... -blacklist?

# Automatically add BOGONs to your firewall's address lists.
# Only works with 6.43 and up.
#
# Please do not fetch more often than the listed update interval, for the
# lists that are updated only as IANA allocations change, please do not fetch
# more than once per day.
#
# by Phillip Stromberg
# 2018-11-07
# uses team-cymru.org BOGON lists
{
:global content;
:local url;
:local addressListName;

:set addressListName "BogonsFullIPv4"

####################### UNCOMMENT THE URL YOU NEED: #######################

### This is the list of bit notation bogons, aggregated, in text format.
### Updated as IANA allocations and special prefix reservations are made.

# :set url "https://www.team-cymru.org/Services/Bog ... bn-agg.txt"

### The traditional bogon prefixes, plus prefixes that have been allocated to RIRs
### but not yet assigned by those RIRs to ISPs, end-users, etc.
### Updated every four hours.

#:set url "https://www.team-cymru.org/Services/Bog ... s-ipv4.txt"
:set url "https://www.talosintelligence.com/docum ... -blacklist"

###########################################################################

:local result [/tool fetch url=$url as-value output=user];

:if ($result->"status" = "finished") do={
:set content ($result->"data");
}
:global contentLen [ :len $content ];
:global lineEnd 0;
:global line "";
:global lastEnd -1;

/ip firewall address-list remove [find list=$addressListName];

:do {
:set lineEnd [:find $content "\n" $lastEnd ];
:set line [:pick $content $lastEnd $lineEnd];
:set lastEnd ( $lineEnd + 1 );
:if ( [:pick $line 0] = "#" ) do={
} else={
# :put $line;
/ip firewall address-list add address=$line list=$addressListName;
}

} while ($lineEnd < $contentLen - 2)
}
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Download list from https similar to team-cymru

Sun May 01, 2022 12:48 pm

Maybe because of this:
Screenshot_20220501_114629.jpg
You do not have the required permissions to view the files attached to this post.
 
texmeshtexas
Member Candidate
Member Candidate
Topic Author
Posts: 151
Joined: Sat Oct 11, 2008 11:17 pm

Re: Download list from https similar to team-cymru

Sun May 01, 2022 5:31 pm

works from my web browser just fine.

I think it has to do with the initial site has 301/302 redirects to https://snort.org/downloads/ip-block-list

which then redirects to https://snort-org-site.s3.amazonaws.com ... 2489a7eb44

even putting that last one in my web browser works fine the first time.
I think the Signature is dynamic

Then I tried this
https://snort-org-site.s3.amazonaws.com ... filter.blf
Fetch no longer gets an error but returns nothing. but his works consistently in my web browser.
Last edited by texmeshtexas on Sun May 01, 2022 6:07 pm, edited 1 time in total.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Download list from https similar to team-cymru

Sun May 01, 2022 5:46 pm

I can't help any further because can't open stuff that is on AmazonAWS . Look at my signature.
 
texmeshtexas
Member Candidate
Member Candidate
Topic Author
Posts: 151
Joined: Sat Oct 11, 2008 11:17 pm

Re: Download list from https similar to team-cymru

Sun May 01, 2022 6:15 pm

ok. thanks for offering.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Download list from https similar to team-cymru

Sun May 01, 2022 6:56 pm

Fetch do not support redirect.
Testing https://snort.org/downloads/ip-block-list on script based on
viewtopic.php?f=2&t=178355&p=878643#p878643
this is the result:

terminal code

Waiting the end of process for file testfetch.txt to be ready, max 20 seconds...
waiting... 1
waiting... 2
Done. Elapsed Seconds: 2

Result of Fetch:
****************************
  status: connecting

  status: failed

failure: closing connection: <302 Found "https://snort-org-site.s3.....34c73e4c5a"> 104.18.139.9:443 (4)

****************************
when the error is 302, using :pick and :find inside the script over the variable $filecontent you can read the redirected URL

Removing all debug info on the code:
/file remove [find where name="testfetch.txt"]
{
    :local jobid [:execute file=testfetch.txt script="/tool fetch url=\"https://snort.org/downloads/ip-block-list\""]
    :local Gltesec 0
    :while (([:len [/sys script job find where .id=$jobid]] = 1) && ($Gltesec < 20)) do={
        :set Gltesec ($Gltesec + 1)
        :delay 1s
    }
    :if ([:len [/file find where name="testfetch.txt"]] = 1) do={
        :local filecontent [/file get [/file find where name="testfetch.txt"] contents]
        :put [:pick $filecontent ([:find $filecontent "<302 Found \"" -1] + 12 ) [:find $filecontent "\"> " -1]]
    }
}

you obtain the correct URL: (all on same line, but formatted with enter at the "?" and "&" for readability on forum)

terminal code

https://snort-org-site.s3.amazonaws.com ... filter.blf
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=AKIAU7AK5ITMFGQS4MSV%2F20220501%2Fus-east-1%2Fs3%2Faws4_request
&X-Amz-Date=20220501T154240Z&X-Amz-Expires=3600
&X-Amz-SignedHeaders=host
&X-Amz-Signature=fba15d0995a1c49f48588ce7b34a3178495769e5391e28c6a8e00f8a0fcd7dc9
if the URL is not updated on this way, using only ".blf" link, you obtain

html code

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>G7395GJY4J7KEGQF</RequestId>
<HostId>
ml9x/fMPByX01HgzN1qYz3wNIHpcrArojvm4KPHdmSAlq/taVatI4ISPwWKt3fn0FEmxOIjAYP0=
</HostId>
</Error>
 
texmeshtexas
Member Candidate
Member Candidate
Topic Author
Posts: 151
Joined: Sat Oct 11, 2008 11:17 pm

Re: Download list from https similar to team-cymru  [SOLVED]

Sun May 01, 2022 7:33 pm

Ok. so I took that approach and generated a variable called newurl and tried to use that in the fetch to download the list
in the end I get this

https://snort-org-site.s3.amazonaws.com ... S4-HMAC-SH
A256&X-Amz-Credential=AKIAU7AK5ITMFGQS4MSV%2F20220501%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220501T163250Z&X-Amz-Expires=3600&X-
Amz-SignedHeaders=host&X-Amz-Signature=736deb7481413a0e32106687f80271b6ca53dfbefe88972f7827d0bb972e4235
failure: invalid URL protocol

/file remove [find where name="testfetch.txt"]
{
    :local jobid [:execute file=testfetch.txt script="/tool fetch url=\"https://snort.org/downloads/ip-block-list\""]
    :local Gltesec 0
    :while (([:len [/sys script job find where .id=$jobid]] = 1) && ($Gltesec < 20)) do={
        :set Gltesec ($Gltesec + 1)
        :delay 1s
    }
    :if ([:len [/file find where name="testfetch.txt"]] = 1) do={
        :local filecontent [/file get [/file find where name="testfetch.txt"] contents]
        :put [:pick $filecontent ([:find $filecontent "<302 Found \"" -1] + 12 ) [:find $filecontent "\"> " -1]]
       :local newurl [:pick $filecontent ([:find $filecontent "<302 Found \"" -1] + 12 ) [:find $filecontent "\"> " -1]];
   :log info "newurl=$newurl";
    }
}
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Download list from https similar to team-cymru

Sun May 01, 2022 9:13 pm

Using your link I have different error, why? ROS version?

terminal code

  status: failed

failure: cannot open file
 
texmeshtexas
Member Candidate
Member Candidate
Topic Author
Posts: 151
Joined: Sat Oct 11, 2008 11:17 pm

Re: Download list from https similar to team-cymru

Mon May 02, 2022 1:35 am

using 6.49.6 final full code
# Automatically add Talos Blacklist to firewall address list
# 
#
# Please do not fetch more often than the listed update interval, for the 
# lists that are updated only as IANA allocations change, please do not fetch 
# more than once per day.
#
# by Phillip Stromberg
# 2018-11-07
# uses team-cymru.org BOGON lists
{
    :global Tcontent;
    :local starturl;
    :local addressListName;
    
    :set addressListName "Talos"
    
    ####################### UNCOMMENT THE URL YOU NEED: #######################
    
 #   :set starturl "https://www.talosintelligence.com/documents/ip-blacklist"
    :set starturl "https://snort.org/downloads/ip-block-list"
 #    :set starturl "https://snort-org-site.s3.amazonaws.com/production/document_files/files/000/014/565/original/ip_filter.blf"
    
    ###########################################################################
### code to process web redirect by rextended on MT forum ###  
/file remove [find where name="testfetch.txt"]
{
    :local jobid [:execute file=testfetch.txt script="/tool fetch url=\"$starturl\""]
    :local Gltesec 0
    :while (([:len [/sys script job find where .id=$jobid]] = 1) && ($Gltesec < 20)) do={
        :set Gltesec ($Gltesec + 1)
        :delay 1s
    }
    :if ([:len [/file find where name="testfetch.txt"]] = 1) do={
        :local filecontent [/file get [/file find where name="testfetch.txt"] contents]
        :put [:pick $filecontent ([:find $filecontent "<302 Found \"" -1] + 12 ) [:find $filecontent "\"> " -1]]
       :local newurl [:pick $filecontent ([:find $filecontent "<302 Found \"" -1] + 12 ) [:find $filecontent "\"> " -1]];
   :log info "newurl=$newurl";
    }
}
    :global Tresult [/tool fetch url=$newurl as-value output=user];
    
    :if ($Tresult->"status" = "finished") do={
        :set Tcontent ($Tresult->"data");
    }
    :global TcontentLen [ :len $Tcontent ];
    :global lineEnd 0;
    :global line "";
    :global lastEnd -1;
    
    /ip firewall address-list remove [find list=$addressListName];
    
    :do {
        :set lineEnd [:find $Tcontent "\n" $lastEnd ];
        :set line [:pick $Tcontent $lastEnd $lineEnd];
        :set lastEnd ( $lineEnd + 1 );
        :if ( [:pick $line 0] = "data" ) do={
        } else={
            # :put $line;
            /ip firewall address-list add address=$line list=$addressListName;
        }
        
    } while ($lineEnd < $contentLen - 2)
}
Last edited by texmeshtexas on Mon May 02, 2022 2:05 am, edited 1 time in total.
 
texmeshtexas
Member Candidate
Member Candidate
Topic Author
Posts: 151
Joined: Sat Oct 11, 2008 11:17 pm

Re: Download list from https similar to team-cymru

Mon May 02, 2022 1:42 am

Actually, I changed the :local newurl to :global newurl and script is working and downloading the list now!!!

Thanks for the help!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Download list from https similar to team-cymru

Mon May 02, 2022 2:50 am

Just a reformatted style:
# Automatically add Talos Blacklist to firewall address list
#
#
# Please do not fetch more often than the listed update interval, for the
# lists that are updated only as IANA allocations change, please do not fetch
# more than once per day.
#
# by Phillip Stromberg
# 2018-11-07
# uses team-cymru.org BOGON lists
{
    :global Tcontent
    :local  starturl
    :local  addressListName "Talos"

####################### UNCOMMENT THE URL YOU NEED: #######################
#    :set starturl "https://www.talosintelligence.com/documents/ip-blacklist"
    :set starturl "https://snort.org/downloads/ip-block-list"
#    :set starturl "https://snort-org-site.s3.amazonaws.com/production/document_files/files/000/014/565/original/ip_filter.blf"
###########################################################################

### code to process web redirect by rextended on MT forum ###
    /file remove [find where name="testfetch.txt"]
    {
        :local jobid [:execute file=testfetch.txt script="/tool fetch url=\"$starturl\""]
        :local Gltesec 0
        :while (([:len [/sys script job find where .id=$jobid]] = 1) && ($Gltesec < 20)) do={
            :set Gltesec ($Gltesec + 1)
            :delay 1s
        }
        :if ([:len [/file find where name="testfetch.txt"]] = 1) do={
            :local filecontent [/file get [/file find where name="testfetch.txt"] contents]
            :local newurl [:pick $filecontent ([:find $filecontent "<302 Found \"" -1] + 12 ) [:find $filecontent "\"> " -1]]
            :log info "newurl=$newurl"
        }
    }

    :global Tresult [/tool fetch url=$newurl as-value output=user]

    :if ($Tresult->"status" = "finished") do={
        :set Tcontent ($Tresult->"data")
    }
    :global TcontentLen [:len $Tcontent]
    :global lineEnd     0
    :global line        ""
    :global lastEnd     -1
    /ip firewall address-list
    remove [find where list=$addressListName]
    :do {
        :set lineEnd [:find $Tcontent "\n" $lastEnd]
        :set line    [:pick $Tcontent $lastEnd $lineEnd]
        :set lastEnd ($lineEnd + 1)
        :if ([:pick $line 0] != "data") do={
            # :put $line
            add address=$line list=$addressListName
        }
    } while ($lineEnd < $contentLen - 2)
}

Who is online

Users browsing this forum: No registered users and 26 guests