I found this script and haven been using it successfully to download bogon lists.
However, I'm trying to use the same script to download a different list from Talos that is offered in same exact format but there is something about it that the fetch command does not like.
the list is located here https://www.talosintelligence.com/documents/ip-blacklist
anyone know why fetch is happy with https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt but not https://www.talosintelligence.com/documents/ip-blacklist?
Automatically add BOGONs to your firewall's address lists.
Only works with 6.43 and up.
Please do not fetch more often than the listed update interval, for the
lists that are updated only as IANA allocations change, please do not fetch
more than once per day.
by Phillip Stromberg
2018-11-07
{
:global content;
:local url;
:local addressListName;
:set addressListName "BogonsFullIPv4"
####################### UNCOMMENT THE URL YOU NEED: #######################
This is the list of bit notation bogons, aggregated, in text format.
Updated as IANA allocations and special prefix reservations are made.
The traditional bogon prefixes, plus prefixes that have been allocated to RIRs
but not yet assigned by those RIRs to ISPs, end-users, etc.
Updated every four hours.
#:set url "https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt"
:set url "https://www.talosintelligence.com/documents/ip-blacklist"
###########################################################################
:local result [/tool fetch url=$url as-value output=user];
:if ($result->"status" = "finished") do={
:set content ($result->"data");
}
:global contentLen [ :len $content ];
:global lineEnd 0;
:global line "";
:global lastEnd -1;
/ip firewall address-list remove [find list=$addressListName];
:do {
:set lineEnd [:find $content "\n" $lastEnd ];
:set line [:pick $content $lastEnd $lineEnd];
:set lastEnd ( $lineEnd + 1 );
:if ( [:pick $line 0] = "#" ) do={
} else={
:put $line;
/ip firewall address-list add address=$line list=$addressListName;
}
} while ($lineEnd < $contentLen - 2)
}
I can’t help any further because can’t open stuff that is on AmazonAWS . Look at my signature.
Fetch do not support redirect.
Testing https://snort.org/downloads/ip-block-list on script based on
http://forum.mikrotik.com/t/fetch-capable-of-following-redirects/151723/7
this is the result:
Waiting the end of process for file testfetch.txt to be ready, max 20 seconds…
waiting… 1
waiting… 2
Done. Elapsed Seconds: 2
Result of Fetch:
status: connecting
status: failed
failure: closing connection: <302 Found “https://snort-org-site.s3…34c73e4c5a”> 104.18.139.9:443 (4)
when the error is 302, using :pick and :find inside the script over the variable $filecontent you can read the redirected URL
Removing all debug info on the code:
/file remove [find where name="testfetch.txt"]
{
:local jobid [:execute file=testfetch.txt script="/tool fetch url=\"https://snort.org/downloads/ip-block-list\""]
:local Gltesec 0
:while (([:len [/sys script job find where .id=$jobid]] = 1) && ($Gltesec < 20)) do={
:set Gltesec ($Gltesec + 1)
:delay 1s
}
:if ([:len [/file find where name="testfetch.txt"]] = 1) do={
:local filecontent [/file get [/file find where name="testfetch.txt"] contents]
:put [:pick $filecontent ([:find $filecontent "<302 Found \"" -1] + 12 ) [:find $filecontent "\"> " -1]]
}
}
you obtain the correct URL: (all on same line, but formatted with enter at the “?” and “&” for readability on forum)
https://snort-org-site.s3.amazonaws.com/production/document_files/files/000/014/566/original/ip_filter.blf?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=AKIAU7AK5ITMFGQS4MSV%2F20220501%2Fus-east-1%2Fs3%2Faws4_request
&X-Amz-Date=20220501T154240Z&X-Amz-Expires=3600
&X-Amz-SignedHeaders=host
&X-Amz-Signature=fba15d0995a1c49f48588ce7b34a3178495769e5391e28c6a8e00f8a0fcd7dc9
if the URL is not updated on this way, using only “.blf” link, you obtain
AccessDenied
Access Denied
G7395GJY4J7KEGQF
ml9x/fMPByX01HgzN1qYz3wNIHpcrArojvm4KPHdmSAlq/taVatI4ISPwWKt3fn0FEmxOIjAYP0=
Ok. so I took that approach and generated a variable called newurl and tried to use that in the fetch to download the list
in the end I get this
https://snort-org-site.s3.amazonaws.com/production/document_files/files/000/014/567/original/ip_filter.blf?X-Amz-Algorithm=AWS4-HMAC-SH
A256&X-Amz-Credential=AKIAU7AK5ITMFGQS4MSV%2F20220501%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220501T163250Z&X-Amz-Expires=3600&X-
Amz-SignedHeaders=host&X-Amz-Signature=736deb7481413a0e32106687f80271b6ca53dfbefe88972f7827d0bb972e4235
failure: invalid URL protocol
/file remove [find where name="testfetch.txt"]
{
:local jobid [:execute file=testfetch.txt script="/tool fetch url=\"https://snort.org/downloads/ip-block-list\""]
:local Gltesec 0
:while (([:len [/sys script job find where .id=$jobid]] = 1) && ($Gltesec < 20)) do={
:set Gltesec ($Gltesec + 1)
:delay 1s
}
:if ([:len [/file find where name="testfetch.txt"]] = 1) do={
:local filecontent [/file get [/file find where name="testfetch.txt"] contents]
:put [:pick $filecontent ([:find $filecontent "<302 Found \"" -1] + 12 ) [:find $filecontent "\"> " -1]]
:local newurl [:pick $filecontent ([:find $filecontent "<302 Found \"" -1] + 12 ) [:find $filecontent "\"> " -1]];
:log info "newurl=$newurl";
}
}
Using your link I have different error, why? ROS version?
status: failed
failure: cannot open file
using 6.49.6 final full code
# Automatically add Talos Blacklist to firewall address list
#
#
# Please do not fetch more often than the listed update interval, for the
# lists that are updated only as IANA allocations change, please do not fetch
# more than once per day.
#
# by Phillip Stromberg
# 2018-11-07
# uses team-cymru.org BOGON lists
{
:global Tcontent;
:local starturl;
:local addressListName;
:set addressListName "Talos"
####################### UNCOMMENT THE URL YOU NEED: #######################
# :set starturl "https://www.talosintelligence.com/documents/ip-blacklist"
:set starturl "https://snort.org/downloads/ip-block-list"
# :set starturl "https://snort-org-site.s3.amazonaws.com/production/document_files/files/000/014/565/original/ip_filter.blf"
###########################################################################
### code to process web redirect by rextended on MT forum ###
/file remove [find where name="testfetch.txt"]
{
:local jobid [:execute file=testfetch.txt script="/tool fetch url=\"$starturl\""]
:local Gltesec 0
:while (([:len [/sys script job find where .id=$jobid]] = 1) && ($Gltesec < 20)) do={
:set Gltesec ($Gltesec + 1)
:delay 1s
}
:if ([:len [/file find where name="testfetch.txt"]] = 1) do={
:local filecontent [/file get [/file find where name="testfetch.txt"] contents]
:put [:pick $filecontent ([:find $filecontent "<302 Found \"" -1] + 12 ) [:find $filecontent "\"> " -1]]
:local newurl [:pick $filecontent ([:find $filecontent "<302 Found \"" -1] + 12 ) [:find $filecontent "\"> " -1]];
:log info "newurl=$newurl";
}
}
:global Tresult [/tool fetch url=$newurl as-value output=user];
:if ($Tresult->"status" = "finished") do={
:set Tcontent ($Tresult->"data");
}
:global TcontentLen [ :len $Tcontent ];
:global lineEnd 0;
:global line "";
:global lastEnd -1;
/ip firewall address-list remove [find list=$addressListName];
:do {
:set lineEnd [:find $Tcontent "\n" $lastEnd ];
:set line [:pick $Tcontent $lastEnd $lineEnd];
:set lastEnd ( $lineEnd + 1 );
:if ( [:pick $line 0] = "data" ) do={
} else={
# :put $line;
/ip firewall address-list add address=$line list=$addressListName;
}
} while ($lineEnd < $contentLen - 2)
}
Actually, I changed the :local newurl to :global newurl and script is working and downloading the list now!!!
Thanks for the help!
Just a reformatted style:
# Automatically add Talos Blacklist to firewall address list
#
#
# Please do not fetch more often than the listed update interval, for the
# lists that are updated only as IANA allocations change, please do not fetch
# more than once per day.
#
# by Phillip Stromberg
# 2018-11-07
# uses team-cymru.org BOGON lists
{
:global Tcontent
:local starturl
:local addressListName "Talos"
####################### UNCOMMENT THE URL YOU NEED: #######################
# :set starturl "https://www.talosintelligence.com/documents/ip-blacklist"
:set starturl "https://snort.org/downloads/ip-block-list"
# :set starturl "https://snort-org-site.s3.amazonaws.com/production/document_files/files/000/014/565/original/ip_filter.blf"
###########################################################################
### code to process web redirect by rextended on MT forum ###
/file remove [find where name="testfetch.txt"]
{
:local jobid [:execute file=testfetch.txt script="/tool fetch url=\"$starturl\""]
:local Gltesec 0
:while (([:len [/sys script job find where .id=$jobid]] = 1) && ($Gltesec < 20)) do={
:set Gltesec ($Gltesec + 1)
:delay 1s
}
:if ([:len [/file find where name="testfetch.txt"]] = 1) do={
:local filecontent [/file get [/file find where name="testfetch.txt"] contents]
:local newurl [:pick $filecontent ([:find $filecontent "<302 Found \"" -1] + 12 ) [:find $filecontent "\"> " -1]]
:log info "newurl=$newurl"
}
}
:global Tresult [/tool fetch url=$newurl as-value output=user]
:if ($Tresult->"status" = "finished") do={
:set Tcontent ($Tresult->"data")
}
:global TcontentLen [:len $Tcontent]
:global lineEnd 0
:global line ""
:global lastEnd -1
/ip firewall address-list
remove [find where list=$addressListName]
:do {
:set lineEnd [:find $Tcontent "\n" $lastEnd]
:set line [:pick $Tcontent $lastEnd $lineEnd]
:set lastEnd ($lineEnd + 1)
:if ([:pick $line 0] != "data") do={
# :put $line
add address=$line list=$addressListName
}
} while ($lineEnd < $contentLen - 2)
}
