Page 1 of 1

HELP with error in script to import the IPv4 full bogons list from www.team-cymru.org

Posted: Wed Oct 25, 2017 10:47 am
by LoveLife
Hello,

I am a total novice at scripting and simply duplicated someone elses script (https://wiki.polaire.nl/doku.php?id=rou ... gon_update) but with a small change.

The change I've made is, to instead fetch the complete IP4 bogons list from http://www.team-cymru.org/Services/Bogo ... s-ipv4.txt on a nightly schedule which Team Cymru update every 4hrs or so.

The issue I am having is when I run the modified script and a duplicate IP range is detected in the txt file (which there must be I guess), the script fails and displays "failure: already have such entry" in the Terminal Window.

Can anyone suggest a work around to modify the following script so I can avoid this error and allow the import to continue on?

Ideally if a duplicate entry is found in the txt file and it tries to add it to the "bogons" address list it simply ignores adding it to the address list and moves on to the next line thus continue to import the rest of the bogons in the txt file.

NOTE: The first line in that txt file which is being fetched contains "# last updated 1508907008 (Wed Oct 25 04:50:08 2017 GMT)", would this cause an issue? If so could the first line be ignored?
## Builds an address list with bogons based on:
## http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt

:log info "Fetching bogon list." 
/tool fetch url="http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt" mode=http

:log info "Removing all bogons." 
/ip firewall address-list remove [/ip firewall address-list find list=bogons]

:global content [/file get [/file find name=fullbogons-ipv4.txt] contents] ;
:global contentLen [:len $content];

:global lineEnd 0;
:global line "";
:global lastEnd 0;

:log info "Adding bogons from file." 
:do {
       :set lineEnd [:find $content "\n" $lastEnd ] ;
       :set line [:pick $content $lastEnd $lineEnd] ;
       :set lastEnd ( $lineEnd + 1 ) ;

       :if ( $line != "" ) do={
         /ip firewall address-list add list=bogons address=$line;
       }
} while ($lineEnd < $contentLen)
I'm using this address list along with other firewall entries to try and avoid being subject to DARKNET troubles such as this https://www.theregister.co.uk/2017/10/2 ... ansomware/. It may help a little, then again it may not, but it's also nice to stop this type of traffic going out or in to the local networks.

Re: HELP with error in script to import the IPv4 full bogons list from www.team-cymru.org

Posted: Wed Apr 21, 2021 11:51 pm
by texmeshtexas
This does not work because reading the file to the variable is too large. Would work for much smaller file but the bogon list is now over 1300 entries and about 21KB in size.

Re: HELP with error in script to import the IPv4 full bogons list from www.team-cymru.org

Posted: Thu Apr 22, 2021 1:25 am
by msatter
This does not work because reading the file to the variable is too large. Would work for much smaller file but the bogon list is now over 1300 entries and about 21KB in size.
Less than 64KB file: viewtopic.php?f=9&t=152632&p=758435

Re: HELP with error in script to import the IPv4 full bogons list from www.team-cymru.org

Posted: Thu Apr 22, 2021 3:02 am
by rextended
search tag # rextended convert to ip prefix


Reordered and corrected some aspects...

/ip firewall address-list
{
:log info "Remove old bogon list"
remove [find where list="bogons"]

:log info "Fetching bogon list"
:local content ([/tool fetch url="https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt" mode=https output=user as-value]->"data")

:local contentLen [:len $content]
:local lineEnd -1
:local line ""
:local lastEnd -1

:log info "Adding bogons from memory"
:do { :set lineEnd [:find $content "\n" $lastEnd]
      :set line    [:pick $content $lastEnd $lineEnd]
      :set lastEnd ($lineEnd + 1)

      :if ([:typeof [:toip $line]] = "ip") do={
          add list="bogons" timeout=1w address=$line
      } else={
          :do {
              :local xparse ([[:parse ":return $line"]])
              :if ([:typeof $xparse] = "ip-prefix") do={
                  add list="bogons" timeout=1w address=$line
              }
          } on-error={}
      }

} while=($lineEnd < ($contentLen - 1))

:log info "Done."
}

eh eh eh eh.... :))

add only if is valid ip-prefix like 6.6.6.0/24 or if a single IP like 6.6.6.1 or for some reason on the file are wited 6.6.6.2/32