Using fetch to retrieve IP lists

RouterOS Scripting
1

From mikrotik newsletter 8
Fetch Tool
For some time now, RouterOS has a neat little feature - it’s called Fetch. It is a HTTP and FTP client program that allows you to download files to your router.
For example fetch in action:
[admin@I] /tool> fetch address=www.mikrotik.com path=index.php
host=forum.mikrotik.com
status: connecting
status: downloading
downloaded: 7KiB
duration: 1s
You can use Fetch in scripting - for example to download auto-config files and new RouterOS versions.

If I retrieve a plaintext list of IP’s from a website, is there any way using the MT scripting that I could add these to an address list?
Can’t seem to find any commands to actually parse a file or break it into strings…

also, nothing in the 3.0 documentation about scripting at all.. other than the mikrotik newsletter I would never have even known that a fetch command existed :-/

2

String parsing like that would be best done elsewhere. I’d have a php or perl process on a linux box elsewhere do the work and return a prepared list to the mikrotik. The fetch cmd can, effectively, remotely execute the linux process.

3

Hmm, the list I’m referring to would already be preformatted, just looking for the commands I’d use to actually read the file using the mikrotik? or should I get the script to prepare it in the form

/ip firewall address-list add address= address-list=abc

what command could I then use to run the file as a script or such?

4

can’t you do it the other way - instead of making the router connect somewhere, make some nix box connect to the routers and execute some commands on the ROS?

5

I could but I figured there may be something i can use to do it on the router(s)
that way I could have multiple routers grabbing different IP lists on a regular basis.

Either way I might as well make it a request… the ability to read a file (or the ability to read address lists from one ROS box on another)

I know that other people do similar things on Linux boxes as normis mentioned (normally for iptables on their actuall box thou) so I figure it would be a good feature to have on the MT’s either for system-wide black lists, adding unmetered IP ranges etc etc.

6

Okay … here is an example of something I am working on. This works under 3.10, I havent tested earlier versions. I also cannot figure out how to use ‘/tool fetch’ in a script and have it do anything afterwards, so I believe you have to use 2 scripts, one to fetch, and another to parse. If someone wants to explain that let me know.

## Parse DSHIELD feed and build an address-list.
## Written by Sam Norris, ChangeIP.com 2008
## Any comments or suggestions welcome in the forums.
##
## 06/03/08 - Initial list parsing.

#/tool fetch address=feeds.dshield.org host=feeds.dshield.org mode=http path=block.txt

:if ( [/file get [/file find name=block.txt] size] > 0 ) do={ 

  /ip firewall address-list remove [/ip firewall address-list find list=dshield]

  :global content [/file get [/file find name=block.txt] contents] ;
  :global contentLen [ :len $content ] ;

  :global lineEnd 0;
  :global line "";
  :global lastEnd 0;

  :do {
       :set lineEnd [:find $content "\n" $lastEnd ] ;
       :set line [:pick $content $lastEnd $lineEnd] ;
       :set lastEnd ( $lineEnd + 1 ) ;

       :if ( [:pick $line 0 1] != "#" ) do={

	 :if ([:typeof [:toip [:pick $line 0 [:find $line "\t"] ] ] ] != "nil") do={
		:local pos1 [:find $line "\t" 0]
		:local pos2 [:find $line "\t" $pos1]
		:local pos3 [:find $line "\t" $pos2]
		:log info ( "DShield Entry: " . [:pick $line 0 $pos1 ] . "/" . [:pick $line ($pos2+1) $pos3 ] )
		/ip firewall address-list add list=dshield address=( [:pick $line 0 $pos1 ] . "/" . [:pick $line ($pos2+1) $pos3 ] )
         }  

       }

  } while ($lineEnd < $contentLen)

}

Which results in this:

[xxx@rancheros-AP] > /ip firewall address-list print from=[/ip firewall address-list find list=dshield]

Flags: X - disabled, D - dynamic 
 #   LIST                             ADDRESS                        
 0   dshield                          125.211.198.0/24               
 1   dshield                          193.164.131.0/24               
 2   dshield                          81.3.247.0/24                  
 3   dshield                          81.3.248.0/24                  
 4   dshield                          220.128.206.0/24               
 5   dshield                          81.3.244.0/24                  
 6   dshield                          81.3.249.0/24                  
 7   dshield                          81.192.191.0/24                
 8   dshield                          81.3.243.0/24                  
 9   dshield                          81.3.250.0/24                  
10   dshield                          70.21.127.0/24                 
11   dshield                          218.6.8.0/24                   
12   dshield                          202.138.172.0/24               
13   dshield                          81.3.246.0/24                  
14   dshield                          202.99.254.0/24                
15   dshield                          202.6.225.0/24                 
16   dshield                          99.129.23.0/24                 
17   dshield                          76.88.8.0/24                   
18   dshield                          60.222.231.0/24                
19   dshield                          222.187.221.0/24

Once I clean this up a little, and add some other list parsing I will add to the wiki.

Thx,
Sam

7

Here’s the latest with Spamhaus drop.lasso capability. These could be refined if anyone wants to help.

## Parse DSHIELD & Spamhaus feed and build an address-list.
## Written by Sam Norris, ChangeIP.com 2008
## Any comments or suggestions welcome in the forums.
##
## 06/03/08 - Initial list parsing.

#/tool fetch address=feeds.dshield.org host=feeds.dshield.org mode=http path=block.txt
#/tool fetch address=www.spamhaus.org host=www.spamhaus.org mode=http path=drop/drop.lasso

##
## DSHIELD Drop List
##

:if ( [/file get [/file find name=block.txt] size] > 0 ) do={ 

  /ip firewall address-list remove [/ip firewall address-list find list=dshield]

  :global content [/file get [/file find name=block.txt] contents] ;
  :global contentLen [ :len $content ] ;

  :global lineEnd 0;
  :global line "";
  :global lastEnd 0;

  :do {
       :set lineEnd [:find $content "\n" $lastEnd ] ;
       :set line [:pick $content $lastEnd $lineEnd] ;
       :set lastEnd ( $lineEnd + 1 ) ;

       :if ( [:pick $line 0 1] != "#" ) do={

	 :if ([:typeof [:toip [:pick $line 0 [:find $line "\t"] ] ] ] != "nil") do={
		:local pos1 [:find $line "\t" 0]
		:local pos2 [:find $line "\t" $pos1]
		:local pos3 [:find $line "\t" $pos2]
		:log info ( "DShield Entry: " . [:pick $line 0 $pos1 ] . "/" . [:pick $line ($pos2+1) $pos3 ] )
		/ip firewall address-list add list=dshield address=( [:pick $line 0 $pos1 ] . "/" . [:pick $line ($pos2+1) $pos3 ] )
         }  

       }

  } while ($lineEnd < $contentLen)

}

##
## SPAMHAUS.ORG Drop List
##

:if ( [/file get [/file find name=drop.lasso] size] > 0 ) do={ 

  /ip firewall address-list remove [/ip firewall address-list find list=spamhaus.lasso]

  :global content [/file get [/file find name=drop.lasso] contents] ;
  :global contentLen [ :len $content ] ;

  :global lineEnd 0;
  :global line "";
  :global lastEnd 0;

  :do {
       :set lineEnd [:find $content "\n" $lastEnd ] ;
       :set line [:pick $content $lastEnd $lineEnd] ;
       :set lastEnd ( $lineEnd + 1 ) ;

       :if ( [:pick $line 0 1] != ";" ) do={

	 :if ([:len [:pick $line 0 [:find $line ";"] ] ] > 0 ) do={
		:local pos1 [:find $line ";" 0]
		:local entry [:pick $line 0 ($pos1-1) ]
		:if ( [:len $entry ] > 0 ) do={
			:log info "Lasso Entry: $entry"
			/ip firewall address-list add list=spamhaus.lasso address=$entry
		}
         }  

       }

  } while ($lineEnd < $contentLen)

}
8

You sir, are a champion :smiley:

9

if you have any other text lists you want help making a script for let me know. post the URL or a sample.

10 
## Generic IP address list input
## Based on a script written by Sam Norris, ChangeIP.com 2008
:if ( [/file get [/file find name=node-unmetered-ips.txt] size] > 0 ) do={

 /ip firewall address-list remove [/ip firewall address-list find list=Node-Unmetered]

 :global content [/file get [/file find name=node-unmetered-ips.txt] contents] ;
 :global contentLen [ :len $content ] ;

 :global lineEnd 0;
 :global line "";
 :global lastEnd 0;

 :do {
      :set lineEnd [:find $content "\n" $lastEnd ] ;
      :set line [:pick $content $lastEnd $lineEnd] ;
      :set lastEnd ( $lineEnd + 1 ) ;

      :if ( [:pick $line 0 1] != "#" ) do={

     :local entry [:pick $line 0 ($lineEnd -1) ]
     :if ( [:len $entry ] > 0 ) do={
        /ip firewall address-list add list=Node-Unmetered address=$entry
     }
   }
 } while ($lineEnd < $contentLen)
}

This will do the same thing for a simple list in the format

#This is a comment
1.1.1.1
2.2.2.0/24
3.3.3.3
4.4.4.128/26

Just substitute the address-list and filename you want to pull from (for anyone else who wants to use it with their own generated lists)

11

Thanks changeip, tks for sharing your knowlege .
Max

12

Hmm, okay I’ve got one for you..

Trying to extract addresses from http://tools.pipenetworks.com/lists/qld.txt

## QLD PIPE Address List
## Based on a script written by Sam Norris, ChangeIP.com 2008
:if ( [/file get [/file find name=qld.txt] size] > 0 ) do={

  /ip firewall address-list remove [/ip firewall address-list find list=qld-pipe]

  :global content [/file get [/file find name=qld.txt] contents] ;
  :global contentLen [ :len $content ] ;

  :global lineEnd 0;
  :global line "";
  :global lastEnd 0;

  :do {
       :set lineEnd [:find $content "\n" $lastEnd ] ;
       :set line [:pick $content $lastEnd $lineEnd] ;
       :set lastEnd ( $lineEnd + 1 ) ;

       :if ( [:pick $line 0 1] != "#" ) do={

    :if ([:len [:pick $line 0 [:find $line ","] ] ] > 0 ) do={
      :local pos1 [:find $line "," 0]
      :local entry [:pick $line 0 ($pos1) ]
      :if ( [:len $entry ] > 0 ) do={
         :log info "QLDPIPE: $entry"
         /ip firewall address-list add list=qld-pipe address=$entry
      }
         } 

       }

  } while ($lineEnd < $contentLen)

}
13

Have added this to the wiki - http://wiki.mikrotik.com/wiki/Using_Fetch_and_Scripting_to_add_IP_Address_Lists

14

Just minor syntax thing.

Change path=folder/anotherfolder/ipaddress.txt to src-path=folder/anotherfolder/ipaddress.txt


Thanx for great script, Changeip!

15

can’t happen to see what I’m doing wrong in my above script eh? :slight_smile:

16

:wink:

17

Hi to all.
Can somebody help me with this script?
I’m trying to modify it to process file from http://www.colocall.net/uaix/prefixes.txt
For fetch script I use “/tool fetch address=www.colocall.net src-path=/uaix/prefixes.txt host=www.colocall.net mode=http” and it works and creates file prefixes.txt
But I can’t manage to add info from this file to address list :frowning:
It seems that content from file can’t be retrieved.
This part of code doesn’t work and return in log $content=nothing and $contentLen=0 even so I opened file and it is full of lines :frowning:

:global content [/file get [/file find name=prefixes.txt] contents] ;
:log info $content;
:global contentLen [ :len $content ] ;
:log info $contentLen;

I have Mikrotik 3.13

18

I am not certain you will consider this help, but take a look at
/file print detail
If you do not see a ‘contents’ listing associated with the filename, it is probably because the file is larger than 4095 bytes.
By experience, if there is no ‘contents’ listing, it cannot be loaded/accessed/edited by a script. Maybe someone else can be more help.

19

Thank you SurferTim for your help.
It was the size of file, when I edited it to 2049 bytes the script correctly processed it and added to address list. But how this limit can be overcomed? I need this big list of networks.

20

Hi sergeda,

Others have the same challenge. I was hoping there was a “readline” command for text files, where you could read a line a a time from large files (like C and other languages), but have not found anything close. Maybe the MikroTik Team will read this and include something in a future release.

ADD: Wouldn’t this be great?

:set linecount [:len [:readline test.txt]]
:set thisline [:readline test.txt 20]
:set nextline [:readline test.txt]

It would read the number of lines in test.txt into $linecount and the 20th (ok, maybe 21st) line from test.txt into $thisline, and subsequent calls without a ‘line number’ parameter retrieves the next line. So the last line is the same as
:set nextline [:readline test.txt 21]

JUST WISHING!! :smiley: