Page 1 of 1

Import large list of IP's to Firewall

Posted: Thu Mar 18, 2010 4:39 pm
by mhosts
Is there a way to Import a long list of CIDR IP ranges into the Firewall in ROS?

For example:

If I wanted to Deny the following ranges in a Rule. Do I need to create 1 rule for each range? or is there a way to have the rule refer to a text file that I upload with all of the ranges? or Can I import all of them into the single rule?

xxx.xxx.xxx.xxx/16
xxx.xxx.xxx.xxx/24
xxx.xxx.xxx.xxx/22
etc...

Thanks,

Re: Import large list of IP's to Firewall

Posted: Thu Mar 18, 2010 4:44 pm
by fewi
There's many ways to approach that. You could create a script and copy/paste it in, or upload it and import the script. You could write a script that parses lines in a file and inserts firewall rules based on them.

But if you need to apply the same rules to just a lot of IP ranges, the best approach is to create firewall address-lists and refer to src-address-list or dst-address-list in the one firewall rule.

Re: Import large list of IP's to Firewall

Posted: Thu Mar 18, 2010 4:47 pm
by mhosts
There's many ways to approach that. You could create a script and copy/paste it in, or upload it and import the script. You could write a script that parses lines in a file and inserts firewall rules based on them.

But if you need to apply the same rules to just a lot of IP ranges, the best approach is to create firewall address-lists and refer to src-address-list or dst-address-list in the one firewall rule.
I need to apply the same rule to a lot of IP ranges.

If I was to create firewall address-lists and refer to src-address-list as you suggested, would that be a file that is uploaded via ftp?

Re: Import large list of IP's to Firewall

Posted: Thu Mar 18, 2010 4:54 pm
by fewi
No. But as I said, you can externally create a script of format:
/ip firewall address-list
add list=listName address=10.1.0.0/24
add list=listName address=10.2.0.0/24 
and run it after you upload it. If you already have a list of just the networks it'd be easy to use Perl or any other scripting language to generate such a script.

Re: Import large list of IP's to Firewall

Posted: Thu Mar 18, 2010 4:55 pm
by mhosts
No. But as I said, you can externally create a script of format:
/ip firewall address-list
add list=listName address=10.1.0.0/24
add list=listName address=10.2.0.0/24 
and run it after you upload it. If you already have a list of just the networks it'd be easy to use Perl or any other scripting language to generate such a script.
Thanks, I will try it out. I didn't know ROS had Perl capabilities.

Re: Import large list of IP's to Firewall

Posted: Thu Mar 18, 2010 5:01 pm
by fewi
It doesn't. You would do that on an external host and import the script generated on the external host.

RouterOS does have a built in scripting language, but it's considerably more unwieldy. You could use it to parse a file, but it's more complicated.

Re: Import large list of IP's to Firewall

Posted: Thu Mar 18, 2010 5:22 pm
by fewi
FWIW the below seems to work for the RouterOS scripting language. Adjust the first two non-comment lines to your environment, and adjust the line below if you're not using Windows line delimiters:
# set some parameters
:local fileName "address-ranges.txt";
:local addressList "myListName";
# set to "\r\n" below for Windows text files, "\n" for everything else
:local lineDelimiter "\r\n";

# get a hold on the file
:local filePointer [/file find name=$fileName];
# check if it exists
:if ($filePointer != "") do={
# file exists, start a counter for the entries inserted
	:local added 0;
# read the configuration file into a buffer
	:local buffer [/file get $filePointer contents];
# record the total buffer size for later use in do loop cycling through the lines
	:local bufferSize [:len $buffer];
# create variable to hold buffer offset from start to current line end
	:local lineEnd 0;
# create variable that will hold the current line
	:local line "";
# create variable that keeps track of current position in buffer
	:local position 0;
# go through buffer
	:do {
# find next line end
		:set lineEnd [:find $buffer $lineDelimiter $position];
# fetch the line
		:set line [:pick $buffer $position $lineEnd];
# advance position skipping the newlines
		:set position ($lineEnd + [:len $lineDelimiter]);
# verify that the line isn't empty
		:if ([:len $line] > 0) do={
			/ip firewall address-list add list="$addressList" address="$line";
			:set added ($added + 1);
		} else={
# invalid entry in file, log that fact
			:log warning "Cannot insert address-list entry, corrupt line in file!"
		}
	} while=($lineEnd < $bufferSize);
# check how many entries were inserted - if any, log for informational purposes
	:if ($added > 0) do={
		:log info "Inserted $added new address-list entries."
	} else={
# if 0, check if the buffer read from the file was larger than 0. if so we should have had inserts
		:if ($bufferSize > 0) do={
			:log warning "address-list insertion file had content, but none were inserted!"
		}
	}
} else={
# file could not be opened. log that fact
	:log warning "Unable to open file $fileName!"
}

Re: Import large list of IP's to Firewall

Posted: Thu Mar 18, 2010 5:48 pm
by Chupaka
of you may use Micro$oft Office Excel to make a list of necessary commands )))

Re: Import large list of IP's to Firewall

Posted: Thu Mar 18, 2010 9:53 pm
by rmichael
I curious, what is the practical limit on number of entries per access list?

Re: Import large list of IP's to Firewall

Posted: Thu Mar 18, 2010 10:43 pm
by fewi
I curious, what is the practical limit on number of entries per access list?
I would also like to know that.

Re: Import large list of IP's to Firewall

Posted: Fri Mar 19, 2010 1:02 am
by Chupaka
seems like it uses hash tables, so... the limit is free memory? =) and almost no performance degradation

Re: Import large list of IP's to Firewall

Posted: Fri Mar 19, 2010 12:57 pm
by janisk
yes, the limitation is available memory.

Re: Import large list of IP's to Firewall

Posted: Sun Aug 09, 2015 12:18 pm
by PtDragon
Sadly it does not work for me.
I tried it but 6 hours of waiting(one core is occupied by it still) and still nothing.
What am i dong wrong?
RouterOS - 6.30.2

Re: Import large list of IP's to Firewall

Posted: Mon Aug 10, 2015 2:42 am
by Chupaka
Sadly it does not work for me.
I tried it
what 'it'?

Re: Import large list of IP's to Firewall

Posted: Mon Aug 10, 2015 8:04 pm
by PtDragon
Sadly it does not work for me.
I tried it
what 'it'?
Script in this topic.
I tried to convert lists from iblocklist to CDIR format and import one of lists.

Re: Import large list of IP's to Firewall

Posted: Tue Aug 11, 2015 2:19 am
by Chupaka
so what exactly happens? how long is your list? it's not possible to work with files > 4KiB in RouterOS, for example

Re: Import large list of IP's to Firewall

Posted: Tue Aug 11, 2015 9:19 pm
by PtDragon
so what exactly happens? how long is your list? it's not possible to work with files > 4KiB in RouterOS, for example
It was for big enough files or i misunderstood it?
My list is surely MUCH bigger than 4kib.
It has 257969 lines (4341454 bytes).
Also i tried to make it with bash(to prepare rsc for import, on bash all works good) but it goes wrong(on mikrotik) and stops on hitting something in other ban list(which is dynamic list of attackers).

Re: Import large list of IP's to Firewall

Posted: Tue Aug 11, 2015 10:27 pm
by Chupaka
It was for big enough files or i misunderstood it?
My list is surely MUCH bigger than 4kib.
It has 257969 lines (4341454 bytes).
yes, RouterOS Scripting cannot read so big files
and stops on hitting something in other ban list(which is dynamic list of attackers).
what?.. (O_o)

what attackers? what do you mean?
Also i tried to make it with bash(to prepare rsc for import, on bash all works good)
that's more or less correct way. you can also split rsc to small parts and import them one by one

Re: Import large list of IP's to Firewall

Posted: Wed Aug 12, 2015 1:14 am
by PtDragon
what?.. (O_o)

what attackers? what do you mean?
Guys trying to brute passwords, guys trying to put services down by DDoS and so on.
that's more or less correct way. you can also split rsc to small parts and import them one by one
Is there a way to keep importing on errors(just skipping error line and going further)?