Community discussions

MUM Europe 2020
 
mhosts
newbie
Topic Author
Posts: 36
Joined: Tue Nov 03, 2009 4:43 pm

Import large list of IP's to Firewall

Thu Mar 18, 2010 4:39 pm

Is there a way to Import a long list of CIDR IP ranges into the Firewall in ROS?

For example:

If I wanted to Deny the following ranges in a Rule. Do I need to create 1 rule for each range? or is there a way to have the rule refer to a text file that I upload with all of the ranges? or Can I import all of them into the single rule?

xxx.xxx.xxx.xxx/16
xxx.xxx.xxx.xxx/24
xxx.xxx.xxx.xxx/22
etc...

Thanks,
 
fewi
Forum Guru
Forum Guru
Posts: 7734
Joined: Tue Aug 11, 2009 3:19 am

Re: Import large list of IP's to Firewall

Thu Mar 18, 2010 4:44 pm

There's many ways to approach that. You could create a script and copy/paste it in, or upload it and import the script. You could write a script that parses lines in a file and inserts firewall rules based on them.

But if you need to apply the same rules to just a lot of IP ranges, the best approach is to create firewall address-lists and refer to src-address-list or dst-address-list in the one firewall rule.
 
mhosts
newbie
Topic Author
Posts: 36
Joined: Tue Nov 03, 2009 4:43 pm

Re: Import large list of IP's to Firewall

Thu Mar 18, 2010 4:47 pm

There's many ways to approach that. You could create a script and copy/paste it in, or upload it and import the script. You could write a script that parses lines in a file and inserts firewall rules based on them.

But if you need to apply the same rules to just a lot of IP ranges, the best approach is to create firewall address-lists and refer to src-address-list or dst-address-list in the one firewall rule.
I need to apply the same rule to a lot of IP ranges.

If I was to create firewall address-lists and refer to src-address-list as you suggested, would that be a file that is uploaded via ftp?
 
fewi
Forum Guru
Forum Guru
Posts: 7734
Joined: Tue Aug 11, 2009 3:19 am

Re: Import large list of IP's to Firewall

Thu Mar 18, 2010 4:54 pm

No. But as I said, you can externally create a script of format:
/ip firewall address-list
add list=listName address=10.1.0.0/24
add list=listName address=10.2.0.0/24 
and run it after you upload it. If you already have a list of just the networks it'd be easy to use Perl or any other scripting language to generate such a script.
 
mhosts
newbie
Topic Author
Posts: 36
Joined: Tue Nov 03, 2009 4:43 pm

Re: Import large list of IP's to Firewall

Thu Mar 18, 2010 4:55 pm

No. But as I said, you can externally create a script of format:
/ip firewall address-list
add list=listName address=10.1.0.0/24
add list=listName address=10.2.0.0/24 
and run it after you upload it. If you already have a list of just the networks it'd be easy to use Perl or any other scripting language to generate such a script.
Thanks, I will try it out. I didn't know ROS had Perl capabilities.
 
fewi
Forum Guru
Forum Guru
Posts: 7734
Joined: Tue Aug 11, 2009 3:19 am

Re: Import large list of IP's to Firewall

Thu Mar 18, 2010 5:01 pm

It doesn't. You would do that on an external host and import the script generated on the external host.

RouterOS does have a built in scripting language, but it's considerably more unwieldy. You could use it to parse a file, but it's more complicated.
 
fewi
Forum Guru
Forum Guru
Posts: 7734
Joined: Tue Aug 11, 2009 3:19 am

Re: Import large list of IP's to Firewall

Thu Mar 18, 2010 5:22 pm

FWIW the below seems to work for the RouterOS scripting language. Adjust the first two non-comment lines to your environment, and adjust the line below if you're not using Windows line delimiters:
# set some parameters
:local fileName "address-ranges.txt";
:local addressList "myListName";
# set to "\r\n" below for Windows text files, "\n" for everything else
:local lineDelimiter "\r\n";

# get a hold on the file
:local filePointer [/file find name=$fileName];
# check if it exists
:if ($filePointer != "") do={
# file exists, start a counter for the entries inserted
	:local added 0;
# read the configuration file into a buffer
	:local buffer [/file get $filePointer contents];
# record the total buffer size for later use in do loop cycling through the lines
	:local bufferSize [:len $buffer];
# create variable to hold buffer offset from start to current line end
	:local lineEnd 0;
# create variable that will hold the current line
	:local line "";
# create variable that keeps track of current position in buffer
	:local position 0;
# go through buffer
	:do {
# find next line end
		:set lineEnd [:find $buffer $lineDelimiter $position];
# fetch the line
		:set line [:pick $buffer $position $lineEnd];
# advance position skipping the newlines
		:set position ($lineEnd + [:len $lineDelimiter]);
# verify that the line isn't empty
		:if ([:len $line] > 0) do={
			/ip firewall address-list add list="$addressList" address="$line";
			:set added ($added + 1);
		} else={
# invalid entry in file, log that fact
			:log warning "Cannot insert address-list entry, corrupt line in file!"
		}
	} while=($lineEnd < $bufferSize);
# check how many entries were inserted - if any, log for informational purposes
	:if ($added > 0) do={
		:log info "Inserted $added new address-list entries."
	} else={
# if 0, check if the buffer read from the file was larger than 0. if so we should have had inserts
		:if ($bufferSize > 0) do={
			:log warning "address-list insertion file had content, but none were inserted!"
		}
	}
} else={
# file could not be opened. log that fact
	:log warning "Unable to open file $fileName!"
}
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Import large list of IP's to Firewall

Thu Mar 18, 2010 5:48 pm

of you may use Micro$oft Office Excel to make a list of necessary commands )))
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
rmichael
Forum Veteran
Forum Veteran
Posts: 718
Joined: Sun Mar 08, 2009 11:00 pm

Re: Import large list of IP's to Firewall

Thu Mar 18, 2010 9:53 pm

I curious, what is the practical limit on number of entries per access list?
 
fewi
Forum Guru
Forum Guru
Posts: 7734
Joined: Tue Aug 11, 2009 3:19 am

Re: Import large list of IP's to Firewall

Thu Mar 18, 2010 10:43 pm

I curious, what is the practical limit on number of entries per access list?
I would also like to know that.
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Import large list of IP's to Firewall

Fri Mar 19, 2010 1:02 am

seems like it uses hash tables, so... the limit is free memory? =) and almost no performance degradation
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: Import large list of IP's to Firewall

Fri Mar 19, 2010 12:57 pm

yes, the limitation is available memory.
 
PtDragon
Frequent Visitor
Frequent Visitor
Posts: 78
Joined: Sun Apr 26, 2009 8:52 pm

Re: Import large list of IP's to Firewall

Sun Aug 09, 2015 12:18 pm

Sadly it does not work for me.
I tried it but 6 hours of waiting(one core is occupied by it still) and still nothing.
What am i dong wrong?
RouterOS - 6.30.2
CCR1036-12G-4S +6x100Mbit ^_^
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Import large list of IP's to Firewall

Mon Aug 10, 2015 2:42 am

Sadly it does not work for me.
I tried it
what 'it'?
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
PtDragon
Frequent Visitor
Frequent Visitor
Posts: 78
Joined: Sun Apr 26, 2009 8:52 pm

Re: Import large list of IP's to Firewall

Mon Aug 10, 2015 8:04 pm

Sadly it does not work for me.
I tried it
what 'it'?
Script in this topic.
I tried to convert lists from iblocklist to CDIR format and import one of lists.
CCR1036-12G-4S +6x100Mbit ^_^
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Import large list of IP's to Firewall

Tue Aug 11, 2015 2:19 am

so what exactly happens? how long is your list? it's not possible to work with files > 4KiB in RouterOS, for example
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
PtDragon
Frequent Visitor
Frequent Visitor
Posts: 78
Joined: Sun Apr 26, 2009 8:52 pm

Re: Import large list of IP's to Firewall

Tue Aug 11, 2015 9:19 pm

so what exactly happens? how long is your list? it's not possible to work with files > 4KiB in RouterOS, for example
It was for big enough files or i misunderstood it?
My list is surely MUCH bigger than 4kib.
It has 257969 lines (4341454 bytes).
Also i tried to make it with bash(to prepare rsc for import, on bash all works good) but it goes wrong(on mikrotik) and stops on hitting something in other ban list(which is dynamic list of attackers).
CCR1036-12G-4S +6x100Mbit ^_^
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8342
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Import large list of IP's to Firewall

Tue Aug 11, 2015 10:27 pm

It was for big enough files or i misunderstood it?
My list is surely MUCH bigger than 4kib.
It has 257969 lines (4341454 bytes).
yes, RouterOS Scripting cannot read so big files
and stops on hitting something in other ban list(which is dynamic list of attackers).
what?.. (O_o)

what attackers? what do you mean?
Also i tried to make it with bash(to prepare rsc for import, on bash all works good)
that's more or less correct way. you can also split rsc to small parts and import them one by one
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
PtDragon
Frequent Visitor
Frequent Visitor
Posts: 78
Joined: Sun Apr 26, 2009 8:52 pm

Re: Import large list of IP's to Firewall

Wed Aug 12, 2015 1:14 am

what?.. (O_o)

what attackers? what do you mean?
Guys trying to brute passwords, guys trying to put services down by DDoS and so on.
that's more or less correct way. you can also split rsc to small parts and import them one by one
Is there a way to keep importing on errors(just skipping error line and going further)?
CCR1036-12G-4S +6x100Mbit ^_^

Who is online

Users browsing this forum: Google [Bot], JohnNL and 87 guests