Community discussions

MikroTik App
 
User avatar
BartoszP
Forum Guru
Forum Guru
Topic Author
Posts: 2855
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

How to opitimize list of IP4 addresses

Thu May 02, 2019 11:43 pm

I was thinking how to optimize big IP lists before importing them to Mikrotik. It ended as this program.
Feel free to use it. Comments welcome.

Written with GNU Linux and gcc. Standard usage ... takes data from stdin and outputs to stdout

Program tries to merge consecutive IP addresses or IP ranges.
This list
10.10.10.4
10.10.10.5
10.10.10.6
10.10.10.7
192.168.1.128/32
192.168.1.129/32
192.168.1.130
192.168.1.131
192.168.1.132
192.168.1.133
192.168.1.134/32
192.168.1.135/32
192.168.1.136/29
192.168.2.128/31
192.168.2.130
192.168.2.131
192.168.2.132
192.168.2.133
192.168.2.134/32
192.168.2.135/32
192.168.2.136/29
192.168.2.144/28
172.16.1.1
1.1.1.1
8.8.8.8
9.9.9.9
172.16.1.2
172.16.2.0
172.16.2.1
172.16.1.3
is optimized to
192.168.2.128/27
192.168.1.128/28
10.10.10.4/30
172.16.1.2/31
172.16.2.0/31
1.1.1.1
8.8.8.8
9.9.9.9
172.16.1.1
Source:
optimizeip.c
Exapmple ip list:
testip.txt
Optimized list:
optimized.txt
You do not have the required permissions to view the files attached to this post.
 
nostromog
Member Candidate
Member Candidate
Posts: 226
Joined: Wed Jul 18, 2018 3:39 pm

Re: How to opitimize list of IP4 addresses

Sat May 04, 2019 5:53 pm

I think it is not working 100% right.

Example. Llet's get all facebook IPv4 address ranges and process them with your program:
$ (for orig in AS32934 AS63293 AS54115; do whois -h whois.radb.net -- "-i origin $orig"; done) | grep route: | awk '{print $2}' >facebook4.txt
$ gcc -o optimizeip optimizeip.c
$ ./optimizeip <facebook4.txt | grep 129
129.134.0.0/16
129.134.0.0/17
129.134.128.0/23
I still have had no time to study the code to look for the bug, but I wanted to warn you of this behaviour. The way I understand it 129.134.0.0/16 should be the only one left in the output
 
User avatar
BartoszP
Forum Guru
Forum Guru
Topic Author
Posts: 2855
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: How to opitimize list of IP4 addresses

Sat May 04, 2019 8:09 pm

Thank you for the report.
It is example of situation when one subnet is fully included in another. I do not look for such optimization ... yet :)
IMHO it is not "a bug" .. output is fully valid however not optimized to "deep roots".
 
ludvik
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Mon May 26, 2008 4:36 pm

Re: How to opitimize list of IP4 addresses

Fri May 17, 2019 10:42 pm

 
taichinhhanaHN
just joined
Posts: 2
Joined: Wed Jan 31, 2018 10:10 am
Contact:

Re: How to opitimize list of IP4 addresses

Tue May 28, 2019 10:59 am

How many possible host addresses are available for an IPv4 address?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: How to opitimize list of IP4 addresses

Tue May 28, 2019 11:16 am

How many possible host addresses are available for an IPv4 address?
Depends on how IPv4 is divided to subnets. Theoretical that's 2^32 addresses (slightly less than 4.2*10^9). Some are used for network addresses (and same number for broadcast addresses) - exact number depends on sizes of subnets and with CIDR addressing you can't know how some particular address range might be divided into subnets just by look of it. Some addresses are private (10/8, 172.16/12, 192.168/16) and don't really count. Some addresses are used for multicasts (224/4) and don't count either. My guess is that realistic number is around 3.9 billion (3.9*10^9) of IPv4 host addresses. Give or take a few millions.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 207
Joined: Tue May 05, 2015 11:12 am
Location: 74, FR / SA48, UK
Contact:

Re: How to opitimize list of IP4 addresses

Tue May 28, 2019 12:33 pm

Also you can use https://github.com/snar/bgpq3, for example:
bgpq3 -A -4 -j AS-FACEBOOK
Here I'm using JSON output format so that I can have access to a prefix-length range, but you could equally use
bgpq3 -A -4 AS-FACEBOOK
and get a Cisco-style output:
ip prefix-list NN permit 31.13.24.0/21
ip prefix-list NN permit 31.13.64.0/18 le 19
ip prefix-list NN permit 31.13.64.0/19 ge 24 le 24
ip prefix-list NN permit 45.64.40.0/22
ip prefix-list NN permit 66.111.48.0/22
ip prefix-list NN permit 66.111.48.0/23 ge 24 le 24
…snip…
ip prefix-list NN permit 199.201.64.0/22
ip prefix-list NN permit 199.201.64.0/22 ge 24 le 24
ip prefix-list NN permit 204.15.20.0/22
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: How to opitimize list of IP4 addresses

Fri Jun 25, 2021 6:37 pm

search tag # rextended short address list ip address aggregator

Thanks for this share!!!

Who is online

Users browsing this forum: No registered users and 15 guests