Community discussions

MikroTik App
 
Sipmud
just joined
Topic Author
Posts: 1
Joined: Thu Sep 17, 2020 11:20 am

Importing IP List from file

Thu Sep 17, 2020 11:32 am

Hello.

I'm struggling with getting to run script that I found to import IP addresses from file and adding them to my address list.
Idea is to fetch file with all official country IP addresses from Web page (file name is "local.net") and add them to list.
Fetching script is working fine, bet when I run the script to add IP to list I get only first number added to my address list.
Screenshot 2020-09-17 at 11.27.53.png
Also, is there "4096 byte read/write limit" workaround?

Maybe there is some scripting guru that can help me?

Script that I am using is:
  :if ( [/file get [/file find name=local.net] size] > 0 ) do={
    /ip firewall address-list remove [/ip firewall address-list find list=Test]
    :local content [/file get [/file find name=local.net] contents] ;
    :local contentLen [ :len $content ] ;
    :put ( "=>" . $content . "<=" )
    :local lineEnd 0
    :local line ""
    :local lastEnd [ :tonum 0 ]


    :do {
      :set line [ :pick $content $lastEnd [ :find $content "\r\n" $lastEnd ] ]
      :set lineEnd [:find $content "\n" $lastEnd ] ;
      :put "lineEnd=$lineEnd< - lastEnd=$lastEnd - line=$line<"
    :if ( [ :len $content ] = 0 ) do={
      :put "UNIX"
      :set line [ :pick $content $lastEnd [ :find $content "\r\n" $lastEnd ] ]
       :set lineEnd [:find $content "\n" $lastEnd ] ;
    }
      :set lastEnd ( $lineEnd + 1 ) ;

      :if ( [:pick $line 0 1] != "\n" ) do={
      :put ">$line<"
        :if ( [:len $line ] > 0 ) do={ /ip firewall address-list add list=Test address=$line }
      }
    } while ($lineEnd < $contentLen)
  }
You do not have the required permissions to view the files attached to this post.
 
Lelik200
just joined
Posts: 22
Joined: Fri Jul 24, 2020 1:48 pm

Re: Importing IP List from file

Fri Oct 09, 2020 9:49 pm

Please post local.net file example.
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 2865
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Importing IP List from file

Sat Oct 10, 2020 10:09 am

You should prepare your list before import.
Read this topic: viewtopic.php?p=606832#p606832
 
gsbiz
just joined
Posts: 20
Joined: Sat Nov 17, 2018 5:18 pm

Re: Importing IP List from file

Sat Oct 10, 2020 5:37 pm

Try these 5 lines.
just add your own TLD's, it will add an address list for each TLD. Run it again to update them. You will need to create a firewall rule to drop the TLD lists. Beware the lists are huge (who woulda guessed) and if you load them into memory they will take it all, so select your TLD's well and don't go for too many, or at least monitor your memory level.

All the best.
foreach i in={ "NL"; "CN"; "RU"; "DE"; "UA"} do={
/tool fetch url="http://www.iwik.org/ipcountry/mikrotik/$i" dst-path=/DISK/downloads/$i;
/ip firewall address-list remove [/ip firewall address-list find list=$i];
/import file-name=/DISK/downloads/$i;
/file remove /DISK/downloads/$i;
}
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Importing IP List from file

Sat Oct 10, 2020 5:48 pm

Reading pure IP adresses is possible up to 64KB large files.

viewtopic.php?f=9&t=152632

I am on the moment busy to create backup/restore for adresslists present in the router and it will export a .RSC file that smaller than the normal export.
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 2865
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Importing IP List from file

Sat Oct 10, 2020 6:32 pm

I recall this ... not perfect but it could help
viewtopic.php?f=23&t=148187&p=729368#p729368
 
garethiowc
just joined
Posts: 19
Joined: Wed Dec 03, 2014 12:33 am

Re: Importing IP List from file

Tue Apr 27, 2021 12:39 pm

Try these 5 lines.
just add your own TLD's, it will add an address list for each TLD. Run it again to update them. You will need to create a firewall rule to drop the TLD lists. Beware the lists are huge (who woulda guessed) and if you load them into memory they will take it all, so select your TLD's well and don't go for too many, or at least monitor your memory level.

All the best.
foreach i in={ "NL"; "CN"; "RU"; "DE"; "UA"} do={
/tool fetch url="http://www.iwik.org/ipcountry/mikrotik/$i" dst-path=/DISK/downloads/$i;
/ip firewall address-list remove [/ip firewall address-list find list=$i];
/import file-name=/DISK/downloads/$i;
/file remove /DISK/downloads/$i;
}
When I run this script I get the following error - expected end of command (line 2 column 86)
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3291
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Importing IP List from file

Tue Apr 27, 2021 1:18 pm

There is a bug in folder part of the script. Folder not needed so this does work fine.
foreach i in={ "NL"; "CN"; "RU"; "DE"; "UA"} do={
	/tool fetch url="http://www.iwik.org/ipcountry/mikrotik/$i" dst-path=$i
	/ip firewall address-list remove [/ip firewall address-list find list=$i]
	/import file-name=$i
	/file remove $i
}
 
garethiowc
just joined
Posts: 19
Joined: Wed Dec 03, 2014 12:33 am

Re: Importing IP List from file

Tue Apr 27, 2021 1:21 pm

Awesome thanks
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Mon Aug 09, 2021 3:24 am

I consider a true security breach import any file downloaded from 3rd party site with /import file-name=$scriptName command.

If some hacker or the author itself put some commands inside the downloaded file, can do anything he want with the RouterBOARD...
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Mon Aug 09, 2021 11:05 am

True, one should always be very vigilant when obtaining some stuff from a 3e party.
A blunt "import" and just executing some imported stuff are not best practices for sure.
As these are block-lists, you could argue that IF the list was maliciously modified and for example legit systems added you could also potentially "disrupt" legit traffic.

I have to say I also do this for this specific list with TLD's.

So in fact the best thing is to sort of combine existing scripts.

foreach i in={ "CN"; "RU"; "UA"} do={
/tool fetch url="http://www.iwik.org/ipcountry/mikrotik/$i" dst-path=$i
/ip firewall address-list remove [/ip firewall address-list find list=$i]
/import file-name=$i
/file remove $i
}

and then below a typical script used for importing others (eg. here a TOR-exit nodes list, similar concept)
The difference is that the IP's are without any CIDR notation at the end (eg /24 /22 or something) and offcourse do not contain all other routerOS commands, but the regex should extraxt only IP info anyway.
So perhaps the logic of the script below needs to be slightly adapted to form a combined one. This method is safe as IP's are placed and handled/parsed). Any fake command in the list would cause an error anyway.

-> No need to fetch them, as the above script has the iteration and downloads separate files for the different TLD's (CN / RU / UA in this example) so $i could be used as the source.

{
/ip firewall address-list
:local update do={
:do {
:local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
:do { remove [find list=$blacklist comment!="Optional"] } on-error={}

:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^([0-2]{0,1}[0-9]{1,2}\\.){3}[0-2]{0,1}[0-9]{1,2}(\\/[0-3]{0,1}[0-9]{1,1}){0,1}") do={
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) } on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
} ; :log warning "Imported address list < $blacklist> from file: $url"
} else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
} on-error={:log warning "Address list <$blacklist> update failed"}
}
$update url=https://check.torproject.org/torbulkexitlist blacklist="TOR-Nodes" delimiter=("\n")
}
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Mon Aug 09, 2021 11:21 am

This must also be considered:
viewtopic.php?f=13&t=146615#p871846
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Mon Aug 09, 2021 2:54 pm

Anybody wants to give a go at this below ?
So basically these TLD's come from the onsite website in this form for IPv4 and IPv6
Let's start simple and only look at IPv4

/log info "Loading GR ipv4 address list"
/ip firewall address-list remove [/ip firewall address-list find list=GR]
/ip firewall address-list
:do { add address=2.84.0.0/14 list=GR } on-error={}
:do { add address=5.54.0.0/15 list=GR } on-error={}
:do { add address=5.144.192.0/18 list=GR } on-error={}
:do { add address=5.172.192.0/20 list=GR } on-error={}
:do { add address=5.203.0.0/16 list=GR } on-error={}
:do { add address=31.14.168.0/21 list=GR } on-error={}


I don't have the impression that the logic below handles it well. It should throw away/ignore anything besides X.X.X.X/Y and then import it into a ACL.
Regex guru's here ?

:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^([0-2]{0,1}[0-9]{1,2}\\.){3}[0-2]{0,1}[0-9]{1,2}(\\/[0-3]{0,1}[0-9]{1,1}){0,1}") do={
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) } on-error={}
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Mon Aug 09, 2021 3:16 pm

Ehm...just search "ip regex"...
"Someone" has already wroten something... :lol:
viewtopic.php?f=9&t=152632&p=871284&hil ... ex#p871284

remember than this are wroten for be put directly on script,
if tested on terminal you must add \ before the ?
if tested on regex101 must be removed \ before \. and \/


What you have post is my version that uses less CPU because it approximates the value, with or without zero, but it is quite reliable.
0.0.0.0/0 to 299.299.299.299/39
000.000.000.000/00 to 299.299.299.299/39

This is the most precise version I have made, but use more CPU, really not soo much:
((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\/(3[0-2]|[0-2]?[0-9])
This match exactly IP-prefix (on the link on this post are present alternative "IP only" and "IP with optional prefix")
0.0.0.0/0 to 255.255.255.255/32
with or without the 0 like 008.000.000.000/08
Last edited by rextended on Mon Aug 09, 2021 4:21 pm, edited 5 times in total.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Mon Aug 09, 2021 3:44 pm

Doesn't seem to do anything. I get the message in my logs (like with other updates)

Imported address list < GR-TLD> from file: https://www.iwik.org/ipcountry/mikrotik/GR

But the ACL is not created on my RB3011
I've tried the code below, I've omitted the "^" because we will not see any match at the beginning of a string anyway.
Tried it on regex101.com but I had to modify it in order to capture/match an IP like 10.20.10.0/12

((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9]).){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\/(3[0-2]|[012]?[0-9])

{
/ip firewall address-list
:local update do={
:do {
:local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
:do { remove [find list=$blacklist comment!="Optional"] } on-error={}

:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\/(3[0-2]|[012]?[0-9])") do={
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) } on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
} ; :log warning "Imported address list < $blacklist> from file: $url"
} else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
} on-error={:log warning "Address list <$blacklist> update failed"}
}
$update url=https://www.iwik.org/ipcountry/mikrotik/GR blacklist="GR-TLD" delimiter=("\n")
}

Not there yet ;-)
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Mon Aug 09, 2021 3:49 pm

I add a warning for be clear, also on other post, thanks for make me notice that!!! :lol:

remember than this are wroten for be put directly on script,
if tested on terminal you must add \ before the ?
if tested on regex101 must be removed \ before \. and \/


On regex101 you must remove MikroTik required \ on terminal, but you must not remove the \ on \. and on \/ because must be literal . and / , not any character....

from

MikroTik Script POSIX regex code

((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\/(3[0-2]|[0-2]?[0-9])
to

MikroTik CLI code

((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\/(3[0-2]|[0-2]\?[0-9])
and to

regex101 Standard POSIX regex code

((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\/(3[0-2]|[0-2]?[0-9])


test_regex.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Mon Aug 09, 2021 6:23 pm

Hmm, don't get it. It creates no ACL at all, yet it reports no issues ;-(
So the only way to troubleshoot is execute on CLI and see what comes on the screen ?
But if tested on CLI, then regex-code differs slightly.

{
/ip firewall address-list
:local update do={
:do {
:local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
:do { remove [find list=$blacklist comment!="Optional"] } on-error={}

:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\/(3[0-2]|[0-2]?[0-9])") do={
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]])} on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
} ; :log warning "Imported address list < $blacklist> from file: $url"
} else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
} on-error={:log warning "Address list <$blacklist> update failed"}
}
$update url=https://www.iwik.org/ipcountry/mikrotik/GR blacklist="GRTLD" delimiter=("\n")
}
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Mon Aug 09, 2021 6:38 pm

First of all you must split on three parts because the IP is not at the start and not at the end, like other lists...
Space replaced with "·"

split code

:do·{·add·address=|2.84.0.0/14|·list=GR·}·on-error={}
left delimiter:
address=
right delimiter:
·list=
including space

Example for CLI, I use global just for the test, on script can/must be used local.
# test string
:global teststr ":do { add address=2.84.0.0/14 list=GR } on-error={}\r\n:do { add address=5.54.0.0/15 list=GR } on-error={}"

# remove head and tail, used + 8 because is the character lenght of "address="
# instead of a fixed value can be set also, for example, with [:len $rightstrdelimiter] on script
# notice the needed space before list
:global testip [:pick $teststr ([:find $teststr "address=" -1] + 8) [:find $teststr " list=" -1]]

# now I chech against regexp if is valid IP-prefix
:if ($testip~"((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\/(3[0-2]|[0-2]\?[0-9])") do={ \
    :put "$testip is a IP-prefix"
} else={:put "$testip is NOT a IP-prefix"}

Doing again adding last "right" position value (on example 29), give next IP, without split the file on lines and without modify everytime the data value.
:global testip [:pick $teststr ([:find $teststr "address=" 29] + 8) [:find $teststr " list=" 29]]
:if ($testip~"((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\/(3[0-2]|[0-2]\?[0-9])") do={ \
    :put "$testip is a IP-prefix"
} else={:put "$testip is NOT a IP-prefix"}
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Mon Aug 09, 2021 7:57 pm

Thx for the feedback, I'll play with that and try to get it working.
I was under the impression that perhaps the parsing was done a bit like "sed" (streaming editor) in Linux/Unix so it would scan the line (until the delimeter \n) and capture the regex logic in there without
the need to split it yourself etc.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Mon Aug 09, 2021 8:07 pm

if on mikrotik the find can accept the regexp as search string, the code are more easy to do...
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Mon Aug 09, 2021 10:13 pm

Now I understand why I'm not a programmer ... AT ALL...I can't wrap my head around such simple logic.
Please follow my reasoning here and feel free to correct me, I'll post inline comment starting @@@

{
/ip firewall address-list
:local update do={
:do {
:local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
@@@ this line fetches the bottom defined URL and if value of the "downloaded" is not 63 it maps each line received from the $result variable into another local variable "data" @@@

:do { remove [find list=$blacklist comment!="Optional"] } on-error={}
@@@ find an existant ACL and remove entries which do not have a comment-value set "Optional" @@@

:while ([:len $data]!=0) do={
@@@ So as long as the $data is not empty (as it contained the freshly loaded info from the URL perform statements below @@@
:if ([:pick $data ([:find $data "address=" -1] + 8) [:find $data " list=" -1]]~"((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\/(3[0-2]|[0-2]\?[0-9])") do={
@@@ So scan each line looking for what is between the head & tail of each message. @@@
@@@ Example below of the fetch @@@
/log info "Loading GR ipv4 address list"
/ip firewall address-list remove [/ip firewall address-list find list=GR]
/ip firewall address-list
:do { add address=2.84.0.0/14 list=GR } on-error={}
:do { add address=5.54.0.0/15 list=GR } on-error={}
:do { add address=5.144.192.0/18 list=GR } on-error={}
:do { add address=5.172.192.0/20 list=GR } on-error={}


:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]])} on-error={}
}
@@@ populate the $blacklist (called here "GRTLD") and this what I don't understand => The $data in the above only contain the data between address=X.X.X.X.X/3 list=GR ???
@@@ Why this construction address=([:pick $data 0 [:find $data $delimiter]]) to populate the actual IP/MASK in the ACL ? Probably the $delimeter is of no use here anymore

:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
@@@ Why the above rule ? Why do you have to "set" data ? You just want to parse (as long as $data!=0 ? right?)

} ; :log warning "Imported address list < $blacklist> from file: $url"
} else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
} on-error={:log warning "Address list <$blacklist> update failed"}
}
@@@ above some generic messages depending on exit/error-codes I guess, not really mandatory anyway @@@

$update url=https://www.iwik.org/ipcountry/mikrotik/GR blacklist="GRTLD" delimiter=("\n")

Thanks for any feedback!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Mon Aug 09, 2021 10:26 pm

Do not think badly about yourself, on other posts I read about a user who called himself an expert programmer and did not solve a minimum problem of logic...

Now I read what you write and I reply.
If you can next time put the script inside [] code block or is hard to read (for me) thanks.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Mon Aug 09, 2021 10:55 pm

I indent all because without indenting is near unreadable
I only explain, not correct anything

WARNING for other users: do not use this, is useless
# open the bracket { to test inside a terminal, remove on script
{

# put the context in right... context (bad, must be defined inside function, but now no matter)
/ip firewall address-list

# define update function
:local update do={

    # bad start rely immediately on "on-error"...
    :do {
        :local result [/tool fetch url=$url as-value output=user]
# TRUE: @@@ this line fetches the bottom defined URL and if value of the "downloaded" is not 63 it maps each line
# received from the $result variable into another local variable "data" @@@

        :if ($result->"downloaded" != "63") do={
            :local data ($result->"data")
# another on-Orror, the remove function in this case can't do any error...
            :do { remove [find list=$blacklist comment!="Optional"] } on-error={}
# TRUE: @@@ find an existant ACL and remove entries which do not have a comment-value set "Optional" @@@

            :while ([:len $data]!=0) do={
# TRUE: @@@ So as long as the $data is not empty (as it contained the freshly loaded info from the URL perform statements below @@@
                :if ([:pick $data ([:find $data "address=" -1] + 8) [:find $data " list=" -1]]~"((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\/(3[0-2]|[0-2]\?[0-9])") do={
# TRUE: but is better to split on two passages, is more readable @@@ So scan each line looking for what is
# between the head & tail of each message. @@@

# another on-Orror
                   :do { add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]])} on-error={}
# TRUE: @@@ populate the $blacklist (called here "GRTLD") @@@
               # end of if ([:pick $data ...
               }


# @@@ and this what I don't understand => The $data in the above only contain the data between address=X.X.X.X.X/3 list=GR
# You do not understand it, because everytime a record is saved, the remaining data replace "data",
# this is why before are present a ^ on front of the regex
# I use another method, instead to lost time modifing everytime the data,
# i move the current "pointer" of the start and the end of the point where regex try to find...
# @@@ Why this construction address=([:pick $data 0 [:find $data $delimiter]]) to populate the actual IP/MASK in the ACL \?
# Probably the $delimeter is of no use here anymore
# I do not write this functions, but all is a mess...

               :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
# @@@ Why the above rule - Why do you have to "set" data - You just want to parse (as long as $data!=0  right)
# parsing do not decrease data, I explain cut and paste of data lines before
           # end of while
           }
           :log warning "Imported address list < $blacklist> from file: $url"
       # end of :if ($result->"downloaded" != "63")
       } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }


    # end of general function update
    } on-error={ :log warning "Address list <$blacklist> update failed" }
# TRUE: @@@ above some generic messages depending on exit/error-codes I guess, not really mandatory anyway @@@

# end of update function
}

# launch the function update with parameters
# better use " " everytime is not clearly a number, tue or false, yes or not, IP or IP-prefix, and something other now I miss for sure...
$update url=https://www.iwik.org/ipcountry/mikrotik/GR blacklist="GRTLD" delimiter=("\n")

# close the script for the terminal
}
Last edited by rextended on Thu Aug 12, 2021 12:39 am, edited 1 time in total.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Mon Aug 09, 2021 11:17 pm

Thanks for sure for spending the time to educate me a bit on this!
Appreciated!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Tue Aug 10, 2021 12:47 am

I'll give you another piece of code... it's almost finished, actually there is no need for any "on-errors"...

What is missing:
1) Read the file from server and handle the missing file, wrong response from server, etc. (the hard part)
2) Create a function and pass the parameters to it
3) Check the passed parameters if they are correct
4) Create whitelist, before add the IP / IP prefix check if it is on whitelist, then if is it, no add
5) Check on add if the ip-prefix is already present inside other IP-prefix already on address-list
6) Check on add if the ip-prefix is comprehensive of one or more IP-prefix on address-list, remove old(s) and add new bigger.
7) for security accept only from /12 to /32 prefix. /11 or less on IPv4 is too much big for be true...
8 ) Set an option for put the IP on the address-list but on temporary way (Dynamic) for specified time (from 1 second to near 35 weeks),
this do not export this type of IP on address-list on export or backup
whith this option set, if the address is found again on the imported list, instead to delete it and re-import, have time resetted again (from 1 second to near 35 weeks)

For now the script is intended to work on terminal only, after completion, it needs to be converted to be used on scripts.

Some item are modified for generate invalid values like 5.203.0.0/00 and 31.14.168.0/0 (formwerly valid IP with prefix, but /0 mean "all IPs"...)
# simulation of reading from a file
:global teststr (":do { add address=2.84.0.0/14 list=GR } on-error={}\r\n:do { add address=5.54.0.0/15 list=GR } on-error={}\r\
                \n:do { add address=5.144.192.0/18 list=GR } on-error={}\r\n:do { add address=5.172.192.0/32 list=GR } on-error={}\r\
                \n:do { add address=5.203.0.0/00 list=GR } on-error={}\r\n:do { add address=31.14.168.0/0 list=GR } on-error={}")

# manually defined, but in the future read as parameters of the function
:global addlist "test"
# add parameter if the entry must be dynamic (only on volatile memory, self-destructing after x seconds/hours/days/etc.)
# or static (keeped on reboot)
# must be added the option to accept from the downloaded address list only IP, only IP prefixes or both. For now it accepts only IP prefixes
# keep previous entries in the address-list or not
:global keep true
:global head "address="
:global tail " list="

# initializing variables (on global because we want test it on terminal, on script can/must be local
:global regexipwithsubnet "((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\/(3[0-2]|[0-2]\?[0-9])"
:global lenght [:len $teststr]
:global offset [:len $head]
:global actualhead -1
:global actualtail -1
:global testip ""

# move all to the right context to shorten the commands
/ip firewall address-list

# if the previous content is not to be kept, it removes all entries
:if (!($keep)) do={ remove [find where list=$addlist] }

:while ([:typeof $actualtail] != "nil") do={
    :set actualhead ([:find $teststr $head $actualtail] + $offset)
    :set actualtail  [:find $teststr $tail $actualhead]
    :if ([:typeof $actualtail] != "nil") do={
        :set testip [:pick $teststr $actualhead $actualtail]
# if must be imported a list of IP without prefix, simply check
#        :if ([:typeof [:toip $testip]] = "ip") do={
# because for a bug added on newer versions,
# can not test directly if a string is a ip-prefix and ip-prefix do not have function like :toip
# I invented this walkthrough for not use regex, but is hard to understand and I don't know if it stop working on future versions
#        :if ([:typeof [[:parse ":return $testip"]] ] = "ip-prefix") do={
        :if ($testip ~ $regexipwithsubnet) do={
            :if ($testip ~ "\\/0(0|\$)") do={
                :log warning "Invalid IP-prefix >$testip<"
            } else={
                # address list save IP/32 without /32, must search for duplicate without the /32, adding with or without /32 not matter
                :if ($testip ~ "\\/32") do={ :set testip [:pick $testip 0 [:find $testip "/32" -1]] }
                :if ([:len [find where list=$addlist and address=$testip]] = 0) do={ add list=$addlist address=$testip }
            }
        }
    }
}
Last edited by rextended on Thu Aug 12, 2021 1:20 pm, edited 13 times in total.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Wed Aug 11, 2021 12:28 pm

I'm probably going to look for a slightly modified way outside of RouterOS, as this RouterOS scripting gives me permanent brain-freeze. :shock: :(
I've some 24/7 NAS running anyway that can handle something simple like :

curl -v --stderr - https://www.iwik.org/ipcountry/mikrotik/GR |grep -E -o '((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\/(3[0-2]|[0-2]?[0-9])'
(I can easily have an iteration loop fetching more TLD's where required)

This gives me a curated output with what is needed, plain IPv4 prefixes.
Then it's a matter of "fetching" this curated list on the Mikrotik using the most simple "importer" as there is not that much voodoo to be done.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Wed Aug 11, 2021 1:02 pm

Sorry for the headache ;)
I tried to be as clear as possible... : mrgreen:

If it were possible to use the metarouter to have a full "busybox"...
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Wed Aug 11, 2021 2:00 pm

Stupid me, apparently on the same host the "regular" CIDR prefixes voor IPv4/IPv6 can be found.
The specific URL used by me points to a "mikrotik" section with indeed RouterOS import "ready" entries.
So yeah, we only need to fetch the list below (or any other TLD-code). This is then "safe" (well, as safe as it can be...) to process with the existing scripts as you are not executing any CLI directly.

https://www.iwik.org/ipcountry/GR.cidr
or
https://www.iwik.org/ipcountry/GR.ipv6

:lol: :D
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Wed Aug 11, 2021 2:41 pm

By opening this, you can also see if the size is too large for RouterOS (65535 bytes / 64K)
https://www.iwik.org/ipcountry/
For example, ITaly is 47K, ok,
but USa is 942K and is excesive, can not be imported without first dividing the file into 15 parts

I open the file on hex editor, each ip-prefix are separated with
0x0A / char 10 / \n / New Line
and is not present MS-DOS style
0x0D / char 13 / \r / Carriage Return

The file end with one \n

At this point split the file for each \n and test if the field is a valid ip-prefix before import.

Must be added an extra \n at start and one \n at the end for read also the first and the last line.
Because before first line \n are not present and is not guaranted the \n at the end

The script for terminal I wrote before is still valid,
just specify \n as head and tail (I removed all previous comments for better reading, to see them go to previous post)
:global teststr ("# Generated 2021-08-10 12:01:25.213173\n2.84.0.0/14\n5.54.0.0/15\n5.144.192.0/18\n5.172.192.0/20\n\
    5.203.0.0/32\n31.14.168.0/0\n31.14.208.0/00\n")

:global addlist "test"
:global keep true
# head and tail for ip-prefix are \n
:global head "\n"
:global tail "\n"

:global regexipwithsubnet "((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\/(3[0-2]|[0-2]\?[0-9])"
:global lenght [:len $teststr]
:global offset [:len $head]
:global actualhead -1
:global actualtail -1
:global testip ""

/ip firewall address-list

:if (!($keep)) do={ remove [find where list=$addlist] }

:while ([:typeof $actualtail] != "nil") do={
    :set actualhead ([:find $teststr $head $actualtail] + $offset)
    :set actualtail [:find $teststr $tail $actualhead]
    :if ([:typeof $actualtail] != "nil") do={
        :set testip [:pick $teststr $actualhead $actualtail]
        :if ($testip ~ $regexipwithsubnet) do={
            :if ($testip ~ "\\/0(0|\$)") do={
                :log warning "Invalid IP-prefix >$testip<"
            } else={
                :if ($testip ~ "\\/32") do={ :set testip [:pick $testip 0 [:find $testip "/32" -1]] }
                :if ([:len [find where list=$addlist and address=$testip]] = 0) do={ add list=$addlist address=$testip }
            }
        }
    }
}
 
szaboistvan007
just joined
Posts: 8
Joined: Mon Dec 28, 2020 9:52 pm

Re: Importing IP List from file

Wed Aug 11, 2021 6:40 pm

Hi guys, I have a different issue, using this example script: https://wiki.mikrotik.com/wiki/Using_Fe ... ress_Lists with IP source list downloaded from here: https://s3-eu-west-1.amazonaws.com/plex ... kiqIPs.txt
My script looks like this:
   ## Generic IP address list input
   ## Based on a script written by Sam Norris, ChangeIP.com 2008
   ## Edited by Andrew Cox, AccessPlus.com.au 2008
   :if ( [/file get [/file find name=sidekiqIPs.txt] size] > 0 ) do={
   # Remove exisiting addresses from the current Address list
   /ip firewall address-list remove [/ip firewall address-list find list=Plex]
   
   :global content [/file get [/file find name=sidekiqIPs.txt] contents] ;
   :global contentLen [ :len $content ] ;
   
   :global lineEnd 0;
   :global line "";
   :global lastEnd 0;
   
   :do {
         :set lineEnd [:find $content "\n" $lastEnd ] ;
         :set line [:pick $content $lastEnd $lineEnd] ;
         :set lastEnd ( $lineEnd + 1 ) ;
         #If the line doesn't start with a hash then process and add to the list
         :if ( [:pick $line 0 1] != "#" ) do={
   
        :local entry [:pick $line 0 $lineEnd ]
        :if ( [:len $entry ] > 0 ) do={
           /ip firewall address-list add list=Plex address=$entry
        }
      }
   } while ($lineEnd < $contentLen)
   }
My issue is the following:
URL has the following (example) IPs
54.170.120.91
46.51.207.89
(empty line)

Script generates this:
Flags: X - disabled, D - dynamic 
 #   LIST                        ADDRESS                                          CREATION-TIME        TIMEOUT             
 0   Plex                        54.170.120.91                                    aug/11/2021 18:28:58
 1   Plex                        46.51.207.89                                     aug/11/2021 18:28:58
 2   Plex                        4.170.120.91                                     aug/11/2021 18:28:58
So first line is duplicated on last line, with first character truncated.
I am not so good in scripting, and haven't managed to figure out why this happens.
Also I tried a lot of variants for this script without success.

Can you please advise?

Thanks!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Wed Aug 11, 2021 7:48 pm

Why advise you to fix a script already borned wrong?

I'll finish my example first than correct that sht.

My example do exactly what you want... from file of IP to address list.

@jvanhambelgium has abandoned, @Jotne doesn't say anything, at this point I finish the script, just wait
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3291
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Importing IP List from file

Wed Aug 11, 2021 8:14 pm

Found an error when you like to delete a large number of imported IP.
/ip dns static remove [find address=127.0.0.1]
action timed out - try again, if error continues contact MikroTik support and send a supout file (13)
It takes some minute to delete a big list, so I guess the limit for a command to run is passed, so it gives error.
Running the command again remove the rest of the lost. Here MT should increase the timeout to not get this message.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Wed Aug 11, 2021 8:18 pm

search tag # rextended delete without action timed out

Hi!

DNS static? Is about DNS static, not IP address-list

I just write a workaround... for you :mrgreen:
/ip dns static
{
    :local testcounter 0
    :put "Deletion in progress, please wait..."
    :foreach dns in=[find where address=127.0.0.1] do={
        :if (($testcounter % 10) = 0) do={ :put "deleted till now: $testcounter..." }
        :set testcounter ($testcounter + 1)
        remove [find where .id=$dns]
    }
    :put "Done.\r\nTotal deleted: $testcounter"
}
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Wed Aug 11, 2021 8:46 pm

@jvanhambelgium has abandoned, @Jotne doesn't say anything, at this point I finish the script, just wait
Nah, not really abandoned ;-) Just letting thing settle a bit....

For the previous poster on Plex, I do exactly the same using this script (that I recycle and use as generic list-loader)
The only thing is I've added that the script only deletes entries not having the comment "Static".
My Plex ACL contains the AWS IP's, but also entries that I've added manually (guests with fixed IP)
As you see "dynamic=no" I also do not touch/delete any dynamic entries that are created with for example "port-knock" sequences for temporary access.
So this works just fine.


{
/ip firewall address-list
:local update do={
:do {
:local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
:do { remove [find list=$blacklist comment!="Static" dynamic=no] } on-error={}

:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^([0-2]{0,1}[0-9]{1,2}\\.){3}[0-2]{0,1}[0-9]{1,2}(\\/[0-3]{0,1}[0-9]{1,1}){0,1}") do={
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) } on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
} ; :log warning "Imported address list < $blacklist> from file: $url"
} else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
} on-error={:log warning "Address list <$blacklist> update failed"}
}
$update url=https://s3-eu-west-1.amazonaws.com/plex ... kiqIPs.txt blacklist="Plex" delimiter=("\n")
}
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Wed Aug 11, 2021 8:52 pm

If you read post #25
viewtopic.php?f=9&t=166293#p872049

I have added some to-do in the list ;)

When I have time I finish the script.
That script can import any type of IP list, and help to not insert duplicate or double matched subnet, shrinking also the address list, like:

new 10.0.0.0/24 replace both already existant 10.0.0.0/25 and 10.0.0.0/28 because are included on /24

and do not add, for example, if 10.0.0.0/25 and 10.0.0.0/28 10.0.0.0/24 are already present on addres-list, because /24 include /25 and /28
Last edited by rextended on Thu Aug 12, 2021 1:12 pm, edited 3 times in total.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Wed Aug 11, 2021 9:16 pm

1) Read the file from server and handle the missing file, wrong response from server, etc. (the hard part)
2) Create a function and pass the parameters to it
3) Check the passed parameters if they are correct
4) Create whitelist, before add the IP / IP prefix check if it is on whitelist, then if is it, no add
5) Check on add if the ip-prefix is already present inside other IP-prefix already on address-list
6) Check on add if the ip-prefix is comprehensive of one or more IP-prefix on address-list, remove old(s) and add new bigger.
7) for security accept only from /12 to /32 prefix. /11 or less on IPv4 is too much big for be true...
And what about the splicing to fix the 64K limit ? Eg. if you look at the CN file (and US probably even bigger), it downloads a 530KBytes files with plenty of lines in it.
On Linux a very simple "split -l 3000 <filename>" would create a bunch of cna,cnb,cnc,... files and you can choose so it remains within the 64K limit of RouterOS
Is that even possible in RouterOS ? I think many people in the past have already looked at this but I never found and way for it searching through the forum.
I don't think its possible because you always have the start loading the file from the beginning, hence you'll run out of the first 64K entries and cannot look further?
 
szaboistvan007
just joined
Posts: 8
Joined: Mon Dec 28, 2020 9:52 pm

Re: Importing IP List from file

Wed Aug 11, 2021 10:24 pm

@jvanhambelgium has abandoned, @Jotne doesn't say anything, at this point I finish the script, just wait
Nah, not really abandoned ;-) Just letting thing settle a bit....

For the previous poster on Plex, I do exactly the same using this script (that I recycle and use as generic list-loader)
The only thing is I've added that the script only deletes entries not having the comment "Static".
My Plex ACL contains the AWS IP's, but also entries that I've added manually (guests with fixed IP)
As you see "dynamic=no" I also do not touch/delete any dynamic entries that are created with for example "port-knock" sequences for temporary access.
So this works just fine.


{
/ip firewall address-list
:local update do={
:do {
:local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
:do { remove [find list=$blacklist comment!="Static" dynamic=no] } on-error={}

:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^([0-2]{0,1}[0-9]{1,2}\\.){3}[0-2]{0,1}[0-9]{1,2}(\\/[0-3]{0,1}[0-9]{1,1}){0,1}") do={
:do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) } on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
} ; :log warning "Imported address list < $blacklist> from file: $url"
} else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
} on-error={:log warning "Address list <$blacklist> update failed"}
}
$update url=https://s3-eu-west-1.amazonaws.com/plex ... kiqIPs.txt blacklist="Plex" delimiter=("\n")
}
Thanks, this works like a charm!
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Importing IP List from file

Wed Aug 11, 2021 11:13 pm

This all has been adressed in one of the largest treads her about blacklisting.Then we got the flexible way by Shumkov but it limited to 64 KB file.

viewtopic.php?t=98804https://forum.mikr ... hp?t=98804

Removing one address in a big address-list is very inefficient. If you make changes in a current address-list create 'delta' with you changes.

Importing a new list is removing the whole list, import the delta and then the new list. The delta entries are no overwritten when you use on-error.

O, if you want to exclude a number of countries then you could turn that around and just define which countries you want to accept and exclude the rest.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Wed Aug 11, 2021 11:14 pm

And what about the splicing to fix the 64K limit ?
I think you do not read
viewtopic.php?f=9&t=166293#p872295

But about the 64K limits...
You understand if I write one solution for that they make me a monument on the forum ...
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Wed Aug 11, 2021 11:25 pm

This all has been adressed in one of the largest treads her about blacklisting.Then we got the flexible way by Shumkov but it limited to 64 KB file.

viewtopic.php?t=98804https://forum.mikr ... hp?t=98804

No one link longer work, all is abandoned because no one pay the service...
But you don't check what you recommend, do you put random links?


I share all the code, @IntrusDave even if it has stopped, don't want to share it.
I do it for free, @IntrusDave saw it only as an opportunity to make money ...

@IntrusDave 13 Jun 2019
Sorry, but I feel no need to disclose my stats and financial needs for a service that is free.
I can tell that you 4 servers, 120 honeypots, a CDN, storage and the bandwidth needed for all of it is quite a lot.
I wont be open sourcing the code either. it's 100% written by my with no use of any open source code.
It has a use to me still and I will be keeping it for myself.


Removing one address in a big address-list is very inefficient. If you make changes in a current address-list create 'delta' with you changes.
Importing a new list is removing the whole list, import the delta and then the new list. The delta entries are no overwritten when you use on-error.
My script do not remove previous entry (unless specified), and check for duplicates, and do more without use on-error on any point.
Also I develop a way to shrink address list removing overlapping entries, if you read my previous posts , you see the scripts and the descriptions.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Thu Aug 12, 2021 12:08 am

And what about the splicing to fix the 64K limit ?
I think you do not read
viewtopic.php?f=9&t=166293#p872295

But about the 64K limits...
You understand if I write one solution for that they make me a monument on the forum ...
For sure this import-to-variables-cannot-exceed-64K limit within RouterOS has driven quite some people mad.
At this point the only alternative was some separate processing and then splitted loading back into RouterOS
I wonder what you have pulled out of the sleeve 8)
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Thu Aug 12, 2021 12:41 am

Aaaaahhhhhh.... did you expect it ??? :lol: :lol: :lol:

viewtopic.php?f=9&t=177530#p872372
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Importing IP List from file

Thu Aug 12, 2021 12:26 pm

This is an example so can load HUGE address-lists. I prepare them on a Linux system with a script. Then I download them in files with schedule.

viewtopic.php?f=2&t=157591&p=775032&hil ... st#p775032

It will not overwrite any existing addresses.

The stated time is the longest possible time in ROS.

This format is made to have small files and so the easier to transfer.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Thu Aug 12, 2021 12:29 pm

I have solved the problem, without transit from another machine:
viewtopic.php?f=9&t=177530#p872372
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Importing IP List from file

Thu Aug 12, 2021 12:44 pm

I have solved the problem, without transit from another machine:
viewtopic.php?f=9&t=177530#p872372
Perhaps also an option like @mkx did with the time-out value set to highest possible value (= so it becomes a dynamic entry) so indeed it does not get written to nvram and only consumes memory.
So add some true/false settable variable in the script?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Thu Aug 12, 2021 1:11 pm

Added
8 ) Set an option for put the IP on the address-list but on temporary way (Dynamic) for specified time (from 1 second to near 35 weeks),
this do not export this type of IP on address-list on export or backup
whith this option set, if the address is found again on the imported list, instead to delete it and re-import, have time resetted again (from 1 second to near 35 weeks)
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Importing IP List from file

Thu Aug 12, 2021 3:03 pm

@rextended, "ignore list" means I can't see and won't open the hidden content, being your postings.

Over and out.
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3291
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Importing IP List from file

Fri Aug 13, 2021 11:50 am

HMM, delete 10000 IP using command
/ip dns static remove [find address=127.0.0.1]
Gives 10000 log lines like this
system,info MikroTik: static dns entry changed by xxx
system,info MikroTik: static dns entry changed by xxx
system,info MikroTik: static dns entry changed by xxx
Instead of just one log.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Fri Aug 13, 2021 12:08 pm

quote

You try my procedure? I think probably do the log...
viewtopic.php?f=9&t=166293#p872347
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3291
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Importing IP List from file

Fri Aug 13, 2021 12:25 pm

Will test it, but since it just run the same remove command, but at different bulks, it will give one log for each line it deletes.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Importing IP List from file

Fri Aug 13, 2021 2:04 pm

A few years logging this was being addressed in a tread about large list import. Then the way was to specific disable logging causing these entries.

Search result: search.php?keywords=logging&t=98804&sf=msgonly
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Fri Aug 13, 2021 2:37 pm

Will test it, but since it just run the same remove command, but at different bulks, it will give one log for each line it deletes.
They'll log for sure, I was wondering if you tried it for that timeout issue ...

The only way not to log in is ... temporarily disable the log,

BUT if the script exits by mistake, the logs are not re-enabled,
AND if at that moment there is something important to log, do not log...


13:40:37 system,info static dns entry added by admin
13:40:39 system,info static dns entry changed by admin
13:40:40 system,info static dns entry removed by admin

Just ask and I try to solve the problem...

something like this:
:global loglist [/system logging find where disabled=no topics~"((^|,)info|(^|,)system)"]
/system logging disable $loglist

# ... all the script ...

/system logging enable $loglist

why (^|,)info|(^|,)system ???
because must match where info OR system are active BUT MUST NOT MATCH where !info and/or !system are present!!!

match where on topic are the first item (or the only item) where the "," are not present
topics=info
topics=system

and match where the info or system are not the first item:
topics=wireless,info
topics=wireless,system

but do not match if the info or system must be excluded, like
topics=wireless,!info
topics=wireless,!system

this match system, but not info:
topics=wireless,!info,system
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Importing IP List from file

Sun Aug 15, 2021 1:03 pm

This all has been adressed in one of the largest treads her about blacklisting.Then we got the flexible way by Shumkov but it limited to 64 KB file.

viewtopic.php?t=98804https://forum.mikr ... hp?t=98804

No one link longer work, all is abandoned because no one pay the service...
But you don't check what you recommend, do you put random links?


I share all the code, @IntrusDave even if it has stopped, don't want to share it.
I do it for free, @IntrusDave saw it only as an opportunity to make money ...

@IntrusDave 13 Jun 2019
Sorry, but I feel no need to disclose my stats and financial needs for a service that is free.
I can tell that you 4 servers, 120 honeypots, a CDN, storage and the bandwidth needed for all of it is quite a lot.
I wont be open sourcing the code either. it's 100% written by my with no use of any open source code.
It has a use to me still and I will be keeping it for myself.
Removing one address in a big address-list is very inefficient. If you make changes in a current address-list create 'delta' with you changes.
Importing a new list is removing the whole list, import the delta and then the new list. The delta entries are no overwritten when you use on-error.
My script do not remove previous entry (unless specified), and check for duplicates, and do more without use on-error on any point.
Also I develop a way to shrink address list removing overlapping entries, if you read my previous posts , you see the scripts and the descriptions.
Have are read here and you don't know happened for the last years in this forum.

viewtopic.php?f=9&t=136666&p=724664&hilit=ill#p714663

The way to read and optimize import of address list was alway an obstacle for huge lists. Several member here, worked on that for several months to find the optimum way to import and this was the result.

I have later written a way to exchange separate lists be exchanged between router based on this way of working. But now, I keep that also for myself, no interest anymore on my side to share that.

What David did not wanted share was the eco-system behind creating the lists and the distribution. What scripts do, did not belong to him and was a group effort. You are free to use the scripts or not.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Importing IP List from file

Sun Aug 15, 2021 4:51 pm

Your linked post: 12 Feb 2019, 02:21

You do not have read this from @IntrusDave?
viewtopic.php?f=9&t=136666&p=732876#p732876
28 May 2019, 04:43
I regret to inform you all that I have shut down the servers.
With over 35,000 routers hitting the server every few hours, and only 10 supporters (totally $50/month), the expenses are not even remotely being covered. I'm putting out almost $500/month now just in bandwidth costs.

I'm moving the code to my in home server where it will just be supporting my personal units now.

it was a good run. I tried to keep it going, I thought that more people would be willing to help, but sadly not.
**************************************

@msatter wrote
I have later written a way to exchange separate lists be exchanged between router based on this way of working.
But now, I keep that also for myself, no interest anymore on my side to share that.
I find this "I have it and I don't give it to you" attitude very rude and disrespectful for all forum users.
If you have something and you don't want to share it, keep it to yourself.
Are you trying to get the users of the froum to PRAY you because you release the "script"?
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Importing IP List from file

Sun Aug 15, 2021 7:06 pm


**************************************

@msatter wrote
I have later written a way to exchange separate lists be exchanged between router based on this way of working.
But now, I keep that also for myself, no interest anymore on my side to share that.
I find this "I have it and I don't give it to you" attitude very rude and disrespectful for all forum users.
If you have something and you don't want to share it, keep it to yourself.
Are you trying to get the users of the froum to PRAY you because you release the "script"?
The only who is disrespectfull and rude is you. So please, PISS OFF.
 
jult
Frequent Visitor
Frequent Visitor
Posts: 52
Joined: Sat Dec 26, 2020 1:16 am

Re: Importing IP List from file

Mon Oct 18, 2021 1:03 pm


**************************************

@msatter wrote

I find this "I have it and I don't give it to you" attitude very rude and disrespectful for all forum users.
If you have something and you don't want to share it, keep it to yourself.
Are you trying to get the users of the froum to PRAY you because you release the "script"?
The only who is disrespectfull and rude is you. So please, PISS OFF.
I, and probably most others, would disagree. You are a selfish cunt for not sharing it.
 
texmeshtexas
Member Candidate
Member Candidate
Posts: 151
Joined: Sat Oct 11, 2008 11:17 pm

Re: Importing IP List from file

Wed May 11, 2022 11:58 pm

I know this post is a little stale but I'm trying to import this list
http://blacklists.co/download/all.txt

Its seems to be too long to fit in a variable method like used in this post.
The file is only a list of IPs, not in .rsc format.
can get the fine onto MT but how to process and covert to an Address list?
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Importing IP List from file

Thu May 12, 2022 12:04 am

Follow the link in posting five by me.

Who is online

Users browsing this forum: norepto and 13 guests