Community discussions

MikroTik App
 
cairoapcampos
just joined
Topic Author
Posts: 16
Joined: Sat Nov 25, 2023 2:58 am

How to Read line by line from a file using a script?

Mon Nov 04, 2024 9:47 pm

I want a MikroTik script to read a '.txt' file and add the found IPs to a list used by a firewall rule. However, the addresses are not added to the list. Can you help me?

Code:
:local fileName "iplist.txt"
:local fileContents [/file get $fileName contents]
:local addressListName "MyAddressList" 

:foreach ip in=$fileContents do={
    /ip firewall address-list add list=$addressListName address=$ip
}

iplist.txt:

192.168.10.20
192.168.11.50
192.168.13.12

Obs: The IPs in the example are generic. This list has a larger number of IPs and is constantly updated automatically.
Last edited by cairoapcampos on Tue Nov 05, 2024 10:53 pm, edited 2 times in total.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4263
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: How to Read line by line from a file using a script?

Mon Nov 04, 2024 10:21 pm

[lost by accident - I meant reply not edit - but was just example script two post below now]
Last edited by Amm0 on Tue Nov 05, 2024 8:16 pm, edited 2 times in total.
 
cairoapcampos
just joined
Topic Author
Posts: 16
Joined: Sat Nov 25, 2023 2:58 am

Re: How to Read line by line from a file using a script?

Tue Nov 05, 2024 7:46 pm

Your suggestion worked. However, the ip 0.0.0.0, which did not exist in the file, was added. There were no empty lines.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4263
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: How to Read line by line from a file using a script?

Tue Nov 05, 2024 8:16 pm

Your suggestion worked. However, the ip 0.0.0.0, which did not exist in the file, was added. There were no empty lines.
The :deserialize is very new, so it may a bug in that actually. i.e. inserting an extra array element, that would unconverted to an IP type, which defaults to 0.0.0.0.

Do you have a sample file you can post?

But I think this should protect against that case:
{
    :local filebody [/file/get iplist.txt contents]
    :local addressListName "MyAddressList"
    :local expire 1h
    
    :foreach ip in=[ :deserialize [:tolf $filebody] delimiter="\n" from=dsv options=dsv.plain ] do={
        /log debug "adding $ip to $addressListName"
        :if ([:tostr $ip] != "0.0.0.0") do={
             :onerror e in={
                  /ip firewall address-list add list=$addressListName address=$ip timeout=$expire
              } do={/log debug "skipping $ip ($e)"}
        }
    }
}
(updated with error checking for duplicates)

The "old school" way of doing this is to read each char using [:pick] and using two arrays to store, one for the currently in-progress parse one, and another a list of all IP found (when a \n is the char found in buffer). There is nothing like a readline() from file in RouterOS otherwise.
Last edited by Amm0 on Wed Nov 06, 2024 10:12 am, edited 2 times in total.
 
cairoapcampos
just joined
Topic Author
Posts: 16
Joined: Sat Nov 25, 2023 2:58 am

Re: How to Read line by line from a file using a script?

Tue Nov 05, 2024 10:45 pm

The ip 0.0.0.0 was set when I created a test file on Linux. When I downloaded the file from a URL I had no problem.

Script:
# Variables
:local url "https://myurl/blocklist.txt"
:local fileName "blocklist.txt"
:local addressListName "blocklist"
:local filebody
:local delayTime 5s

# Download the specified file
/tool fetch url=$url mode=https dst-path=$fileName

# 5 second delay
:delay $delayTime

# Remove all addresses from the specified list
/ip firewall address-list remove [find list=$addressListName]

# Reads the contents of the file and stores it in the local variable filebody
:set filebody [/file get $fileName contents]

# Iterate over each IP from the file contents and add it to the specified list of addresses
:foreach ip in=[:deserialize $filebody delimiter="\n" from=dsv options=dsv.plain] do={
    :put $ip
    /ip firewall address-list add list=$addressListName address=$ip
}

# Command to remove the file
/file remove $fileName
Last edited by cairoapcampos on Tue Nov 05, 2024 10:48 pm, edited 1 time in total.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4263
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: How to Read line by line from a file using a script?

Tue Nov 05, 2024 10:47 pm

Ah, you might have needed a delimiter="\r\n" if the file was created on RouterOS since it uses windows line-endings.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12530
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: How to Read line by line from a file using a script?

Wed Nov 06, 2024 1:40 am

All the problems already emerged to import IPs from a file, present in other posts, will also be duplicated in this topic.

Here it seems that everything starts from scratch, ignoring what has ALREADY been done.

It is true that now there is :deserialize, but all the rest of the problems remain the same...
(like avoid block 0.0.0.0, avoid block own IPs, avoid unprotect the network during import***, different delimiter, etc.)


*** do not delete/remove, simply set dynamic timeout > update interval and import just what not already exist...


Avoid add/remove from permanent config, use dynamic entry instead for preserve NAND/Flash
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4263
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: How to Read line by line from a file using a script?

Wed Nov 06, 2024 3:07 am

Not wrong... See https://help.mikrotik.com/docs/spaces/R ... ress-lists
If the timeout parameter is not specified, then the address will be saved to the list permanently on the disk. If a timeout is specified, the address will be stored on the RAM and will be removed after a system's reboot.
And actually duplicates be another one. Updated example script to cover these.

I was more providing an example of the read line-by-line, than using address-list. But if the list is really long... @rextended is right with larger list more care may be needed. Like you might want to consolidate prefixes before adding them to the address-list.
 
User avatar
eworm
Forum Guru
Forum Guru
Posts: 1090
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: How to Read line by line from a file using a script?

Wed Nov 06, 2024 9:58 am

Ah, you might have needed a delimiter="\r\n" if the file was created on RouterOS since it uses windows line-endings.
There's a function to convert CRLF to just LF. Added that it should handle both:
[...]
:foreach ip in=[ :deserialize [ :tolf $filebody ] delimiter="\n" from=dsv options=dsv.plain ] do={
[...]
 
optio
Forum Veteran
Forum Veteran
Posts: 914
Joined: Mon Dec 26, 2022 2:57 pm

Re: How to Read line by line from a file using a script?

Wed Nov 06, 2024 6:20 pm

As I see :deserialize from dsv always split by new line regardless which line ending and delimiter is used, no need for conversion:
iplist.txt (LF or CRLF):
1.1.1.1
2.2.2.2

same results regardless if is line ending LF or CRLF:
 > :put [:deserialize [/file/get iplist.txt contents] delimiter="\n" from=dsv options=dsv.plain]
1.1.1.1;2.2.2.2;
 > :put [:deserialize [/file/get iplist.txt contents] delimiter="\r\n" from=dsv options=dsv.plain]
1.1.1.1;2.2.2.2;
 > :put [:deserialize [/file/get iplist.txt contents] delimiter="." from=dsv options=dsv.plain]
1;1;1;1;2;2;2;2;
 > :put [:deserialize [/file/get iplist.txt contents] delimiter="foo" from=dsv options=dsv.plain]
1.1.1.1;2.2.2.2;
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12530
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: How to Read line by line from a file using a script?

Wed Nov 06, 2024 7:30 pm

no matter what you put, everytime \r or \n (or both) are considered as delimiter, also if delimiter is something other...
[rex@v7] > :put [:deserialize "test\rtest" delimiter="<!>" from=dsv options=dsv.plain]   
test;test
[rex@v7] > :put [:deserialize "test\ntest" delimiter="<!>" from=dsv options=dsv.plain]   
test;test
[rex@v7] > :put [:deserialize "test\r\ntest" delimiter="<!>" from=dsv options=dsv.plain]   
test;test

[rex@v7] > :put [:deserialize "test\rte<!>st\ntest" delimiter="<!>" from=dsv options=dsv.plain]
test;te;st;test
 
optio
Forum Veteran
Forum Veteran
Posts: 914
Joined: Mon Dec 26, 2022 2:57 pm

Re: How to Read line by line from a file using a script?

Wed Nov 06, 2024 7:51 pm

exactly I wanted to prove with foo delimiter
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4263
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: How to Read line by line from a file using a script?

Wed Nov 06, 2024 7:51 pm

@optio has a good point. delimiter= is the "field separator", not the "record separator" (if I borrow awk's terms). The default "record separator" is a "newline" it seems.. So the delimiter does not matter if there only one "field" per row, so the delimiter= does not need to look for the newline, since that's implicit in a CSV-like thing.


Also looking at what it does is hard to see in a :put... since that doesn't show the actual array structure :deserialize generates. So taking one of @rextended's examples...
:put [:deserialize "test\rte<!>st\ntest" delimiter="<!>" from=dsv options=dsv.plain]
test;te;st;test
Using JSON let's you see the the above "test;te;st;test" is not actually a simple list, rather a list of lists.
:put [:serialize to=json options=json.pretty [:deserialize "test\rte<!>st\ntest" delimiter="<!>" from=dsv options=dsv.plain]] 
[
    [
        "test"
    ],
    [
        "te",
        "st"
    ],
    [
        "test"
    ]
]

Or, more simply:
:put ([:deserialize "test\rte<!>st\ntest" delimiter="<!>" from=dsv options=dsv.plain]->1->1) 
st
Last edited by Amm0 on Wed Nov 06, 2024 7:58 pm, edited 2 times in total.
 
optio
Forum Veteran
Forum Veteran
Posts: 914
Joined: Mon Dec 26, 2022 2:57 pm

Re: How to Read line by line from a file using a script?

Wed Nov 06, 2024 7:53 pm

It make sence to be 2-dimensional array (fields per lines). When new line separator is used or delimiter which doesn't exists in text it will be always single filed per line and such can be used for full line reading...
Last edited by optio on Wed Nov 06, 2024 8:01 pm, edited 1 time in total.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4263
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: How to Read line by line from a file using a script?

Wed Nov 06, 2024 7:58 pm

It make sence to be 2-dimensional array (fields per lines)
Yeah I kinda abused it here for the more simple case (one field)... but it's more designed to import a CSV file.
 
optio
Forum Veteran
Forum Veteran
Posts: 914
Joined: Mon Dec 26, 2022 2:57 pm

Re: How to Read line by line from a file using a script?

Wed Nov 06, 2024 8:01 pm

agree, same concluded when editing prev. post
 
cairoapcampos
just joined
Topic Author
Posts: 16
Joined: Sat Nov 25, 2023 2:58 am

Re: How to Read line by line from a file using a script?

Fri Nov 08, 2024 9:16 pm

Thanks for the tips. It would be possible to adapt the script I posted to work on RouterOS 6 and 7. It doesn't work on version 6.

The ip 0.0.0.0 was set when I created a test file on Linux. When I downloaded the file from a URL I had no problem.

Script:
# Variables
:local url "https://myurl/blocklist.txt"
:local fileName "blocklist.txt"
:local addressListName "blocklist"
:local filebody
:local delayTime 5s

# Download the specified file
/tool fetch url=$url mode=https dst-path=$fileName

# 5 second delay
:delay $delayTime

# Remove all addresses from the specified list
/ip firewall address-list remove [find list=$addressListName]

# Reads the contents of the file and stores it in the local variable filebody
:set filebody [/file get $fileName contents]

# Iterate over each IP from the file contents and add it to the specified list of addresses
:foreach ip in=[:deserialize $filebody delimiter="\n" from=dsv options=dsv.plain] do={
    :put $ip
    /ip firewall address-list add list=$addressListName address=$ip
}

# Command to remove the file
/file remove $fileName

Who is online

Users browsing this forum: No registered users and 9 guests