How to Read line by line from a file using a script?

RouterOS Scripting
1

I want a MikroTik script to read a ‘.txt’ file and add the found IPs to a list used by a firewall rule. However, the addresses are not added to the list. Can you help me?

Code:

:local fileName "iplist.txt"
:local fileContents [/file get $fileName contents]
:local addressListName "MyAddressList" 

:foreach ip in=$fileContents do={
    /ip firewall address-list add list=$addressListName address=$ip
}

iplist.txt:

192.168.10.20
192.168.11.50
192.168.13.12

Obs: The IPs in the example are generic. This list has a larger number of IPs and is constantly updated automatically.

2

[lost by accident - I meant reply not edit - but was just example script two post below now]

3

Your suggestion worked. However, the ip 0.0.0.0, which did not exist in the file, was added. There were no empty lines.

4

The :deserialize is very new, so it may a bug in that actually. i.e. inserting an extra array element, that would unconverted to an IP type, which defaults to 0.0.0.0.

Do you have a sample file you can post?

But I think this should protect against that case:

{
    :local filebody [/file/get iplist.txt contents]
    :local addressListName "MyAddressList"
    :local expire 1h
    
    :foreach ip in=[ :deserialize [:tolf $filebody] delimiter="\n" from=dsv options=dsv.plain ] do={
        /log debug "adding $ip to $addressListName"
        :if ([:tostr $ip] != "0.0.0.0") do={
             :onerror e in={
                  /ip firewall address-list add list=$addressListName address=$ip timeout=$expire
              } do={/log debug "skipping $ip ($e)"}
        }
    }
}

(updated with error checking for duplicates)

The “old school” way of doing this is to read each char using [:pick] and using two arrays to store, one for the currently in-progress parse one, and another a list of all IP found (when a \n is the char found in buffer). There is nothing like a readline() from file in RouterOS otherwise.

5

The ip 0.0.0.0 was set when I created a test file on Linux. When I downloaded the file from a URL I had no problem.

Script:

# Variables
:local url "https://myurl/blocklist.txt"
:local fileName "blocklist.txt"
:local addressListName "blocklist"
:local filebody
:local delayTime 5s

# Download the specified file
/tool fetch url=$url mode=https dst-path=$fileName

# 5 second delay
:delay $delayTime

# Remove all addresses from the specified list
/ip firewall address-list remove [find list=$addressListName]

# Reads the contents of the file and stores it in the local variable filebody
:set filebody [/file get $fileName contents]

# Iterate over each IP from the file contents and add it to the specified list of addresses
:foreach ip in=[:deserialize $filebody delimiter="\n" from=dsv options=dsv.plain] do={
    :put $ip
    /ip firewall address-list add list=$addressListName address=$ip
}

# Command to remove the file
/file remove $fileName
6

Ah, you might have needed a delimiter=“\r\n” if the file was created on RouterOS since it uses windows line-endings.

7

All the problems already emerged to import IPs from a file, present in other posts, will also be duplicated in this topic.

Here it seems that everything starts from scratch, ignoring what has ALREADY been done.

It is true that now there is :deserialize, but all the rest of the problems remain the same…
(like avoid block 0.0.0.0, avoid block own IPs, avoid unprotect the network during import***, different delimiter, etc.)


*** do not delete/remove, simply set dynamic timeout > update interval and import just what not already exist…


Avoid add/remove from permanent config, use dynamic entry instead for preserve NAND/Flash

8

Not wrong… See https://help.mikrotik.com/docs/spaces/ROS/pages/130220135/Address-lists

If the timeout parameter is not specified, then the address will be saved to the list permanently on the disk. If a timeout is specified, the address will be stored on the RAM and will be removed after a system’s reboot.

And actually duplicates be another one. Updated example script to cover these.

I was more providing an example of the read line-by-line, than using address-list. But if the list is really long… @rextended is right with larger list more care may be needed. Like you might want to consolidate prefixes before adding them to the address-list.

9

There’s a function to convert CRLF to just LF. Added that it should handle both:

[...]
:foreach ip in=[ :deserialize [ :tolf $filebody ] delimiter="\n" from=dsv options=dsv.plain ] do={
[...]
10

As I see :deserialize from dsv always split by new line regardless which line ending and delimiter is used, no need for conversion:
iplist.txt (LF or CRLF):

1.1.1.1
2.2.2.2

same results regardless if is line ending LF or CRLF:

 > :put [:deserialize [/file/get iplist.txt contents] delimiter="\n" from=dsv options=dsv.plain]
1.1.1.1;2.2.2.2;
 > :put [:deserialize [/file/get iplist.txt contents] delimiter="\r\n" from=dsv options=dsv.plain]
1.1.1.1;2.2.2.2;
 > :put [:deserialize [/file/get iplist.txt contents] delimiter="." from=dsv options=dsv.plain]
1;1;1;1;2;2;2;2;
 > :put [:deserialize [/file/get iplist.txt contents] delimiter="foo" from=dsv options=dsv.plain]
1.1.1.1;2.2.2.2;
11

no matter what you put, everytime \r or \n (or both) are considered as delimiter, also if delimiter is something other…

[rex@v7] > :put [:deserialize "test\rtest" delimiter="<!>" from=dsv options=dsv.plain]   
test;test
[rex@v7] > :put [:deserialize "test\ntest" delimiter="<!>" from=dsv options=dsv.plain]   
test;test
[rex@v7] > :put [:deserialize "test\r\ntest" delimiter="<!>" from=dsv options=dsv.plain]   
test;test

[rex@v7] > :put [:deserialize "test\rte<!>st\ntest" delimiter="<!>" from=dsv options=dsv.plain]
test;te;st;test
12

exactly I wanted to prove with foo delimiter

13

@optio has a good point. delimiter= is the “field separator”, not the “record separator” (if I borrow awk’s terms). The default “record separator” is a “newline” it seems.. So the delimiter does not matter if there only one “field” per row, so the delimiter= does not need to look for the newline, since that’s implicit in a CSV-like thing.


Also looking at what it does is hard to see in a :put… since that doesn’t show the actual array structure :deserialize generates. So taking one of @rextended’s examples…

:put [:deserialize "test\rte<!>st\ntest" delimiter="<!>" from=dsv options=dsv.plain]
test;te;st;test

Using JSON let’s you see the the above “test;te;st;test” is not actually a simple list, rather a list of lists.

:put [:serialize to=json options=json.pretty [:deserialize "test\rte<!>st\ntest" delimiter="<!>" from=dsv options=dsv.plain]] 
[
    [
        "test"
    ],
    [
        "te",
        "st"
    ],
    [
        "test"
    ]
]

Or, more simply:

:put ([:deserialize "test\rte<!>st\ntest" delimiter="<!>" from=dsv options=dsv.plain]->1->1) 
st
14

It make sence to be 2-dimensional array (fields per lines). When new line separator is used or delimiter which doesn’t exists in text it will be always single filed per line and such can be used for full line reading…

15

Yeah I kinda abused it here for the more simple case (one field)… but it’s more designed to import a CSV file.

16

agree, same concluded when editing prev. post

17

Thanks for the tips. It would be possible to adapt the script I posted to work on RouterOS 6 and 7. It doesn’t work on version 6.