Script error: already have such entry (address lists)

sjoram · Tue Mar 18, 2025 7:36 pm

I have a scheduled script which calls several rsc files to add a bunch of IPs to address lists on a dynamic basis.
The rsc files are compiled through a powershell script running separately on a Windows server, but if the same or an overlapping IP/subnet exists in multiple lists, it is not de-duplicated.
Is there any easy way to set the script to continue to run on error?

Strangely, I have 2 x RouterOS devices on separate networks using the same script and .rsc files and only one of the two is generating the error.
I've seen this error previously but always on both devices and next time the underlying internet list was updated, it seemed to disappear.

/tool fetch address=<redacted> host=<redacted> mode=https src-path=/blacklists/firehol1.rsc keep-result=yes;
/import firehol1.rsc;
/tool fetch address=<redacted> host=<redacted> mode=https src-path=/blacklists/firehol2.rsc keep-result=yes;
/import firehol2.rsc;
/tool fetch address=<redacted> host=<redacted> mode=https src-path=/blacklists/firehol3.rsc keep-result=yes;
/import firehol3.rsc;
/tool fetch address=<redacted> host=<redacted> mode=https src-path=/blacklists/fireholwebclient.rsc keep-result=yes;
/import fireholwebclient.rsc;
 /tool fetch address=<redacted> host=<redacted> mode=https src-path=/public/aws.rsc keep-result=yes;
/import aws.rsc;

Each rsc file only contains a whole bunch of these:

/ip firewall address-list remove [/ip firewall address-list find list=firehol_L1]
/ip firewall address-list add addres=1.10.16.0/20 list=firehol_L1 timeout=2d
/ip firewall address-list add addres=1.19.0.0/16 list=firehol_L1 timeout=2d
/ip firewall address-list add addres=1.32.128.0/18 list=firehol_L1 timeout=2d
/ip firewall address-list add addres=2.56.192.0/22 list=firehol_L1 timeout=2d
etc...

The powershell script contains:

# Delete old blacklist files
Remove-Item "C:\inetpub\wwwroot\blacklists\firehol_L1.txt"
Remove-Item "C:\inetpub\wwwroot\blacklists\firehol_L2.txt"
Remove-Item "C:\inetpub\wwwroot\blacklists\firehol_L3.txt"
Remove-Item "C:\inetpub\wwwroot\blacklists\firehol_client.txt"
Remove-Item "C:\inetpub\wwwroot\blacklists\uptimerobot.txt"
# Force TLS1.2
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
# Download latest blacklist files
$WebClient = New-Object System.Net.WebClient
$WebClient.DownloadFile("https://iplists.firehol.org/files/firehol_level1.netset","C:\inetpub\wwwroot\blacklists\firehol_L1.txt")
$WebClient.DownloadFile("https://iplists.firehol.org/files/firehol_level2.netset","C:\inetpub\wwwroot\blacklists\firehol_L2.txt")
$WebClient.DownloadFile("https://iplists.firehol.org/files/firehol_level3.netset","C:\inetpub\wwwroot\blacklists\firehol_L3.txt")
$WebClient.DownloadFile("https://iplists.firehol.org/files/firehol_webclient.netset","C:\inetpub\wwwroot\blacklists\firehol_webclient.txt")
$WebClient.DownloadFile("https://uptimerobot.com/inc/files/ips/IPv4.txt","C:\inetpub\wwwroot\blacklists\uptimerobot.txt")
# Delete old import scripts
Remove-Item "C:\inetpub\wwwroot\blacklists\firehol1.rsc"
Remove-Item "C:\inetpub\wwwroot\blacklists\firehol2.rsc"
Remove-Item "C:\inetpub\wwwroot\blacklists\firehol3.rsc"
Remove-Item "C:\inetpub\wwwroot\blacklists\fireholwebclient.rsc"
Remove-Item "C:\inetpub\wwwroot\blacklists\uptimerobot.rsc"
# Create Firehol L1 import script
$script= "C:\inetpub\wwwroot\blacklists\firehol1.rsc"
"/ip firewall address-list remove [/ip firewall address-list find list=firehol_L1]" | Add-Content $script
$blacklist= Get-Content "C:\inetpub\wwwroot\blacklists\firehol_L1.txt" | Where-Object {$_ -notmatch "^#|0.0.0.0|10.0.0.0|172.16.0.0/12|192.168.0.0|224.0.0.0"}
ForEach ($address in $blacklist){
"/ip firewall address-list add addres=$address list=firehol_L1 timeout=2d" | Add-Content $script
}
etc...

rextended · Tue Mar 18, 2025 8:03 pm

full of useless...

/ip firewall address-list remove [/ip firewall address-list find list=firehol_L1]
/ip firewall address-list add addres=1.10.16.0/20 list=firehol_L1 timeout=2d
/ip firewall address-list add addres=1.19.0.0/16 list=firehol_L1 timeout=2d
/ip firewall address-list add addres=1.32.128.0/18 list=firehol_L1 timeout=2d
/ip firewall address-list add addres=2.56.192.0/22 list=firehol_L1 timeout=2d

do not remove spaces where are presents.

/ip firewall address-list
remove [find where list=firehol_L1]
:do {add addres=1.10.16.0/20 list=firehol_L1 timeout=2d} on-error={ }
:do {add addres=1.19.0.0/16 list=firehol_L1 timeout=2d} on-error={ }
:do {add addres=1.32.128.0/18 list=firehol_L1 timeout=2d} on-error={ }
:do {add addres=2.56.192.0/22 list=firehol_L1 timeout=2d} on-error={ }

also this:
address=<redacted> host=<redacted> mode=https src-path=/blacklists/firehol1.rsc

full useless if you can do something like:
url="https://www.mysite.net/blacklists/firehol1.rsc"

full useless also all the useless ;

so the script change (also the IPs.....)

untested and unchecked part of PowerShell code

"/ip firewall address-list" | Add-Content $script
"remove [find where list=firehol_L1]" | Add-Content $script
$blacklist= Get-Content "C:\inetpub\wwwroot\blacklists\firehol_L1.txt" | Where-Object {$_ -notmatch "^#|0.0.0.0|10.0.0.0|100.64.0.0|127.0.0.0|169.254.0.0|172.16.0.0|192.168.0.0|224.0.0.0"}
ForEach ($address in $blacklist){
":do {add addres=$address list=firehol_L1 timeout=2d} on-error={ }" | Add-Content $script
}