I am trying to use the deserialize command (which previously did work), but today it is throwing errors.
I can't remember if I updated ROS before or after successfully using deserialize, but I am getting an error related to json now when I am specifying dsv explicitly.
Command:
:put [:deserialize [/file/get unix_line_endings.txt contents] delimiter="\n" from=dsv options=dsv.plain]
Error:
invalid json
What is the deal with this?
I believe line length is problem here.
I have 2 files, one with few lines, and one with many lines. The files are the same except for length. The long file fails with JSON error, the short file has no issues.
commented on similar issue here: How to Read line by line from a file using a script? - #35 by andar1an
Looking into:
The JSON error message is a bug, it always says "invalid JSON" for all errors, even if it's a "DSV".
I have not tested the limit, but there is one. Certainly no variables can exceed 64K, so if your file is better, that's the problem.
Thank you @Amm0.
Do you know if it is possible to split a downloaded file with scripting without having to do the fetch byte portions linked above.
E.g. is it possible to just count lines and split file with cli tools.
Alternatively, is it possible to access a linux shell in ROS?
I see there is an API, but I am configuring new hardware.
Optimally I would love to just build an immutable ROS image in linux and deploy, but I dont know if that is possible.
Nope. Most things are limited to 64Kb. So basically two other choices (beyond HTTP range + fetch approach):
/container to process it, so that's how you'd get a shell.:import, will allow very large file (AFAIK) but it has to be RouterOS commands. For example, if this CSV is eventually going into an address-list, you can use awk to generate the entire RouterOS command then import the awk-generate RouterOS commands, and by-pass scripting functions like serialize/convert/etc.Thank you. Im trying to avoid containers on the network hardware. I was initially using import 
, I saw deserialize and thought it may limit work haha. I can process on my server and fetch from as you said.
Does anyone install and run Alpine as OS on Mikrotik hardware instead of ROS? I was looking into that a bit today, but am worried it is not trivial.
Here is something that I created for POC for parsing quite large IP list and adding them into address list (shell script generates rsc for import), and it is fastest approach compared with file chunked read with pure RSC because it uses multiple cores for parsing by awk (controlled by maxParallelJobs variable):
:local listUrl "https://public-dns.info/nameservers-all.txt"
:local listName "PUBLIC_DNS"
:local listComment "[dnslist]"
:local listTimeout 35w
:local validRecordRegex "^[^#][0-9]{0,2}\\\\.[0-9]{1,3}\\\\.[0-9]{1,3}\\\\.[0-9]{1,3}"
:local maxParallelJobs 3
:local containerName "linux"
:local containerMount "linux_share"
:local containerWaitTimeoutSec 60
:local stopContainerWhenDone false
:local rand [:rndnum from=1000 to=9999]
:local shellScriptName "import_list_$rand.sh"
:local importRscName "import_list_$rand.rsc"
:local containerTmpDir "/tmp"
:local successMessagePrefix "RSC import file generated with number of records: "
:local shellScript "#!/bin/bash
LIST_URL='$listUrl'
WORK_DIR=\"import_list_$rand\"
DOWNLOAD_FILE='import_list.txt'
CHUNK_PREFIX='import_list_'
CWD=\"\$(pwd)\"
cleanup() {
cd \"\${CWD}\"
rm -rf \"\${WORK_DIR}\"
}
mkdir -p \"\${WORK_DIR}\"
if [ \$? -ne 0 ]; then
echo \"Unable to create work dir: \${WORK_DIR}\"
exit 1
fi
cd \"\${WORK_DIR}\"
curl -s -o \"\${DOWNLOAD_FILE}\" \"\${LIST_URL}\"
if [ \$? -ne 0 ]; then
echo \"Error downloading list from: \${LIST_URL}\"
cleanup
exit 1
fi
split -e -d -a 1 -n l/$maxParallelJobs --additional-suffix='.txt' \"\${DOWNLOAD_FILE}\" \"\${CHUNK_PREFIX}\"
if [ \$? -ne 0 ]; then
echo \"Error splitting list file: \${DOWNLOAD_FILE}\"
cleanup
exit 1
fi
chunk_count=0
for filename in \"\${CHUNK_PREFIX}\"*.txt; do
awk -v valid_record_regex='$validRecordRegex' '{
for (line=1; line<=NF; line++) {
if (\$line ~ valid_record_regex) {
print \":onerror e {/ip/firewall/address-list/add list=\\\"$listName\\\" address=\"\$line\" timeout=$listTimeout comment=\\\"$listComment\\\"} do={:log debug \\\"Unable to add address \"\$line\" to address list $listName, reason: \$e\\\"} \"
}
}
}' \"\${filename}\" > \"\${CHUNK_PREFIX}\${chunk_count}.rsc\" &
chunk_count=\$((chunk_count + 1))
done
wait
find . -type f -name \"\${CHUNK_PREFIX}*.rsc\" -exec cat {} + >> \"$importRscName\"
if [ \$? -ne 0 ]; then
echo \"Error concatenating parsed chunks\"
cleanup
exit 1
fi
RECORDS=\$(wc -l \"$importRscName\" | cut -d ' ' -f 1)
cp \"$importRscName\" \"\${CWD}/\"
cleanup
echo \"$successMessagePrefix\${RECORDS}\"
"
:log info "Public DNS servers list import started"
:local quitWIthError do={
:log error $1
:error $1
}
/container
:if ([:len [find name=$containerName mounts=[
:local findMount do={
:foreach m in=$2 do={ :if ($m = $1) do={ :return true } }
:return false
}
:if ([$findMount $containerMount $mounts]) do={ :return $mounts }
]]] = 0) do={ $quitWIthError ("Container: $containerName not found or mount: $containerMount is not assigned to container") }
:if (![get $containerName running]) do={
:if ([get $containerName stopping]) do={
:local sec 0
:local timeout ($containerWaitTimeoutSec * 2)
:while (![get $containerName stopped] && $sec < $timeout) do={
:delay .5
:set sec ($sec + 1)
}
:if ($sec >= $timeout) do={ $quitWIthError ("Timeout while waiting container: $containerName") }
}
:if (![get $containerName starting]) do={ start $containerName }
:local sec 0
:local timeout ($containerWaitTimeoutSec * 2)
:while (![get $containerName running] && $sec < $timeout) do={
:delay .5
:set sec ($sec + 1)
}
:if ($sec >= $timeout) do={ $quitWIthError ("Timeout while waiting container: $containerName") }
}
:local srcDir [mounts/get $containerMount src]
/file/add contents=$shellScript name="$srcDir/$shellScriptName"
:local dstDir [mounts/get $containerMount dst]
:local res [shell $containerName cmd="2>&1 mv -f \"$dstDir/$shellScriptName\" \"$containerTmpDir\" && 2>&1 cd \"$containerTmpDir\" && 2>&1 chmod 755 \"$shellScriptName\" && 2>&1 ./\"$shellScriptName\" && 2>&1 mv -f \"$importRscName\" \"$dstDir/\" && 2>&1 rm -f \"$shellScriptName\"" as-value]
:if (!($res ~ "^$successMessagePrefix[0-9]+")) do={ $quitWIthError ("Error while executing script in container: $containerName, reason: \"$res\"") }
:local records [:tonum [:pick $res [:len $successMessagePrefix] [:len $res]]]
:local importScriptPath "$srcDir/$importRscName"
:if ($records > 0) do={
/ip/firewall/address-list
remove [find dynamic list=$listName comment=$listComment]
:local jobId [:execute ":import file-name=\"$importScriptPath\" verbose=yes"]
/system/script/job
:while ([:len [find .id=$jobId]] != 0) do={ :delay .2 }
} else={
:log warn "Public DNS servers list was parsed empty. Check for script errors!"
}
:if ([:pick $importScriptPath 0 1] = "/") do={ :set importScriptPath [:pick $importScriptPath 1 [:len $importScriptPath]] }
/file/remove "$importScriptPath"
:if ($stopContainerWhenDone) do={ /container/stop $containerName }
:log info "Public DNS servers list import finished, records processed: $records"
Tested with linuxserver/openssh-server container which is Alpine based. For script to work container requires mount for in/out file sharing with container (variable containerMount).
They working on /app and have a /container/run, see 7.21+, which makes will make using a temporary container easier for one-off UNIX shell things. MikroTik does not yet have a basic Alpine shell (nor some busybox thing) for /app, which be a bit simpler. But can use /container to add the alpine image (search forum for instructions), but you do not have to start it – you can use /container/run [find...] cmd="awk ..." which start the container to do the command and stop when done. Container on RouterOS are more similar to starting a process (think chroot and cgroups), so should not be big impact for task since it basically just starting busybox which has the UNIX tools.
But Alpine is more useful as container than a base OS....since the idea of Alpine is very lightweight in CPU/memory/disk but has all standard shell things, not drivers. OpenWrt is option, in some case, I believe, although that not common and not tested.
I use alpine as my daily driver for all of my machines, it can handle anything I have thrown at it to date and is often used for network hardware. I would love to build images as I do for my awesome mikrotik hardware like I do for my servers.
I love mikrotik, i just think it would be wonderful if it was easier to replace RouterOS at least on arm based hardware with a linux distro like Alpine.
There is a lot of functionality (examples like http/2 for doh or easy secret management) that is late or missing compared to other options like openWRT (which can even use alpine packages), and I am finding I am wasting many hours with the scripting.
I am getting through with many workarounds, I just wish I knew where to look to see what kernel compilation looks like for hardware and how to backup license properly so I could try.
Hopefully I will find some OS related build code or some example when I look.
The RouterBOOT is restricted, so that's main thing. I know they add some support for loading OpenWRT, but not sure anyone tried but I'm not on OpenWrt forum/etc.
MikroTik has made a lot of progress in /container, so some of the complexity highlighted by @optio's excellent script can be avoid in upcoming stable. So I think there can be better blending of the Linux tasks to /container leaving RouterOS the complex use of the routing protocols, bridging and HW offloading switch chips (the latter is where the rubber meets the road in trying to use Linux on MikroTik hardware, since fully utilizing the specific SoC is not easy).
I have example in my TIKOCI repo of building RouterOS containers. I mainly use GitHub Actions to do it since you can then just use http://ghcr.io/... as a URL in container to access built containers. See https://tikoci.github.io and make.d(GitHub - tikoci/make.d: RouterOS-friendly Alpine /container, managed by `make`) which is an experimental multi-process Alpine container (using Makefile) or my netinstall container which is simpler and uses Alpine too.
Darn about the routerBoot, that sucks, I wonder if one can request an unlock, maybe it is locked for security but not as blocker.
Optios script is pretty serious! That is along lines of what I am doing, but I am avoiding containers on the router for security reasons and because im not a huge container fan. I have array of embedded systems downstream to run things.
I am more familiar with Alpine and would be more comfortable configuring that safely. I can try reaching out and see what support says. I imagine it is not optimistic but worth an ask. Mikrotik mainly makes money on hardware I would imagine? I dk.