Splitting/parsing variable data

upnort · Tue Aug 20, 2019 7:43 pm

What is the command equivalent of bash awk or cut? I found the ":parse" function but am unable to get that to succeed.

When I run the following command:

:put [/system resource get version]

I see something like:

6.44.5 (long-term)

I want to return only the version number. That is, use the space as the delimiter.

Thank you!

Jotne · Tue Aug 20, 2019 7:57 pm

This should do:

{
:local ver [/system resource get version]
:put [:pick $ver 0 [:find $ver " "]]
}

:local ver [/system resource get version] Get the line
[:find $ver " "] gets position of space
[:pick $ver 0 [:find $ver " "]] get text from position 0 to first space

See manual here:
https://wiki.mikrotik.com/wiki/Manual:Scripting

find :find <arg> <arg> <start> return position of substring or array element :put [:find "abc" "a" -1]

pick :pick <var> <start>[<end>] return range of elements or substring. If end position is not specified, will return only one element from an array. :put [:pick "abcde" 1 3]

pe1chl · Tue Aug 20, 2019 8:50 pm

It would be nice when the scripting language was extended with some features known from e.g. Perl, like "split" and "regexp match with capture".
These operations would take a line of characters and either a splitting character (like a space in your case) or a regular expression describing the line and what you want to get from it, and return the results as an array holding the extracted fields.
Until then, indeed you will have to use (possibly repeated) use of find and pick.

upnort · Tue Aug 20, 2019 9:07 pm

Thank you!

Raffa2023 · Wed Nov 08, 2023 2:38 am

Hi all,,
I also need a script to parse data from comments in secrets.

As an example:

Comment secret A:
081234567890|West Java|John Due

Comment secret B:
08123456789|East Java|Michael Jackson

Comment secret C:
0812345678901|Central Java|Clara Bernadeth

And using "|" as a separator.

How do I retrieve telephone, address and name data if the data is random like the example above?

Thank you

rextended · Wed Nov 08, 2023 4:12 pm

example without any cycle code

{
:local test "081234567890|West Java|John Due"
:put [:pick $test (                                   -1    + 1)                  [:find $test "|" -1]  ]
:put [:pick $test (                  [:find $test "|" -1]   + 1) [:find $test "|" [:find $test "|" -1]] ]
:put [:pick $test ( [:find $test "|" [:find $test "|" -1]]  + 1)                  [:len  $test       ]  ]
}

diamuxin · Thu Nov 09, 2023 1:00 am

example without any cycle code

{
:local test "081234567890|West Java|John Due"
:put [:pick $test                                           0  [:find $test "|"                   -1] ]
:put [:pick $test ([:find $test "|"                   -1] + 1) [:find $test "|" [:find $test "|" -1]] ]
:put [:pick $test ([:find $test "|" [:find $test "|" -1]] + 1) [:len  $test                         ] ]
}

Hi Rex,

Could you explain this part of the script? (I see that it is to get the second character "|" but I do not understand it.

[:find $test "|" [:find $test "|" -1]]

Tnx.

rextended · Thu Nov 09, 2023 1:35 am

The spaces were not placed by chance...

If [:find $test "|" -1] identify the position of the first "|"

([:find $test "|" -1] + 1) is the next character from the |

so if you start the next :find from ([:find $test "|" -1] + 1) you obtain the next "|", etc...

{
:local test "081234567890|West Java|John Due"
:put [:pick $test (-1 + 1) [:find $test "|" -1] ]
:put [:pick $test ([:find $test "|" -1] + 1) [:find $test "|" [:find $test "|" -1]] ]
:put [:pick $test ([:find $test "|" [:find $test "|" -1]] + 1) [:len $test ] ]
}

so, if you add another item:

{
:local test "081234567890|West Java|John Due|another one"
:local s    "|" ; # separator
:put [:pick $test (                                                -1    + 1)                                 [:find $test $s -1]   ]
:put [:pick $test (                                [:find $test $s -1]   + 1)                 [:find $test $s [:find $test $s -1]]  ]
:put [:pick $test (                [:find $test $s [:find $test $s -1]]  + 1) [:find $test $s [:find $test $s [:find $test $s -1]]] ]
:put [:pick $test ([:find $test $s [:find $test $s [:find $test $s -1]]] + 1)                                 [:len  $test      ]   ]
}

diamuxin · Thu Nov 09, 2023 1:51 am

A little difficult but with your excellent explanation I understand.
Thank you magister!

rextended · Thu Nov 09, 2023 1:54 am

Is just an example for a little complex string, if the fields are more, a cycle is better...

diamuxin · Fri Nov 17, 2023 3:30 pm

Is just an example for a little complex string, if the fields are more, a cycle is better...

What would be the way to do it without pick?

I have tried as an array but only "the comma" works as a separator between values. The vertical bar does not work.

{
:local test [:toarray "081234567890|West Java|John Due|Another one|Second item"]
:put ($test->0) 
:put ($test->1)
:put ($test->2)
:put ($test->3)
:put ($test->4)
}

excuse my ignorance.

BR.

Amm0 · Fri Nov 17, 2023 4:20 pm

Using :toarray "trick" to parse a delimated string doesn't know about | being a separator. By "cycle", that means a ":for" loop with a :pick and :find inside...since :toarray isn't really a parser, it just conveniently use comma in syntax to define elements .

rextended · Fri Nov 17, 2023 4:44 pm

I can (uselessly) write a method for not use at all pick for split a string, but if pick exist, why not use it?

For convert x|y|z on one array of 3 elements, two way are simpler: first replace inside the string all | with , and use :toarray
or use a "while" until you do not process the string from the start to the end and create a new string inside the array fore each | and the last at the end of the string.

pe1chl · Fri Nov 17, 2023 4:48 pm

Normally the reason to use some separator is because it is known that this separator cannot occur in the data.
I often use <TAB> but in this case apparently | was chosen. I would not want to change all | to , and then use :toarray, it may well be that some fields between the | separators can have , in the value, and chaos would result.
What we really need is script functions that can convert a line to an array using user-defined separator or better: regexp with capture.
At one time we were promised a script library with useful functions like that...

Amm0 · Fri Nov 17, 2023 5:05 pm

What we really need is script functions that can convert a line to an array using user-defined separator or better: regexp with capture.

FWIW, I put in a feature request on the v7.13beta's :serialize and :deserialize to add support for "delimited file" (CSV/TSV/etc). While the new commands add JSON in beta, they do support a from= and to=<type>...

But... I'd agree a :regexp command (with capture/grouping) that be similar to `sed` be useful & also solve parsing string more generically.

rextended · Fri Nov 17, 2023 6:30 pm

it may well be that some fields between the | separators can have , in the value, and chaos would result.

Not, just simply replace that "," with (for example) $$$comma$$$ and when array is created, convert back $$$comma$$$ to "," on each item.

diamuxin · Fri Nov 17, 2023 7:28 pm

Thank you all for your advice.

BR.

diamuxin · Fri Nov 17, 2023 7:54 pm

For convert x|y|z on one array of 3 elements, two way are simpler: first replace inside the string all | with , and use :toarray
or use a "while" until you do not process the string from the start to the end and create a new string inside the array fore each | and the last at the end of the string.

Something like that, right?

{
:local string "081234567890|West Java|John Due|Another one|Second item"
:local reEncoded

:for i from=0 to=([:len $string] - 1) do={ 
    :local char [:pick $string $i]
    :if ($char = "|") do={:set $char ","}
    :set reEncoded ($reEncoded . $char) 
}

:local test [:toarray $reEncoded]
:put ($test->0) 
:put ($test->1)
:put ($test->2)
:put ($test->3)
:put ($test->4)
}

Output:
081234567890
West Java
John Due
Another one
Second item

I have adapted this function that I have found in the Scripting forum and that is valid for what I need, I share it.

:global strEncode do={
    :local string $1
    :local reEncoded
    :local char
    :if (([:typeof $string] != "str") or ($string = "")) do={ 
        :return "" 
    } else={
        :for i from=0 to=([:len $string] -1) do={ 
            :set char [:pick $string $i]
            :if ($char = "|") do={:set $char ","}
            :if ($char = "!") do={:set $char ","}
            :set reEncoded ($reEncoded . $char) 
        }
        :return $reEncoded
    }
}

Use:
:put [$strEncode "081234567890|West Java|John Due|Another one!Second item"]
Output:
081234567890,West Java,John Due,Another one,Second item

pe1chl · Fri Nov 17, 2023 10:33 pm

Remember that this is a recipe for "injection attacks" similar to the wellknown "SQL injection attacks" that are so often used to hack websites, online stores, etc., and also like using eval() to quickly parse JSON data into JavaScript.
This can really only be used when the data to be parsed can be fully trusted, not when there is any possibility that an outsider could modify the data e.g. via a form or some other data entry.

diamuxin · Sat Nov 18, 2023 12:08 am

The data used is for learning use only.

pe1chl · Sat Nov 18, 2023 11:17 am

The data used is for learning use only.

Doesn't matter. What matters is the SOURCE of the data. Is it from a file or database you edit yourself or is it somehow imported from another system that you cannot fully trust. That is what matters.

diamuxin · Sat Nov 18, 2023 12:42 pm

Ok I understand. The data comes from computers INSIDE my LAN. They are not external data. I'm careful with that evidently,
But thanks for the advice.

Splitting/parsing variable data [SOLVED]

Splitting/parsing variable data [SOLVED]

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

example without any cycle code

Re: Splitting/parsing variable data

example without any cycle code

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Re: Splitting/parsing variable data

Who is online