Using two arrays, process the text and create a third array

[rextended]

Step 1: clean frills, fix concept erros, and add something that display the result:
{
:local smsStatus [:toarray “REC UNREAD,REC READ,STO UNSENT,STO SENT,ALL”]
:local modemOutput ([/interface lte at-chat lte1 wait=yes input=“AT+CMGL=4” as-value]->“output”)
:local search1 “+CMGL:”
:local search2 “\r\n”
:local matchPosition -1
:local matchCount 0
:local arrayStat [:toarray “”]

while ([:typeof [:find $modemOutput $search1 $matchPosition]] != “nil”) do={
:set matchPosition [:find $modemOutput $search1 $matchPosition]
:local tmpStat [:toarray [:pick $modemOutput ($matchPosition + 7) [:find $modemOutput $search2 $matchPosition]]]
:local subarray [:toarray “”]
:set ($subarray->“index” ) [:tonum ($tmpStat->0)]
:set ($subarray->“status”) ($smsStatus->[:tonum ($tmpStat->1)])
:if ([:len $tmpStat] = 3) do={
:set ($subarray->“alpha” ) “”
:set ($subarray->“length”) [:tonum ($tmpStat->2)]
} else={
:set ($subarray->“alpha” ) [:tonum ($tmpStat->2)]
:set ($subarray->“length”) [:tonum ($tmpStat->3)]
}
:set matchPosition [:find $modemOutput $search2 $matchPosition]
:set ($subarray->“sms”) ([:pick $modemOutput ($matchPosition + 2) [:find $modemOutput $search2 $matchPosition]])
:set ($arrayStat->$matchCount) $subarray
:set matchCount ($matchCount + 1)
}

:foreach item in=$arrayStat do={
:put “SMS index=$($item->“index”), status=$($item->“status”), phonebook name=$($item->“alpha”), pdu length=$($item->“length”)\r\n*** START PDU "
:put ($item->“sms”)
:put " END PDU ***\r\n\r\n”
}

}
If I continue to fix your code, at the end is identical to the mine.
Better if you go to Problem 6) Subproblem A), you need to write one separate function…
At the start try to convert just the string
07912180958739F1040B917120069876F000009140503223218A21D4F29C0E6A97E7F3F0B90CA2BF41412A68F86EB7C36E32885A9ED3CB72
ignoring all previous code, is useless at this step…

[optio]

If I continue to fix your code, at the end is identical to the mine.

And to continue to do that for several hundreds of lines for proper implementation… this will be a long thread

[DyadyaGenya]

Results:
ERROR: missing everytime last PDU hex value…

Very strange. I check the result of your script and mine and they have identical results in my opinion. Could you show what exactly is missing in my version? What symbols?

At the start try to convert just the string
07912180958739F1040B917120069876F000009140503223218A21D4F29C0E6A97E7F3F0B90CA2BF41412A68F86EB7C36E32885A9ED3CB72
ignoring all previous code, is useless at this step…

I have not yet begun to decode the body of the SMS itself, that is, such an excerpt as you showed. But I can even translate it a little with my hands. For example, the phone is SCA 12085978931, and the sender is 17026089670, TP-DCS is a 7-bit compressed encoding, the possible date of sending is 2019/04/05 23:32:12 A 8 time zone (probably China). Naturally, I’m not an expert and I’m not very good at decoding and I can make mistakes in this data, but I just started doing it.
I can select all these parts of the SMS using the script, including the SMS text itself, and at this stage I cannot yet decode the SMS text itself. The rest are no problems so far.

[rextended]

Results:
ERROR: missing everytime last PDU hex value…

Very strange. I check the result of your script and mine and they have identical results in my opinion. Could you show what exactly is missing in my version? What symbols?

Example:
using my code
07912180958739F1040B917120069876F000009140503223218A21D4F29C0E6A97E7F3F0B90CA2BF41412A68F86EB7C36E32885A9ED3CB72
usin your (unaltered or clean, no difference) code
07912180958739F1040B917120069876F000009140503223218A21D4F29C0E6A97E7F3F0B90CA2BF41412A68F86EB7C36E32885A9ED3CB7
do not notice the mising 2 at the end?

( 8A is GMT -7 (28 times -15) as explained before, is on USA , not China… )

[optio]

Very strange. I check the result of your script and mine and they have identical results in my opinion. Could you show what exactly is missing in my version? What symbols?

Could be a problem at line that matches expression

.*\$matchPosition[[:space:]]][[:space:]]-[[:space:]]1[[:space:]]\)[[:space:]]][[:space:]]\)$

which extracts one character less at the end…

[DyadyaGenya]

If I do such a check, then these gaps are also clearly visible.

foreach key,sms in=$smsArray do={
    :put "$key=$sms";
}

Even if I replace I will do this replacement:

        :set smsArray ( $smsArray , [ :pick $modemOutput ( $matchPosition+1) ( [ :find $modemOutput $search2 $matchPosition ] - 1 ) ] );
 to
         :set smsArray ( $smsArray , [ :pick $modemOutput $matchPosition ( [ :find $modemOutput $search2 $matchPosition ] - 1 ) ] );

[DyadyaGenya]

Example:
using my code
07912180958739F1040B917120069876F000009140503223218A21D4F29C0E6A97E7F3F0B90CA2BF41412A68F86EB7C36E32885A9ED3CB72
usin your (unaltered or clean, no difference) code
07912180958739F1040B917120069876F000009140503223218A21D4F29C0E6A97E7F3F0B90CA2BF41412A68F86EB7C36E32885A9ED3CB7
do not notice the mising 2 at the end?

You are really right. This is my carelessness. I corrected the starting point in one place, because I began to use “\r\n” instead of “\n” and forgot in another. This is how it works right

:global printValue;
:local smsStatus [:toarray "REC UNREAD,REC READ,STO UNSENT,STO SENT,ALL"]
:local modemOutput ( [ /interface lte at-chat lte1 wait=yes input="AT+CMGL=4" as-value ]->"output" );
:local search1 "+CMGL:";
:local search2 "\r\n";
:local matchPosition -1;
:local matchCount 0;
:local arrayStat [:toarray ""]

:local continue true;
while ( $continue ) do={
    :set matchPosition [ :find $modemOutput $search1 $matchPosition ];
    :if ( [ :typeof $matchPosition ] != "nil" ) do={


        :local tmpStat [:toarray [:pick $modemOutput ($matchPosition+7) [ :find $modemOutput $search2 $matchPosition ] ] ]
        :local subarray [:toarray ""]
            :if ( [:len $tmpStat]=3 ) do={
                :set ($subarray->"index" ) [:tonum ($tmpStat->0)]
                :set ($subarray->"status") ($smsStatus->[:tonum ($tmpStat->1)])
                :set ($subarray->"length") [:tonum ($tmpStat->2)]
                :set ($subarray->"alpha") ("")
            } else={
                :set ($subarray->"index" ) [:tonum ($tmpStat->0)]
                :set ($subarray->"status") ($smsStatus->[:tonum ($tmpStat->1)])
                :set ($subarray->"length") [:tonum ($tmpStat->3)]
                :set ($subarray->"alpha") [:tonum ($tmpStat->2)]
            }

        :set matchPosition [ :find $modemOutput $search2 $matchPosition ];
        :set ($subarray->"sms" )  [ :pick $modemOutput ( $matchPosition+2) ( [ :find $modemOutput $search2 $matchPosition ]  ) ]
        :set ($arrayStat->$matchCount) $subarray
        :set matchCount ( $matchCount + 1 );
    } else={ :set continue false; };
};

:put $arrayStat

Unfortunately, I have not yet had time to see what my conceptual errors are, which you noticed in the redone version. I hope to be able to take a look later tonight.

Could be a problem at line that matches expression

.*\$matchPosition[[:space:]]][[:space:]]-[[:space:]]1[[:space:]]\)[[:space:]]][[:space:]]\)$

which extracts one character less at the end…

I think I went down the same path that you suggested looking for my mistake.

[DyadyaGenya]

:set ($subarray->“index” ) [:tonum ($tmpStat->0)]
:set ($subarray->“status”) ($smsStatus->[:tonum ($tmpStat->1)])
:if ([:len $tmpStat] = 3) do={
:set ($subarray->“alpha” ) “”
:set ($subarray->“length”) [:tonum ($tmpStat->2)]
} else={
:set ($subarray->“alpha” ) [:tonum ($tmpStat->2)]
:set ($subarray->“length”) [:tonum ($tmpStat->3)]
}

Managed to watch. Here you are absolutely right. So the code will be more correct and shorter. I should have guessed it myself, even though I did a draft.

while ([:typeof [:find $modemOutput $search1 $matchPosition]] != “nil”) do={
:set matchPosition [:find $modemOutput $search1 $matchPosition]

I read somewhere that the “find” command is the most expensive and energy intensive. In my version, it is used only once.

[rextended]

I read somewhere that the “find” command is the most expensive and energy intensive. In my version, it is used only once.

We are talking about at most 25 SMS, not a log of 1000 items or a firewall with 10,000 items…
But is easy solveable…

Fixed the cleaned version with same fix you do at your code
http://forum.mikrotik.com/t/using-two-arrays-process-the-text-and-create-a-third-array/166923/41

[rextended]

I’ve updated my script by changing the variable names to be more consistent,
I’ve reformatted the code for clarity,
optimized the internal operations to execute as few instructions as possible
and make everything easier to understand how it works.

{
:local CMGLstatus [:toarray “REC UNREAD,REC READ,STO UNSENT,STO SENT,ALL”]
:local CMGLoutput ([/interface lte at-chat ([find]->0) input=“AT+CMGF=0;+CMGL=$[:find $CMGLstatus “ALL” -1]” wait=yes as-value ]->“output”)

Future check: If CMGLoutput value is “OK” only, there is no results.

:set CMGLoutput "$[:pick $CMGLoutput 7 ([:len $CMGLoutput] - 2)]+CMGL: "
:local SMSarray [:toarray “”]
:local separator "\r\n+CMGL: "
:local lenCMGLsep [:len $separator]
:local findStart (0 - $lenCMGLsep) ; :local tempPos 0 ; :local smsIndex 0 ; :local alphaIndex 0 ; :local rawIndex 0

:local findPosSeparator [:find $CMGLoutput $separator $findStart]
:while ([:typeof $findPosSeparator] = “num”) do={
:set tempPos $findPosSeparator
:local tempArray [:toarray [:pick $CMGLoutput ($findStart + $lenCMGLsep) $tempPos]]
# Future check: if on SIM phonebook the name containing comma, and the Modem reply with the name on SIM phonebook, prevent this problem
:if ([:len $tempArray] = 4) do={:set alphaIndex 2 ; :set rawIndex 3} else={:set alphaIndex 4 ; :set rawIndex 2 ; :set ($tempArray->4) “”}
:local subArray [:toarray “”]
:set ($subArray->“index” ) [:tonum ($tempArray->0)]
:set ($subArray->“status” ) ($CMGLstatus->[:tonum ($tempArray->1)])
:set ($subArray->“alpha” ) “"$($tempArray->$alphaIndex)"”
:set ($subArray->“length” ) [:tonum [:pick ($tempArray->$rawIndex) 0 [:find ($tempArray->$rawIndex) “\r\n” -1]]]
:set ($subArray->“PDU” ) [:pick ($tempArray->$rawIndex) ([:find ($tempArray->$rawIndex) “\r\n” -1] + 2) [:len ($tempArray->$rawIndex)]]
:set ($SMSarray->$smsIndex) $subArray
:set smsIndex ($smsIndex + 1)
:set findStart $tempPos
:set findPosSeparator [:find $CMGLoutput $separator $findStart]
}

:foreach item in=$SMSarray do={
:put “SMS index=$($item->“index”), status=$($item->“status”), phonebook name=$($item->“alpha”), pdu length=$($item->“length”)”
:put “*** START PDU "
:put ($item->“PDU”)
:put " END PDU ***\r\n”
}

}

[DyadyaGenya]

You need to decode PDU line from sms key in $arrayStat values map from offsets described at post #23 or see here

Do you have more examples explaining how to decipher some of the meanings? I can’t figure out how to deal with the sender number and SMS type (SMS-DELIVER,SMS-STATUS REPORT,SMS-SUBMIT REPORT,RESERVED)
Or, for example, what is the type of SMS if it is encrypted as 44 or 60 or 40?

[DyadyaGenya]

I’ve updated my script by changing the variable names to be more consistent,
I’ve reformatted the code for clarity,
optimized the internal operations to execute as few instructions as possible
and make everything easier to understand how it works.

Your code is really good, I didn’t argue with that. But my task was first of all to learn different ways of working with arrays using the example of working with SMS. And I always keep your option in front of my eyes. If I understand correctly, then with the correction of my mistake about cutting off the last SMS character, there are no special comments on my version?
If so, then it remains for me to deal with some of the nuances of recoding. I already know how to make an additional local function that would turn an encrypted phone into a normal one. Still need to figure out how to work with the letter number. And some other things.

[DyadyaGenya]

As already written on another post, never use ({}) the result is unexpected.
The only way to create empty array is [:toarray “”] and without ( ).

By the way, can you tell me the link where it is written about this?

[Amm0]

As already written on another post, never use ({}) the result is unexpected.
The only way to create empty array is [:toarray “”] and without ( ).

Just try it. It’s a syntax error in recent V7 at least. But {} is [:nothing] (or nil), not an array.

By the way, can you tell me the link where it is written about this?

https://wiki.mikrotik.com/wiki/Manual:Scripting_Tips_and_Tricks#How_to_define_empty_array

[rextended]

Do you have more examples explaining how to decipher some of the meanings? I can’t figure out how to deal with the sender number and SMS type (SMS-DELIVER,SMS-STATUS REPORT,SMS-SUBMIT REPORT,RESERVED)
Or, for example, what is the type of SMS if it is encrypted as 44 or 60 or 40?

I know only what is already wrote on Problem 6) here. Google is your friend…
http://forum.mikrotik.com/t/using-two-arrays-process-the-text-and-create-a-third-array/166923/1

(44? 60? 40? in those case is better if the number is > of 9 specify if are decimal numbers or hexadecimal, etc. but I do not know what correspondence is…)

[DyadyaGenya]

TP-UDL: 21 (message lenght on septept, because is on GSM-7 format. Are octects if the mesage is on UCS-2)
TP-UD: D4F29C0E6A97E7F3F0B90CA2BF41412A68F86EB7C36E32885A9ED3CB72
(message encoded on octect, but internally is on septept for GSM-7, is on octect if is ASCII-8 or UCS-2)

Decode the GSM-7 or eventually the UCS-2 part is easy with my already existent functions.
Function to convert Hex GSM7 to CP1252 string (for decode the PDU message part on GSM7)
✂ Rextended Fragments of Snippets
Function to convert UCS-2 to UTF-8 (for decode the PDU message part on UCS-2)
Convert any text to UNICODE - #18 by rextended

It’s not entirely clear how I’m supposed to figure out which encoding to choose for my language. Here, it seems that the message is written in English, but it can only be translated by your UCS2toUTF8 function into some other numbers and percent signs, and the second function says that there is an error. Although the text in the SMS is written in English.
Here is the SMS text: “C8329BFD065DDF72363904”
Before that, it says that some kind of 7-bit encoding is used, but I don’t understand what that means.
SMS from your example also does not display in the terminal in normal English or something else. Only numbers and percent signs

[rextended]

Before it can be used on my program for encode/decode GSM-7 the data must be converted from 7 bit PDU to 8 bit hex…

RouterOS can display only 7 bit ascii standard characters, not GSM-7 UCS-2 or UTF-8
The %encode is for send the message using fetch to twitter, whatsapp, etc.

[optio]

RouterOS can display only 7 bit ascii standard characters, not GSM-7 UCS-2 or UTF-8

Even from utf8 terminal over ssh? I’ve seen characters correctly displayed from scripts which are not in 7 bit ascii when connecting over ssh, still cli input is ascii always…

[rextended]

http://forum.mikrotik.com/t/rextended-fragments-of-snippets/151033/1

C8329BFD065DDF72363904 => 48656C6C6F20576F726C6421 => Hello World!

:put [$pdutogsm7 (“\C8\34\88\FE\06\05\D9\EC\50\28\04”)]
:put [$HexGSM7toCP1252 (“48656C6C6F20576F726C6421”)]

[rextended]

RouterOS can display only 7 bit ascii standard characters, not GSM-7 UCS-2 or UTF-8

Even from utf8 terminal over ssh? I’ve seen characters correctly displayed from scripts which are not in 7 bit ascii when connecting over ssh, still cli input is ascii always…

I’m not talking about third-party programs.
It’s that program that shows them to you, not RouterOS…