Send SMS messages via Huawei LTE modem API (tested with E3372)

diamuxin · January 24, 2023, 6:51pm

Any progress Nikolay?

BR.

Sertik · February 2, 2023, 6:34am

Well, I have already received my Huawei E3372h-320 modem that incorporates an API that allows to manage some functions of the modem.

So does this script only work with this version of the modem ?

diamuxin · February 2, 2023, 6:57am

I have tested it on two models:

E8372h-320 → Does not work.
E3372h-320 → Yes it works (only to send SMS).

I have tested (read back) to receive SMS and process the content, without success.

I am waiting to see if the author of the script gives us an idea on how to receive SMS using the Huawei API.

Sertik · February 2, 2023, 9:10am

Yes, I would also wait for the author to tell me exactly which versions of the firmware and WEB interface his script works with? And where to get the API documentation for these modems …

pkt · February 2, 2023, 10:57pm

These scripts are contributed unselfishly by its creators. It’s unfair to expect or demand them any kind of guarantee about its applicability. For this kind of info you should contact the modem manufacturer or vendor.

I think that there are two kind of HiLink devices:

single-user devices: with only one usb connection like E3372h
multi-user devices: like full-fledged routers with ethernet and/or WiFi, and also MiFi devices with usb connection and WiFi hotspot, like E8372h

Single-user devices (at least older ones) are only connected to other one device via a physical link, so access is somewhat controlled and use a simple way of authentication consisting of retrieving a session id and token thru a http request.

Multi-user devices are exposed to arbitrary users and therefore, API access is authenticated with user and password. The problem is that user-password authentication returns access tokens in the headers of the http response and /tool/fetch has no way to get them, so it’s no possible to access these devices from RoS scripts.

The best documentation links for HiLink API that I know are:

They are not 100% accurate, but are a good source of information.

Here is a discussion of the several authentication methods:

https://github.com/arska/e3372/issues/1

pkt · February 2, 2023, 11:23pm

The recvSMS works well, but retrieves a bunch of sms messages at a time. The getBetween parser is very limited and can only process the first appearance ot each tag.

I’ve a more advanced parser that walks incrementally across the xml:

# TOKEN PARSER LIBRARY
# v1.0.0 pkt

# :put [($tokenParser->"getTag") source=$xml tag="SessionInfo"]

# ($tokenParser->"getBetween")
#  get delimited value
#    source - source string
#    fromTok - (optional) text AFTER this token (or from source beginning) will be returned
#    toTok - (optional) text BEFORE this token (or until source finish) will be returned
#    startPos - (optional) start position (default 0 = beginning)
#  returns an array with fields data and pos
# 
# ($tokenParser->"getTag")
#  get value for XML tag
#    source - xml string
#    tag - tag which value is to be returned
#    startPos - (optional, text index) start position (if not specified, it will search from the beginning of the string)
#  returns the tag content
# 
# ($tokenParser->"getTagDetailed")
#  get value and end position for XML tag
#    source - xml string
#    tag - tag which value is to be returned
#    startPos - (optional, text index) start position (if not specified, it will search from the beginning of the string)
#  returns an array with fields "data" (tag content) and "pos" (where tag ends)
# 
# ($tokenParser->"getTagList")
#  get a list with the content of each appearance of tag
#    source - xml string
#    tag - tag which value is to be returned
#  returns an array with tag contents
# 
# ($tokenParser->"forEachTag")
#  incremental parser that calls the callback with the content of each appearance of tag
#    source - xml string
#    tag - tag which value is to be returned
#    callback - callback (with param content) to be called for each appearance of tag
#    callbackArgs - callback will be called passing this value in param args


:global tokenParser ({})

:set ($tokenParser->"getBetween") do={ # get delimited value
  # source - source string
  # fromTok - (optional) text AFTER this token (or from source beginning) will be returned
  # toTok - (optional) text BEFORE this token (or until source finish) will be returned
  # startPos - (optional) start position (default 0 = beginning)
  
  # returns an array with fields data and pos
  # if fromTok and/or toTok are specified and neither of them appear in source, empty string "" will be returned as data

  # based on function getBetween by CuriousKiwi, modified by pkt

  :local posStart
  if ([:len $startPos] = 0) do={
    :set posStart -1
  } else={
    :set posStart ($startPos-1)
  }

  :local found true
  :local data 

  :local resultStart
  :if ([:len $fromTok] > 0) do={
    :set resultStart [:find $source $fromTok $posStart]
    :if ([:len $resultStart] = 0) do={ # start token not found
      :set found false
      :set data ""
    }
    :set resultStart ($resultStart + [:len $fromTok])
  } else={
    :set resultStart 0
  }

  :local resultEnd
  :if (found = true && [:len $toTok] > 0) do={
    :set resultEnd [:find $source $toTok ($resultStart-1)]
    :if ([:len $resultEnd] = 0) do={ # end token not found
      :set found false
      :set data ""
    }
  } else={
    :set resultEnd [:len $source]
  }

  :if ($found = true) do={ :set data [:pick $source $resultStart $resultEnd] }

  :return { data=$data; pos=$resultEnd }
}

:set ($tokenParser->"getTag") do={ # get value for XML tag
  # source - xml string
  # tag - tag which value is to be returned
  # startPos - (optional, text index) start position (if not specified, it will search from the beginning of the string)

  # returns the tag content

  :global tokenParser
  :return ([($tokenParser->"getBetween") source=$source fromTok=("<$tag>") toTok=("</$tag>") startPos=$startPos]->"data")
}

:set ($tokenParser->"getTagDetailed") do={ # get value and end position for XML tag
  # source - xml string
  # tag - tag which value is to be returned
  # startPos - (optional, text index) start position (if not specified, it will search from the beginning of the string)

  # returns an array with fields "data" (tag content) and "pos" (where tag ends)

  :global tokenParser
  :return [($tokenParser->"getBetween") source=$source fromTok=("<$tag>") toTok=("</$tag>") startPos=$startPos]
}

:set ($tokenParser->"getTagList") do={ # get a list with the content of each appearance of tag
  # source - xml string
  # tag - tag which value is to be returned

  # returns an array with tag contents

  :global tokenParser

  :local result ({})
  :local doneTags false
  :local startPos 0

  :do {
    :local tagContent [($tokenParser->"getTagDetailed") source=$source tag=$tag startPos=$startPos]

    :local content ($tagContent->"data")
    :if ($content != "") do={
      :set ($result->[:len $result]) $content

      # advance start pos to search for next tag
      :set startPos ($tagContent->"pos")
    } else={
      :set doneTags true
    }
  } while=($doneTags = false)

  :return $result
}

:set ($tokenParser->"forEachTag") do={ # incremental parser that calls the callback with the content of each appearance of tag
  # source - xml string
  # tag - tag which value is to be returned
  # callback - callback (with param content) to be called for each appearance of tag
  # callbackArgs - callback will be called passing this value in param args

  :global tokenParser

  :local doneTags false
  :local startPos 0

  :do {
    :local tagContent [($tokenParser->"getTagDetailed") source=$source tag=$tag startPos=$startPos]

    :local content ($tagContent->"data")
    :if ($content != "") do={
      [$callback tagContent=$content args=$callbackArgs]

      # advance start pos to search for next tag
      :set startPos ($tagContent->"pos")
    } else={
      :set doneTags true
    }
  } while=($doneTags = false)
}

The receive script will be something as this:

:global recvSMS do={
  :local lteIP "192.168.8.1"

  :global tokenParser

  # get SessionID and Token via LTE modem API
  :local urlSesTokInfo "http://$lteIP/api/webserver/SesTokInfo"
  :local api [/tool fetch $urlSesTokInfo output=user as-value]
  :local apiData  ($api->"data")

  # parse SessionID and Token from API session data 
  :local apiSessionID [($tokenParser->"getTag") source=$apiData tag="SesInfo"]
  :local apiToken [($tokenParser->"getTag") source=$apiData tag="TokInfo"]

  # header and data config
  :local apiHead "Content-Type:text/xml,Cookie: $apiSessionID,__RequestVerificationToken:$apiToken"
  :local recvData "<?xml version=\"1.0\" encoding=\"UTF-8\"?><request><PageIndex>1</PageIndex><ReadCount>20</ReadCount><BoxType>1</BoxType><SortType>0</SortType><Ascending>0</Ascending><UnreadPreferred>1</UnreadPreferred></request>"

  # recv SMS via LTE modem API with fetch
  :return [/tool fetch http-method=post http-header-field=$apiHead url="http://$lteIP/api/sms/sms-list" http-data=$recvData output=user as-value]
}

:local xmlSmsList ([$recvSMS]->"data")

:global tokenParser

:local smsList [($tokenParser->"getTagList") source=$xmlSmsList tag="Message"]

:foreach tagContent in=$smsList do={
  # tagContent - content of tag

  :local index ([($tokenParser->"getTag") source=$tagContent tag="Index"])
  :local phone ([($tokenParser->"getTag") source=$tagContent tag="Phone"])
  :local content ([($tokenParser->"getTag") source=$tagContent tag="Content"])
  :local read ([($tokenParser->"getTag") source=$tagContent tag="Smstat"] = 1)

  :put "$index $read $phone $[pick $content 0 50]"
}

With this you can try to made some progress.

But you will encounter more problems, such as that some characters are escaped (< is < …), how to mark the sms as processed so next time it will be ignored, how to avoid skipping sms if they arrive when the inbox is being processed…

I am working (in my very limited spare time) in a more powerful library for HiLink access in RouterOS scripts that will try to solve these problems, call a function when new sms arrives, and even implements automatic execution of RoS scripts when receiving sms with :cmd, as Mikrotik’s own SMS support do.

I except to have a test version the next week or maybe the next one.

diamuxin · February 3, 2023, 8:08am

Fantastic @pkt!

Thank you very, very much for helping us with the Huawei API.
We’ll keep an eye out for your review.

Best regards.

diamuxin · February 3, 2023, 10:35am

Hi @pkt

You have made a very important breakthrough in research, thank you for your work.

Your script works very fine.

Unread messages:

[admin@MikroTik] > sys scr run test-sms
40002 false +34XXXXXXXXX Received message test 2
40001 false +34XXXXXXXXX Received message test 1

Already read messages: (Changes the state of the variable “$read” to true)

[admin@MikroTik] > sys scr run test-sms
40002 true +34XXXXXXXXX Received message test 2
40001 true +34XXXXXXXXX Received message test 1

It would be great, that once we get the messages, we can use the content of the $phone and $[pick $content 0 50] variables independent of the $read condition and be able to forward it for example by email. After sending it, apply a :relay 5s and delete the inbox.

With a scheduler task, I would check the inbox for X minutes.

At the time I was trying to delete messages and I created this script, but does not delete them.

# Original code by @pkt

:global delSMS do={
  :local lteIP "192.168.8.1"

  :global tokenParser

  # get SessionID and Token via LTE modem API
  :local urlSesTokInfo "http://$lteIP/api/webserver/SesTokInfo"
  :local api [/tool fetch $urlSesTokInfo output=user as-value]
  :local apiData  ($api->"data")

  # parse SessionID and Token from API session data 
  :local apiSessionID [($tokenParser->"getTag") source=$apiData tag="SesInfo"]
  :local apiToken [($tokenParser->"getTag") source=$apiData tag="TokInfo"]

  # header and data config
  :local apiHead "Content-Type:text/xml,Cookie: $apiSessionID,__RequestVerificationToken:$apiToken"
  :local delData "<?xml version:\"1.0\" encoding=\"UTF-8\"?><request><Index>10</Index></request>"

  # delete SMS via LTE modem API with fetch
  :return [/tool fetch http-method=post http-header-field=$apiHead url="http://$lteIP/api/sms/delete-sms" http-data=$delData output=user as-value]
}

BR.

pkt · February 4, 2023, 9:29am

Afaik, you can’t delete the entire inbox with one command. But can do it one message at a time using its index.

In the delete request, instead of 10 use $index.

And then (not tested):

...

:local index ([($tokenParser->"getTag") source=$tagContent tag="Index"])

[:delSMS index=$index]

Note that you can’t know if email is sent successfully, so, if there is an error, you have a risk of deleting a sms not really sent and you will lost it.

One approach could be to include the sms index in the email subject so you can easily note in received email the hole in numbering if one is missing (assuming hilink increments index with each sms} if e.g. you receive 40001, 40002 and then 40004, then 40003 is missing and you can check mikrotik log for errors, and also read the sms in modem’s web interface.

So you should not delete it immediately, but after a few days. For this it is needed to parse the date time of the sms so it can be deleted when is older than a given elapsed time. I have some functions for this, but are not finished yet.

For ignoring messages already sent by email but yet not deleted and avoid sending it in each scheduler talk execution, the only way I can imagine is sending by email only not read messages, and marking it as read after email is sent. No delay is needed.

rextended · February 4, 2023, 12:17pm

The SMSs can be stored locally:

Example:

{
:local SMSfolder "SMS" ; # must already exist, do not count "flash", is added automatically if is needed
:local SMSfileend "backup_sms.txt" ; # only .txt file, the date and the time is added automatically at the start

:local arrMonths {an="01";eb="02";ar="03";pr="04";ay="05";un="06";ul="07";ug="08";ep="09";ct="10";ov="11";ec="12"}
/system clock ; # instead of use local date and time, if can be obtained, the SMS date and time can be used.
:local date [get date]
:local time [get time]
:local SMSfntime  "$[:pick $date 7 11]$($arrMonths->[:pick $date 1 3])$[:pick $date 4 6]$[:pick $time 0 2]$[:pick $time 3 5]"
:local SMSfpath   "$SMSfolder/$SMSfntime_$SMSfileend"
:local SMSnumber  "+18005551234567890"
:local SMSmessage "123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
/file
:put $SMSfpath
:if ([:len [find where name="flash" and type="disk"]] = 1) do={:set SMSfpath "flash/$SMSfpath"}
:put $SMSfpath
:if ([:len [find where name=$SMSfpath]] = 0) do={print file=$SMSfpath; :delay 5s; set $SMSfpath contents=""}
:put $SMSfpath
:delay 2s
set $SMSfpath contents="$[get $SMSfpath contents]$SMSnumber $SMSmessage\r\n"
}

pkt · February 5, 2023, 8:55am

Using local storage is great idea. This way there are no risk of exceeding the (not documented) storage capacity of the modem.

Thanks for the code, I will make good use of it. I have already reused successfully some of your snippets in my hilink routines..

diamuxin · February 5, 2023, 9:35pm

Afaik, you can’t delete the entire inbox with one command. But can do it one message at a time using its index.

In the delete request, instead of 10 use $index.

And then (not tested):
...
:local index ([($tokenParser->"getTag") source=$tagContent tag="Index"])
[:delSMS index=$index]

Hello, thank you for replying @pkt.

But, I don’t quite understand your approach. In the script I use called test-sms has the following content:

:global recvSMS do={
  :local lteIP "192.168.8.1"

  :global tokenParser

  # get SessionID and Token via LTE modem API
  :local urlSesTokInfo "http://$lteIP/api/webserver/SesTokInfo"
  :local api [/tool fetch $urlSesTokInfo output=user as-value]
  :local apiData  ($api->"data")

  # parse SessionID and Token from API session data 
  :local apiSessionID [($tokenParser->"getTag") source=$apiData tag="SesInfo"]
  :local apiToken [($tokenParser->"getTag") source=$apiData tag="TokInfo"]

  # header and data config
  :local apiHead "Content-Type:text/xml,Cookie: $apiSessionID,__RequestVerificationToken:$apiToken"
  :local recvData "<?xml version=\"1.0\" encoding=\"UTF-8\"?><request><PageIndex>1</PageIndex><ReadCount>20</ReadCount><BoxType>1</BoxType><SortType>0</SortType><Ascending>0</Ascending><UnreadPreferred>1</UnreadPreferred></request>"

  # recv SMS via LTE modem API with fetch
  :return [/tool fetch http-method=post http-header-field=$apiHead url="http://$lteIP/api/sms/sms-list" http-data=$recvData output=user as-value]
}

:global tokenParser
:local xmlSmsList ([$recvSMS]->"data")
:local smsList [($tokenParser->"getTagList") source=$xmlSmsList tag="Message"]
:local sendthisuniquelist ""

:foreach tagContent in=$smsList do={
  # tagContent - content of tag
  
  :local index [($tokenParser->"getTag") source=$tagContent tag="Index"]
  :local phone [($tokenParser->"getTag") source=$tagContent tag="Phone"]
  :local content [($tokenParser->"getTag") source=$tagContent tag="Content"]
  :local read ([($tokenParser->"getTag") source=$tagContent tag="Smstat"] = 1)

  :if ($content != "") do={
  :put "$index $read $phone $content"
  :set sendthisuniquelist ($sendthisuniquelist."$index $read $phone $content\r\n")
  }
}
/tool e-mail send to=to@gmail.com subject="SMS List" body=$sendthisuniquelist

And the script with the function “delSMS” has the following content:

:global delSMS do={
  :local lteIP "192.168.8.1"

  :global tokenParser

  # get SessionID and Token via LTE modem API
  :local urlSesTokInfo "http://$lteIP/api/webserver/SesTokInfo"
  :local api [/tool fetch $urlSesTokInfo output=user as-value]
  :local apiData  ($api->"data")

  # parse SessionID and Token from API session data 
  :local apiSessionID [($tokenParser->"getTag") source=$apiData tag="SesInfo"]
  :local apiToken [($tokenParser->"getTag") source=$apiData tag="TokInfo"]

  # header and data config
  :local apiHead "Content-Type:text/xml,Cookie: $apiSessionID,__RequestVerificationToken:$apiToken"
  :local index [($tokenParser->"getTag") source=$tagContent tag="Index"]
  :local delData "<?xml version:\"1.0\" encoding=\"UTF-8\"?><request><Index>$index</Index></request>"

  # delete SMS via LTE modem API with fetch
  :return [/tool fetch http-method=post http-header-field=$apiHead url="http://$lteIP/api/sms/delete-sms" http-data=$delData output=user as-value]
}

If I run the first script “test-sms” I get the two test messages on screen and by email:

40001 false +34XXXXXXXXX Lorem ipsum 2
40000 false +34XXXXXXXXX Lorem ipsum 1

and then, if I run the “delSMS” script it does not delete any message.

This line does not work:

[:delSMS index=$index]

Can you help me?

I would like that once the messages are read, I can delete them.

BR.

pkt · February 5, 2023, 10:48pm

I thought that you wanted to send one email for each sms. In that case the script would be something like this:

...

:foreach tagContent in=$smsList do={
  # tagContent - content of tag
  
  :local index [($tokenParser->"getTag") source=$tagContent tag="Index"]
  :local phone [($tokenParser->"getTag") source=$tagContent tag="Phone"]
  :local content [($tokenParser->"getTag") source=$tagContent tag="Content"]
  :local read ([($tokenParser->"getTag") source=$tagContent tag="Smstat"] = 1)

  :if ($content != "") do={
    :put "$index $read $phone $content"
    /tool e-mail send to=to@gmail.com subject="SMS List" body="$index $read $phone $content"
    [$delSMS index=$index]
  }
}

But if you want to send all sms grouped in only one email, then it is better to first concatenate all sms and send the email (you are already doing this), and then do a second foreach to delete all sms:

...

:local sendthisuniquelist ""

:foreach tagContent in=$smsList do={
  # tagContent - content of tag
  
  :local index [($tokenParser->"getTag") source=$tagContent tag="Index"]
  :local phone [($tokenParser->"getTag") source=$tagContent tag="Phone"]
  :local content [($tokenParser->"getTag") source=$tagContent tag="Content"]
  :local read ([($tokenParser->"getTag") source=$tagContent tag="Smstat"] = 1)

  :if ($content != "") do={
    :put "$index $read $phone $content"
    :set sendthisuniquelist ($sendthisuniquelist."$index $read $phone $content\r\n")
  }
}

/tool e-mail send to=to@gmail.com subject="SMS List" body=$sendthisuniquelist

:foreach tagContent in=$smsList do={
  :local index [($tokenParser->"getTag") source=$tagContent tag="Index"]
  [$delSMS index=$index]
}

Correct syntax is [$delSMS index=$index] no [:delSMS index=$index] as I posted before. My fault.

diamuxin · February 6, 2023, 9:37am

The second foreach does not work, tested as follows:

{
:global tokenParser
:local xmlSmsList ([$recvSMS]->"data")
:local smsList [($tokenParser->"getTagList") source=$xmlSmsList tag="Message"]

:foreach tagContent in=$smsList do={
  :local index [($tokenParser->"getTag") source=$tagContent tag="Index"]
  [$delSMS index=$index]
}
}

SMS messages are not deleted, verified:

[admin@MikroTik] > :put ([$recvSMS]->"data")

<?xml version="1.0" encoding="UTF-8"?>
<response>
  <Count>2</Count>
  <Messages>
    <Message>
      <Smstat>1</Smstat>
      <Index>40002</Index>
      <Phone>+34XXXXXXXXX</Phone>
      <Content>Received message test 2</Content>
      <Date>2023-02-03 16:33:01</Date>
      <Sca></Sca>
      <SaveType>0</SaveType>
      <Priority>0</Priority>
      <SmsType>1</SmsType>
    </Message>
    <Message>
      <Smstat>1</Smstat>
      <Index>40001</Index>
      <Phone>+34XXXXXXXXX</Phone>
      <Content>Received message test 1</Content>
      <Date>2023-02-03 14:02:27</Date>
      <Sca></Sca>
      <SaveType>0</SaveType>
      <Priority>0</Priority>
      <SmsType>1</SmsType>
    </Message>
  </Messages>
</response>

Any ideas please?

BR.

Sertik · February 6, 2023, 12:05pm

Well , then they started tweeting without me … In order not to disassemble all the posts, can someone clearly write how to receive SMS and how to send it and on which firmware it will work?

diamuxin · February 6, 2023, 12:09pm

http://forum.mikrotik.com/t/send-sms-messages-via-huawei-lte-modem-api-tested-with-e3372/143316/1

https://forum.mikrotik.com/viewtopic.php?p=982481#p981623

Device name: E3372h-320
Hardware version: CL4E3372HM
Software version: 11.0.1.1(H697SP4C983)
Web UI version: WEBUI 11.0.1.1(W13SP4C7110)

Some info can be found here http://forum.jdtech.pl/Watek-hilink-api-dla-urzadzen-huawei (in polish)

Sertik · February 6, 2023, 12:19pm

Well , then they started tweeting without me … In order not to disassemble all the posts, can someone clearly write how to receive SMS and how to send it and on which firmware it will work?
None of the above scripts for sending/receiving SMS are working for me …

diamuxin · February 6, 2023, 12:24pm

You will need to look for a Huawei Modem model that supports the API. Mine if you read backwards is the E3372h-320 model that I can send and receive messages via MIkrotik fetch (I just need to know how to delete them once read).

rextended · February 6, 2023, 12:24pm

At least you have Send SMS messages via Huawei LTE modem API (tested with E3372 ) ???

Sertik · February 6, 2023, 12:26pm

I have two E3372 modems here are their data:

Device name: E3372
Hardware version: CL2E3372HM
Software version: 22.315.01.00.00
Web UI version: WEBUI 17.100.13.01.03-Mod1.10

Device name: E3372
Hardware version: CL2E3372HM
Software version: 22.333.01.00.00
Web UI version: WEBUI 17.100.13.112.03 (17.100.13.01.03_Mod1.12)