Community discussions

MikroTik App
 
arfoll
newbie
Topic Author
Posts: 28
Joined: Mon Sep 24, 2012 8:24 pm

Knot bluetooth PDU format

Sun Apr 17, 2022 3:06 pm

I'm using a Knot to capture bluetooth payloads from a xiaomi temp sensor, all the filtering is working well and I get adverisements as I'd expect. I can forward those via MQTT, but they are not in the format I would expect. Note I'm really not a bluetooth expert so I'm probably missing something super obvious.

[xxx@xxx] /iot/bluetooth> scanners/advertisements/print follow
0 bt1 adv-ind apr/17/2022 13:28:44 public A4:C1:38:2F:29:19 -37dBm 22 151695fe50585b059719292f38c1a40d1004e4005c01

Looks ok, but the packets I would expect would be way longer, like this one (identical packet captured via a python script on a laptop next to the knot:
043e220201000019292f38c1a416151695fe50585b059919292f38c1a40d1004e4005d01ce

I know this is a good packet because I can decode it:
>>> sensor_msg, tracker_msg = ble_parser.parse_data(bytes(bytearray.fromhex("043e220201000019292f38c1a416151695fe50585b059919292f38c1a40d1004e4005d01ce")));print(sensor_msg)
{'rssi': -50, 'mac': 'A4C1382F2919', 'type': 'LYWSD03MMC', 'packet': 153, 'firmware': 'Xiaomi (MiBeacon V5)', 'data': True, 'temperature': 22.8, 'humidity': 34.9}

Clearly the two payloads look really similar:
043e220201000019292f38c1a416151695fe50585b059919292f38c1a40d1004e4005d01ce
                            151695fe50585b059719292f38c1a40d1004e4005c01
I had a good read of https://help.mikrotik.com/docs/display/ROS/Bluetooth and I figure maybe i'm only getting that payload data and not the raw PDU. It seems to me for whatever reason mikrotik strip the payload and leave me to reconstruct it? Seems like a waste of time? How can I get back the full payload?
 
denissMT
MikroTik Support
MikroTik Support
Posts: 49
Joined: Wed May 26, 2021 12:00 pm

Re: Knot bluetooth PDU format

Mon May 02, 2022 5:11 pm

In the current Bluetooth implementation, the "data" parameter shows the "AdvData" (0...31 octets) part of the whole "PDU payload".
 
lvlaster
just joined
Posts: 7
Joined: Mon Apr 20, 2020 10:40 am

Re: Knot bluetooth PDU format

Wed Aug 10, 2022 10:03 pm

Is there any roadmap to extend PDU support, also very interested to connect Xiaomi BLE sensors to the KNOT.
As Mikrotik TAG only have temp sensor, and not humidity - there could be strong interest to integrate other vendors.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Knot bluetooth PDU format

Wed Aug 10, 2022 11:13 pm

PDU code

043e220201000019292f38c1a416151695fe50585b059919292f38c1a40d1004e4005d01ce
VRllkkjjiihhggffeeddccbbaaPLdl??????ctflTYPEpk????????????objxOLtempHUMXrs
                            151695fe50585b059719292f38c1a40d1004e4005c01

VR = 0x04 version (1 Byte) = 4
gg:hh:ii:jj:kk:ll = {!!! supposed !!!} 00:00:01:02:22:3E dst MAC (6 Bytes) 00:00:01 = Broadcast
aa:bb:cc:dd:ee:ff = A4:C1:38:2F:29:19 src MAC (6 Bytes) A4:C1:38 = Telink Semiconductor
PL = 0x16 payload length (1 Byte) = 22 Byte
dl = 0x15 data lengt (1 Byte) = 21 Byte (excluding this starting data byte)
ctfl = 0x5850 = 0101100001010000 = control flags (2 Bytes)
                ^^^^             0101 version          = "MiBeacon V" + 5
                    ^            1 ?                   = yes ?
                     ^^          0 aut mode            = none
                       ^         0 solicited           = none
                        ^        0 registered          = no
                         ^       0 mesh device         = no
                          ^      1 object included     = yes
                           ^     0 capability included = no
                            ^    1 MAC included        = yes
                             ^   0 encrypted           = no
                              ^  0 ?                   = no ?
                               ^ 0 timing request      = no

TYPE = 0x055B type (2 Byte) = 1371 LYWSD03MMC (Xiaomi)
pk = 0x99 packet (1 Byte)  = 153
objx = 0x100d object (2 Byte) = 0001 0000 0000 1101 = temperature & humidity
OL = 0x04 object length (1 Byte) = 4 Byte
temp = 0x00e4 [temperature × 10] (2 Byte float) = 22,8°C
HUMX = 0x015d [humidity × 10] (2 Byte float) = 34,9%
rs = 0xce rssi (1 Byte signed) = -50
I do not know what containing the undecoded part, but is like the removed parts do not containing so much...
 
lvlaster
just joined
Posts: 7
Joined: Mon Apr 20, 2020 10:40 am

Re: Knot bluetooth PDU format

Fri Aug 12, 2022 12:49 am

Thank you rextended for very useful information!
Actually I have modified (quick and dirty) standard Mikrotik mqtt script with addition of some parsing and your float2deg function.
Now reading temp from TG-BT5-OUT and temp+humidity from LYWSDCGQ.

The only issue - maybe you can comment on that - there two sort of ads from Xiaomi device - one is in same format as you stated :
020106151695fe5020aa01f54e4fdba8654c0d1004fb003702
second type is similar but shorter - 020106131695fe5020aa013c4e4fdba8654c041002fb00 - and slightly different ids inside.
I'm ignoring second type of packets for now...
VRllkkjjiihhggffeeddccbbaaPL????????ctflTYPEpk??????????????????tempHUMXrs
                      020106151695fe5020aa01f54e4fdba8654c0d1004fb003702
                      020106131695fe5020aa013c4e4fdba8654c041002fb00
Script with parsing:
# Required packages: iot

# rextended // modified
# fload Divide function float2deg ($1 - 0x value, $2 - decimal signs, $3 = offset 256 -> for Mikrotik or 10 -> for Xiaomi
#

:local float2deg do={
    :local float [:tonum $1]
    :local offset [:tonum $3]
    :if ($offset<=0) do {:set offset 256} 

    :local decimalsign "." ; # in Italy it is "," in other countries can be "."
    :local sign        ""  ; # if wanted can be "+"
    :if ($float > 32767) do={:set float (($float - 65536) * -1); :set sign "-"}
    :local forthousand ($float * 1000)
    :local forthousand ($forthousand / $offset) ; # any offset used
    :local ftstring "00$[:tostr $forthousand]"
    :local pickpos  ([:len $ftstring] - 3)
    :local decimals [:pick $ftstring $pickpos ($pickpos + [:tonum $2])]
    :local celsius  ($forthousand/1000)
    :if ([:tonum $2] > 0) do={:set celsius "$celsius$decimalsign$decimals" }
    :return "$sign$celsius"
}


################################ Configuration ################################
# Name of an existing MQTT broker that should be used for publishing
:local broker "mqtt1"

# MQTT topic where the message should be published
:local topic "v1/mikrotik"

# Interface whose MAC should be used as 'Locator ID'
:local locatorIface "ether1"

# POSIX regex for filtering advertisement Bluetooth addresses. E.g. "^BC:33:AC"
# would only include addresses which start with those 3 octets.
# To disable this filter, set it to ""
#
# now filtering done on KNOT side
:local addressRegex ""

# POSIX regex for filtering Bluetooth advertisements based on their data. Same
# usage as with 'addressRegex'.
:local advertisingDataRegex ""

# Signal strength filter. E.g. -40 would only include Bluetooth advertisements
# whose signal strength is stronger than -40dBm.
# To disable this filter, set it to ""
:local rssiThreshold ""

#################################### System ###################################
:put ("[*] Gathering system info...")
:local ifaceMac [/interface get [/interface find name=$locatorIface] mac-address]
:local cpuLoad [/system resource get cpu-load]
:local freeMemory [/system resource get free-memory]
:local usedMemory ([/system resource get total-memory] - $freeMemory)
:local rosVersion [/system package get value-name=version \
    [/system package find where name ~ "^routeros"]]
:local model [/system routerboard get value-name=model]
:local serialNumber [/system routerboard get value-name=serial-number]
# Health is a bit iffy since '/system health' does not have 'find' in ROS6
:local health [/system health print as-value]
:local supplyVoltage 0
:local boardTemp 0
:foreach entry in=$health do={
    :if ($entry->"name" = "voltage") do={:set $supplyVoltage ($entry->"value")}
    :if ($entry->"name" = "board-temperature1") do={:set $boardTemp ($entry->"value")}
}

################################## Bluetooth ##################################
:put ("[*] Gathering Bluetooth info...")
:global btOldestAdvertisementTimestamp
:if ([:typeof $btOldestAdvertisementTimestamp] = "nothing") do={
    # First time this script has been run since booting, need to initialize
    # persistent variables
    :set $btOldestAdvertisementTimestamp 0
}
:local btProcessingStart [/system clock get time]
:local advertisements [/iot bluetooth scanners advertisements print detail \
    as-value where \
        epoch > $btOldestAdvertisementTimestamp and \
        address ~ $addressRegex and \
        data ~ $advertisingDataRegex and \
        rssi > $rssiThreshold
]
:local advJson ""
:local advCount 0
:local advSeparator ""
:local lastAdvTimestamp 0
# Remove semicolons from MAC/Bluetooth addresses
:local minimizeMac do={
    :local minimized
    :local lastIdx ([:len $address] - 1)
    :for idx from=0 to=$lastIdx step=1 do={
        :local char [:pick $address $idx]
        :if ($char != ":") do={
            :set $minimized "$minimized$char"
        }
    }
    :return $minimized
}

:foreach adv in=$advertisements do={
    :local address ($adv->"address")
    :local ts ($adv->"epoch")
    :local rssi ($adv->"rssi")
    :local ad ($adv->"data")

#
# Temperature parsing for TAG-OUT and XIAOMI
#
    :local payload "\"ad\":\"$ad\"";
    :local temp "0"
    :local hum "0"

    # Mikrotik format
    :if ([:pick $ad 0 8] = "15ff4f09" ) do={ 
	:set temp [:pick $ad 30 32] 
	:set temp ("0x" . $temp . [:pick $ad 28 30])
	:set temp [$float2deg $temp 2] 
        :set payload "$payload,\ \"temp\":$temp\ ";
    } else={
        # Xiaomi format
        :if ([:pick $ad 0 14] = "020106151695fe" ) do={ 
    	   :set temp [:pick $ad 44 46] 
    	   :set temp ("0x" . $temp . [:pick $ad 42 44])
	   :set temp [$float2deg $temp 2 10]
    	   :set hum [:pick $ad 48 50] 
    	   :set hum ("0x" . $hum . [:pick $ad 46 48])
	   :set hum [$float2deg $hum 2 10]
           :set payload "$payload,\ \"temp\":$temp,\ \"humidity\":$hum\ ";
	}
    }

    :local obj "\
        {\
            \"id\":\"$[$minimizeMac address=$address]\",\
            \"ts\":$ts,\
            \"rssi\":$rssi,\
            \"ed\":{\
              $payload
            }\
        }"

    :set $advCount ($advCount + 1)
    :set $lastAdvTimestamp $ts
    # Ensure that the last object is not terminated by a comma
    :set $advJson "$advJson$advSeparator$obj"
    :if ($advSeparator = "") do={
        :set $advSeparator ","
    }
}

:if ($advCount > 0) do={
                                                      
    :set $btOldestAdvertisementTimestamp $lastAdvTimestamp

}

:put ("[*] Found $advCount new advertisements \
    (processing time: $[([/system clock get time] - $btProcessingStart)])")

#################################### MQTT #####################################
:local message \
    "{\
        \"clientId\":\"$[/iot mqtt brokers get value-name=client-id \
            [/iot mqtt brokers find name=$broker]]\",\
        \"t\":0,\
        \"v\":1,\
        \"OldestAdvertisementTimestamp\":$btOldestAdvertisementTimestamp,\
        \"locs\":[{\
            \"id\":\"$[$minimizeMac address=$ifaceMac]\",\
            \"tags\":[$advJson],\
            \"ed\":{\
                \"model\":\"$model\",\
                \"sn\":\"$serialNumber\",\
                \"ros\":\"$rosVersion\",\
                \"cpu\":$cpuLoad,\
                \"umem\":$usedMemory,\
                \"fmem\":$freeMemory,\
                \"psu\":$supplyVoltage,\
                \"temp\":$boardTemp\
            }\
        }]\
    }"
:log info "$message";
:put ("[*] Total message size: $[:len $message] bytes")
:put ("[*] Sending message to MQTT broker...")
/iot mqtt publish broker=$broker topic=$topic message=$message
#:put ("[*] /iot mqtt publish broker=$broker topic=$topic message=$message")
:put ("[*] Done")
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Knot bluetooth PDU format

Fri Aug 12, 2022 3:02 am

I'm not an expert but...

added data length and object type and length on previous post
dl = 0x15 data lengt (1 Byte) = 21 Byte (excluding this starting data byte)
objx = 0x100d object (2 Byte) = 0001 0000 0000 1101 = temperature & humidity
OL = 0x04 object length (1 Byte) = 4 Byte

PDU code

VRmeshdl??????ctflTYPEpk????????????objxOLtempHUMXrs
020106151695fe5020aa01f54e4fdba8654c0d1004fb003702

VR = 0x02 version (1 Byte) = 2 mesh???
capability = 0x01 0x06 ??? 2 item 1 Byte each, or 1 item 2 Byte (0x0601), or 2 Byte binary flags 011000000001 ???
dl = 0x15 data lengt (1 Byte) = 21 Byte (excluding this starting data byte)

ctfl = 0x5850 = 0010000001010000 = control flags (2 Bytes)
                ^^^^             0100 version          = "MiBeacon V" + 2
                    ^            0 ?                   = no ?
                     ^^          0 aut mode            = none
                       ^         0 solicited           = none
                        ^        0 registered          = no
                         ^       1 mesh device         = yes (header 02)
                          ^      0 object included     = no
                           ^     1 capability included = yes (01 06 ???)
                            ^    1 MAC included        = yes (no MAC!!!)
                             ^   0 encrypted           = no
                              ^  0 ?                   = no ?
                               ^ 0 timing request      = no
TYPE = 0x01AA type (2 Byte) = 426 LYWSDCGQ (Xiaomi)
pk = 0xf5 packet (1 Byte)  = 245
temp = 0x00fb [temperature × 10] (2 Byte float) = 25,1°C
objx = 0x100d object (2 Byte) = 0001 0000 0000 1101 = temperature & humidity
OL = 0x04 object length (1 Byte) = 4 Byte
HUMX = 0x0237 [humidity × 10] (2 Byte float) = 56,7%

PDU code

VRmeshdl??????ctflTYPEpk????????????objxOLtempHUMXrs
020106131695fe5020aa013c4e4fdba8654c041002fb00

VR = 0x02 version (1 Byte) = 2 mesh???
capability = 0x01 0x06 ??? 2 item 1 Byte each, or 1 item 2 Byte (0x0601), or 2 Byte binary flags 011000000001 ???
dl = 0x13 data lengt (1 Byte) = 19 Byte (excluding this starting data byte)

ctfl = 0x5850 = 0010000001010000 = control flags (2 Bytes)
                ^^^^             0100 version          = "MiBeacon V" + 2
                    ^            0 ?                   = no ?
                     ^^          0 aut mode            = none
                       ^         0 solicited           = none
                        ^        0 registered          = no
                         ^       1 mesh device         = yes (header 02)
                          ^      0 object included     = no
                           ^     1 capability included = yes (01 06 ???)
                            ^    1 MAC included        = yes (no MAC!!!)
                             ^   0 encrypted           = no
                              ^  0 ?                   = no ?
                               ^ 0 timing request      = no
TYPE = 0x01AA type (2 Byte) = 426 LYWSDCGQ (Xiaomi)
pk = 0x3c packet (1 Byte)  = 60
objx = 0x1004 object (2 Byte) = 0001 0000 0000 0100 = temperature only
OL = 0x02 object length (1 Byte) = 2 Byte
temp = 0x00fb [temperature × 10] (2 Byte float) = 25,1°C

Who is online

Users browsing this forum: Bing [Bot], johnson73, StephenDig and 75 guests