shkhmaaz
Posts: 43
Joined: Mon Aug 31, 2020 2:30 pm

Need help reading hex data

Wed Aug 04, 2021 6:57 am

So an IoT device that I am currently using stores payloads as a base64 string. this payload along with other data is stored in a .json file which is then transferred to a MQTT broker. The broker then decodes the payload from the base64 string to a hexadecimal string. The hex string has 7 bytes. The documentation of the device says that bytes 3,4 specify angle (ranging from -180 to 180) and they are of type int16. While bytes 5 and 6 represent a percentage from 0 to 100 and they are of type uint16. All values are stored in little endian format.

An example reading i got was "230112effff303". How do i convert these values into angle and percentage?

The description for the angle :

angle
This signed integer provides angle vector measurement
from a closed position in units of 0.1° from event detection.
This can be used to indicate how far the valve is open. It
depends on the type and size of the valve. It can be ignored
if not needed.
Range: -180.0° -- 180.0°


for percentage :

The magnitude ratio compared to calibration vector
magnitude in % with steps of 0.1% . This can be used to
indicate how far the valve is open. It depends on the type
and size of the valve. It can be ignored if not needed.
Range: 0.0% .. 100.0%

Also says 0.1% per LSB

PiGraham
Posts: 4847
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Need help reading hex data

Wed Aug 04, 2021 7:32 am

I'm not a Python expert, but possibly, convert the hex string to a byte buffer then unpack the byte data into a struct.

https://pypi.org/project/PyByteBuffer/

https://docs.python.org/3/library/struct.html

You could also slice the hex into substrings for each data field and convert those to integers with int()

shkhmaaz
Posts: 43
Joined: Mon Aug 31, 2020 2:30 pm

Re: Need help reading hex data

Wed Aug 04, 2021 7:49 am

PiGraham wrote:
Wed Aug 04, 2021 7:32 am
I'm not a Python expert, but possibly, convert the hex string to a byte buffer then unpack the byte data into a struct.

https://pypi.org/project/PyByteBuffer/

https://docs.python.org/3/library/struct.html

You could also slice the hex into substrings for each data field and convert those to integers with int()
Hi! thanks for the links and the information. I just want to be able to understand how to convert the values i have in readable form manually first so then i can program it later.

So for example in the value i got, byte numbers 5 and 6 are "f303". if i convert them to uint16, i get "1101" , which doesnt make sense because it is over the percentage range.

PiGraham
Posts: 4847
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Need help reading hex data

Wed Aug 04, 2021 8:48 am

shkhmaaz wrote:
Wed Aug 04, 2021 7:49 am
PiGraham wrote:
Wed Aug 04, 2021 7:32 am
I'm not a Python expert, but possibly, convert the hex string to a byte buffer then unpack the byte data into a struct.

https://pypi.org/project/PyByteBuffer/

https://docs.python.org/3/library/struct.html

You could also slice the hex into substrings for each data field and convert those to integers with int()
Hi! thanks for the links and the information. I just want to be able to understand how to convert the values i have in readable form manually first so then i can program it later.

So for example in the value i got, byte numbers 5 and 6 are "f303". if i convert them to uint16, i get "1101" , which doesn't make sense because it is over the percentage range.
f303 hex is 62211 decimal.

23 01 12 ef ff f3 03

there is a question of endianness, whether bytes in memory at a lower address represent the most or least significant part of multi-byte values.
It could be f303 (big endian) or 03f3 (little endian)
03f3 is 1011
That might be percentage x 10 or 101.1%

cleverca22
Posts: 4363
Joined: Sat Aug 18, 2012 2:33 pm

Re: Need help reading hex data

Wed Aug 04, 2021 9:24 am

figuring out how to turn that hex into a usable number will first require the datasheet for whatever sensor is being used

shkhmaaz
Posts: 43
Joined: Mon Aug 31, 2020 2:30 pm

Re: Need help reading hex data

Wed Aug 04, 2021 10:04 am

PiGraham wrote:
Wed Aug 04, 2021 8:48 am
shkhmaaz wrote:
Wed Aug 04, 2021 7:49 am
PiGraham wrote:
Wed Aug 04, 2021 7:32 am
I'm not a Python expert, but possibly, convert the hex string to a byte buffer then unpack the byte data into a struct.

https://pypi.org/project/PyByteBuffer/

https://docs.python.org/3/library/struct.html

You could also slice the hex into substrings for each data field and convert those to integers with int()
Hi! thanks for the links and the information. I just want to be able to understand how to convert the values i have in readable form manually first so then i can program it later.

So for example in the value i got, byte numbers 5 and 6 are "f303". if i convert them to uint16, i get "1101" , which doesn't make sense because it is over the percentage range.
f303 hex is 62211 decimal.

23 01 12 ef ff f3 03

there is a question of endianness, whether bytes in memory at a lower address represent the most or least significant part of multi-byte values.
It could be f303 (big endian) or 03f3 (little endian)
03f3 is 1011
That might be percentage x 10 or 101.1%

Thanks for the explanation! thats kinda how i took it too. The device is a valve with a position sensor so sometimes for the percentage i would be getting values like 101.1% when its "closed" but if i would turn it to an open position i would get a higher value, which doesnt make sense to me :/



cleverca22 wrote:
Wed Aug 04, 2021 9:24 am
figuring out how to turn that hex into a usable number will first require the datasheet for whatever sensor is being used

hello, thanks for the response. The documentation is in this link : https://github.com/TWTG-R-D-B-V/neon-pr ... 20v2.2.pdf

Im looking at the "Application event" message at the moment.

cleverca22
Posts: 4363
Joined: Sat Aug 18, 2012 2:33 pm

Re: Need help reading hex data

Wed Aug 04, 2021 10:16 am

Code: Select all

clever@c2d ~ $ node
> Buffer("230112effff303","hex")
<Buffer 23 01 12 ef ff f3 03>
the first byte (0x23) is made up of 2 nibbles
the 0x3 is the message type, application event, the 0x2 is the protocol version

then we skip down to page 14/27
the 0x01 byte is the current state, 1 meaning closed
then the 0x12 is a counter of how many transitions have happened, so you can see if you missed an event
then you have a 16 bit angle held in the 0xef and 0xff, covering the range of -180 to +180
then you have a 16 bit magnitude value, 0xf3 and 0x03

assuming its big-endian, you would then treat it as:

Code: Select all

state = 0x1
counter = 0x12
angle = 0xefff
magnitude = 0xf303
and you must test the message type in the 1st byte before parsing anything, because it could be a different message!

the next quesiton then, is how to convert the angle into a number in the -180 to +180 range...
the pdf isnt as clear on how that is done
Last edited by cleverca22 on Wed Aug 04, 2021 10:22 am, edited 1 time in total.

shkhmaaz
Posts: 43
Joined: Mon Aug 31, 2020 2:30 pm

Re: Need help reading hex data

Wed Aug 04, 2021 10:22 am

cleverca22 wrote:
Wed Aug 04, 2021 10:16 am

Code: Select all

clever@c2d ~ $ node
> Buffer("230112effff303","hex")
<Buffer 23 01 12 ef ff f3 03>
the first byte (0x23) is made up of 2 nibbles
the 0x3 is the message type, application event, the 0x2 is the protocol version

then we skip down to page 14/27
the 0x01 byte is the current state, 1 meaning closed
then the 0x12 is a counter of how many transitions have happened, so you can see if you missed an event
then you have a 16 bit angle held in the 0xef and 0xff, covering the range of -180 to +180
then you have a 16 bit magnitude value, 0xf3 and 0x03

assuming its big-endian, you would then treat it as:
[code]
state = 0x1
counter = 0x12
angle = 0xefff
magnitude = 0xf303
and you must test the message type in the 1st byte before parsing anything, because it could be a different message!

the next quesiton then, is how to convert the angle into a number in the -180 to +180 range...
the pdf isnt as clear on how that is done
That makes sense. The bytes are in little endian order though! and yeah, the documentation is not that clear on how it works :roll:

PiGraham
Posts: 4847
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Need help reading hex data

Wed Aug 04, 2021 10:23 am

Note position ranges -180° to +180° in 0.1° steps.

So not percentage and actual range of values depends on the valve type.
It might also depend on whether the sensor is properly zeroed.
Value is supposed to be "angle from a closed position" so we would expect ~0° when closed and larger values + or - when open.
If you get 101.1° when closed maybe the sensor (a potentiometer?) is mounted rotated from zero. Is that adjustable?

cleverca22
Posts: 4363
Joined: Sat Aug 18, 2012 2:33 pm

Re: Need help reading hex data

Wed Aug 04, 2021 10:23 am

try graphing how the angle changes over time, as it makes a complete rotation, and see what the min/max are

shkhmaaz
Posts: 43
Joined: Mon Aug 31, 2020 2:30 pm

Re: Need help reading hex data

Wed Aug 04, 2021 10:42 am

PiGraham wrote:
Wed Aug 04, 2021 10:23 am
Note position ranges -180° to +180° in 0.1° steps.

So not percentage and actual range of values depends on the valve type.
It might also depend on whether the sensor is properly zeroed.
Value is supposed to be "angle from a closed position" so we would expect ~0° when closed and larger values + or - when open.
If you get 101.1° when closed maybe the sensor (a potentiometer?) is mounted rotated from zero. Is that adjustable?
So i multiply whatever value i get for the angle with 0.1? And yes the sensor is adjustable. I probably will have to re-calibrate it and test it again. Just wanted to check if i was reading them the right way.




cleverca22 wrote:
Wed Aug 04, 2021 10:23 am
try graphing how the angle changes over time, as it makes a complete rotation, and see what the min/max are

yeah i probably will do that. Thank you!

PiGraham
Posts: 4847
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Need help reading hex data

Wed Aug 04, 2021 12:08 pm

shkhmaaz wrote:
Wed Aug 04, 2021 10:42 am

So i multiply whatever value i get for the angle with 0.1? And yes the sensor is adjustable. I probably will have to re-calibrate it and test it again. Just wanted to check if i was reading them the right way.

Probably. If you can command the valve to move to positions measure the physical angle moved. My guess is that the number is x10 (fixed point decimal). You can use the value as-is or scale to floating point degrees by multiplying by 0.1 (or dividing by 10.0)

Return to “Beginners”