[solved] Puzzled by two simple hex-dec-string mixup

[sid_snake]

I'm puzzled by two simple hex-dec-string mixups. (Python 3.7.3)

I discover (using Thonny) for it to accept a hex number in my .py code I need to use 0x and not \x, but then when the results are shown in the Shell, it shows the same number but with the \x prefix.

So I try this in the Shell

Code: Select all

>>> int("\xc0",16)
Traceback (most recent call last):
  File "<pyshell>", line 1, in <module>
ValueError: invalid literal for int() with base 16: 'À'

(I'm assuming ascii code for A is C0)

so I try this in the Shell

Code: Select all

>>> int("0xc0",16)
192

which works. So why to both \x and 0x work but in different context? Is that normal?

My next puzzle relates to how to get the hex numbers I extract from a file into decimal. I set the file up as binary, used read(2) to get at two hex numbers, that seems to work thus:

Code: Select all

position = fo.seek(0x0C,0) # put pointer to 12
my_bpm =  fo.read(2)
print("Look now at BPM as two hx numbers ", my_bpm)

that gets the two hex numbers and displays them in the Shell thus:

Code: Select all

Look now at BPM as two hx numbers  b'\x03\xc0'

this shows my first puzzle, the Shell shows the hex numbers as \xc0 whereas in my py code I need to write 0xc0 - why the change?

And now to the big puzzle, exactly what data type is my_bpm returned when using the read() function? As string?
So I use

Code: Select all

my_bpm_number = int(my_bpm, 16)
print("BPM from hex to decimal - ", str(my_bpm_number) )

thinking the first line will convert the 2 byte hex number (string) and convert it to a number.
I want to print the number so I can inspect it so I convert the number to a string for printing, but alas errors.

Code: Select all

Look now at BPM as two hx numbers  b'\x03\xc0'
Traceback (most recent call last):
  File "/home/pi/tester_1/File_Handling/file_experiments_1.py", line 21, in <module>
    my_bpm_number = int(my_bpm, 16)
ValueError: invalid literal for int() with base 16: b'\x03\xc0'

I can only guess I've got my strings mixed with numbers somehow.

My objective is simply to extract 2 adjacent hex numbers from a binary file and use the resulting (decimal) number in further number munching.

[rkn704]

I use the binascii module for converting serial data to Hex or Ascii representation.

The following has been chopped from a program that monitors a serial port and then saves the data to two separate text files. One with the Hex data the other with the Ascii. You can easily manipulate the data once it is in a string.

Code: Select all

import binascii
import serial
rx_channel1 = serial.Serial('COM25', 2400, interCharTimeout= 0.012)

rx_data1 = rx_channel1.read(400)
    

    if len(rx_data1) >0:
        hexpart1 = binascii.hexlify(rx_data1)
        ascpart1 = binascii.unhexlify(hexpart1)
        decodehexData1 =  hexpart1.upper()
        decodeASCData1 =  ascpart1
        print decodehexData1
        print decodeASCData1

[sid_snake]

After reading various forums and py docs, I am becoming to think perhaps I don't know/understand exactly what a binary file is in Python. I thought it would be just a whole list of numbers from which I could pick out using an index pointer but perhaps it's not a simple as that. BTW: my source data files are small 1k to 60k

I've found this: https://stackoverflow.com/a/8711061
which refers to needing struct to perhaps put the source data into a buffer and then access using a format specifier as that buffer image will have added various extra bytes.

Then in Py docs is see this: https://docs.python.org/3/library/struc ... uct.unpack - on how to go about the job.

This all seems very complicated for such a simple task.

Thanks for your suggestion rkn704 I will give it a close look.

I've now found this on the subject, time to see if I can pick out what I need.
https://stackoverflow.com/questions/103 ... ect=1&lq=1

[billio]

\xC0 is a representation of a character not a number, it is upper-case A grave. See the Latin-1 character set. This is what the error message is saying. I am not sure why you think \xC0 should be a number.

0xC0 is the manner in which python represents hex numbers in a program, just the same as 1, 3.162 etc.. The purpose of int() is to convert integers expressed as character strings to real numbers in memory, it is not to provide the numeric value of a character, the function ord() does that .

When reading bytes from a file and converting to numbers, there are some things to think about :

- is it just one byte being converted to a number or two or more
- can the number be negative
- is it big endian or little endian

The documentation of the application writing the file should give information about this.

But, bearing in mind "My objective is simply to extract 2 adjacent hex numbers from a binary file and use the resulting (decimal) number in further number munching." Try the following :

Create a file "binfile.bin" with a random set of characters and use this code to read the file.

Code: Select all

f=open("binfile.bin","rb") # read as binary
num=list(f.read()) # read(x) will read x bytes from the file
print (num)
f.close()

The list shows the characters in the file as decimal numbers, you can select the bytes you want by indexing them. I hope this helps.

[sid_snake]

Thanks yes your suggestions help. I see now the difference between 0x and \x when referring to hex numbers. I've seen references in many documents to where a hex number is preceded by \x and probably why many people like myself are getting confused.

I ran the code you suggested, make a "list" object, yes that ran and printed all decimal numbers in the Shell window. As an experiment I changed it to a tuple and it printed in the Shell window hex numbers preceded with \x but also it printed ascii characters where they existed. So I'm thinking the Shell/Python will use \x before a number to show a hex number. Maybe, it will show the hex number from the Latin-1 character set (or a subset thereof) if not then display as a hex number by using \x.

[Paeryn]

Python will use the prefix \x if it is printing a character that doesn't have printable glyph in the encoding, it will use the prefix 0x if it is printing an integer in hexadecimal. It all depends on what type the value has.

\x is used inside strings to denote that it and the hexadecimal value straight after are to be replaced by the character with the ordinal value given.

0x is used in front of a number to denote that the number is in hexadecimal rather than decimal.

Code: Select all

ord('\x0f') == 0x0f == 15
chr(0x0f) == chr(15) == '\x0f'

[sid_snake]

Thanks for your clarification Paeryn and the code snippet. It's one of those small things that needs careful study to be sure how it works.

After reading your message 20 times and tried a few Shell entry codes with respect to the Latin-1 table, I see now how it works, thanks. I think your wording is perfect and should be added to the official Python docs!

[Paeryn]

Thank you, it took several re-writes before I was happy that I hadn't made things more confusing for you.

Playing around making small changes and seeing what difference it makes (and whether that is what you are expecting) is a good way of learning, programming languages are (for the most part) very logical.