User avatar
jbeale
Posts: 3367
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

writing I32 to binary file: endian-ness on R-Pi

Mon Apr 16, 2018 6:44 am

I have an ADC sending 24-bit samples that I want to record and later examine using Audacity. To avoid having to create a WAV or other standard format file, I'm just writing raw I32 values and using the Audacity "raw input" option, File -> Import -> Raw Data -> Encoding: "Signed 32-bit PCM". I didn't have any trouble doing a similar thing, writing raw 32-bit floats on a Windows PC, but the R-Pi seems to work differently. Here's my code:

Code: Select all

#!/usr/bin/python
import time, serial, struct

ser = serial.Serial('/dev/ttyUSB0',115200)
bcount = 0
inWord = ser.read(4)
ba = bytearray(inWord)
print(ba)  # show first 4 bytes to confirm good start
f = open('log1.raw','wb')  # raw output log file

inWord = ser.read(4)  # read 4 bytes
tLast = time.time()

while inWord is not None:
    dat = struct.unpack('l', inWord)[0]  # one I32 integer
    dat2 = dat * 512                           # scale up to avoid loss in float.pt. project
    inWord2 = struct.pack('l', dat2)
    if ((bcount % 3000) == 0):
      tNow = time.time()
      dT = tNow-tLast
      tLast = tNow
      hstr = ':'.join(x.encode('hex') for x in inWord2)
      print( "dT: %.1f  val: %d , %d , %s" % (dT,dat,dat2,hstr) )

    bcount += 1
    f.write(inWord2)
    inWord = ser.read(4)  # read 4 bytes
Every 30 seconds I print out some status info including an I32 value and the 4 bytes in hex.
The program prints -155136 (decimal) and "00:a2:fd:ff" (hex) which is straightforward little-endian representation of the signed 32-bit integer, as b0 b1 b2 b3. But using 'od -x log1.raw | head' I see what is on disk is:
"a200 fffd" in other words b1 b0 b3 b2. What's the right way to do this?

Edit: Oops, I see with 'od -l t3.raw | head' indeed the correct I32 value is printed.

User avatar
OutoftheBOTS
Posts: 674
Joined: Tue Aug 01, 2017 10:06 am

Re: writing I32 to binary file: endian-ness on R-Pi

Mon Apr 16, 2018 8:47 am

Ok I am a little confused about 1 part of your code

Code: Select all

    dat = struct.unpack('l', inWord)[0]  # one I32 integer
    dat2 = dat * 512                           # scale up to avoid loss in float.pt. project
    inWord2 = struct.pack('l', dat2)
You seem to unpack a long (4 byte int) then multiply it by 512 (shift it << 9 bits) then pack it back as a long (4 byte int). This juts doesn't seem to make any sense to me. All your doing is throwing away the top 9 bits.

You have a commit "# scale up to avoid loss in float.pt. project" but if you have unpacked the data as a signed 4 byte whole number data type there won't be any floating point.

Also if you do want to move the data by 9 bits to the right use "<<9" instead of "*512" because for the CPU to do multiplication require a lot of clock cycles. For the CPU to do multiply what it does is a loop over addition e.g 3 * 5 does 3+3+3+3+3 and addition is done by looping over just adding 1 at a time e.g 3+3 is done by 3+1+1+1. So if you have a large number and multiply it by 512 then it require a hell of a lot of cycles to do the job compared to just shifting the bit 9 places with <<9 as this is by by shifting the bits 1 place 9 times over and is completed in 9 cycles.

User avatar
Paeryn
Posts: 2215
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: writing I32 to binary file: endian-ness on R-Pi

Mon Apr 16, 2018 11:20 am

OutoftheBOTS wrote:
Mon Apr 16, 2018 8:47 am
For the CPU to do multiply what it does is a loop over addition e.g 3 * 5 does 3+3+3+3+3 and addition is done by looping over just adding 1 at a time e.g 3+3 is done by 3+1+1+1. So if you have a large number and multiply it by 512 then it require a hell of a lot of cycles to do the job compared to just shifting the bit 9 places with <<9 as this is by by shifting the bits 1 place 9 times over and is completed in 9 cycles.
Multiplication isn't done by adding the multiplicand multiplier times, that's a horribly inefficient way of doing it, and to suggest that addition is done by repeatedly incrementing by 1 is, to honest, absurd. Obviously it could be done that way but only by a very naive implementation.
She who travels light — forgot something.

User avatar
paddyg
Posts: 2243
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: writing I32 to binary file: endian-ness on R-Pi

Mon Apr 16, 2018 11:39 am

And, apart from that, I had also assumed that bit shifting was quicker for powers of two but, in python3 anyway, it doesn't seem to be

Code: Select all

import timeit
setup = '''
a = 123
'''
fns = ['''
b = a * 512
''', '''
b = a << 9
''']
for f in fns:
  print(timeit.timeit(f, setup=setup, number=10000000))
0.5207787250001275
0.5899600220000139

PS interesting bit of recent R&D on stochastic processing in which multiplication is done by ANDing two streams of random bits where the values to be multiplied are represented by the probability of a 1 in each stream. Very simple and error tolerant (but obviously low precision) [sorry @jbeale this isn't helping your question]
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

User avatar
jbeale
Posts: 3367
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: writing I32 to binary file: endian-ness on R-Pi

Mon Apr 16, 2018 1:57 pm

Audacity can import I32, but the project edit and display mode is reported as 32-bit float. If it is representing numbers as 32-bit floats internally, that means 1 bit sign, 8 bits exponent and 23 bits mantissa. So it seemed to me (late last night when my brain wasn't so great) that the low-order 9 bits of an I32 could get thrown away when converted to float, and I only have a 24-bit ADC not a 32-bit ADC anyway.

User avatar
paddyg
Posts: 2243
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: writing I32 to binary file: endian-ness on R-Pi

Mon Apr 16, 2018 2:04 pm

what do you get if you change the endianness with > or < (in the struct pack formatter)?

PS or when importing into audacity?
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

User avatar
OutoftheBOTS
Posts: 674
Joined: Tue Aug 01, 2017 10:06 am

Re: writing I32 to binary file: endian-ness on R-Pi

Mon Apr 16, 2018 6:53 pm

@Paeryn

It is quite interesting how a computer is built and how the computer maths. For a small inside see this project where a guy builds his own computer for adding https://blog.hackster.io/redditor-build ... 07dae08c00

A computer is only capable of binary operations, in fact it is only capable of NAND binary operations and from a NAND operation it can create all the other binary operations. From these biniary operations you can create what's called an adder which can add 1 to any binary number and this is the base of what's called an ALU (arithmetic logic unit) which is the core hardware of a CPU. From this adder all other math operations can be created: adding larger numbers is created by adding 1 many times over, minus is created by adding a negative number and multiplication is created by adding many times over.

@Paddyg

If python is doing mulyplying by 2 and <1 in the same time this would indicate that python is smart and is doing multiplying by 2 by changing it to <1.


The bottom line is binary operations are always faster than other math functions as the computer is a binary operator.

User avatar
OutoftheBOTS
Posts: 674
Joined: Tue Aug 01, 2017 10:06 am

Re: writing I32 to binary file: endian-ness on R-Pi

Mon Apr 16, 2018 6:58 pm

@Jbeale

If you data is coming as a float but you need to export it as a int then just unpack it as a float then pack it as an int and the struct libiary will do the rest.

dat = struct.unpack('f', inWord)[0]
inWord2 = struct.pack('l', dat)

User avatar
paddyg
Posts: 2243
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: writing I32 to binary file: endian-ness on R-Pi

Mon Apr 16, 2018 7:28 pm

Not sure about your multiplication machine but the standard hardware works along these lines i.e there is addition but just like you do with 'long' multiplication where you add up the columns. Yes, at the machine level, multiplication is a lot more expensive than bit shifting and all modern compilers will spot powers of two and use bit shifting... Which means that generally you should leave that optimization to the compiler!

The bit shifting v multiplication does seem to be due to optimization see here. However it's not, as you suggest, because python spots 2**n. When I run the timeit comparison with other numbers multiplication is still quite a bit faster.

The import into audacity is so flexible that I wouldn't attempt to do any conversion in python. It seems to be able to cope with different signed or unsigned int, 32 or 64 bit floats as well as four different endianness options and up to 16 channels.
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

User avatar
Paeryn
Posts: 2215
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: writing I32 to binary file: endian-ness on R-Pi

Mon Apr 16, 2018 9:42 pm

<This is off-topic to the original post so I won't continue after this>
OutoftheBOTS wrote:
Mon Apr 16, 2018 6:53 pm
@Paeryn

It is quite interesting how a computer is built and how the computer maths. For a small inside see this project where a guy builds his own computer for adding https://blog.hackster.io/redditor-build ... 07dae08c00

A computer is only capable of binary operations, in fact it is only capable of NAND binary operations and from a NAND operation it can create all the other binary operations. From these biniary operations you can create what's called an adder which can add 1 to any binary number and this is the base of what's called an ALU (arithmetic logic unit) which is the core hardware of a CPU. From this adder all other math operations can be created: adding larger numbers is created by adding 1 many times over, minus is created by adding a negative number and multiplication is created by adding many times over.
I'm well aware of how CPUs work. And just because NAND gates can be used to create all other logic gates it doesn't mean you can only use NAND gates (indeed all logic gates can also be made using only NOR gates). An NOR gate would take more transistors to build if you had to make it out of the required 4 NAND gates.

No real-world processor adds two numbers by repeatedly adding 1. Even a naive implementation could just use a cascade of full-adders, time required dependant on bit-width as you have to wait for the carry signal to propagate along the width. Carry look-ahead methods can reduce that by calculating the carries in parallel up-front then generate the result bits in parallel so you don't have to wait for the carry to ripple along the width.

Look up the TI 74181, a 4-bit ALU from 1970, so nearly 50 years ago they had ICs that could add/sub/inc/dec (and more) in 4-bit chunks and not using many logic gates at all. Modern CPUs do a full 32-bit or 64-bit add/sub in a single cycle.

Multiplication is more complex but at most you need one add and one shift per bit-width of the multiplier. If you throw enough logic at it you can reduce it further, the ARM in the Pi0 can multiply two 32-bit numbers returning a 64-bit answer in just 5 cycles.
She who travels light — forgot something.

User avatar
scruss
Posts: 1863
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: writing I32 to binary file: endian-ness on R-Pi

Mon Apr 16, 2018 9:48 pm

While I can't disagree with using Audacity as a large ADC data set viewer, once you get your import parameters sorted it I find it much quicker to use SoX (in the standard Raspbian repos: sudo apt install sox libsox-fmt-all) to do the conversion. As it works from the command line, you can ferret the parameters in a shell script and never worry about remembering them again. It knows about all endianesses.

SoX has suffered a bit from feature creep over the years, so to say its command line options are extensive are an understatement. You can do everything from simple file conversions to massive synthesized effects like this glissando.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

User avatar
OutoftheBOTS
Posts: 674
Joined: Tue Aug 01, 2017 10:06 am

Re: writing I32 to binary file: endian-ness on R-Pi

Tue Apr 17, 2018 2:32 am

@Paddyg
The bit shifting v multiplication does seem to be due to optimization see here.
Ok after reading that link then multiplying by 255 isn't as heavy as I first through but it is definitely still much heavier than just shifting to the left.

User avatar
jbeale
Posts: 3367
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: writing I32 to binary file: endian-ness on R-Pi

Fri Apr 20, 2018 6:04 pm

scruss wrote:
Mon Apr 16, 2018 9:48 pm
While I can't disagree with using Audacity as a large ADC data set viewer, once you get your import parameters sorted it I find it much quicker to use SoX (in the standard Raspbian repos: sudo apt install sox libsox-fmt-all) to do the conversion. As it works from the command line, you can ferret the parameters in a shell script and never worry about remembering them again. It knows about all endianesses.
Thank you for the note about sox, I had not fully appreciated what it can do. It's an excellent tool and very fast to use. It has no problems reading my raw single-channel I32 files on the R-Pi and it can even generate spectrograms, which means I don't need Audacity at all for my present purposes.

Return to “Python”