Page 1 of 1

Write delays on piZero / Data stack / Async writing advice sought

Posted: Fri Apr 05, 2019 10:27 pm
by robjharrison
I am wondering if there is anyone who is perhaps more capable with regards either python threading, async data writing or understanding some of the limitations of an SD card (write speeds?) who might be able to comment/suggest ways forward on the script i have and/or why i might be seeing this unexpected time step in records.

In particular, I am trying to get to the bottom of why my script is only recording data every 3-4 seconds when i should be recording it every 1 second ~(see python script driveData.py - there's a 1 second sleep).

I've put the python files on: https://github.com/robjharrison/MovementDataAnalysis

But a clipped version of the output below: It's the seconds that i'm most interested in here.
timestamp,temp_h,temp_p,humidity,pressure,pitch,roll,yaw,mag_x,mag_y,mag_z,accel_x,accel_y,accel_z,gyro_x,gyro_y,gyro_z
2019-04-04 21:38:00,17.973743438720703,
2019-04-04 21:38:04,18.10059356689453,
2019-04-04 21:38:08,18.154956817626953,
2019-04-04 21:38:12,18.10059356689453,
2019-04-04 21:38:16,18.028106689453125,
2019-04-04 21:38:20,18.263683319091797,
2019-04-04 21:38:24,18.245563507080078,
2019-04-04 21:38:28,18.2999267578125,
2019-04-04 21:38:32,18.354290008544922,
2019-04-04 21:38:35,18.372411727905273,
2019-04-04 21:38:38,18.571746826171875,
2019-04-04 21:38:41,18.4630184173584,
2019-04-04 21:38:44,18.4992618560791,
2019-04-04 21:38:47,18.571746826171875,
2019-04-04 21:38:50,18.48114013671875,
2019-04-04 21:38:53,18.589866638183594,
2019-04-04 21:38:56,18.64423179626465,
2019-04-04 21:38:59,18.626110076904297,
2019-04-04 21:39:02,18.662353515625,
2019-04-04 21:39:05,18.68047332763672,
2019-04-04 21:39:08,18.734838485717773,
2019-04-04 21:39:11,18.789201736450195,
2019-04-04 21:39:13,18.807323455810547,
2019-04-04 21:39:16,18.970415115356445,

I have seen a similar delay with the pi3 (although the steps do seem to be reduced to 2-3seconds). Current hardware is:
Pi zero, Sensehat, pi camera

Perhaps worth noting that i am collecting video footage, saved in 5min segments, motion data from that and the sensor data headings seen above here, so i guess there is a fair bit of data being written - but it is batched into minute sections and the video into 5mins. But all should be clear from the example scripts on Github. Looking for any advice regarding the scripts whether specific to this issue or not . thank you.

Re: Write delays on piZero / Data stack / Async writing advice sought

Posted: Fri Apr 05, 2019 11:00 pm
by Andyroo
I would add a set of print statements between each line in your while loop to see where the delay actually happens.

For example, is the show_message command blocking?

As for threading, you could create a queue to store the data on and have threads that:

1) Read the data from the seance hat and store it on the queue
2) Check if anything is on the queue and write out the data

Re: Write delays on piZero / Data stack / Async writing advice sought

Posted: Sun Apr 07, 2019 3:37 pm
by robjharrison
Andyroo wrote:
Fri Apr 05, 2019 11:00 pm
For example, is the show_message command blocking?
All good advice, Thx. You were right about this. Removing this within the loop reduces the delay and so clearly has an impact .

As for the queue/threads - I havent done this before and so will have to look further into how etc.

Re: Write delays on piZero / Data stack / Async writing advice sought

Posted: Sun Apr 07, 2019 3:52 pm
by Andyroo
I posted a very very simple thread example viewtopic.php?t=235173#p1438374

Also have a look at viewtopic.php?p=1444310 where I have issues and solve keyboard interruptions (ctrl-c) inside the threads

That should get you started with threads - they are complex and avoids the whole multiprocessing module :lol:

Queues on the other hand are simpler and https://pymotw.com/2/Queue/ is OK to start with and has more links.

Re: Write delays on piZero / Data stack / Async writing advice sought

Posted: Wed Apr 10, 2019 11:01 pm
by robjharrison
Andyroo your post was (again!) really useful, much appreciated.