io.FileIO blocking read

[Hove]

I'm using Python io.FileIO to read raw binary data from a file. The read() always returns even if the returned data is zero in length. A block of 1680 bytes of data is written to the file every 0.1s.

I'd like the read to block pending these updates. "print" diagnostics clearly show it isn't - a lot of the reads return 0 bytes

Searches of the python.org site and google only talk about non-blocking as though blocking is perhaps the default, but that's not what I'm seeing.

Any ideas how to make this file access blocking?

Just FYI, the 'file' is in fact a stream of data from picamera writing the macro-blocks to a shared memory file (/dev/shm/filename), and the reading side is a separate thread reading it to do motion processing with each frame's macro blocks. Frame rate is 10fps, frame size = 320 x 320 (21 x 20 macro-blocks)

[sprinkmeier]

Everything in unix is a file, I'd have a look at the "select" module (i.e."import select" and then call teh blocking "select.select" to wait for data to arrive)

If the file is growing by 16KB per second how long until you run out fo memory?

[Hove]

Thanks. I hadn't realized I could mix io.FileIO() with select(). Sadly, the select() doesn't block either - the code hard loops, occasionally returning the expected 1680 bytes, but mostly returning 0. Any ideas?

Code: Select all

#!/usr/bin/python
import io
import time
import picamera
import curses
import thread
import select

####################################################################################################
#
# Kitty++.py - motion tracker based upon video-frame macro-blocks
#
####################################################################################################

def stream_processor():
    stream = io.FileIO("/dev/shm/motion_stream", "r")
    total_bytes = 0
    while True:
        select.select([stream], [], [])
        bytes = stream.readall()
        total_bytes += len(bytes)
        print "num bytes = %d, total %d" % (len(bytes), total_bytes)

#-----------------------------------------------------------------------------------------------
# Setup a shared memory based data stream for the PiCamera video motion output
#-----------------------------------------------------------------------------------------------
stream = io.FileIO("/dev/shm/motion_stream", "wb")

#-----------------------------------------------------------------------------------------------
# Set up the thread for reading and processing the video macro-block data
#-----------------------------------------------------------------------------------------------
m_thread = thread.start_new_thread(stream_processor, ())

#--------------------------------------------------------------------------------------------------
# Video for 10 seconds at 10fps = 100 frames each with 420 x  4 byte blocks per frame = 1680 bytes per frame
#--------------------------------------------------------------------------------------------------
total_bytes = 0
with picamera.PiCamera() as camera:
    camera.resolution = (320, 320)  
    camera.framerate = 10
    camera.start_recording('/dev/null', format='h264', motion_output=stream, quality=23)
    camera.wait_recording(10)
    camera.stop_recording()

stream.close()

[sprinkmeier]

you coudl use a named pipe rather than a file, something like:

Code: Select all

FIFO='/dev/shm/foo'
os.mkfifo(FIFO)
rw=open(FIFO,'w+')
# in the thread
rw.read(1680)
# in the main body
camera.start_recording(rw, format='h264', quality=23)

/dev/shm is a RAM disk, so any files you write there uses RAM. Run for long enough and you're likely to exhaust your RAM.
a fifo does not permanently store the data so you should be able to run for as long as you want.

[Hove]

you coudl use a named pipe rather than a file, something like:

Code: Select all

FIFO='/dev/shm/foo'
os.mkfifo(FIFO)
rw=open(FIFO,'w+')
# in the thread
rw.read(1680)
# in the main body
camera.start_recording(rw, format='h264', quality=23)

Thanks, I'll definitely try that - the critical thing is that select.select() blocks and with io.FileIO is isn't, even if there is no data in the stream. Is it better to pass the rw fd through to the thread that reads it? Currently, the reading thread opens the same file as readonly.

/dev/shm is a RAM disk, so any files you write there uses RAM. Run for long enough and you're likely to exhaust your RAM.
a fifo does not permanently store the data so you should be able to run for as long as you want.

I _believe_ (i.e. not definite fact but strong suspicion) that becase the io.FileIO is a stream , each read of the data clears the data from the fd, regarless whether its a device (like an IP socket), or a file (like /dev/shm/filename). Certainly that's what I'm seeing - the reads of 0 data length suggest that the stream is empty since last time and is now 0 bytes long.

[Hove]

@sprinkmeier: all working now, thank you - now to get on with the processing of the motion data!

Code: Select all

#!/usr/bin/python
import os
import io
import time
import picamera
import curses
import thread
from Queue import Queue as FIFO
import select

####################################################################################################
#
# Kitty++.py - motion tracker based upon video-frame macro-blocks
#
####################################################################################################


def stream_processor():
    global thread_running

    py_fifo = open("/dev/shm/motion_stream", "r")
    total_bytes = 0
    while thread_running:
        select.select([py_fifo], [], [])
        bytes = py_fifo.read(1680)
        total_bytes += len(bytes)
        print "num bytes = %d, total %d" % (len(bytes), total_bytes)
    py_fifo.close()

#-----------------------------------------------------------------------------------------------
# Setup a shared memory based data stream for the PiCamera video motion output
#-----------------------------------------------------------------------------------------------
os.mkfifo("/dev/shm/motion_stream")

#-----------------------------------------------------------------------------------------------
# Set up the thread for reading and processing the video macro-block data
#-----------------------------------------------------------------------------------------------
thread_running = True
thread.start_new_thread(stream_processor, ())

#--------------------------------------------------------------------------------------------------
# Video for 10 seconds at 10fps = 100 frames. Each frame is 320 x 320 pixels.  Each macro-block is
# 16 x 16 pixels.  Due to an extra column of macro-blocks (dunno why), that means each frame breaks
# down into 420 macro-blocks, each of which is 4 bytes - 1 byte X, 1 byte Y and 2 bytes SAD
#--------------------------------------------------------------------------------------------------
with picamera.PiCamera() as camera:
    camera.resolution = (320, 320)
    camera.framerate = 10
    camera.start_recording('/dev/null', format='h264', motion_output="/dev/shm/motion_stream", quality=23)
    camera.wait_recording(10)
    camera.stop_recording()

thread_running = False

[sprinkmeier]

Nice Program!
I haven't done much with the camera myself, hope you post the result somewhere.

I think the "stream" part of FileIO refers to the interface you get, much like the Java IO streams.
Underneath it's still just a file and, being on /dev/shm, it's RAM.
os.stat() the file and watch it grow...

[Hove]

Underneath it's still just a file and, being on /dev/shm, it's RAM.
os.stat() the file and watch it grow...

It's deliberately /dev/shm - as you say it's ram, and in another application I've locked the program + data to not page ram to disk, so I should get the best performance possible. I'll double check with os.stat() but it does seem that the combination of writing and reading to the ram based FIFO fixed the select() problem, and the fifo 'file' shows 0 bytes after each read and at the end of the test.

I'll add the non-paging to this code too once I've got to a stage where I'm getting meaningful motion vectors out. Maybe this afternoon if the kids behave themselves!

[sprinkmeier]

Underneath it's still just a file and, being on /dev/shm, it's RAM.
os.stat() the file and watch it grow...

It's deliberately /dev/shm - as you say it's ram, and in another application I've locked the program + data to not page ram to disk, so I should get the best performance possible.

Last time I tried to 'optimise' things like this (overriding the scheduler to lock tasks to a core) I ended up making things worse.
Boot into console mode (great way to get rid of all the gooey stuff).
Next I'd try disabling swap. If you still need it to go faster make careful measurements to see if what you do actually helps.

[jojopi]

Searches of the python.org site and google only talk about non-blocking as though blocking is perhaps the default, but that's not what I'm seeing.

To try to clarify this, blocking I/O is the default. There is the concept of whether a file descriptor is "ready" for read or write, which you can test with select or poll. With blocking I/O, read or write will block until the file is ready. With non-blocking I/O they will return immediately.

A pipe, fifo, socket, or a device file such as a terminal or serial port is ready for read if there is at least some data in the input buffer, and is ready for write if there is at least some space left in the output buffer. Thus blocking I/O may wait, and non-blocking I/O will not wait.

However, a regular file is always considered ready for both read and write. There is no difference between blocking and non-blocking. If you happen to be at the end of the file, you will read zero bytes. If the disk is full, writes will fail. There is no option to wait in the hope that some other process makes the file longer or frees up disk space.

A regular file is simply not suitable for streaming data from one application to another. Even if you wanted to keep all the data permanently on disk as well, you should write it to a file and pass it through a pipe.

(There is also a potential issue that regular files are buffered in large blocks by libc, while pipes and such are normally line buffered. Despite all this it is sometimes desirable to monitor the output of a program without having to rebuild it to output to a pipe. For this a workaround is "tail -f", which tries its best to get data in real time by monitoring for changes in the file length.)

By the way, /dev/shm (tmpfs) is not backed by RAM but by virtual memory, which includes swap. Even it is swaps, though, it should be much faster than any disk-based filesystem would be, because it does not survive a reboot, so it stores no permanent metadata. Now that you have switched to a fifo, it takes up no space anyway.

Disabling swap will almost always reduce performance, because the Foundation's images are already configured to swap only as a last resort. So your application may crash because there is not enough memory, or it may be slower because the kernel cannot swap other programs out to make more room. It cannot be faster because the kernel was swapping it just for fun.

[Hove]

It's always interesting to know the underlying details as well as the correct solution, so thanks for sharing on /dev/shm, paging and blocking fd's. 45 and still learning