lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Minimal latency for still image

Fri Feb 07, 2014 10:45 am

I'd like to use the CSi cam for a unicycling sports event. When a racer passes the finish line, I'd like to take a still image with minimal latency (as a record to settle possible disputes). The racers might pass the line with 30 km/h, and because of that I'd like to have a snapshot within 10 ms after triggering. Quicker response (like 1 ms) would even be better.

I think my best bet is to use the same API as RaspiStill (and borrow most of its code ;) ), but I am not sure about a couple of things.
  • If there is a preview, the camera is streaming, right? So I cannot capture my still at the best time, but need to depend on the best matching image from that stream? So it seems a requirement to not do a preview, or am I wrong?
  • Is it possible to prepare the camera, let it do some auto-exposure etc. and then put it in some kind of "wait mode" and trigger it to take stills with minimal latency?
Any comments or suggestions welcome!

Alex

ethanol100
Posts: 589
Joined: Wed Oct 02, 2013 12:28 pm

Re: Minimal latency for still image

Fri Feb 07, 2014 11:59 am

I think it will be very difficult to reach 10 ms. I have played a bit with the signal method of raspistill and the best delay i have got is about 160 ms. Even if you do not use the preview, taking an image using the still port will take some time to switch to capture mode. You could use the preview port, like i.e. raspimjpeg does, to capture images, but even this will just capture the "next" frame which will be ~34ms delayed.

I would try to record a video with raspivid and use the circular buffer which was introduced recently. With this you can capture a small time duration(i.e. 10s duration 5sec before and 5 sec after the event). You would get ~34 ms between frames at 30fps.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: Minimal latency for still image

Fri Feb 07, 2014 12:15 pm

Thanks ethanol!

What I guess from reading the RaspiStill code, is that it sets the camera to some streaming mode, because of the preview. I hope it is possible to start the device, configure it, let it do calculations on exposure etc. and than have it sit still, and be ready for action. If needed, I could have it take some sample shots every second or so to maintain correct exposure levels, and only put it in "ready for a still" a second or so before the stills are needed (could just set up another beam breaker or other motion detector a few meters before the finish line for that).

If that is all not possible, I could also let the camera stream images, and continuously keep "the last frame". After the trigger, I could also capture the next frame, compare the capture times with the trigger time, and keep the one closest to the trigger time. That would give me a timing error of at most 1/2 of the time between frames (i.e. max 17 ms for 30 fps). The downside is of course the lower resolution (video vs still), but maybe that is not a problem at all. (Will need to do some live shots to see how much resolution we really need).

What I also would like to know is some more details about how the still port really works. Is the camera put in a special state? I could not find documentation on that yet!

ethanol100
Posts: 589
Joined: Wed Oct 02, 2013 12:28 pm

Re: Minimal latency for still image

Fri Feb 07, 2014 1:24 pm

I think it works similar to this(but only guessing):

The program consists of "components" which transport the image data via a buffer which is piped over "ports" between these components.
The basic component is the camera. The camera is capturing continuous images at some fixed frame rate(as requested with -fps) and resolution(only 2 resolutions are available (full frame 2592x1944 with max of 15fps and video frames at 1920x1080 at max of 30fps, all other sizes are scaled later).
The camera has three "output" ports, it can deliver the image to three other components.
In raspivid and raspistill these ports are connected to the "preview", "video" and "still" components which have "input" ports. The "still" component is a jpeg encoder, it takes the uncompressed image, compresses it as an jpeg image and outputs it to a callback function, which writes the jpeg data to a file. The "video" component is a h264 encoder, which will create a videostream of a series of images and again it is connected to a callback function which will just write the output of the encoder to a file. The "preview" port just "copy" the buffer to the monitor.
The difference between raspistill and raspivid is that raspistill only uses the jpeg encoder component and raspivid the h264encoder component.

The preview window usually displays only 1920x1080 at 30fps. The camera can only deliver one size of a frame at a time, and it needs to be changed to 1920x1080 for the preview. Now if the program requests a still image, it needs to change the resolution to the full sensor size, and this full frame is then send to the jpeg encoder. The change of the "mode" will take some time, and therefore an additional delay will occur. There is an option "--fullpreview" which will change the preview to display the full frame at 15 fps where no mode change should be necessary. And an option "--nopreview" where the buffer is just send to a null sink (just a component which does nothing with the buffer) and will not be shown. But in my tests I have not seen much difference between these preview settings.

The problem is the preview frames will always be generated, there are just not send in all cases. The image processing in the gpu will calculate exposure and iso using this continuous stream of images. And I think you can not "pause" this stream. And an image or video is just created, when the programm tells the camera component to deliver these frames to the "encoding" components.

Only JamesH and a few other will really know how it is working in the videocore gpu.

But for two full frame stills you will get always a delay of 67ms between frames. Therefore I would not be very optimistic for triggering full frame images.

I would also like to know if there is a difference between taking a picture from the preview port and taking the picture of the still port. There should be some, I guess.

Interesting link: http://inrng.com/2012/04/photo-finish-camera/
So the usual way is to take 10,000 frames per second video. Then you can really decide who has won a photo finish.
The idea is funny, to take a single column of a video frame and create an artificial image by putting these columns side by side to create a picture with time as the x-direction.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: Minimal latency for still image

Mon Feb 10, 2014 3:06 pm

Yeah, those multi-thousands frames per second camera's have drawn my attention. Even thought the frames are only 1 - 3 pixels wide, they are very interesting for such a project (or for photographing e.g. entire trains while passing at 160 km/h, but that is another project). Those fast line scan camera's make a huge difference according to the CSi cam of the RPi, also in price :)

Anyway, thanks for the further explanation of the inner working of the GPU video processing. I already gathered most of it, but this sums it up nicely for me.

I also read about a 720p/60 mode "in the make", that would reduce the maximum error of images from a stream to 1/120 s = 8.3 ms. Sounds nice enough - anyone knows if this is achievable yet?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24556
Joined: Sat Jul 30, 2011 7:41 pm

Re: Minimal latency for still image

Mon Feb 10, 2014 9:37 pm

Yes, we got 58fps out of it today.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: Minimal latency for still image

Tue Feb 11, 2014 9:59 am

Hi James, this is great news! And thank you for the great work (JamesH a.o.?)!

I suppose this is still lab work, and may need changes in the firmware? The event is in May, so I can wait. And oh, otherwise we'll just have to do with a little less this year...

Can you (or anyone) shed some light (or give a pointer) about the resolutions, fps's, speed/bandwidth limitations for the various encoders on the video and still ports, etc.
It would be great to have a list of tested (or otherwise known working) resolutions and fps together with the used encoders. This could of course be a community effort, but maybe there is already a list? I couldn't find one yet...

Again: thanks James a.o. for your efforts, much appreciated!

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: Minimal latency for still image

Tue Feb 11, 2014 10:08 am

And another question... can I set the camera imaging chip to full resolution, and have a ROI of 720p AND stream images somewhere near 60 fps? Or is the camera chip always sending all pixels, and is the down scale done in the GPU?
So mainly my question is: where are the bottlenecks? Is it the imaging chip not being able to read so many pixels in a certain time frame, or is it in the transport, or is it in the GPU?

Anyone with pointers to more technical information like this (not as deep down technical stuff as chip datasheets, but 'just a bit more processed information'), please let me know!

ethanol100
Posts: 589
Joined: Wed Oct 02, 2013 12:28 pm

Re: Minimal latency for still image

Tue Feb 11, 2014 11:10 am

The main bottleneck is the sensor:

According to this link: http://ovt.com/products/sensor.php?id=66
The sensor is capable of getting these resolutions and framerates:
  • 720p @ 60 fps
  • 1080p @ 30 fps
  • Full @ 15 fps
  • VGA @ 90 fps
  • 960p @ 45 fps
You can only get higher frame rates if you request a smaller resolution. Basically it always takes a specific amount to read a single line of the sensor, and if you skip some lines you can get to smaller times needed to read one image. So at the moment you can not even specify a ROI because only a central crop can be read. There are some tries to get full field of view resized to [email protected] by only reading each second line(2x2 binning as described in the link).

To your other question regarding the encoder and its limitations:

The gpu uses H.264 Level 4.0, this limits the bitrate to 25 Mbit/s in Profile "High". And it just allows maximum resolutions of 1920x1080(or 1088). This also gives some limitations for framerate and sizes see "levels": http://de.wikipedia.org/wiki/H.264
By default the bitrate is set to 17Mbit/s in raspivid.

If you would need to capture full frames you need to switch to the older MJPEG encoder( And then you could have problems with io-bandwidth).
Last edited by ethanol100 on Tue Feb 11, 2014 11:34 am, edited 1 time in total.

gordon77
Posts: 4367
Joined: Sun Aug 05, 2012 3:12 pm

Re: Minimal latency for still image

Tue Feb 11, 2014 11:22 am

ethanol100 wrote:I think it will be very difficult to reach 10 ms. I have played a bit with the signal method of raspistill and the best delay i have got is about 160 ms. Even if you do not use the preview, taking an image using the still port will take some time to switch to capture mode. You could use the preview port, like i.e. raspimjpeg does, to capture images, but even this will just capture the "next" frame which will be ~34ms delayed.

I would try to record a video with raspivid and use the circular buffer which was introduced recently. With this you can capture a small time duration(i.e. 10s duration 5sec before and 5 sec after the event). You would get ~34 ms between frames at 30fps.
Could you explain the 'signal method' of raspistill

Thanks

Gordon77

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: Minimal latency for still image

Tue Feb 11, 2014 11:54 am

RaspiStill has several methods of triggering when to take an image. Default it displays a preview for some time, and after that time elapses, it takes a snapshot. Other modes are e.g. time lapse, but also "wait for a unix interrupt signal", I think ethanol refers to this last method.

I want more control, so I'm trying to incorporate some of the functionality of RaspiStill in my own program. But I found some work of Chris Cummings. He's written a C++ class to take YUV uncompressed images from the video port, see http://robotblogging.blogspot.co.uk/201 ... i-for.html. For my purpose this seems a good option, especially when the cam can support 60 fps in the near future. I'll just copy every image in a CPU memory buffer (overwriting the last one), keep the current time (perhaps minus some offset) as capture time. On the signal of a finish line pass, I'll keep that last buffered frame and keep the next frame, and let e.g. OpenCV encode the best matching image (or both) to JPEG, write them to some storage.
In this setup I have no control on the exact moment the image is captured, but it is at most 1000 / ( 2 * fps) ms off.

I'll start coding! :)

ethanol100
Posts: 589
Joined: Wed Oct 02, 2013 12:28 pm

Re: Minimal latency for still image

Tue Feb 11, 2014 12:35 pm

Yes exactly, I have used this "wait for a unix interrupt signal". It just waits until receiving a "USR1" signal through the kill command which was send by an other program, starting a stop watch on the monitor to measure the time between sending the signal and the time the image was captured.

But the arm is too slow to handle the yuv stream at that resolution and framerate (that is the reason why "PiCam API/Samples" uses only [email protected]). The data needs to be copied from the gpu memory to the arm memory. And if you want to do some processing it will just not have enough calculation power. Why not use raspivid to record and i.e. "https://www.cabrillo.edu/~dbrown/tracker/" to analyze it. Then you can even extract the velocity of the bicycles and estimate the exact finishing time by linear fit of the surrounding images? What do you expect to archive, which additional control do you want to get? Everything on the arm side will be really slow...

User avatar
waveform80
Posts: 315
Joined: Mon Sep 23, 2013 1:28 pm
Location: Manchester, UK

Re: Minimal latency for still image

Tue Feb 11, 2014 1:01 pm

I think ethanol mentioned this above, but you probably want to look into capturing stills via the video port. With the picamera library I can (for short bursts) manage full frame 1024x768 captures at 30fps, and that's in Python (not exactly a quick language). The quality isn't as good as stills via the still port, but they'd be good enough for a photo finish.

That said, I still think the suggestion of a circular buffer for video recording is the best option. There's enough memory in a model B to record a full minute to RAM at 720p30 (assuming not much else is running - no X and so on), so when 720p60 comes along you'd be able to stuff up to 30 seconds in there and select the frames you want for a photo finish.


Dave.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: Minimal latency for still image

Tue Feb 11, 2014 2:15 pm

I've done some homework. From the datasheet of the OV5647, imaging chip I gather there are only a distinct number of modes:

Code: Select all

    format         resolution    fps  scaling method        pixel clock
    QSXGA (5 Mpx)  2592 x 1944    15  full resolution       80   MHz
    1080p          1920 x 1080    30  cropping              68   MHz
    960p           1280 x  960    45  cropping,             91.2 MHz
                                      subsampling/ binning
    720p           1280 x  720    60  cropping,             92   MHz
                                      subsampling/ binning
    VGA             640 x  480    90  cropping,             46.5 MHz
                                      subsampling/ binning
    QVGA            320 x  240   120  cropping,             32.5 MHz
                                      subsampling/ binning
JamesH e.a. have apparently now managed (or are very close) to utilise the 720p mode; QSXGA (full) and 1080p already worked. Need to check 960p and VGA.
I think / guess the fps is more or less flexible, the datasheet lists the maximum values. The driver needs to compute a whole bunch of timing values (e.g. for the pixel clock) based on the requested fps, and program the image sensor accordingly.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: Minimal latency for still image

Tue Feb 11, 2014 2:52 pm

The event has various types of races. Some races are short, (e.g. 100m) and in those cases racers pass the line only once. Others are longer (e.g. 5km or 10km), a race may take 30 minutes, and multiple rounds. Esp. for the shorter distances, finishing times might be very close. There may also be multiple heats, sequential or only slightly shifted in time (racers from multiple heats can be on the track simultaneously).
There are all kinds of unicycling rules that makes correct automated interpretation very difficult. For instance, a racer may fall, and after falling s/he may cross the finish line. In that case, s/he needs to walk back and re-mount - it is unicycling after all :).
In addition, much as I'm trying to get time tracking to a higher lever, budget is limited. E.g. racer identification is currently done manually (RF-ID is something we'd like to add next year, but that equipment is costly).

To keep things simple, I'd like to track start signals, line passes, and images independent. Store data fairly raw, and be able to see and reconstruct the actual events later on. All events (signal or image) will be connected both to a "race time" with millisecond resolution, and to the normal time-of-day time (for images I could use e.g. EXIF data and/or file names to store this metadata). This data is to be stored on a local storage (e.g. USB stick. This is also a backup - laptops may crash after all), and send to a laptop. On that laptop, race monitor software allows a human to attach racer's numbers to line passes, discard invalid passes, etc. To help that human, I'd like to be able to display stills from 2 cameras (front view and over the finish line), taken during the line pass. This displaying of images should be done during the race, i.e. very short after the pass happened. So taking a video for later processing is not really an option.

That software may do other stuff, like display racer's data (e.g. group, class, age, sex), and for spectators it may show data on a larger display (e.g. round times, preliminary rankings etc.). I expect there will be a more or less continuous stream of events that need to be recorded, interpreted, and browsed (in case of a dispute). And to settle the possible disputes (if any :)) I need stills orthogonal to the race track, in focus and with fair resolution, taken over the finish line, and as closely to the moments the finish line is crossed.

ethanol100
Posts: 589
Joined: Wed Oct 02, 2013 12:28 pm

Re: Minimal latency for still image

Tue Feb 11, 2014 3:43 pm

What about using QR codes on the bikes/racers? Similar to:
http://www.dw.de/mass-experiment-helps- ... a-16922008
Than automatic realtime processing on a pc with a live stream from the raspberry pi could manage to track line crosses. And then use a second pi to display lap times and other things?

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: Minimal latency for still image

Tue Feb 11, 2014 4:01 pm

Funny idea (both the research and applying it to this event) :). But the riding style of competitors may vary quite considerably. Maybe a camera looking on the backs may work. Will consider this for next year, thanks!

davewh
Posts: 2
Joined: Sat Mar 15, 2014 5:30 pm

Re: Minimal latency for still image

Sat Mar 15, 2014 6:22 pm

lxh wrote:Funny idea (both the research and applying it to this event) :). But the riding style of competitors may vary quite considerably. Maybe a camera looking on the backs may work. Will consider this for next year, thanks!
I'm interested in finish results as well. Please let me know how you get on. I am already using a 120fps camera and have a working system, but they are $650 USD each. I have been using the RPi camera and the GPU but just started.

Stage 1 is to get the 30 fps working creating jpg images of every 4 seconds. Depending on the distance the camera is from the finish line, you need to change the pixels / ms. @5 meters from the line (further back the better when using low fps) I capture 45 pixels. I am still converting from dotnet and need help :)

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: Minimal latency for still image

Wed Apr 30, 2014 8:52 pm

OK, back again. I am still looking for a working solution, but life and other work have kept me a little bit too busy.

The status: I get 90 frames per second all right, in MMAL_ENCODING_I420 format (I presume, have not yet displayed it)

What I'm trying to achieve is to get a JPEG from the video stream (a few images per second at most), but with minimal latency.
My current thinking:
  • set up camera with e.g. 10 buffers
  • in the callback, hold on for the last buffer for a short while, release it on the next frame
  • when triggered, store both the last frame and the next frame to files/memory stream (JPEG)
I don't have a display attached, so in my opinion all GPU power can be used for handling the video/stills/encoding. It seems wasteful to have every frame encoded in JPEG, but I wouldn't mind if that is the solution. I tried setting the video port encoding to MMAL_ENCODING_JPEG, but that did not work (I'm not really surprised :) ).

Is there an easy way to get the I420 data (or other supported formats coming from the video port) and have it compressed and packed in a JPEG? The mmal library is loaded, of course, but I'm open to suggestions for other libraries (such as OpenCV), another requirement is grabbing stills (not too time critical) from an external webcam.

Any suggestions? Thanks in advance!
Alex

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: Minimal latency for still image

Thu May 01, 2014 12:15 pm

Answering my own question: yes, there is a way to compress data to jpeg on the GPU, the mmal encoder component does that.

I wish I could find some decent documentation about mmal, it seems such a powerful thing!

Return to “Camera board”