User avatar
TheAlbatross
Posts: 7
Joined: Wed Jul 23, 2014 7:10 pm

still capture rate (still FPS)

Wed Jul 23, 2014 7:49 pm

About the frame rate for capturing still images:

I have tried with bash raspistill, and python picamera capture().

My framerate with picamera is 10 frames in about 6.25 seconds. With a bash script it was 10 frames in 14.5 seconds -- without the -o option. (Under 1 frame a second...)

I tried all different modes (no preview, awb off, fast shutter speed, small image size, default jpg encoding, etc.). No resource drags are running... Here is one iteration of the bash line I was using, followed by the picamera python:

Code: Select all

bash: raspistill --nopreview -awb off -t 1 -tl 0
python picamera: camera.capture('/home/pi/cam/images/' + str(x) +'.jpeg')
Again, the latter was more successful at almost 2 fps! But I suspect I am missing something.

"2592×1944 1-15fps, video or stills mode, Full sensor full FOV, default stills capture" -- is 15fps possible in stills mode?

I love it anyway, just trying to learn the thing. Thanks!
Last edited by TheAlbatross on Thu Jul 24, 2014 12:46 pm, edited 1 time in total.

User avatar
jbeale
Posts: 3499
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: still capture rate (still FPS)

Thu Jul 24, 2014 12:05 am

The OmniVision sensor chip itself is, in theory capable of full-resolution stills at 15 fps. However, I do not believe that frame rate has been demonstrated on a Raspberry Pi with the current configuration of image pipeline firmware and raspistill software. In still capture mode, not only is there some delay for setup of autoexposure and color balance each time, but I also suspect the pipeline may average together some frames for noise reduction purposes. I find stills have a notably better image quality than still frames from video, even at the same pixel resolution, and even if you tell MMAL to compress video frames directly to JPEG instead of going through H.264, and that temporal noise reduction may be why.

There are examples in Python (picamera) of connecting a still image encoder to the video "port" internally, see for example
http://picamera.readthedocs.org/en/rele ... -streaming

User avatar
TheAlbatross
Posts: 7
Joined: Wed Jul 23, 2014 7:10 pm

Re: still capture rate (still FPS)

Fri Jul 25, 2014 2:12 pm

Thanks John! I was half hoping to hear that I was doing something obviously wrong.

I will try to figure out how to minimize the delaying factors. I tried to minimize the camera's "spin up" time, disabling nonessential features. Please share any thoughts that spring to mind!

Please share more thoughts on the frame averaging theory. This could indeed decrease noise, but unless the camera and subject are completely still, it will blur the two frames, like having a longer exposure. But again I'm probably missing something. I'm sure you're on to something, and I'd love to hear more. Quite an interesting subject! Thanks again!
Last edited by TheAlbatross on Mon Aug 04, 2014 3:41 am, edited 1 time in total.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23915
Joined: Sat Jul 30, 2011 7:41 pm

Re: still capture rate (still FPS)

Fri Jul 25, 2014 3:03 pm

We don't average between frames. We do bin pixels in the lower resolutions which has a similar effect.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7431
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: still capture rate (still FPS)

Fri Jul 25, 2014 3:09 pm

Just to quash the rumour, there is no frame averaging on stills captures. There is a more intensive denoise applied to captures which slows it down, but actually most of the delay is due to having to change the sensor mode, which means:
  • Waiting for the end of frame
  • stopping the sensor
  • reprogramming it
  • restarting it
  • waiting most of an exposure time for the first frame, but then dropping it as it is normally corrupted
  • reading out the desired frame
This typically all adds up to at least 90ms, and potentially much much more if you're on more than a 30ms exposure.
There is a way around this, but not currently in the released apps.

I've just played with V4L2 using the commands:
  • sudo modprobe bcm2835-v4l2 max_video_width=2592 max_video_height=1944
  • v4l2-ctl -p 15
  • v4l2-ctl -v width=2592,height=1944,pixelformat=I420
  • v4l2-ctl --stream-mmap=3 --stream-to=/dev/null --streamcount=1000
and that is reading out at a claimed 15 or 16fps. :D
I have checked and that is genuinely reading the full 5MPix off the sensor and processing it to produce the output frames.
Share and enjoy.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jbeale
Posts: 3499
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: still capture rate (still FPS)

Fri Jul 25, 2014 4:07 pm

Thank you 6by9! Looks like I was wrong on both counts. Impressive that we can do 15 fps at full res. Sorry for spreading the rumour- I guess my frames with blurry motion were simply the result of indoor lighting = longer shutter time.

gapwedge
Posts: 10
Joined: Mon Jul 28, 2014 4:14 pm

Re: still capture rate (still FPS)

Mon Jul 28, 2014 9:13 pm

Noob question: How do you play a file produced by v4l2-ctl on the RPi on a Windows PC?

I followed 6x9's instructions and produced a file like this:

Code: Select all

[email protected] ~ $ v4l2-ctl --stream-mmap=3 --stream-to=./myfile --stream-count=10
<<<<<<< 6 fps
<<<
[email protected] ~ $
I then FTP'd myfile to my Windows box and tried playing it with ffplay.exe and VLC, but neither understood the format.

ethanol100
Posts: 587
Joined: Wed Oct 02, 2013 12:28 pm

Re: still capture rate (still FPS)

Mon Jul 28, 2014 10:30 pm

The file is raw yuv420 file. This file has no information on resolution, this needs to be specified. You could try ffplay:

Code: Select all

ffplay -f rawvideo -pix_fmt yuv420p -video_size 2592x1944 -i video.yuv
Or you can split the files to frames and convert them to jpeg, i.e.:

Code: Select all

split -d -b $((3*2592*1944/2)) video.yuv frame-
for i in frame-??; do 
mv $i $i.yuv;
convert -size 2592x1944 -depth 8 -sampling-factor 2x2 $i.yuv $i.jpg;
done
(with imagemagick).

Both not tested, but I hope they work ;)

gapwedge
Posts: 10
Joined: Mon Jul 28, 2014 4:14 pm

Re: still capture rate (still FPS)

Mon Jul 28, 2014 10:57 pm

Thanks! your ffplay params worked. It was very big though at max resolution--75 megabytes for 10 frames.

I'm hoping I can change the pixel format to mjpeg to reduce the size, and then figure out the params to use for ffplay.

-- edit --

Just in case anyone is interested, here's the commands for mjpeg:

Code: Select all

 v4l2-ctl -v width=2592,height=1944,pixelformat=MJPG
v4l2-ctl --stream-mmap=3 --stream-to=./video.mjpg --stream-count=50
[Download much smaller file per frame to Windows PC]

Code: Select all

./ffplay -f mjpeg -i video.mjpg -framerate 10
I had set my RPi camera framerate to 10 with v4l2-ctl

Am I losing some image detail/sharpness when going from the raw data to mjpeg?

User avatar
jbeale
Posts: 3499
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: still capture rate (still FPS)

Tue Jul 29, 2014 2:42 am

MJPEG is a series of JPEG images. JPEG uses lossy compression to reduce file size, so certainly some information is going away. JPEG compression is adjustable and if the settings are high enough, it becomes "visually lossless" that is to say, you don't notice the missing information by eye. I don't know what default compression settings are used in the R-Pi output, but by all means, do some tests and see for yourself if it looks different from the uncompressed version.

Also note that there is some noise reduction in the R-Pi before you ever get the image, even in uncompressed form so the loss due to JPEG may be even less significant.

gapwedge
Posts: 10
Joined: Mon Jul 28, 2014 4:14 pm

Re: still capture rate (still FPS)

Tue Jul 29, 2014 3:05 am

Thanks for the info. The image quality of the mjpeg video looks great to the eye. The main goal of my project is to do some object detection/recognition on the image frames coming from the RPi. We're hoping the RPi's 5 megapixel camera provides a lot more detail than our current 2mp webcam for things like feature detection and template matching.

I just want to make sure the mjpeg compression doesn't ruin the gains we see from 2mp to 5mp. I'll do some tests as you suggest.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7431
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: still capture rate (still FPS)

Tue Jul 29, 2014 7:20 am

MJPEG, being a video codec, has a target bitrate rather than specifying the normal Q-factor. It tweaks the Q-factor per frame based on the bitrate achieved so far.
As long as you set the bitrate high enough(*), then the image quality should be maintained reasonably. MJPEG is not an efficient codec, but it's the only option as H264 on Pi maxes out at 1920x1080.

(*) Sorry, can't remember the v4l2-ctl command off the top of my head. It'll be v4l2-ctl --set-ctrl=<something>=<value>. v4l2-ctl --list-ctrls would list all the values allowed for <something>, one of which will be video bitrate.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

gapwedge
Posts: 10
Joined: Mon Jul 28, 2014 4:14 pm

Re: still capture rate (still FPS)

Wed Jul 30, 2014 3:41 pm

One last set of questions, sorry for kind of de-railing this thread.

I see there are a lot of posts in this forum about streaming. People recommend gstreamer, mjpg-streamer, netcat, etc. My current plan is to capture each frame at max res with MJPEG format using v4l2-ctl, and then 'netcat' the v4l2-ctl output over our LAN to an image processing server. A friend recommended configuring netcat to use UDP, he thought it would make a big performance difference. Then on the image server pull out the jpeg frames and process them individually. Is this a stupid plan? :lol:

I've never used netcat before. Is it good at throttling data, throwing stuff away when it needs to? I assume at times the camera will be producing data a lot faster than netcat can send depending on network speed.

I'm new to streaming--It seems like most of the streaming threads here first save to the SD Card and then stream over the network. And then they never talk about cleaning up the file(s). How do most people handle that part of it? I'm assuming if I can figure out how to send the output of v4l2 directly to netcat, then I won't need to write anything to the SD Card, but I'm not sure.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7431
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: still capture rate (still FPS)

Wed Jul 30, 2014 4:42 pm

UDP is not a guaranteed transmission protocol, whereas TCP is. UDP packets could be reordered and/or not be delivered at all. I would only suggest you use it if both your devices are connected on the same wired LAN. Any wireless links or routers in the way could otherwise cause issues.
Sorry, I'm no expert on streaming the data and sensible ways to do it.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
waveform80
Posts: 305
Joined: Mon Sep 23, 2013 1:28 pm
Location: Manchester, UK

Re: still capture rate (still FPS)

Thu Jul 31, 2014 9:46 am

jbeale wrote:Thank you 6by9! Looks like I was wrong on both counts. Impressive that we can do 15 fps at full res. Sorry for spreading the rumour- I guess my frames with blurry motion were simply the result of indoor lighting = longer shutter time.
Actually the rumour's probably from my docs - I'll get those fixed up for 1.7 now I know the real reason! :)

Dave.

Kozuch
Posts: 64
Joined: Sun Aug 10, 2014 10:35 am

Re: still capture rate (still FPS)

Tue Sep 16, 2014 4:05 pm

Does anyone know how many FPS is it possible to achieve in case where I want to trigger every single frame via a wired GPIO external trigger (remote shutter)? My problem is that I need to synchronize two cameras for (possibly) perfect stereo capture. Ideal solution would of course be those 15 FPS in full resolution mode. I am not talking about the stereo sync solution now since that is a different story. I am simply interested in how big is the delay after a single GPIO triggered still shot until a new trigger can be accepted.

You may say I could use video and trigger it only once for start but the problem with video usually is that it will most likely drift out of stereo sync after some time during its capture. So my idea is to use still photos and trigger this every time again and minimize any possible (video) drift like this. The problem I fear is the speed of saving those individual frames to SD card. I plan to get class 10 card (10MB/s write speed) to minimize the writing time.

Also, I dont mind saving some kind of raw sensor data (like YUV format) to offload the CPU but in case JPEG can be encoded quickly enough by HW than it would probably be saved faster to sd card than an uncompressed image file. I will appreciate any comments! :)

User avatar
TheAlbatross
Posts: 7
Joined: Wed Jul 23, 2014 7:10 pm

Re: still capture rate (still FPS)

Tue Sep 16, 2014 8:33 pm

I think I read on this forum somewhere that the USB itself is the bottleneck.

Not sure about this -- but if I remember/understood correctly: Information from the camera must pass through the internal USB line to get to the SD card.

Another question is, is there a "camera ready" state you can check? Maybe the two cameras in your setup could communicate with each other, so they know when they're both ready.

EDIT: As noted upthread, a "spin up" before the camera fires, even if the preview delay is zero. See the list upthread to learn what happens in preparation before the exposure is captured; the Pi will do several steps, and maybe they can get out of sync, if the spin up doesn't always take the same amount of time. In my tests (firing as fast as permitted), there were sometimes big pauses between shots, so yes the spin-up may be variable. The spin-up doesn't happen between frames, in video capture mode.
Last edited by TheAlbatross on Tue Sep 16, 2014 10:19 pm, edited 1 time in total.

Kozuch
Posts: 64
Joined: Sun Aug 10, 2014 10:35 am

Re: still capture rate (still FPS)

Tue Sep 16, 2014 9:32 pm

Thanks for an insight. I guess I simply may just need to try this out - feed a GPIO input signal of certain frequency and see if shots are dropped or not (simply if all photos were taken or not). If shots are dropped, I may need to go for a lower frequency. I am ready to explore the raspistill source and modify it since it probably is not prepared for such scenario (I havent examined its code yet).

This guy modified raspivid to get possibly better sync for video start so I guess I will need to do something similar. I do not own the Pi yet (will arrive tomorrow though) and I know nothing about the camera "workflow" - if there are some stages like "camera ready" etc. Will try to dive into it a bit soon. So far I hope on both units to behave the same way since the cameras will see almost the same scene.

User avatar
waveform80
Posts: 305
Joined: Mon Sep 23, 2013 1:28 pm
Location: Manchester, UK

Re: still capture rate (still FPS)

Wed Sep 17, 2014 1:27 am

Kozuch wrote:Thanks for an insight. I guess I simply may just need to try this out - feed a GPIO input signal of certain frequency and see if shots are dropped or not (simply if all photos were taken or not). If shots are dropped, I may need to go for a lower frequency. I am ready to explore the raspistill source and modify it since it probably is not prepared for such scenario (I havent examined its code yet).

This guy modified raspivid to get possibly better sync for video start so I guess I will need to do something similar. I do not own the Pi yet (will arrive tomorrow though) and I know nothing about the camera "workflow" - if there are some stages like "camera ready" etc. Will try to dive into it a bit soon. So far I hope on both units to behave the same way since the cameras will see almost the same scene.
If you're intending to use JPEG you will certainly find that the SD card's bandwidth is the limiting factor. I'm not sure whether the SD card path goes via USB or not, but I can say that even with a class 10 card the bandwidth is less than the USB bus' overall bandwidth given that certain operations work happily when streaming over ethernet (which goes via USB on the model B pi) but fail (stalling for IO) when writing to the SD card. I can predict with some confidence that you won't be able to write full resolution JPEGs to a class 10 SD card at anywhere near 15fps. Using the python picamera library, with a class 10 SD card, and a Pi overclocked to "medium", I can just about manage ~1.5fps with JPEGs at full resolution:

Code: Select all

import io
import sys
import time
import picamera

COUNT = 30

with picamera.PiCamera() as camera:
    camera.resolution = camera.MAX_RESOLUTION
    camera.framerate = 15
    if sys.argv[1] == '-':
        output = sys.stdout
    else:
        output = io.open(sys.argv[1], 'wb')
    start = time.time()
    for i in range(COUNT):
        camera.capture(output, 'jpeg', use_video_port=True)
    finish = time.time()
    print('Captured %d images in %.1fs at %.2ffps' % (
        COUNT, finish - start, COUNT / (finish - start)))
Obviously there's some overhead from Python there versus something like C, but mostly it's the SD card holding it back. Writing to /dev/null instead of an actual file, it jumps to ~3fps (Python's probably a more significant factor there):

Code: Select all

[email protected] ~/foo $ python limit.py foo.data
Captured 30 images in 18.5s at 1.62fps
[email protected] ~/foo $ python limit.py /dev/null
Captured 30 images in 10.3s at 2.91fps
At lower resolutions (and lower qualities) you may have more luck. Using the VGA-res high fps modes I've managed individually triggered captures at close to 40fps (again, bear in mind that's Python so you can certainly go faster with C).

In other words, what you're proposing (capturing rapidly in response to GPIO impulses) is possible, but if you're intending to use SD card output I'd temper your expectations of resolution, framerate, or quality accordingly.

Dave.

User avatar
TheAlbatross
Posts: 7
Joined: Wed Jul 23, 2014 7:10 pm

Re: still capture rate (still FPS)

Wed Sep 17, 2014 2:06 am

With all due respect, of course -- I don't know for sure. But I don't think the SD card's bandwidth is the bottleneck. I think I have read that the write-speed of the SD card is faster than an internal Pi bottleneck, in capturing information from the camera. I hope a wiz will clarify (or correct me!).

Kozuch
Posts: 64
Joined: Sun Aug 10, 2014 10:35 am

Re: still capture rate (still FPS)

Wed Sep 17, 2014 6:45 am

Well, the whole issue will probably go down to some basic physics as a primary limiting factor and then the application speed (the code) as a second limiting factor. I dont know how large is full resolution JPEG with default compression from Pi camera but say an image will have 2MB (for example). If I can write to an sd at 10MB/s I could save 5 FPS in theory (at best). If SD card is wired via USB internally then a 2.0 USB gives 480 Mbit/s (60MB/s) in theory but say it could maybe do 40MB/s inpractice, so it should actually be possible to to use an external HDD or SSD in USB enclosure to work around the SD card speed limit.

Also Python really may have more overhead than we think. What is also interesting is the "frame skipping" issue (80 to 40 FPS) you describe at SE. Can this be fixed somehow? Is it a bug in the Picamera library?

User avatar
waveform80
Posts: 305
Joined: Mon Sep 23, 2013 1:28 pm
Location: Manchester, UK

Re: still capture rate (still FPS)

Wed Sep 17, 2014 9:36 am

TheAlbatross wrote:With all due respect, of course -- I don't know for sure. But I don't think the SD card's bandwidth is the bottleneck. I think I have read that the write-speed of the SD card is faster than an internal Pi bottleneck, in capturing information from the camera. I hope a wiz will clarify (or correct me!).
If the SD card in the example isn't the bottleneck ... how does the write speed nearly double when switching the output from an actual file to /dev/null? I'm afraid the SD card is a very real bottleneck in terms of IO.

On the subject of how the SD card IO is wired, it's not via USB but direct to the CPU. Also bear in mind that the Pi doesn't support UHS signalling, and as that last post suggests, the bandwidth claims of SD card classes are often so much marketing waffle, which is most likely why my class 10 card above is achieving nowhere near 10Mb/s (it's more like 4.5Mb/s). Although, in its defence, it's a pretty heavily used card (it's been in my development Pi for a good year now!)

On Python's overhead - very much depends what you're doing. If it's pure IO (which is more or less what the example above is doing), there's very little. Most of Python's IO stuff is written in C, and as long as you're just throwing pointers to memory around (which is mostly what picamera does) without allocating and copying stuff (tricky to tell in Python sometimes), it's pretty fast. For example, the library has no trouble keeping up with callbacks from the H.264 encoder when it's recording at 90fps.

That in turn suggests to me that the halved framerate issue mentioned in that SO answer is something related to the triggering of the capture rather than the IO side of things, but I haven't tested that. I vaguely recall in the early days of picamera development I could capture frames at the full framerate from the video port, and at some point in recent development I found the rate had halved - but whether that was due to a change in firmware or the library I've no idea (unfortunately I've no idea when I'll have time to delve into this, but it's an interesting question).


Dave.

Kozuch
Posts: 64
Joined: Sun Aug 10, 2014 10:35 am

Re: still capture rate (still FPS)

Wed Sep 17, 2014 11:07 am

Well, you are right that the SD card actually is a system drive so maybe one should not use it for saving data where speed is needed because any system IO read/write operation that may interfere (I am not saying there surely will be one but just in case) with the desired data feed from camera will surely slow the big camera write down. Probably a very fast (speed tested) USB flash/HDD/SDD drive may be much better since it wont be used by any other process and still has the nice USB 2.0 bus speed.

Regarding SD speed Classes - they pretty much are correct for continuous writes (which the camera does) so a class 10 card will do very close to 10MB/s. My "stereo Pi" units (2 x Pi + 2 x camera) just arrived today so I hope to start exploring this. Even though the Picamera might have nice features I will most likely dive right into raspistill/raspivid C code to get as close to metal as possible. I hope to be able to hack the pair of these files to get a decent amount of what I need - stereo sync for stills or video.

If rapsivid is friendly enough one may even be able to do a sync via GPIO for each video frame... that would be an absolute dream solution since it could use the H.264 to keep the bandwidth for storage bottleneck low in comparison to JPEG or raw YUV.

User avatar
TheAlbatross
Posts: 7
Joined: Wed Jul 23, 2014 7:10 pm

Re: still capture rate (still FPS)

Wed Sep 17, 2014 11:50 am

waveform80 wrote:If the SD card in the example isn't the bottleneck ... how does the write speed nearly double when switching the output from an actual file to /dev/null?
The diagrams here should help:

http://raspberrypi.stackexchange.com/qu ... figuration

I think I read somewhere that the bottleneck is the internal USB, which links the CPU to the ethernet and USB ports (and SD?). But the person at that link there is speaking of a direct connection between the CPU and SD card, so I really am confused. But I think the facts are available. A few months ago I did research on this, but now I've forgotten the details, and it's quite possible I came to invalid conclusions at that time.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23915
Joined: Sat Jul 30, 2011 7:41 pm

Re: still capture rate (still FPS)

Wed Sep 17, 2014 8:41 pm

The SD card doesn't go via USB. It has its own interface.

SD card can be a bottleneck, certainly in video mode at high bit rates, we can often produce data faster than the SD card can write it.

A mod to Raspistill would be easy to make that caches the images and writes them out later when time is not an issue.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

Return to “Camera board”