New camera mode released

Liz: you’ll notice that this post has no pictures or video. That’s because we’d like you to make some for us, using the new camera mode. Take some 90fps video using our camera board and the information below, slow it down to 30fps and send us a link: if yours is particularly splendid, we’ll feature it here and on the front page. Over to JamesH!

I asked for video, you were forthcoming. Here are two which arrived in the few hours after we posted this: the first video, with the juggling clubs, is from Tobias Huebner; the second, with the bouncing balls, is from JBAnovling. JamesH’s discussion about what’s going on happens after the pretty. Enjoy!

When the Raspberry Pi camera was released, the eagle-eyed among you noticed that the camera hardware itself can support various high frame rate modes, but that the software could ‘only’ manage 30 frames per second in its high-definition video mode.

There’s is no hardware limitation in the Raspberry Pi itself. It’s quite capable of handling these high frame rate modes, but it does require a certain amount of effort to work out these new ‘modes’ inside the camera software. At the original release of the camera, two modes were provided: a stills capture mode, which offers the full resolution of the sensor (2592×1944), and a 1080p video mode (1920x1080p). Those same eagle-eyed people will see that these modes have different aspect ratios – the ratio of width to height. Stills outputs 4:3 (like old school TV), video 16:9 (wide screen).

This creates a problem when previewing stills captures, since the preview uses the video mode so it can run at 30 frames per second (fps) – not only is the aspect of the preview different, but because the video mode ‘crops’ the sensor (i.e. takes a 1920×1080 windows from the centre), the field of view in preview mode is very different from the actual capture.

We had some work to do to develop new modes for high frame rates, and also fix the stills preview mode so that is matches the capture mode.

So now, finally, some very helpful chaps at Broadcom, with some help from Omnivision, the sensor manufacturer, have found some spare time to sort out these modes, and not just that but to add some extra goodness while they were at it. (Liz interjects: The Raspberry Pi Foundation is not part of Broadcom – we’re a customer of theirs – but we’ve got a good relationship and the Foundation’s really grateful for the volunteer help that some of the people at Broadcom offer us from time to time. You guys rock: thank you!)

The result is that we now have a set of mode as follows :

  • 2592×1944 1-15fps, video or stills mode, Full sensor full FOV, default stills capture
  • 1920×1080 1-30fps, video mode, 1080p30 cropped
  • 1296×972 1-42fps, video mode, 4:3 aspect binned full FOV. Used for stills preview in raspistill.
  • 1296×730 1-49fps, video mode, 16:9 aspect , binned, full FOV (width), used for 720p
  • 640×480 42.1-60fps, video mode, up to VGAp60 binned
  • 640×480 60.1-90fps, video mode, up to VGAp90 binned

I’ve introduced a new word in the that list. Binned. This is how we can get high frame rates. Binning means combining pixels from the sensor together in a ‘bin’ in the analogue domain.  As well as reducing the amount of data, this can also improve low light performance as it averages out sensor ‘noise’ in the absence of quantisation noise introduced by the analogue to digital converters (ADCs), which are the bits of electronics in the sensor that convert the analogue information created by incoming photons to digital numbers.

So if we do a 2×2 ‘bin’ on the sensor, it only sends a quarter (2×2 = 4 pixels merged in to one = one quarter!) of the amount of data per frame to the Raspberry Pi. This means we can quadruple (approximately – there are some other issues at play) the frame rate for the same amount of data! So a simple 2×2 bin theoretically means quadruple the frame rate, but at half the X and Y resolution. This is how the 1296×972 mode works – it’s exactly a 2×2 binned mode, so it’s still 4:3 ratio, uses the whole sensor field of view, and makes a perfect preview mode for stills capture.

We also have a very similar mode, which is 1296×730. This is used for 720p video recording (the sensor image is scaled by the GPU to 1280×720). This is a 2×2 binned mode with an additional crop, which also means a slightly increased frame rate as there is less data to transfer.

Now by reducing the resolution output by the sensor even further and by using ‘skipping’ of pixels in combination with binning, we can get even higher frame rates, and this is how the high speed 640×480 VGA modes work. So, the fastest mode is now VGA resolution at 90 frames per second – three times the frame rate of 1080p30.

So, how do we use these new modes?

The demo applications raspistill and raspivid will already work with the new modes. You can specify the resolution you need and the frame rate, and the correct mode will be chosen. You will need to get the newest GPU firmware using sudo rpi-update which contains all these shiny new modes.

One thing to note: the system will always try to run at the frame rate specified in preference to resolution. Therefore if you specify a high rate at a resolution it cannot manage, it will use a low resolution mode to achieve the frame rate and upscale to the requested size – upscaling rarely looks good. It may also be too fast for the video encoder, so some of the extra frames may be skipped. So always ensure the resolution you specify can achieve the required frame rate to get the best results.

So, a quick example, to record a 10s VGA clip at 90fps

raspivid -w 640 -h 480 -fps 90 -t 10000 -o test90fps.h264

There have also been minor changes to the V4l2 driver to support these new modes. These should be included when you do the rpi-update to get the new GPU firmware.

The V4L2 driver supports the new modes too. Just using the normal requests, you can now ask for up to 90fps. So doing the same streaming of VGA at 90fps to H264 would be the following set of v4l2-ctl commands:

v4l2-ctl -p 90
 v4l2-ctl -v width=640,height=480,pixelformat=H264
 v4l2-ctl --stream-mmap=3 --stream-count=900 --stream-to=test90fps.h264

There are a few provisos that you will need to consider when using the faster modes, especially with the V4L driver.

  • They will be increasing the load on the ARM quite significantly as there will more callbacks per second. This may have unpredictable effects on V4L applications so that they may not be able to keep up.
  • The MJPEG codec doesn’t cope above about 720P40 – it will start dropping frames, and above 45fps it seems to be able to lock things solid. You have been warned.
  • H264 will keep up quite happily up to 720P49, or [email protected]

That said, most people should find no problem with these new features, so a big thank you must go to Dave Stevenson and Naush Patuck at Broadcom for finding the time to implement them! Also, thanks to Omnivision for their continued support.