User avatar
jbeale
Posts: 3516
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

new 2x2 bin mode and RAW output

Tue Mar 18, 2014 9:44 pm

Does 'raspistill' always use the 2x2 binning mode if you specify 1296 x 972 pixel resolution? I am asking because after the update, the output of

Code: Select all

raspistill -w 1296 -h 972 --raw -o test.jpg
seems to have 6.4 MB of raw data attached after the JPEG image portion of the file. That is the same size as the raw data has always been with the full 5 Mpixel sensor. If the data is binned 2x2 at the sensor before transfer, I would expect the raw file to be 1/4 the size.

Also, I was able to decode the raw file to Adobe DNG using the raspiraw tool and it is in fact 2592 x 1944 pixels.

I was hoping to use the raw output to compare the effect of 2x2 binning on low-light noise performance, but I'm not sure how to get 2x2 bin mode from Raspistill. Does the --raw option force the sensor to full-resolution regardless of requested JPEG size, which is then done by a GPU downsample ?

fastmapper
Posts: 27
Joined: Wed Mar 27, 2013 3:54 am

Re: new 2x2 bin mode and RAW output

Wed Mar 19, 2014 2:28 am

My understanding is that the binning function simply adds nearby pixels of like color together and does not change the number of pixels produced.

It is subsampling (decimation) that reduces the number of pixels.

I believe that binning and subsampling can be selected independently. Of course, that means they could be selected together.

Even though it can be done in raw mode, adding some pixels together and throwing some away results in an image that isn't really raw anymore. However, it is an effective way to approximate a reduced resolution image capture.

User avatar
jbeale
Posts: 3516
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: new 2x2 bin mode and RAW output

Wed Mar 19, 2014 3:57 am

My understanding is that the reason for using the 2x2 binning mode was that there was not enough bandwidth in the CSI2 serial interface to read out all 5 Mpixels for normal 30 fps video in full-sensor mode. So they cut down the number of pixels transferred by a factor of 4 by using on-chip 2x2 binning, whereby four sensor pixels are combined into one result pixel on the chip, before being transferred over the serial interface to the GPU.

Of course, the GPU could then copy over or upsample the pixels to recreate the full 5Mpixel bitmap on its side by interpolation before further processing, and call that the "raw" image, I suppose that is possible. I was assuming it would be more like the Canon DSLR cameras I have used, which provide the option of several different RAW sizes from the same sensor.

fastmapper
Posts: 27
Joined: Wed Mar 27, 2013 3:54 am

Re: new 2x2 bin mode and RAW output

Wed Mar 19, 2014 4:40 am

I don't know what the maximum operating speed of the CSI-2 interface hardware is, but I expect that the pixel output rate of the image sensor is limited to converting about 96 million pixels (from analog to digital) per second. This means that up to 15 five-megapixel frames can be produced in a second. With subsampling, fewer pixels are processed by the image sensor enabling it to operate at a higher frame rate.

It is true that upsampling may be accomplished using digital zoom, but the result is not a faithful representation of the original image.

What an image sensor collects as a raw image and what is stored in a raw image format are different things. The Raspberry Pi image sensor only collects five megapixel Bayer images. There are different ways to manipulate the resulting pixels, but the resulting images will generally have characteristics that are distinct from the raw image sensing hardware.

ethanol100
Posts: 587
Joined: Wed Oct 02, 2013 12:28 pm

Re: new 2x2 bin mode and RAW output

Wed Mar 19, 2014 8:28 am

In my opinion raspistill only uses the binning mode for preview and not for capturing stills.
In still mode you will get full resolution images and only scale them down to the requested resolution, because it will give generally better image quality(beside the possible noise reduction for low light environments). So the raw data in raspistill should include full sensor data.

You would then need to get the raw data from the video port, but I do not know if this is possible.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7527
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: new 2x2 bin mode and RAW output

Wed Mar 19, 2014 9:02 am

When a stills capture i requested, it always uses the max 2592x1944 mode to maximise the image quality. The preview mode running up to the point that you request the capture will be using the new binned mode as frame rate is more important there.

With the current setups we have, I don't think there is actually a way that you will get anything other than the 5MPix raw image appended to the JPEG. If there is sufficient interest, then it may be possible to add a parameter that would force the sensor mode. It'd be on a "user beware" basis as it would be very easy to do some very weird things.
Let me know if there if that would be of real interest.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24129
Joined: Sat Jul 30, 2011 7:41 pm

Re: new 2x2 bin mode and RAW output

Wed Mar 19, 2014 9:20 am

ethanol100 wrote:In my opinion raspistill only uses the binning mode for preview and not for capturing stills.
In still mode you will get full resolution images and only scale them down to the requested resolution, because it will give generally better image quality(beside the possible noise reduction for low light environments). So the raw data in raspistill should include full sensor data.

You would then need to get the raw data from the video port, but I do not know if this is possible.
This is correct IIRC.

Each of the camera modes has a flag on it that tells the system whether it can be used for video or stills or both. The only stills mode is full resolution, so when taking a still that is what will be used.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24129
Joined: Sat Jul 30, 2011 7:41 pm

Re: new 2x2 bin mode and RAW output

Wed Mar 19, 2014 9:26 am

fastmapper wrote:I don't know what the maximum operating speed of the CSI-2 interface hardware is, but I expect that the pixel output rate of the image sensor is limited to converting about 96 million pixels (from analog to digital) per second. This means that up to 15 five-megapixel frames can be produced in a second. With subsampling, fewer pixels are processed by the image sensor enabling it to operate at a higher frame rate.

It is true that upsampling may be accomplished using digital zoom, but the result is not a faithful representation of the original image.

What an image sensor collects as a raw image and what is stored in a raw image format are different things. The Raspberry Pi image sensor only collects five megapixel Bayer images. There are different ways to manipulate the resulting pixels, but the resulting images will generally have characteristics that are distinct from the raw image sensing hardware.
The SoC support 4 lane CSI-2 which has a huge bandwidth. BUT, the Raspi only has support for two lanes, and the Raspi camera is even slower than that, although it does use both lanes.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

fastmapper
Posts: 27
Joined: Wed Mar 27, 2013 3:54 am

Re: new 2x2 bin mode and RAW output

Wed Mar 19, 2014 2:19 pm

jamesh wrote: The SoC support 4 lane CSI-2 which has a huge bandwidth. BUT, the Raspi only has support for two lanes, and the Raspi camera is even slower than that, although it does use both lanes.
If five megapixel images can be provided at 15 frames per second, then the image sensor is running as fast as it can and the CSI-2 interface is not limiting that speed.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24129
Joined: Sat Jul 30, 2011 7:41 pm

Re: new 2x2 bin mode and RAW output

Wed Mar 19, 2014 3:04 pm

fastmapper wrote:
jamesh wrote: The SoC support 4 lane CSI-2 which has a huge bandwidth. BUT, the Raspi only has support for two lanes, and the Raspi camera is even slower than that, although it does use both lanes.
If five megapixel images can be provided at 15 frames per second, then the image sensor is running as fast as it can and the CSI-2 interface is not limiting that speed.
Generally the limitation isnt the CSI-2 but the ISP. That has a maximum number of pixels/s it can deal with (theoretically about 250Mp/s). But once software overheads are including that drops to about 180Mp/s. Then there are even more software overheads per frame, so the average drops even more.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

User avatar
jbeale
Posts: 3516
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: new 2x2 bin mode and RAW output

Wed Mar 19, 2014 4:07 pm

6by9 wrote:When a stills capture i requested, it always uses the max 2592x1944 mode to maximise the image quality. The preview mode running up to the point that you request the capture will be using the new binned mode as frame rate is more important there.

With the current setups we have, I don't think there is actually a way that you will get anything other than the 5MPix raw image appended to the JPEG. If there is sufficient interest, then it may be possible to add a parameter that would force the sensor mode. It'd be on a "user beware" basis as it would be very easy to do some very weird things. Let me know if there if that would be of real interest.
Well... I think a 2x2 binned raw still mode may be useful for low light, for example astrophotography where you need to add together multiple 1-second exposures to get anything useful. Raw mode avoids noise reduction processing. NR can be quite helpful in normal usage but is a limitation for this use case, as it throws away some genuine image information along with the noise. My hope was that raw 2x2 binned pixels would have a signal-to-noise advantage over raw unbinned pixels in low light, but I don't know if that is true. Moreover, I will admit this may be a "fringe" area of interest, I know time is limited and yours may be better spent on other issues of more general interest.

TommyC
Posts: 2
Joined: Tue May 20, 2014 11:42 am

Re: new 2x2 bin mode and RAW output

Tue May 20, 2014 11:52 am

For what it's worth, I specifically purchased the pi camera NOIR for astrophotography. I just started but been reading every blog and forum available. Binning, long shutter speed, high ISO, gain are most important.

User avatar
jbeale
Posts: 3516
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: new 2x2 bin mode and RAW output

Tue May 20, 2014 6:41 pm

FWIW, 2x2 binning on this device (and probably all Bayer-filter sensors) does not work the way you might at first think. It does NOT bin adjacent pixels together, actually it bins adjacent same-color pixels, to preserve color information in the subsampled output. The Bayer-pattern array has a repeating 2x2 color pattern, for example RGGB with the two green pixels on the diagonal of a 2x2 square of pixels. So to combine 4 red pixels, the chip has to skip over the adjacent green pixels in each direction, likewise for the other colors. So actual resolution after binning is lower than just 1/2 that of the native sensor.

I do appreciate that the 2x2 bin mode works, after much software effort from the R-Pi team. However if you want the best low-light photo, I would not bother with 2x2 raw output; I now suspect the best direction for any further progress is longer exposures at full frame, if possible.

hkoenig
Posts: 14
Joined: Mon Aug 24, 2015 7:36 am
Location: Darmstadt Germany
Contact: Website

Re: new 2x2 bin mode and RAW output

Thu Oct 01, 2015 9:24 am

jbeale wrote:FWIW, 2x2 binning on this device (and probably all Bayer-filter sensors) does not work the way you might at first think. It does NOT bin adjacent pixels together, actually it bins adjacent same-color pixels, to preserve color information in the subsampled output. The Bayer-pattern array has a repeating 2x2 color pattern, for example RGGB with the two green pixels on the diagonal of a 2x2 square of pixels. So to combine 4 red pixels, the chip has to skip over the adjacent green pixels in each direction, likewise for the other colors. So actual resolution after binning is lower than just 1/2 that of the native sensor.
Hey, that was helpful and explained what I just saw in some experiments. When its dark I'll check if improved noise is worth the lower resulution.

Thanks for having explained.

Herbert

Return to “Camera board”