Page 1 of 1

new 2x2 bin mode and RAW output

Posted: Tue Mar 18, 2014 9:44 pm
by jbeale
Does 'raspistill' always use the 2x2 binning mode if you specify 1296 x 972 pixel resolution? I am asking because after the update, the output of

Code: Select all

raspistill -w 1296 -h 972 --raw -o test.jpg
seems to have 6.4 MB of raw data attached after the JPEG image portion of the file. That is the same size as the raw data has always been with the full 5 Mpixel sensor. If the data is binned 2x2 at the sensor before transfer, I would expect the raw file to be 1/4 the size.

Also, I was able to decode the raw file to Adobe DNG using the raspiraw tool and it is in fact 2592 x 1944 pixels.

I was hoping to use the raw output to compare the effect of 2x2 binning on low-light noise performance, but I'm not sure how to get 2x2 bin mode from Raspistill. Does the --raw option force the sensor to full-resolution regardless of requested JPEG size, which is then done by a GPU downsample ?

Re: new 2x2 bin mode and RAW output

Posted: Wed Mar 19, 2014 2:28 am
by fastmapper
My understanding is that the binning function simply adds nearby pixels of like color together and does not change the number of pixels produced.

It is subsampling (decimation) that reduces the number of pixels.

I believe that binning and subsampling can be selected independently. Of course, that means they could be selected together.

Even though it can be done in raw mode, adding some pixels together and throwing some away results in an image that isn't really raw anymore. However, it is an effective way to approximate a reduced resolution image capture.

Re: new 2x2 bin mode and RAW output

Posted: Wed Mar 19, 2014 3:57 am
by jbeale
My understanding is that the reason for using the 2x2 binning mode was that there was not enough bandwidth in the CSI2 serial interface to read out all 5 Mpixels for normal 30 fps video in full-sensor mode. So they cut down the number of pixels transferred by a factor of 4 by using on-chip 2x2 binning, whereby four sensor pixels are combined into one result pixel on the chip, before being transferred over the serial interface to the GPU.

Of course, the GPU could then copy over or upsample the pixels to recreate the full 5Mpixel bitmap on its side by interpolation before further processing, and call that the "raw" image, I suppose that is possible. I was assuming it would be more like the Canon DSLR cameras I have used, which provide the option of several different RAW sizes from the same sensor.

Re: new 2x2 bin mode and RAW output

Posted: Wed Mar 19, 2014 4:40 am
by fastmapper
I don't know what the maximum operating speed of the CSI-2 interface hardware is, but I expect that the pixel output rate of the image sensor is limited to converting about 96 million pixels (from analog to digital) per second. This means that up to 15 five-megapixel frames can be produced in a second. With subsampling, fewer pixels are processed by the image sensor enabling it to operate at a higher frame rate.

It is true that upsampling may be accomplished using digital zoom, but the result is not a faithful representation of the original image.

What an image sensor collects as a raw image and what is stored in a raw image format are different things. The Raspberry Pi image sensor only collects five megapixel Bayer images. There are different ways to manipulate the resulting pixels, but the resulting images will generally have characteristics that are distinct from the raw image sensing hardware.

Re: new 2x2 bin mode and RAW output

Posted: Wed Mar 19, 2014 8:28 am
by ethanol100
In my opinion raspistill only uses the binning mode for preview and not for capturing stills.
In still mode you will get full resolution images and only scale them down to the requested resolution, because it will give generally better image quality(beside the possible noise reduction for low light environments). So the raw data in raspistill should include full sensor data.

You would then need to get the raw data from the video port, but I do not know if this is possible.

Re: new 2x2 bin mode and RAW output

Posted: Wed Mar 19, 2014 9:02 am
by 6by9
When a stills capture i requested, it always uses the max 2592x1944 mode to maximise the image quality. The preview mode running up to the point that you request the capture will be using the new binned mode as frame rate is more important there.

With the current setups we have, I don't think there is actually a way that you will get anything other than the 5MPix raw image appended to the JPEG. If there is sufficient interest, then it may be possible to add a parameter that would force the sensor mode. It'd be on a "user beware" basis as it would be very easy to do some very weird things.
Let me know if there if that would be of real interest.

Re: new 2x2 bin mode and RAW output

Posted: Wed Mar 19, 2014 9:20 am
by jamesh
ethanol100 wrote:In my opinion raspistill only uses the binning mode for preview and not for capturing stills.
In still mode you will get full resolution images and only scale them down to the requested resolution, because it will give generally better image quality(beside the possible noise reduction for low light environments). So the raw data in raspistill should include full sensor data.

You would then need to get the raw data from the video port, but I do not know if this is possible.
This is correct IIRC.

Each of the camera modes has a flag on it that tells the system whether it can be used for video or stills or both. The only stills mode is full resolution, so when taking a still that is what will be used.

Re: new 2x2 bin mode and RAW output

Posted: Wed Mar 19, 2014 9:26 am
by jamesh
fastmapper wrote:I don't know what the maximum operating speed of the CSI-2 interface hardware is, but I expect that the pixel output rate of the image sensor is limited to converting about 96 million pixels (from analog to digital) per second. This means that up to 15 five-megapixel frames can be produced in a second. With subsampling, fewer pixels are processed by the image sensor enabling it to operate at a higher frame rate.

It is true that upsampling may be accomplished using digital zoom, but the result is not a faithful representation of the original image.

What an image sensor collects as a raw image and what is stored in a raw image format are different things. The Raspberry Pi image sensor only collects five megapixel Bayer images. There are different ways to manipulate the resulting pixels, but the resulting images will generally have characteristics that are distinct from the raw image sensing hardware.
The SoC support 4 lane CSI-2 which has a huge bandwidth. BUT, the Raspi only has support for two lanes, and the Raspi camera is even slower than that, although it does use both lanes.

Re: new 2x2 bin mode and RAW output

Posted: Wed Mar 19, 2014 2:19 pm
by fastmapper
jamesh wrote: The SoC support 4 lane CSI-2 which has a huge bandwidth. BUT, the Raspi only has support for two lanes, and the Raspi camera is even slower than that, although it does use both lanes.
If five megapixel images can be provided at 15 frames per second, then the image sensor is running as fast as it can and the CSI-2 interface is not limiting that speed.

Re: new 2x2 bin mode and RAW output

Posted: Wed Mar 19, 2014 3:04 pm
by jamesh
fastmapper wrote:
jamesh wrote: The SoC support 4 lane CSI-2 which has a huge bandwidth. BUT, the Raspi only has support for two lanes, and the Raspi camera is even slower than that, although it does use both lanes.
If five megapixel images can be provided at 15 frames per second, then the image sensor is running as fast as it can and the CSI-2 interface is not limiting that speed.
Generally the limitation isnt the CSI-2 but the ISP. That has a maximum number of pixels/s it can deal with (theoretically about 250Mp/s). But once software overheads are including that drops to about 180Mp/s. Then there are even more software overheads per frame, so the average drops even more.

Re: new 2x2 bin mode and RAW output

Posted: Wed Mar 19, 2014 4:07 pm
by jbeale
6by9 wrote:When a stills capture i requested, it always uses the max 2592x1944 mode to maximise the image quality. The preview mode running up to the point that you request the capture will be using the new binned mode as frame rate is more important there.

With the current setups we have, I don't think there is actually a way that you will get anything other than the 5MPix raw image appended to the JPEG. If there is sufficient interest, then it may be possible to add a parameter that would force the sensor mode. It'd be on a "user beware" basis as it would be very easy to do some very weird things. Let me know if there if that would be of real interest.
Well... I think a 2x2 binned raw still mode may be useful for low light, for example astrophotography where you need to add together multiple 1-second exposures to get anything useful. Raw mode avoids noise reduction processing. NR can be quite helpful in normal usage but is a limitation for this use case, as it throws away some genuine image information along with the noise. My hope was that raw 2x2 binned pixels would have a signal-to-noise advantage over raw unbinned pixels in low light, but I don't know if that is true. Moreover, I will admit this may be a "fringe" area of interest, I know time is limited and yours may be better spent on other issues of more general interest.

Re: new 2x2 bin mode and RAW output

Posted: Tue May 20, 2014 11:52 am
by TommyC
For what it's worth, I specifically purchased the pi camera NOIR for astrophotography. I just started but been reading every blog and forum available. Binning, long shutter speed, high ISO, gain are most important.

Re: new 2x2 bin mode and RAW output

Posted: Tue May 20, 2014 6:41 pm
by jbeale
FWIW, 2x2 binning on this device (and probably all Bayer-filter sensors) does not work the way you might at first think. It does NOT bin adjacent pixels together, actually it bins adjacent same-color pixels, to preserve color information in the subsampled output. The Bayer-pattern array has a repeating 2x2 color pattern, for example RGGB with the two green pixels on the diagonal of a 2x2 square of pixels. So to combine 4 red pixels, the chip has to skip over the adjacent green pixels in each direction, likewise for the other colors. So actual resolution after binning is lower than just 1/2 that of the native sensor.

I do appreciate that the 2x2 bin mode works, after much software effort from the R-Pi team. However if you want the best low-light photo, I would not bother with 2x2 raw output; I now suspect the best direction for any further progress is longer exposures at full frame, if possible.

Re: new 2x2 bin mode and RAW output

Posted: Thu Oct 01, 2015 9:24 am
by hkoenig
jbeale wrote:FWIW, 2x2 binning on this device (and probably all Bayer-filter sensors) does not work the way you might at first think. It does NOT bin adjacent pixels together, actually it bins adjacent same-color pixels, to preserve color information in the subsampled output. The Bayer-pattern array has a repeating 2x2 color pattern, for example RGGB with the two green pixels on the diagonal of a 2x2 square of pixels. So to combine 4 red pixels, the chip has to skip over the adjacent green pixels in each direction, likewise for the other colors. So actual resolution after binning is lower than just 1/2 that of the native sensor.
Hey, that was helpful and explained what I just saw in some experiments. When its dark I'll check if improved noise is worth the lower resulution.

Thanks for having explained.

Herbert