alexamy
Posts: 27
Joined: Thu Aug 08, 2013 1:00 pm

Raw Image & pixel-by-pixel RGB information

Thu Aug 08, 2013 1:12 pm

Hello all,

I would like to use my Raspberry Pi to take a photograph in the most raw format possible (so that no information is lost via compression), and then I would like to be able to examine information on a pixel-by-pixel basis, taking the RGB values for each pixel in turn.

Does anyone know how I could go about doing this please? I'm very much a beginner with all of this I'm afraid.

Thank you very much for any help you can give.

Alex Amy

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23879
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raw Image & pixel-by-pixel RGB information

Thu Aug 08, 2013 2:09 pm

You can specify --raw on the command line and you will get data duirect form the sensor - but this is in a raw bayer format, so not RGB. There's a thread somewhere on converting that to various other formats. Or you can save to png, bmp or gif, all of which are lossless (and you can get RGB out of them), but they are saved after the ISP processes the image, so might not be what you want. Or the latest version of raspistillYUV can also output a raw RGB image after the ISP has processed it.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

User avatar
jbeale
Posts: 3499
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Raw Image & pixel-by-pixel RGB information

Thu Aug 08, 2013 2:17 pm

Have a look at the raw format experiment page here: http://bealecorner.org/best/RPi/

I think you will find, on a raw pixel-by-pixel level, that the sensor has enough noise even in bright light that any single pixel doesn't tell you much. If you debayer and then downscale to 50%, then you get a more reasonable image for that kind of analysis.

alexamy
Posts: 27
Joined: Thu Aug 08, 2013 1:00 pm

Re: Raw Image & pixel-by-pixel RGB information

Thu Aug 08, 2013 3:09 pm

Thank you for your help!

Just to give you a better idea of what I'm aiming for, I'm hoping to take a photograph and then examine the intensity of the red, green and blue channels separately, in as high a resolution as possible.

It would be nice to be able to create images for each channel, extracting the information from the high quality lossless format, so that I can have an image which represents the amount of red light the sensor detected, the amount of green light detected and finally blue in 3 different images.

Thank you once again for your advice, I've already found it very helpful and I really appreciate it.

EDIT: Also, it's worth noting that I don't require a constant stream or high speed processing of such photographs

User avatar
jbeale
Posts: 3499
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Raw Image & pixel-by-pixel RGB information

Thu Aug 08, 2013 5:01 pm

For a start I recommend using the raspistillYUV output, separate the image into color planes, and see if that is adequate for your needs. Depending on exactly what you're looking for, that may actually be better than raw. Remember R,G,B as most people use them is not just one thing, depends on the light spectrum so there is some color adjustment (white balancing) involved to translate raw data into normal RGB values.
Last edited by jbeale on Thu Aug 08, 2013 7:58 pm, edited 2 times in total.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23879
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raw Image & pixel-by-pixel RGB information

Thu Aug 08, 2013 5:10 pm

jbeale wrote:For a start I recommend using the raspiYUV output, separate the image into color planes, and see if that is adequate for your needs. Depending on exactly what you're looking for, that may actually be better than raw. Remember R,G,B as most people use them is not just one thing, depends on the light spectrum so there is color balancing involved to translate raw data into normal RGB values.
The raspistillYUV app should now support a --rgb option, which outputs RGB 8:8:8 which might be a better bet if you really want RGB. (assuming its made it through the release process).

To OP, if you are unfamiliar with YUV, it's a luma channel (grayscale) plus a blue and red chroma channels. It's all on Wikipedia.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

KenT
Posts: 758
Joined: Tue Jan 24, 2012 9:30 am
Location: Hertfordshire, UK
Contact: Website

Re: Raw Image & pixel-by-pixel RGB information

Thu Aug 08, 2013 5:11 pm

Take a look at the Python Imaging Library

www.effbot.org
Pi Presents - A toolkit to produce multi-media interactive display applications for museums, visitor centres, and more
Download from http://pipresents.wordpress.com

Niall101
Posts: 16
Joined: Fri May 10, 2013 2:24 pm
Location: Clinterty Observatories, Aberdeen

Re: Raw Image & pixel-by-pixel RGB information

Thu Aug 08, 2013 5:19 pm

Hi,

You might also want to seriously consider true statistical analysis of the image data - as an astrophotographer, this is what I have to do, even with my iomager costing well over $2000, and with the imager running a a forced-cooled temperature of around -40C !

If you take MANY images (typically 30, or so as a minimum), then you simply 'avrage' these images on a pixel-by-pixel basis (and colour-plane-by-plane basis as well) then you will effectively increase what is known as the SNR (signal-to-noise-ratio) - greatly improving the picture quality.

In the wolrd of astroimaging, we also determine what proportion of the acquired image is purely due to 'therma noise', we acquire enough statistical data to evaluate this noise, and then we, mathemagically, 'remove' it from each and every 'raw' image before we start to average these together. (I could go on - and usually do with little or no provocation :lol: )

It is quite remarkable what can be achieved - nd this is one of the challenges I am looking forward to on my recently acquired Pi and PiCam.

But, I've got to get up to speed on Linux first !!!

Have fun.

Cheers,
Niall
Scatch-built BBC B in 82: blank PCB+scavenged plastics& PSU. Sinclair Calc & ZX81 kits in 74&82. PONG (AY-3-8500) in 77. 1st PC in 89: 386/25+387&10MbHDD. Have previewed pictures with my PiCam. I'll maybe save one next week (new to Linux)!

alexamy
Posts: 27
Joined: Thu Aug 08, 2013 1:00 pm

Re: Raw Image & pixel-by-pixel RGB information

Fri Aug 09, 2013 8:21 am

Ah brilliant, thanks everyone. I'll have a mess around with some of these ideas you've suggested and see what I can achieve. I suspect this will take me quite some time being the beginner that I am, but it should be a fun process.

Just to clarify further on what I'm trying to achieve, I guess it would be easier to explain the scientific side of what I want...

I'm sure most of you already have an understanding of the electromagnetic spectrum and the range of wavelengths that span across the visible part of the spectrum (approx 390nm to 770nm).

As far as I'm aware, a colour camera sensor is made up of an array of little photosensitive squares (pixels) which take in light from the Blue (~470nm), 2xGreen (~530nm) and Red (~660nm) areas of the visible spectrum seperately and combine them to create a colour image??

So what I'm trying to do is essentially take a photograph of an object or scene, and find out how much Blue light it's reflecting, how much Green light it's reflecting and how much Red light it's reflecting all using a single colour photo sensor as opposed to three separate monochrome sensors with filters. I think this would be a nifty little scientific tool.

Thanks for all your advice, certainly given me a lot to start with :D

EDIT: This is probably just me being stupid because I'm a beginner in every sense of the word, but I can't seem to get the 'raspistillYUV' command to work for me, it just says the command is not found??

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23879
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raw Image & pixel-by-pixel RGB information

Fri Aug 09, 2013 9:06 am

Worthwhile looking up how cmos and ccd sensors work especially with reference to bayer patterns. Different sensors have different dyes for the bayer pattern as well, which affects sensitivity to various spectrums.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

User avatar
jbeale
Posts: 3499
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Raw Image & pixel-by-pixel RGB information

Fri Aug 09, 2013 2:35 pm

You might find this image useful for reference. This is the R-Pi camera looking at a continuous-spectrum source (tungsten filament) through a spectroscope with overlaid wavelength reticle, so you can see where the sensitivity of the camera's R,G,B matches up to the wavelength of light. You may notice that the default JPEG shows somewhat different color than my RAW workflow gives... welcome to the world of colorimetry :-).
Image

thatneat
Posts: 4
Joined: Tue Dec 04, 2018 5:52 pm

Re: Raw Image & pixel-by-pixel RGB information

Tue Dec 04, 2018 10:07 pm

My team has been working on a similar problem (analyzing raw data from the pi camera)

We ended up forking the picamera library so that we could pull the RAW data out of the RAW + JPEG files and process that. If you're familiar with the Python numpy/scipy stack, this'll be the best way to go:

Simply take an image with

Code: Select all

raspiraw --raw -o your_output_file.jpeg
Then open that file with picamraw within python: https://github.com/OsmoSystems/picamraw/

Code: Select all

from picamraw import PiRawBayer, PiCameraVersion

raw_bayer = PiRawBayer(
    filepath='path/to/your_output_file.jpeg',
    camera_version=PiCameraVersion.V2,
)
raw_bayer.to_rgb()  # A 16-bit 3D numpy array of raw bayer data collapsed into RGB channels
Note that each physical pixel on the camera is actually red, green or blue, and there are twice as many green pixels as red pixels (same number of red as blue). It uses a Bayer filter: https://en.wikipedia.org/wiki/Bayer_filter.

Code: Select all

to_rgb()
does the absolute minimum required to create an RGB image array - from each logical group of 4 pixels (RGGB) it pulls the R and G value and averages the two G values.

If you're interested in using the pi camera for scientific applications, I also highly recommend checking out this paper by Pagnutti et al which will give you a basic idea of the imperfections of the raw data: https://www.spiedigitallibrary.org/jour ... full?SSO=1

dmatthews
Posts: 2
Joined: Thu Dec 06, 2018 3:18 am

Re: Raw Image & pixel-by-pixel RGB information

Thu Dec 06, 2018 4:07 am

For your application, you need the Bayer data, right? It sounds like the de-Bayering (or demosaicing) process won't interfere with the data you want, so it might be simpler to just stick with minimally-processed RGB. Depends what you want the data for, of course - as someone else said, colorimetry gets complicated!

If it's not rude to stuff my own question in here, I have an application that definitely does need the raw, Bayer data, hopefully without stuffing it into the EXIF in a JPEG like the Python picamera module does. I'm hoping to find a way that's faster than the 1-2 FPS I'm getting currently. Seems like it should be possible, but does anyone know a reason it wouldn't be?

User avatar
HermannSW
Posts: 1581
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Raw Image & pixel-by-pixel RGB information

Thu Dec 06, 2018 9:23 am

dmatthews wrote:
Thu Dec 06, 2018 4:07 am
..., I have an application that definitely does need the raw, Bayer data, hopefully without stuffing it into the EXIF in a JPEG like the Python picamera module does. I'm hoping to find a way that's faster than the 1-2 FPS I'm getting currently. Seems like it should be possible, but does anyone know a reason it wouldn't be?
Use raspiraw, that gives you all standard resolutions with the framerate the camera/raspivid allow for. And you can do high framerate videos as well for reduced 640xH resolutions, up to 750fps/1007fps with v1/v2 camera:
viewtopic.php?t=213482#p1320044

raspiraw stores the raw data frames into distinct files, and with -hd0 option the headers get into a specified file leaving just the raw frame data you are interested in in the frame files:
https://github.com/6by9/raspiraw
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://gitlab.freedesktop.org/HermannSW/gst-template
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7417
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw Image & pixel-by-pixel RGB information

Thu Dec 06, 2018 11:47 am

dmatthews wrote:
Thu Dec 06, 2018 4:07 am
For your application, you need the Bayer data, right? It sounds like the de-Bayering (or demosaicing) process won't interfere with the data you want, so it might be simpler to just stick with minimally-processed RGB. Depends what you want the data for, of course - as someone else said, colorimetry gets complicated!

If it's not rude to stuff my own question in here, I have an application that definitely does need the raw, Bayer data, hopefully without stuffing it into the EXIF in a JPEG like the Python picamera module does. I'm hoping to find a way that's faster than the 1-2 FPS I'm getting currently. Seems like it should be possible, but does anyone know a reason it wouldn't be?
The JPEG+RAW was developed as a hack to be able to get the raw out via Android where the data paths were limited. There isn't an easy path to retrieve the raw images during preview/video modes just because of the way everything is pipelined.
You can enable burst mode, disable EXIF and thumbnails, and make the JPEG small to improve the speed - I get about 3fps with

Code: Select all

raspistill -bm -t 10000 -tl 250 -w 320 -h 240 -th none -o file%d.jpg -a 8 -r
The files come out pretty large, therefore saving to a RAM disk will help.
The other quick change that we ought to make to the stock raspistill is to use more and larger buffers on the output of image_encode. The default is one 80kB buffer, so once it gets passed to the ARM to save to disk the GPU is stalled. Particularly with raw captures that makes around 130 round trips required.

If you want raw then raspiraw gives full control to you/your application, but also requires you to do all the work (no AE/AGC). If you have a moderate level of knowledge then it's of use. If not, then it's probably not the tool for you.
I'm afraid the level of assistance that I can offer on sensor register sets is very limited due to NDAs. The datasheets are out there and allow a moderate amount of tweaking, but it is a case of you needing to be prepared to experiment. HermannSW has been there already and done a fair amount of work so may be prepared to assist if you get stuck.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7417
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw Image & pixel-by-pixel RGB information

Thu Dec 06, 2018 11:57 am

4fps seems to work fairly reliably if using bigger and more buffers in raspistill and saving to /dev/shm. I suspect that is probably about as good as you're going to get through that route.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Qyetnot
Posts: 1
Joined: Mon Apr 08, 2019 8:35 am

Re: Raw Image & pixel-by-pixel RGB information

Mon Apr 08, 2019 8:38 am

I found out a free online tool with RAW format. You could check on https://www.manticonvert.com/svg-to-png.html or https://www.manticonvert.com/raw-to-png.html Hopefully, it can be helpful for you

Return to “Camera board”