Camera module! (And a picture of JamesH)


431 posts   Page 3 of 18   1, 2, 3, 4, 5, 6 ... 18
by jamesh » Sun May 20, 2012 4:39 pm
A few comments to answer some of the Q's above.

To get a sensor manufacturer to even take your calls for a custom device you would need to sell in excess of 500k modules, more than likely over a million, otherwise the dev costs are prohibitive to the supplier. We cannot expect to sell that many. Even at those quantities, you would need to sell at quite a high price. If we did have a custom device there is about 2 man years of work to tune the camera - work that would need to be done at Broadcom - that's where the experts are. So many reasons why this isn't going to happen, and why using an existing sensor is the only way to go.

Technical stuff.
The way these sensors work is you set a load of registers in the sensor to specify pixel clocks, line lengths, blanking etc. The combination of these settings give the exposure time, frame rates etc. In addition there is a analogue gain control (usually one for best picture quality, equivilent to betweem 50 and 100 iso). Depending on the permitted values for these registers to get the frame out size, the FPS, the exposure, and usually, you don't get much more than 2s as a maximum exposure. Note that these sensors are almost always rolling shutter (some do have shutters but they are expensive, and they suffer most of the same limitations), so there are upper limits on what works.

@Hippy. The addition of DSI and CSI connectors cost very little - we still kept to the $25 price point. The Foundation did not sponsor the development of this camera board, so this add on has nothing to do with the educational aspect from their point of view.
Unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11790
Joined: Sat Jul 30, 2011 7:41 pm
by carldani » Sun May 20, 2012 10:50 pm
I had hoped to use one of the OV5650 (5 MP) CSI-2 camera modules you can find extremely cheap on ebay (5 EUR). They are usually sold as "iphone 4 rear camera". That hope probably won't be fulfilled.
Posts: 15
Joined: Wed Jan 11, 2012 12:08 am
by Paul Jurczak » Sun May 20, 2012 11:06 pm
carldani wrote:I had hoped to use one of the OV5650 (5 MP) CSI-2 camera modules you can find extremely cheap on ebay (5 EUR). They are usually sold as "iphone 4 rear camera". That hope probably won't be fulfilled.


I second this request: $4 on eBay with free shipping - can't beat that! It makes sense to support an existing high volume part and iPhone camera seems to be a good choice. Maybe part of the job of fine tuning GPU pipeline for this camera could be offloaded on user community, if appropriate API is provided. Even OpenMAX could be a starting point.
Posts: 30
Joined: Wed Feb 22, 2012 11:46 pm
by reggie » Sun May 20, 2012 11:53 pm
jamesh wrote:A few comments to answer some of the Q's above.

To get a sensor manufacturer to even take your calls for a custom device you would need to sell in excess of 500k modules, more than likely over a million, otherwise the dev costs are prohibitive to the supplier. We cannot expect to sell that many. Even at those quantities, you would need to sell at quite a high price. If we did have a custom device there is about 2 man years of work to tune the camera - work that would need to be done at Broadcom - that's where the experts are. So many reasons why this isn't going to happen, and why using an existing sensor is the only way to go.

Technical stuff.
The way these sensors work is you set a load of registers in the sensor to specify pixel clocks, line lengths, blanking etc. The combination of these settings give the exposure time, frame rates etc. In addition there is a analogue gain control (usually one for best picture quality, equivilent to betweem 50 and 100 iso). Depending on the permitted values for these registers to get the frame out size, the FPS, the exposure, and usually, you don't get much more than 2s as a maximum exposure. Note that these sensors are almost always rolling shutter (some do have shutters but they are expensive, and they suffer most of the same limitations), so there are upper limits on what works.

@Hippy. The addition of DSI and CSI connectors cost very little - we still kept to the $25 price point. The Foundation did not sponsor the development of this camera board, so this add on has nothing to do with the educational aspect from their point of view.
Hi James, thanks for the info, I do appreciate that what I want and what you have are probably going to be far apart. At the moment I am playing with a philips webcam (eol unfortunately, glad I've got 3 :D) which does actually give access to the shutter control pin and the frame readout pins, of course it's all done via hardware mods but the lines are there for the taking. Whilst this camera was probably never intended to take long exposures, I can get 2 minutes out of it at least, although amp glow is a problem at that point but this can also be mitigated with another hardware mod (judicious placement of a zener diode). The software for using it deals with triggering the exposure/frame readout. I don't suppose you know if the csi camera has these connections anywhere that might be accessible? I appreciate that it's beyond what you're doing :)
Posts: 151
Joined: Fri Aug 26, 2011 11:51 am
by AndrewS » Mon May 21, 2012 2:39 am
hippy wrote:I can see it may have uses as part of an application platform but it seems largely redundant and unnecessary extra cost in pursuing the goal of teaching computing to kids.

In including DSI and CSI the R-Pi looked to be hardware more designed towards a tablet / phone wannabe than for education. Though, as the ability is there on the SoC, adding the connectors has not pushed the price up, I'm not objecting to CSI being there.

Having LCD display capability I can more understand but it has always confounded me that not having something was usually answered by pointing to "it's primary purpose is teaching kids to program", and yet things not needed for that have been included. That and promoting the R-Pi as an application platform has always left me feeling that the educational aspirations were actually just a means to an end. Not that I really care because application platform is what I more care about !

Longer-term, I wonder if the camera could be interfaced with Scratch in some suitably high-level way?
Teach kids to make AugmentedReality / EyeToy type games?
Edit: Ah! I've never used Scratch, but just managed to find http://info.scratch.mit.edu/prototype#camerablocks
User avatar
Posts: 3626
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
by RobH » Mon May 21, 2012 11:52 am
Bolt on a Polaroid PoGo and you've got...
Attachments
PIcam.jpg
PIcam.jpg (49.28 KiB) Viewed 2978 times
Posts: 19
Joined: Thu Jul 28, 2011 7:40 pm
by verasen » Mon May 21, 2012 1:25 pm
Guys, can we please have the ability to set the crop rectangle for the raw image? Currently this is only available on some really expensive cameras and it would enable a whole range of applications since we could get any 640x480 area from the 5MP (or higher) at 30 fps.
Posts: 7
Joined: Mon May 21, 2012 1:22 pm
by finnw » Mon May 21, 2012 1:40 pm
One feature I would like to see is the ability to capture an image (in RGB format) to a texture buffer on the GPU.
Posts: 24
Joined: Wed May 16, 2012 7:05 pm
by Danbury Shakes » Mon May 21, 2012 5:53 pm
OK, I know Liz has said that the release model may be 5mp, but if a 14mp camera is being used to capture 1080hs can the GPU use the "extra" pixels as a digital zoom?

Admitedly when it was announced that there would be a high end and low end camera I expected the low end to be VGA.
Posts: 15
Joined: Thu Apr 12, 2012 8:41 am
by Gert van Loo » Mon May 21, 2012 6:25 pm
Danbury Shakes wrote:OK, I know Liz has said that the release model may be 5mp, but if a 14mp camera is being used to capture 1080hs can the GPU use the "extra" pixels as a digital zoom?

Admitedly when it was announced that there would be a high end and low end camera I expected the low end to be VGA.


I have looked at VGA cameras. The problem is that the adapter board has a basic price. (PCB, connector, cable, regulators, manufacturing etc.). It does not make sense to then use a camera which is too cheap.
User avatar
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2060
Joined: Tue Aug 02, 2011 7:27 am
by Danbury Shakes » Mon May 21, 2012 6:46 pm
Please don't get me wrong - I wasn't asking for a VGA version of the camera.

In fact I for one would be extremely happy with whatever camera you go for. Why? Because until the release of this titbit I was thinking "camera - nice but what would I use it for" - I now have a project - watching Tits.

I'm helping a friend redecorate her house and, whilst having a smoke break, I noticed a pair of courting Blue Tits in her garden. The ability to have the camera well away from the nest and use the additional resolution to zoom in and out could be beneficial.
Posts: 15
Joined: Thu Apr 12, 2012 8:41 am
by rew » Tue May 22, 2012 12:53 pm
Danbury, you had me confused a for a short while.
I wasn't aware of the birds called "blue tits". You enhanced my suspicion when your friend was a "her".... A dirty mind is a joy for ever. :-)
Check out our raspberry pi addons: http://www.bitwizard.nl/catalog/
User avatar
Posts: 396
Joined: Fri Aug 26, 2011 3:25 pm
by jamesh » Tue May 22, 2012 1:42 pm
Crop Rectangles/VGA/Digital zoom: Although the sensor might be 14MP, you can set the sensor up to lower resolution modes. The GPU will then fine scale that image to whatever size you want. In addition, I think we are able to specify which are on the resulting image the GPU scales - so it should be possible.

@iPhone rear camera requests. Please read the comments above of requiring a camera that is already tuned by Broadcom top see why using any arbitrary camera is not really possible..
Unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11790
Joined: Sat Jul 30, 2011 7:41 pm
by Danbury Shakes » Tue May 22, 2012 6:29 pm
rew wrote:Danbury, you had me confused a for a short while.
I wasn't aware of the birds called "blue tits". You enhanced my suspicion when your friend was a "her".... A dirty mind is a joy for ever. :-)


Ah yes, my days as a young ornathologist were full of sniggering.

Tits, Boobies and Shags. Of course as I live in the middle of London there wouldn't be any opportunity to video the latter two species. Being a costal species it's hard to find a Shag in London.

;)
Posts: 15
Joined: Thu Apr 12, 2012 8:41 am
by Danbury Shakes » Tue May 22, 2012 6:41 pm
jamesh wrote:Crop Rectangles/VGA/Digital zoom: Although the sensor might be 14MP, you can set the sensor up to lower resolution modes. The GPU will then fine scale that image to whatever size you want. In addition, I think we are able to specify which are on the resulting image the GPU scales - so it should be possibl


Ok so if I read this right the sensor could be set to say 1080 which would be the zoomed out mode. To zoom in the sensor would be set to higher resolution having the same field of view, but a rectangle coresponding to a 1080 frame would be "cut out" of the centre and used. This could be contined at higher resolutions (and zoom level) until the sensor is at 14mp.
Posts: 15
Joined: Thu Apr 12, 2012 8:41 am
by poing » Tue May 22, 2012 7:54 pm
Danbury Shakes wrote:Ok so if I read this right the sensor could be set to say 1080 which would be the zoomed out mode. To zoom in the sensor would be set to higher resolution having the same field of view, but a rectangle coresponding to a 1080 frame would be "cut out" of the centre and used. This could be contined at higher resolutions (and zoom level) until the sensor is at 14mp.


Yes, although depending on sensor size and aperture you might find diffraction (blurring due to the wavelength of the light being bend by the edge of the aperture opening) will give you much less resolution zoomed in compared to using a DSLR with a good lens and a wide aperture. Looking at the size of the module my guess is 100% will show a nice blur :|
Posts: 1097
Joined: Thu Mar 08, 2012 3:32 pm
by carldani » Tue May 22, 2012 9:51 pm
jamesh wrote:@iPhone rear camera requests. Please read the comments above of requiring a camera that is already tuned by Broadcom top see why using any arbitrary camera is not really possible..


We've yet to see any list of camera modules which are tuned by Broadcom, so all we can do is speculate about which camera may be compatible and which camera would be a good choice from a quality POV.

jamesh wrote:How it works : The camera modules outputs raw data direct to the GPU. There it goes through about 17 stages of processing (Debayering, lens correction, black level, gain control, AWB, scaling, cropping, distortion etc). [...] The GPU can then also HW encode to JPEG a single capture at full res, or send a lower rez processed stream to the H264 encoder.


It has been said elsewhere that the full GPU programming docs are only available under NDA, but can you at least tell us what the 17 stages of processing are, and where in that pipeline we could possibly extract data to process it on the ARM?
Posts: 15
Joined: Wed Jan 11, 2012 12:08 am
by jamesh » Tue May 22, 2012 9:58 pm
carldani wrote:
jamesh wrote:@iPhone rear camera requests. Please read the comments above of requiring a camera that is already tuned by Broadcom top see why using any arbitrary camera is not really possible..


We've yet to see any list of camera modules which are tuned by Broadcom, so all we can do is speculate about which camera may be compatible and which camera would be a good choice from a quality POV.

jamesh wrote:How it works : The camera modules outputs raw data direct to the GPU. There it goes through about 17 stages of processing (Debayering, lens correction, black level, gain control, AWB, scaling, cropping, distortion etc). [...] The GPU can then also HW encode to JPEG a single capture at full res, or send a lower rez processed stream to the H264 encoder.


It has been said elsewhere that the full GPU programming docs are only available under NDA, but can you at least tell us what the 17 stages of processing are, and where in that pipeline we could possibly extract data to process it on the ARM?


The iPhone camera are not ones we have tuned. I'm not sure I'm allowed to give a list either, unfortunately.

To extract data at various points of the ISP and send them to the Arm for processing isn't really feasible. There's only a couple of points where it's even possible, and there is not the infrastructure to send the stuff back to the Arm, then back again to the ISP - all that sort of processing is always done on the GPU for performance reasons.
The ISP is pretty good, after all, its the one used in the two best (on image quality) camera phones in the world. The vast majority of people will be able to use it out of the box, just using the provided settings.
Unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11790
Joined: Sat Jul 30, 2011 7:41 pm
by jamesh » Tue May 22, 2012 10:01 pm
poing wrote:
Danbury Shakes wrote:Ok so if I read this right the sensor could be set to say 1080 which would be the zoomed out mode. To zoom in the sensor would be set to higher resolution having the same field of view, but a rectangle coresponding to a 1080 frame would be "cut out" of the centre and used. This could be contined at higher resolutions (and zoom level) until the sensor is at 14mp.


Yes, although depending on sensor size and aperture you might find diffraction (blurring due to the wavelength of the light being bend by the edge of the aperture opening) will give you much less resolution zoomed in compared to using a DSLR with a good lens and a wide aperture. Looking at the size of the module my guess is 100% will show a nice blur :|


Not sure I'm, with you - these sensors don't have movable apertures, so when using a crop of the whole sensors output, you are nowhere near the edges of anything. In fact you are further away from the edges of the sensor.
Unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11790
Joined: Sat Jul 30, 2011 7:41 pm
by poing » Tue May 22, 2012 10:20 pm
jamesh wrote:Not sure I'm, with you - these sensors don't have movable apertures, so when using a crop of the whole sensors output, you are nowhere near the edges of anything. In fact you are further away from the edges of the sensor.


It has to do with the edges of the aperture that have impact on the whole image area. Look at this page: http://www.cambridgeincolour.com/tutorials/diffraction-photography.htm, read down to 'Visual example', click on 'Canon PowerShot G6' to get the pixel grid of the smallest sensor listed and then move the mouse over the listed apertures to see the size of the Airy disc at different apertures.

Then think about the size of the sensor in the Pi camera and the MP count to calculate the pixel size. Next look at the fixed aperture of the lens. I'm not very good at these calculations, but if you have a camera with 1 micron pixels and a f/5.6 aperture my guess is you will only record a few MP of detail in a 14MP sized file.

There's a diffraction limit and if you cross that by adding more pixels to a sensor all you actually record is the same blur with more pixels. Meaning you won't see more detail but you get bigger file sizes. I'm not sure this fact will or should have an impact on your choice of camera, just sayin' ;)

Edit: this diffraction limit is not a hard edge but a slope that you can see here below. This graph is from a 35mm full frame camera with a low pixel density; when the pixel density gets higher the resolution will fall down earlier at larger apertures (smaller number):
Image
Posts: 1097
Joined: Thu Mar 08, 2012 3:32 pm
by poing » Tue May 22, 2012 11:19 pm
From the same page I linked to above I calculated a 14MP camera with a 1/3" sensor. At about f/2.8 it starts to lose resolution. My guess is @ f/5.6 you will see a definite impact and the actual resolution will be much lower than 14MP:

Image
Posts: 1097
Joined: Thu Mar 08, 2012 3:32 pm
by CargoCult » Tue May 22, 2012 11:45 pm
I'd be very happy with just a couple of megapixels as well - talk of 14 megapixels with such a tiny sensor annoys my inner camera nerd as well. (I mean, I need a decent lens and careful shooting to get worthwhile pictures from my 18 megapixel EOS 7D, and that resolution is still overkill for most purposes!)

What would I do with one? I've currently got an Arduino and a tiny serial JPEG camera taking a picture from my front window once every minute. (Nice view of Seattle skyline, complete with ominous construction site preparing to remove said view.) It's been running continuously since early December last year, and the resulting timelapses are almost hypnotic, despite the abysmal picture quality.

Something approaching decent detail and colour would be splendid!
Posts: 25
Joined: Wed Nov 02, 2011 7:06 am
by jamesh » Wed May 23, 2012 2:24 pm
For all those megapixel haters (ie more megapixels in such a small sensor is a bad idea types), I really would suggest looking at some of the pictures taken with the Nokia 808 phone (soon to be released, but there should be pictures out there). This is a 41MP sensor in a phone, and the pictures are stunning.
Also check out the Nokia N8 pictures, which are also very good 0 that s 12MP sensor.

More megapixels DOES result in better quality because with more pixels you have more data and with more data you can apply better post processing to get a good image. You do need to have better post processing and a decent quality sensor which is why some high MP camera are sucky. Dont let those bad ones put you off high MP sensors.
Unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11790
Joined: Sat Jul 30, 2011 7:41 pm
by poing » Wed May 23, 2012 3:11 pm
jamesh wrote:For all those megapixel haters (ie more megapixels in such a small sensor is a bad idea types),..


Haha, this is absolutely the first time someone accuses me of being a 'megapixel hater', as I usually think more MP is better. But this is with DLSRs where I can choose my own aperture, while more MP will never give a worse result and I have enough horse power in the PC to sustain a decent workflow.

jamesh wrote:More megapixels DOES result in better quality because with more pixels you have more data and with more data you can apply better post processing to get a good image.


This is false in the case where your system (lens+sensor) is severely diffraction limited. If you get better results with higher MP sensors while severely diffraction limited it is because the sensor tech is newer, not the higher MP count.

Regarding the camera for the Pi, what do you want to achieve? If it's just a handy thing to make family snaps I'd recommend one of the entry cameras from Nikon or Canon with a load of MP (the higher the better). But if you want folks to do all kinds of fast computer manipulations with the images processed by a tiny CPU I'd say you'd have to carefully balance resolution vs image size. In this case a phone camera with 41MP would be a quite stupid choice IMO, because although the output on screen does look good (new tech), you have way to many pixels compared to the amount of detail to handle quickly by a tiny CPU.

Two pointers:
- Get a resolution chart and learn how to use it: http://www.graphics.cornell.edu/~westin ... chart.html
- Contact the expert I mention in the PB I'm writing to you.
Posts: 1097
Joined: Thu Mar 08, 2012 3:32 pm
by jamesh » Wed May 23, 2012 4:47 pm
poing wrote:
jamesh wrote:For all those megapixel haters (ie more megapixels in such a small sensor is a bad idea types),..


Haha, this is absolutely the first time someone accuses me of being a 'megapixel hater', as I usually think more MP is better. But this is with DLSRs where I can choose my own aperture, while more MP will never give a worse result and I have enough horse power in the PC to sustain a decent workflow.

jamesh wrote:More megapixels DOES result in better quality because with more pixels you have more data and with more data you can apply better post processing to get a good image.


This is false in the case where your system (lens+sensor) is severely diffraction limited. If you get better results with higher MP sensors while severely diffraction limited it is because the sensor tech is newer, not the higher MP count.

Regarding the camera for the Pi, what do you want to achieve? If it's just a handy thing to make family snaps I'd recommend one of the entry cameras from Nikon or Canon with a load of MP (the higher the better). But if you want folks to do all kinds of fast computer manipulations with the images processed by a tiny CPU I'd say you'd have to carefully balance resolution vs image size. In this case a phone camera with 41MP would be a quite stupid choice IMO, because although the output on screen does look good (new tech), you have way to many pixels compared to the amount of detail to handle quickly by a tiny CPU.

Two pointers:
- Get a resolution chart and learn how to use it: http://www.graphics.cornell.edu/~westin ... chart.html
- Contact the expert I mention in the PB I'm writing to you.


I think we have quite enough camera experts thanks. The Broadcom camera/ISP team is one of the best in the world, having produced the two top camera phones, employing real experts in the field (most of the team have PhD's for example). And I work on camera's all the time as part of my job, although I am not at the level of expertise as most of the other people in the team.

With regard to the balance of resolution, you seem to have missed posts above that say you can specify what final resolution you want out of the ISP - the system will then set up the camera in the mode closest to the requirements, then the GPU will scale to the exact size you want. So by the time the image gets to the Arm, it can be big or small, depending on what you ask for.
Unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11790
Joined: Sat Jul 30, 2011 7:41 pm