Camera module! (And a picture of JamesH)


431 posts   Page 1 of 18   1, 2, 3, 4, 5 ... 18
by liz » Fri May 18, 2012 10:45 pm
The guys have got the prototype camera module working. There's a post about it on the front page, which includes some photos of everyone's favourite aggressive mod, JamesH.

http://www.raspberrypi.org/archives/1254

We are really excited about this - the module will be released for sale later in the year (hopefully before the end of summer).
--
Head of Comms, Raspberry Pi Foundation
User avatar
Raspberry Pi Foundation Employee & Forum Moderator
Raspberry Pi Foundation Employee & Forum Moderator
Posts: 4112
Joined: Thu Jul 28, 2011 7:22 pm
by mahjongg » Fri May 18, 2012 10:51 pm
wow! had not expected that already!
So this means GPU code for camera capture is available?
User avatar
Forum Moderator
Forum Moderator
Posts: 5738
Joined: Sun Mar 11, 2012 12:19 am
by teh_orph » Fri May 18, 2012 10:54 pm
What's the lens going to be on this? And what about the IR filter? Will one focal length be chosen for everyone or will it be unscrewable and you buy the one you want?
(btw I can never reply to the posts on the main page, even if I log in, use different browsers etc!)
User avatar
Posts: 345
Joined: Mon Jan 30, 2012 2:09 pm
Location: London
by daveg » Fri May 18, 2012 10:59 pm
Nice work guys, look forward to seeing more as it develops.
I wonder what resolution will end up being "affordable".
User avatar
Posts: 128
Joined: Thu Dec 01, 2011 9:36 am
by cheery » Fri May 18, 2012 11:56 pm
This ought to be useful in many, many applications.

I sort of hope you would provide both cheap and high quality modules. I think both will yield useful.
User avatar
Posts: 219
Joined: Wed Jan 25, 2012 9:39 pm
by jbeale » Sat May 19, 2012 12:10 am
My guess is most, if not all of the camera modules with this type of interface are designed for the cell phone market and hence they are intended for use "as is", with one fixed fairly wide-angle lens. I might be wrong but that's my guess.

It would be great if there was some standardization about the interface so many different modules could be interchanged. But I gather that hasn't happened for this type of cell phone module (?)
User avatar
Posts: 2084
Joined: Tue Nov 22, 2011 11:51 pm
by AndrewS » Sat May 19, 2012 12:25 am
Standardization?! It took the effort of the EU just to standardize the cellphone charger connection ;-)
User avatar
Posts: 3626
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
by liz » Sat May 19, 2012 2:47 am
Yeah - the problem here is that the only parts which are available in this sort of size are usually targeted at mobile phone and webcam vendors. That sort of hardware is minimally customisable; it's vanishingly unlikely that there's anything on the market for us that has some of the features (interchangeable lenses, removable filters) that some of you have been asking for here, in the comments on the original post and on Twitter.

We do hear you, and we'll do our best to find parts that do what you want. But price has to be (as always) the priority; and my strong sense at the moment is that currently, manufacturers don't believe there's a market for such a thing. So prove them wrong, and keep asking them for what you want - and if you're really motivated, there might be a hardware startup in it for one of you.
--
Head of Comms, Raspberry Pi Foundation
User avatar
Raspberry Pi Foundation Employee & Forum Moderator
Raspberry Pi Foundation Employee & Forum Moderator
Posts: 4112
Joined: Thu Jul 28, 2011 7:22 pm
by Paul Jurczak » Sat May 19, 2012 6:53 am
What is the software API to control this camera, V4L2?
Posts: 30
Joined: Wed Feb 22, 2012 11:46 pm
by AndrewS » Sat May 19, 2012 7:20 am
Paul Jurczak wrote:What is the software API to control this camera, V4L2?

I suspect at this very early stage that's totally undefined. Chances are it might be one of the OpenMAX layers again?
User avatar
Posts: 3626
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
by jamesh » Sat May 19, 2012 7:33 am
Focal Length : These sorts of modules hve a fixed lens, so they use something called EDOF to focus (Extended Depth of field). They are quire high quality, but not as good as those with moving lenses. But even those still have a single focal length. There are no cell phone modules that have optical zoom (The Nokia 808 gets round the problem by having a very high MP sensor (41MP) and cropping - that is also run with the VC4 ISP btw)
IR Filters : I don't think any cellphone modules have removable filters. That's not you say you would be unable to butcher one, but on your head be it.
Interface: Currently undefined. All the test pictures were done by code running on the GPU with no Linux interference.
Standard Hardware: afraid not. Modules are all different, requiring (usually) different sockets, and different drivers, and always a different tuning (the set of parameters that converts from Raw sensor data to what you see here)

How it works : The camera modules outputs raw data direct to the GPU. There it goes through about 17 stages of processing (Debayering, lens correction, black level, gain control, AWB, scaling, cropping, distortion etc). This is all done in real time at 30fps because it's all done in HW under control of the GPU software. The GPU can then also HW encode to JPEG a single capture at full res, or send a lower rez processed stream to the H264 encoder which can encode at 1080p30. Once all that is done, the results are/will be sent to Linux running on the Arm. That's the bit that still needs to be sorted out.
Soon to be employed engineer - Hurrah! Volunteer at the Raspberry Pi Foundation, helper at PiAcademy September 2014.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11956
Joined: Sat Jul 30, 2011 7:41 pm
by RaTTuS » Sat May 19, 2012 7:52 am
awesome
http://www.catb.org/esr/faqs/smart-questions.html <- ask smart Questions
"That's not right, the badgers have moved the goalposts."
1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX - Prosliver FTW
User avatar
Posts: 5368
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
by Montala » Sat May 19, 2012 8:30 am
Awesome indeed, and I must say that I am very impressed!
At first I was a bit surprised that the first 'accessory' to be produced by the Foundation was a camera module, but on thinking about it a bit more, I can see that this fits in quite well with its 'educational' objectives, and I am sure that its intended user base will use it for a lot more than just taking pictures of themselves... and each other.... I certainly hope so anyway.
Has any thought been given yet as to how it will be sold? Presumably not as just a bare board again... or will it?
Whatever is decided, I can see a great demand for such a product so, provided that the price is not too prohibitive, I would like to place an advance order for one NOW, please! ;)

P.S.: JamesH... "aggressive"..... surely not... whatever made you say that, Liz? :D
User avatar
Posts: 638
Joined: Mon Mar 05, 2012 11:54 pm
Location: Herefordshire (U.K.)
by jamesh » Sat May 19, 2012 8:40 am
Montala wrote:Awesome indeed, and I must say that I am very impressed!
At first I was a bit surprised that the first 'accessory' to be produced by the Foundation was a camera module, but on thinking about it a bit more, I can see that this fits in quite well with its 'educational' objectives, and I am sure that its intended user base will use it for a lot more than just taking pictures of themselves... and each other.... I certainly hope so anyway.
Has any thought been given yet as to how it will be sold? Presumably not as just a bare board again... or will it?
Whatever is decided, I can see a great demand for such a product so, provided that the price is not too prohibitive, I would like to place an advance order for one NOW, please! ;)

P.S.: JamesH... "aggressive"..... surely not... whatever made you say that, Liz? :D


Oi, watch it you, or there'll be trouble.

;)
Soon to be employed engineer - Hurrah! Volunteer at the Raspberry Pi Foundation, helper at PiAcademy September 2014.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11956
Joined: Sat Jul 30, 2011 7:41 pm
by Norefall » Sat May 19, 2012 10:13 am
Looks great! I want the 14Mpx for sure :)

Can you give us some price hints? Around 35 - 70 usd?
Posts: 43
Joined: Sun Nov 06, 2011 12:19 pm
by jamesh » Sat May 19, 2012 10:22 am
We really don't know about prices at this stage - it's the module that the problem. The PCB price is low, the socket and ribbon cable come to $2/3 (those ribbons are surprisingly expensive). Module? Who knows.
Soon to be employed engineer - Hurrah! Volunteer at the Raspberry Pi Foundation, helper at PiAcademy September 2014.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11956
Joined: Sat Jul 30, 2011 7:41 pm
by spazz » Sat May 19, 2012 10:34 am
jamesh wrote:How it works : The camera modules outputs raw data direct to the GPU. There it goes through about 17 stages of processing (Debayering, lens correction, black level, gain control, AWB, scaling, cropping, distortion etc). This is all done in real time at 30fps because it's all done in HW under control of the GPU software. The GPU can then also HW encode to JPEG a single capture at full res, or send a lower rez processed stream to the H264 encoder which can encode at 1080p30. Once all that is done, the results are/will be sent to Linux running on the Arm. That's the bit that still needs to be sorted out.


Can any kind of access to the IPP be expected or only the resulting data?
Posts: 2
Joined: Sat May 12, 2012 9:02 pm
by jamesh » Sat May 19, 2012 12:04 pm
The only way to access the data whilst its going through the pipeline is with a SW stage running on the GPU, and even then you can only inject them at certain points. BUT, because they run on the GPU, you cannot get at them.

I think we should be able to provide raw data (straight off the sensor so Bayer in whatever format the sensor output), but them you lose all the speed.

It might be possible to give programatic access to turn off various stages in the ISP, but no promises.

Why do you want to get at the data? We may already provide the sort of feature you are looking for.
Soon to be employed engineer - Hurrah! Volunteer at the Raspberry Pi Foundation, helper at PiAcademy September 2014.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11956
Joined: Sat Jul 30, 2011 7:41 pm
by spazz » Sat May 19, 2012 12:35 pm
jamesh wrote:The only way to access the data whilst its going through the pipeline is with a SW stage running on the GPU, and even then you can only inject them at certain points. BUT, because they run on the GPU, you cannot get at them.

I think we should be able to provide raw data (straight off the sensor so Bayer in whatever format the sensor output), but them you lose all the speed.

It might be possible to give programatic access to turn off various stages in the ISP, but no promises.

Why do you want to get at the data? We may already provide the sort of feature you are looking for.


Thanks for the reply.

Ok, that's sort of what I expected.

No particular reason really, I work with image sensors-integration related taks in embedded products so I just thought it might be fun to be able to play around with the image pipeline.
But as you say, for the majority of potential applications I'm sure the standard processing will be absolutely fine, and if you can get at the raw data as well that's excellent.
The only thing I can think of is potentially object tracking and stuff like that where it might be nice to be able to tune sharpness filters etc. Although I guess object tracking would have to be done using the cpu so it might be out of the question anyway.

But I'm definitely looking forward to be able to add a camera module to my RPi, keep up the good work!
Posts: 2
Joined: Sat May 12, 2012 9:02 pm
by kumarnarain » Sat May 19, 2012 1:30 pm
Hi Guys/Gals
Awesome work. Apart form other useful things try to think of a swivel base for this

kumar
Posts: 7
Joined: Sat May 19, 2012 1:27 pm
by Mike Lake » Sat May 19, 2012 2:15 pm
Liz suggested that we should ask for things to bring a bit of pressure on image sensor manufacturers - so here's my input.

Having no filter on an image sensor allows the use of an IR-passing filter instead of the IR-blocking filter which now seems to be standard. Passing IR would lead to all sorts of interesting stuff - like detecting a pulse (or continuous beam) of IR light directly from an IR laser diode or bounced from an IR laser diode. IR laser diodes, as a cylindrical module complete with adjustable collimator lens, are extremely low cost. We found that getting diodes as modules from Taiwan was less expensive than getting diodes alone and making our own modules with them.

IR-passing filters are dirt cheap to make. Take a roll of Ektachrome slide film (still available in professional packs in 35mm upwards), DON'T expose it, and get it developed. Tell the company doing the processing that you know the film appears to contain nothing and you don't want the film cut up - just returned as a roll. You are now in possession of enough IR pass filter film to go in front of thousands of image sensors - if only they did not have an IR blocking filter already on them! ( ... and, no, you can't get the IR blocking filter off.)

A filter-less sensor opens up another requirement which may not be possible with the RPi GPU - getting at the stream of raw pixel data in real time, at real speed, to detect when and where this IR pulse was seen. We do this in one of our products using an Atmel AVR to process the raw data in real time. The speed of the processor is a function of the resolution of the sensor - not everyone needs megapixels when sensing things like IR pulses - we could do it with an AT90S2313 - now the ATtiny2313A.

There used to be loads of low resolution (say, 356 x 292) image sensors that allowed this - they were mainly aimed at the mono security market. Trouble is, the security market has gone colour (maybe burglars look prettier in colour?) and increasingly hi res colour at that, and almost all manufacturers integrate a IR filter.

There are loads of posts on loads of forums all over the Internet asking for image sensors with NO filters so they can be used for all sorts of educational and development projects and new products. Trouble is, manufacturers are deaf to this requirement so the door has been totally slammed on innovation in this area.

The RPi Foundation, with its obvious volumes, could easily persuade a manufacturer like Pixart (http://www.pixart.com.tw) to do a run. We have found them to be an excellent supplier in the past - but they no longer have a part to do the interesting stuff.

Fixed v variable lenses is not such a problem - unless you want to get in really macro-level close - then things get horribly blurry with a standard lens.
User avatar
Posts: 96
Joined: Sun Feb 12, 2012 11:45 am
by Mike Lake » Sat May 19, 2012 3:00 pm
Following up my last post, and because someone has also asked about moving such a sensor, I enclose a photo of part of one of our products.

This shows a visible laser and camera mounted alongside one another. One servo motor is visible, another is on the base of the unit (not shown) which moves in the X (horizontal) axis.

This arrangement allows the system to display a beam of laser light anywhere in front of its location - well, +/- 100 degrees either side of straight ahead, vertically and horizontally.

A pulse of IR laser light (from elesewhere, not part of this module) is fired at the visible laser spot. We use a 30ms IR pulse to be picked up by one frame of the sensor which has an IR-pass filter between it and the lens.

Creating an X/Y platform to move an image sensor is pretty straightforward given a couple of servo motors each of which can be driven directly from an I/O pin.

The photo shows the opened laser/sensor module parked on top of a PiHouse (http://www.ThePiShop.org).

If anyone wants to see the product the module is part of, nothing to do with the RPi I'm afraid, it is at http://www.DryFire.com.
Attachments
ir_laser_sensor_a.jpg
Laser/sensor module
ir_laser_sensor_a.jpg (58.35 KiB) Viewed 29601 times
User avatar
Posts: 96
Joined: Sun Feb 12, 2012 11:45 am
by Burngate » Sat May 19, 2012 3:06 pm
Whilst most phone-cam pictures are postcard-pretty, and require fairly standard processing - auto-black, AWB, gamma, so-on - there are occasion when one wants to change things, and whilst one can redo that on the final Jpeg, that will lose information.
It would be nice if we had a way to alter the parameters that the GPU will use, before they're used. I've no idea about any standard Linux APIs, but rather than "take me a photo and return a Jpeg" it would be nice to have "using parameter file xxx take me a photo and return a Jpeg"
Wyszkowski's Second Law: Anything can be made to work if you fiddle with it long enough.
Brain surgery is easier than psychoanalysis
User avatar
Posts: 2929
Joined: Thu Sep 29, 2011 4:34 pm
Location: Berkshire UK
by kaos » Sat May 19, 2012 5:04 pm
Very interesting stuff!

A few points:

1. About IR filters, do modern phone camera modules invariably incorporate an IR blocking filter? Traditionally this wasn't so; hence the old trick of checking if your TV remote is working by looking at the business end of it through a phone camera. Of course, the quality of phone cameras has been steadily improving, so I guess the high end modules may have this filter, but what about the cheaper modules? Maybe another reason to offer a cheap module as one option?

2. I second the wish for raw camera output. There are two types that could be useful. One would be a completely unprocessed full resolution still image dump; i.e. a raw image. The other one would be a reduced resolution but otherwise unprocessed video stream, resolution to be low enough that the CPU can handle some simple processing of it. Note that even very low resolution could still be useful for some things; even an 8x8 pixel stream would f.x. be useful for a line following robot.

3. Please, pretty please, allow some control over the image processing. Manual override of white balance, plus exposure time correction (EV correction), is, IMO, the absolute minimum. Full manual control of exposure time (F-stop will probably not be adjustable any way on these modules) would be nice. So would control of auto-exposure (center, average, center weighted average).

--
Best regards,
Kári.
Posts: 102
Joined: Mon Mar 26, 2012 8:14 pm
by Paul Jurczak » Sat May 19, 2012 6:10 pm
Image scaling down would be very useful since 14Mp is a lot of image processing for relatively slow CPU. I assume IPP can scale down, but is there any support for pixel binning on image sensor itself?
Posts: 30
Joined: Wed Feb 22, 2012 11:46 pm