Camera module! (And a picture of JamesH)

Hi,

I really don't understand what is the big deal about the camera functionality!!!

Raspberry Pi has a CSI connector camera interface. It means all smart phones' CMOS sensors (cameras) can be connected to the board, as it mentioned before these high quality cameras such as OV5650 which has CSI interface come with a very low price (under 10$). (Even older cameras with I2C interface such as TCM8240MD can be connected by using CSI-2 Camera Bridge IC).

Then, RaspberryPi has a nice GPU which support OpenMAX technology. OpenMAX is programming interface that responsible for processing huge amount of multimedia task such as audio and video encoding,decoding,playing,recording, image processing and ... in an efficient way on the hardware. It means the GPU on the RaspberryPi can access the camera, fetch the raw image, enhances the image, encodes the video and records the video. Actually OpenMAX has 3 layers, the first layer is called "Application Layer" which defines high-level playback and recording API, this layer provides us with an interface to not only fetch raw image data from camera but also access all of the feature of the camera such as flash, focusing, metering, exposure compensation, sensitivity, shutter speed, aperture, white balance, and zoom.

Finally, the conclusion is, if RaspberryPi Foundation provides its customers with the API for OpenMAX AL, and OpenMAX IL, using the camera on the device would be piece of cake.

For all those megapixel haters (ie more megapixels in such a small sensor is a bad idea types), I really would suggest looking at some of the pictures taken with the Nokia 808 phone (soon to be released, but there should be pictures out there). This is a 41MP sensor in a phone, and the pictures are stunning.

Yep, when I first heard about that thing I thought Nokia had gone insane - but it gives some pretty impressive results, especially considering it's a bleddy mobile phone camera. In not so many years they've gone from being the laughing stock of the photographic world to being objects worthy of technological envy. Combination of (for a phone) a very large sensor and a decent lens obviously helps. Example shots I found indicate it's a bit poor at low-light shots, compared with a dSLR. As a super-compact camera? Plain amazing.

I'm rather hoping some of the developments filter their way back up to higher-end cameras. I'm now trying to imagine a full-frame dSLR with a back-illuminated 80-megapixel pixel-binned sensor here. Move over Foveon, the Bayer Mask gang is resolving detail again!

Combination of (for a phone) a very large sensor and a decent lens obviously helps. Example shots I found indicate it's a bit poor at low-light shots, compared with a dSLR. As a super-compact camera? Plain amazing.

I'm rather hoping some of the developments filter their way back up to higher-end cameras. I'm now trying to imagine a full-frame dSLR with a back-illuminated 80-megapixel pixel-binned sensor here. Move over Foveon, the Bayer Mask gang is resolving detail again!

The sensor is not very large, it's just that the pixels are very very small which give the 41MP number. If you understood what I wrote earlier it's obvious that not more detail was captured because of it, they just used a downsizing algorithm which they claim gives better output. The native output is only 5MP that you can change to 3 or 8 depending on your wishes. The shots I've seen were taken in daylight and honestly I doubt they're much better than that of any other high-end phone. Possibly the bulging lens is better than average. Marketing anyone?

When you want to compare the ouput of a phone camera to a full frame DSLR you have to understand that a small sensor gives much more depth of field compared to a large sensor with the same f-stop and the same field of view. So if you would stop down the full frame DSLR to mach the depth of field of the phone camera the quality would be about equal due to diffraction and the higher ISO needed for the full frame. If you compare them using the same f-stop and ISO you change the image parameters between cameras (you take different images) and compare apples to oranges.

If you look at the Nikon1 series, they have a pixel density that equates to 72MP on full frame. So they're fully capable of making such a camera, they just take it step by step. My guess because they want to keep on selling cameras and like to give incremental upgrades to the users.

With regard to the balance of resolution, you seem to have missed posts above that say you can specify what final resolution you want out of the ISP - the system will then set up the camera in the mode closest to the requirements, then the GPU will scale to the exact size you want. So by the time the image gets to the Arm, it can be big or small, depending on what you ask for.

Sorry, but you completely missed the point I was making; but never mind, I'm sure you guys come up with something usable

[jamesh]

Actually, the sensor in the Nokia 808 is pretty big (actually, biggest one I have ever seen), and the pixel density is the same as that in the N8 sensor, which for a mobile phone sensor has pretty big pixels.

[jamesh]

Hi,

I really don't understand what is the big deal about the camera functionality!!!

Raspberry Pi has a CSI connector camera interface. It means all smart phones' CMOS sensors (cameras) can be connected to the board, as it mentioned before these high quality cameras such as OV5650 which has CSI interface come with a very low price (under 10$). (Even older cameras with I2C interface such as TCM8240MD can be connected by using CSI-2 Camera Bridge IC).

Then, RaspberryPi has a nice GPU which support OpenMAX technology. OpenMAX is programming interface that responsible for processing huge amount of multimedia task such as audio and video encoding,decoding,playing,recording, image processing and ... in an efficient way on the hardware. It means the GPU on the RaspberryPi can access the camera, fetch the raw image, enhances the image, encodes the video and records the video. Actually OpenMAX has 3 layers, the first layer is called "Application Layer" which defines high-level playback and recording API, this layer provides us with an interface to not only fetch raw image data from camera but also access all of the feature of the camera such as flash, focusing, metering, exposure compensation, sensitivity, shutter speed, aperture, white balance, and zoom.

Finally, the conclusion is, if RaspberryPi Foundation provides its customers with the API for OpenMAX AL, and OpenMAX IL, using the camera on the device would be piece of cake.

It's a bit more complicated unfortunately. Although almost any CSI camera could be plugged in, all camera are a little bit different, so all require a new driver, which runs on the GPU. The camera's also need to be tuned or the image quality is dreadful. That tuning again lives on the GPU and takes many man months of work to get right.

As to OpenMAX - it's a great idea in principle, but is utterly horrible to actually code with. We do provide a OpenMAX IL API to Linux space, but woe betide anyone who actually tries to use it. I've written stuff for it, and it's was a traumatic experience.

Actually, the sensor in the Nokia 808 is pretty big (actually, biggest one I have ever seen)...

I hope you forgot to add 'in a phone'

...and the pixel density is the same as that in the N8 sensor, which for a mobile phone sensor has pretty big pixels.

Yes, but the pixel size is still only 1.4 micron, roughly the same as your 14MP camera (assuming a 1/3" sensor). That brings me back to why I started all my not-understood postings: Nokia downsamples the 41MP to 5MP and they do that for a reason, mainly because they can use noise reduction more effectively with small pixels so they end up with more detail. The signal is chopped in smaller chunks so detail has a bigger chance to survive the noise reduction algorithm.

It's clear the full 41MP images are basically useless at that size, for they do not contain (much) more detail than the 5MP image. That's why I think a 14MP camera for the Pi does not make sense over a (say) 8MP or 5MP one (same sensor tech), as the actual detail will be roughly equal but the Pi has to crunch through much more pixels which will do nothing else but slow down the process.

While this is not a big deal in a P&S for the 'look mummy' effect, if you want to seriously tinker with an image feed it does make a difference using a slow CPU. If your GPU can downsize the images real time using a state-of-the-art algorithm while mowing the lawn, OK, ignore what I wrote.

On a side note: When you guys were hammering code from magazines into your BBC Micros during the early eighties I was at photography school which has been my day job ever since; and I try to keep up with developments.

[jamesh]

Actually, the sensor in the Nokia 808 is pretty big (actually, biggest one I have ever seen)...

I hope you forgot to add 'in a phone'

...and the pixel density is the same as that in the N8 sensor, which for a mobile phone sensor has pretty big pixels.

Yes, but the pixel size is still only 1.4 micron, roughly the same as your 14MP camera (assuming a 1/3" sensor). That brings me back to why I started all my not-understood postings: Nokia downsamples the 41MP to 5MP and they do that for a reason, mainly because they can use noise reduction more effectively with small pixels so they end up with more detail. The signal is chopped in smaller chunks so detail has a bigger chance to survive the noise reduction algorithm.

It's clear the full 41MP images are basically useless at that size, for they do not contain (much) more detail than the 5MP image. That's why I think a 14MP camera for the Pi does not make sense over a (say) 8MP or 5MP one (same sensor tech), as the actual detail will be roughly equal but the Pi has to crunch through much more pixels which will do nothing else but slow down the process.

While this is not a big deal in a P&S for the 'look mummy' effect, if you want to seriously tinker with an image feed it does make a difference using a slow CPU. If your GPU can downsize the images real time using a state-of-the-art algorithm while mowing the lawn, OK, ignore what I wrote.

On a side note: When you guys were hammering code from magazines into your BBC Micros during the early eighties I was at photography school which has been my day job ever since; and I try to keep up with developments.

Sorry, I am biased, but the 38MP images that come off the Nokia 808 are truly excellent. Noticeably better than the 5MP images - which are also excellent (in fact better in some ways as the scaling down from 38MP reduces noise, just less pixels). Yes, I have seen lab done pictures which prove this (I've been working on that sensor for 2 years). Not as good as a well driven DSLR, but then, this is a mobile phone sensor...
Also, the lens in that sensor is a high enough quality it does not suffer any diffraction limitation (I checked with our expert on this just to be sure).

Yes, the GPU can downsize the large images in real time, although it's less strain on memory bandwidth to used a binned mode on the sensor and scale that. Which is what we do for 1080p for example - pick the closest mode (also depends on zoom level), scale to the size you want - this is done at 30fps.

Actually, the sensor in the Nokia 808 is pretty big (actually, biggest one I have ever seen)...

I hope you forgot to add 'in a phone'

How sweet!



(Full-res example Nokia 808 shots here - definitely not a dSLR, but a surprisingly good camera nevertheless. 38.4MP output is a bit pointless, but downsamples nicely in a hides-the-Bayer-mask-smearing kind of way.)

Hey I work for a defense contractor that makes infrared camera modules and we have a project that is supposed to simply give an engineer a worked example of interfacing our camera with hdmi out and a lcd display. I know the Raspberry doesnt have the screen, but do you think this thing could handle our parallel camera interface that outputs 320x240 digital YUV data?

We are currently using the ti814x / am387x, but TI has everything under NDA and we are having major issues simply getting data out from their device. (I think their cortex M3 is either off or hammering the memory bus where we cant change registers we need to)

[Gert van Loo]

Hey I work for a defense contractor that makes infrared camera modules and we have a project that is supposed to simply give an engineer a worked example of interfacing our camera with hdmi out and a lcd display. I know the Raspberry doesnt have the screen, but do you think this thing could handle our parallel camera interface that outputs 320x240 digital YUV data?

We are currently using the ti814x / am387x, but TI has everything under NDA and we are having major issues simply getting data out from their device. (I think their cortex M3 is either off or hammering the memory bus where we cant change registers we need to)

I need to know a bit more about the parallel camera interface. Width, speed etc. I don't think the Raspberry-Pi system can process the stream. The big question is: do you need the Raspberry-Pi system or are your (future) orders big enough to make your own board which has more pins to connect to. In the latter case you have a very good chance that we can accept your parallel stream.
As alternative you could look at the next chip from Broadcom. I don't know all the details but it that has almost everything the BCM2835 has and more. Among the 'more' is an LCD interface. It is not in the same price class as the BCM2835 though.

[jamesh]

Actually, the sensor in the Nokia 808 is pretty big (actually, biggest one I have ever seen)...

I hope you forgot to add 'in a phone'

How sweet!



(Full-res example Nokia 808 shots here - definitely not a dSLR, but a surprisingly good camera nevertheless. 38.4MP output is a bit pointless, but downsamples nicely in a hides-the-Bayer-mask-smearing kind of way.)

Not entirely pointless, if you want to do you own processing it can come in handy!
Though not DSLR quality, it's certainly up there with high end compact cameras (against which we did test), and it's the big sensor that enables that.

The images in the link were taking pretty early on in the dev process. The latest ones are better. (new version of sensor, newer tuning)

Yes! 38MP and your own processing!

But will it be possible to remove the lens and jury-rig the sensor behind a DSLR lens?

A tip for Eben and anyone other who is going to work on the camera module: Make sure that even the cheaper version is able to read tiny QR and barcodes without trouble. It's even better if it could read just some plain text!

It'd open the raspberry for new sort of commerce applications, as a raspberry system will likely cost much less than a proper barcode scanner.

I'd like to add my vote for no ir blocking filter and a removable lens. Thinking of the possibilities of using the camera/RPi combo as an all in one astrocam mounted on a telescope.

I'm sure the Nokia 41mp sensor is great but you're never going to get it in a camera with a price point of 25 dollars. At that price a sensor with a lower resolution would probably give better results yes?
Not meaning any offence to anybody, but if you can't get a decent image with 5mp you're not going to do any better with 12 or 20 or 41.

All I have to add is I'd really like to see a V4L2 interface, or at least some way of talking to the VideoCore that we could implement that. That means no closed-source userspace libraries and no floating point, please!

And yes, of course I'd be willing to help out with the implementation.

Would it be possible to shoot a movie in over 30 FPS if the resolution is turned down? let's say 60fps 720p, 240FPS SD quality or 50x50px 1000FPS?

ps; would be nice if we can see more puctures taken by the camera. At least a picture outside!

[jamesh]

Would it be possible to shoot a movie in over 30 FPS if the resolution is turned down? let's say 60fps 720p, 240FPS SD quality or 50x50px 1000FPS?

ps; would be nice if we can see more puctures taken by the camera. At least a picture outside!

Yes. I seen the Videocore at 120fps at lower resolutions (not with the sensor you see here). Not sure that's highest ever tried is though. However, whether the sensor we choose will work at those speeds is another matter.

you will need to watch your SD card speed too - I have an in-car camera than can shoot full HD 1920x1080 at 60fps BUT ONLY if I have a Class 10 card in it - and of course we know Class 10 cards are... how shall we say... problematic on the Pi (first use, by me, of this abbreviation )

What I would like to see is some way of getting those ragingly high frames-per-second the BBC use on their Natural World stuff - even at a reduced resolution - can the GPU do this?

[jamesh]

Well, if you assume 25MBits/s as a feasible datarate for high quality at 1080p30 (which is possible using the GPU), you can reduce resolution and increase framerate proportionally. So if you drop the H and V resolution by 2, you can increase the framerate by 4.

Also a vote from me for no IR filter, or an optional/removable one.
A fixed filter rules out many useful technical and educational possibilities.

[jamesh]

Well, if people could give me a list of cheap 5-15MP CSI-2 camera modules that have removable IR filters, that would help....the Foundation certainly doesn't have the time to look for them.

I'm not sure removable IR filter should be a requirement. It'd be nice if in the end we'd be able to do that, but for now, IMHO, the priority should be to get a single camera out in the field. With the proper planning (see other post), it shouldn't be too hard to get other adapters (ie. 15 pin FPC to small-pich-microconnector converter boards) for other cameras.

I found:
http://www.ebay.com/itm/Camera-Part-For ... 27c58fdde1
but only a few suppliers as this is an old (but pretty good) camera. (WAY better than what I have in my current phone).

One of the things I want to try is:

These cameras are intended to be used with a 3.5mm lens or something like that. You'd get a pretty amazing tele-lens-setup if you add a 125mm or 500mm lens... This one is 500mm if I understand things correctly. but here is a 100mm one.

The camera rew mentions is $15 on ebay, so a module including that would cost more. Maybe the first question to ask is, what price point are we targeting?

note: the two lenses rew links to are actually close-up adaptors, which is a simple 1-element magnifying lens element intended to screw on the front of a complete SLR lens, not intended for standalone use.

[jamesh]

I'm not sure removable IR filter should be a requirement. It'd be nice if in the end we'd be able to do that, but for now, IMHO, the priority should be to get a single camera out in the field. With the proper planning (see other post), it shouldn't be too hard to get other adapters (ie. 15 pin FPC to small-pich-microconnector converter boards) for other cameras.

I found:
http://www.ebay.com/itm/Camera-Part-For ... 27c58fdde1
but only a few suppliers as this is an old (but pretty good) camera. (WAY better than what I have in my current phone).

One of the things I want to try is:

These cameras are intended to be used with a 3.5mm lens or something like that. You'd get a pretty amazing tele-lens-setup if you add a 125mm or 500mm lens... This one is 500mm if I understand things correctly. but here is a 100mm one.

As stated above, every camera needs a specific camera driver and tuning, so just getting the HW to match isn't enough.

While I like the idea of a camera module add-on, I haven't quite worked out how it fits with the Foundation's education aim - does taking picies teach me to program?
Given that, I'm also not sure how a removable IR filter would help either.
Yes it would be nice to have, as would a c-mount for all the lenses I've got. But I doubt whether we'll get one.
So what I'm working round to saying is: what sort of parameters are you working to in your selection of cameras (apart from price)?