Complex camera peripherials


28 posts   Page 1 of 2   1, 2
by LloydMoore » Thu Apr 26, 2012 5:55 pm
I'm looking at creating a specialized camera for the Pi and trying to figure out if I actually have the ability to use the camera connector on the board, and if I can get to the peripheral inside the BCM2835 once I get a camera connected. So far I'm not finding much in the way of technical information and the peripheral does not appear to be in the "BCM2835 ARM Peripherals" document. Has anyone else looked at this?

Please note that what I'm thinking of is NOT a straight forward camera application - I could just use a USB connection for that. This would be a more complex device.
Posts: 8
Joined: Mon Jan 16, 2012 9:24 pm
by hzrnbgy » Thu Apr 26, 2012 6:39 pm
Until we get a hold of a more technical datasheet than what it is provided, I can only assume that the camera connector might be of the 8/16 bit parallel digital camera interface.
Posts: 106
Joined: Mon Dec 26, 2011 10:55 pm
by Gert van Loo » Thu Apr 26, 2012 10:00 pm
I would not put any effort in that for two reasons:

1/ The ARM is unlikely to have enough CPU power, unless you are happy with very low quality video.

2/ The foundation has promised that in due time they will provide a camera.

You see, no matter how hard you try you will not be able to beat the camera quality which comes out of the GPU. Phones with this chip in it have won prizes for being 'the best camera phone in the world'. (No boast: check the press releases for that).

That was achieved by a team of people working very hard for many years and on top of that they spend about 6 months per camera device tuning the parameters.

So I think that your time is better spend doing something else and wait for a camera to appear.I know that is frustrating, you can go ahead and try. At least you will learn something. Just don't be disappointed if in a few months a camera appears which wipes the floor with everything you have done.
User avatar
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2078
Joined: Tue Aug 02, 2011 7:27 am
by LloydMoore » Thu Apr 26, 2012 10:17 pm
Ya I'm NOT actually looking at a camera per se - just using one to do other types of sensing. Actually ran though the bandwidth calculations since the original post and I should have plenty of computing power for what I'm looking at, particularly with the resolution that I'm looking at.  It turns out that I actually don't need to use the camera connector either making it available for a high quality camera. You do give me an interesting idea though - what I'm working on would be a perfect complement to the camera that the foundation is going to add. If they are a few months out that would be great, the two systems working together is an excellent idea. <grin> (Ya not telling what I really have in mind as it is coming together as a pretty interesting idea.....)
Posts: 8
Joined: Mon Jan 16, 2012 9:24 pm
by Paul Jurczak » Wed May 02, 2012 5:52 am
I'm thinking about using Raspberry Pi in inexpensive embedded machine vision application. There are many use scenarios, where CPU power is more than sufficient.

The hardware interface to camera on Raspberry Pi seems to be MIPI CSI-2 http://www.mipi.org/specificat.....-interface. There are off-the-shelf camera modules conforming to this specification. The problem is in proprietary image processing hardware and software on BCM2835. For instance, we may not be able to acquire raw images from the sensor, if Broadcom doesn't provide corresponding driver capability. We may be limited only to officially sanctioned camera modules.
Posts: 30
Joined: Wed Feb 22, 2012 11:46 pm
by RaTTuS » Wed May 02, 2012 8:49 am
one of the uses I have for the RPi is an automatic wildlife camera - [or monitor for the garden]
http://www.catb.org/esr/faqs/smart-questions.html <- ask smart Questions
"That's not right, the badgers have moved the goalposts."
1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX - Prosliver FTW
User avatar
Posts: 5339
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
by jamesh » Wed May 02, 2012 9:21 am
Paul Jurczak said:


I'm thinking about using Raspberry Pi in inexpensive embedded machine vision application. There are many use scenarios, where CPU power is more than sufficient.

The hardware interface to camera on Raspberry Pi seems to be MIPI CSI-2 http://www.mipi.org/specificat.....-interface. There are off-the-shelf camera modules conforming to this specification. The problem is in proprietary image processing hardware and software on BCM2835. For instance, we may not be able to acquire raw images from the sensor, if Broadcom doesn't provide corresponding driver capability. We may be limited only to officially sanctioned camera modules.


Correct. Although a device might be MIPI-CSI2, you cannot just plug them in. Each camera module need a different driver to cope with their very different programming characteristics. CSI2 is the hardware spec, and doesn't specify how to use the camera, just how to connect it.

Raw output is perfectly possible - we've use it a lot when tuning the cameras. However, the raw format changes according to the camera module....
Soon to be employed engineer - Hurrah! Volunteer at the Raspberry Pi Foundation, helper at PiAcademy September 2014.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11926
Joined: Sat Jul 30, 2011 7:41 pm
by LloydMoore » Wed May 02, 2012 3:00 pm
Yes completely agree being able to get the raw image is going to be key for doing anything really creative with the camera. At least from what I can infer from the available documentation the camera port is at least somewhat under the control of the GPU, and it appears that we are not going to get much information on that. I'm hoping that at some point there will at least be an API available to give access to the camera and be able to control it in some fashion. For my particular application I've actually taken the route of placing a coprocessor between the camera and the PI to handle the camera directly and then process the data down to a bandwidth that can be sent in over an SPI port. While this actually turns out to be a good solution for what I'm doing, it does add cost, and will not be anywhere ideal in many cases. If anyone has any details on what the API for the camera will be it would be really helpful to make that available as soon as possible.
Posts: 8
Joined: Mon Jan 16, 2012 9:24 pm
by jamesh » Wed May 02, 2012 3:41 pm
There will be an API for the camera board to be developed.

However, I'm always surprised at the persistent desire for RAW data. It's pretty unnecessary with high qualilty JPG's, and the tunings done on the camera at Broadcom are very good indeed.

See the Nokia N8 and 808 examples pictures for some examples of camera output by the same GPU as the Raspi - no raw data involved - this is how they come out of the GPU as JPEG.
Soon to be employed engineer - Hurrah! Volunteer at the Raspberry Pi Foundation, helper at PiAcademy September 2014.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11926
Joined: Sat Jul 30, 2011 7:41 pm
by LloydMoore » Wed May 02, 2012 4:02 pm
The need for raw images (at least for me) comes from the application we are looking at running on the Raspberry Pi. Specifically we are interested in doing image processing on the data collected from the camera BEFORE sending it off to another process. The bulk of this processing needs to be done on uncompressed images so if all we have are the JPEG formatted images the first step is going to be uncompressing them back into a raw form. The image quality isn't actually that big of an issue - in the application I'm looking at 320x240 monochrome is going to be sufficient. As a really simple example consider finding the center of a white ball on a black background - the (very simple for example) image processing pipeline (steps needed to process the image) would be:

1. Collect the image from the camera (if the image is compressed it needs to be uncompressed)

2. If the image is in color, convert it to black and white

3. Threshold the image (force each pixel to 0 or 255 based on a preset level)

4. Scan the image for white pixels and record the coordinates of each

5. Average the coordinates of the white pixels (could combine with step 4)

6. Return the average of the x and y coordinates for the white pixels

In this particular case you are looking at the raw values of each pixel in the image and are more interested in their location than the actual value, or image quality. In the end the only thing that is getting saved out is two integers.

Granted this is a rather simple and contrived example, HOWEVER it is a very fundamental machine vision operation that is taught in machine vision classes, and forms the foundation knowledge for more complex operations such as face tracking, object tracking, object detection, navigation and many of the other operations.
Posts: 8
Joined: Mon Jan 16, 2012 9:24 pm
by jbeale » Wed May 02, 2012 4:06 pm
For most applications JPEG is fine, but the real enthusiasts always want to go to the next level. I think all the modern DSLR cameras provide RAW formats, even some pocket types like my little Canon S95. Have a look at the "supported cameras" for the DCRAW converter: http://www.cybercom.net/~dcoffin/dcraw/

At the level of a cell phone camera, I wouldn't bother with it myself, but I'm sure some applications could benefit. Especially if you're doing something scientific or industrial where you want to turn OFF all automatic features and just make some absolute measurements.
User avatar
Posts: 2076
Joined: Tue Nov 22, 2011 11:51 pm
by hzrnbgy » Thu May 03, 2012 1:52 am
If you are willing to compromise with a different solution other than the RPi, you can use this board

http://www.mouser.com/ProductD.....N%252b9qEw

and this camera

http://www.ebay.com/itm/VGA-OV.....687wt_1163

plus some code and you can have the RGB565-encoded image data in memory, free to do whatever processing you have in mind. You also have access to a rich set of peripherals (timers, PWM, ethernet) that may or may not be useful for your project.

I can share you some of my code if you ask...
Posts: 106
Joined: Mon Dec 26, 2011 10:55 pm
by LloydMoore » Thu May 03, 2012 2:55 pm
Thanks for the links - the STM32f4xx series is actually the processor I'm looking at for being connected directly to the camera and will be doing some of the initial image pipeline work. I also need to be able to run Linux for this application so I'll still need the Pi for that part. Both of them working together should make for a very nice combination.

The camera you have listed is also pretty interesting. I'm looking at a different one, but it is similar. I'll dig down a bit on that one and see how they line up. Thanks for the info!!
Posts: 8
Joined: Mon Jan 16, 2012 9:24 pm
by jamesh » Thu May 03, 2012 3:25 pm
LloydMoore said:


The need for raw images (at least for me) comes from the application we are looking at running on the Raspberry Pi. Specifically we are interested in doing image processing on the data collected from the camera BEFORE sending it off to another process. The bulk of this processing needs to be done on uncompressed images so if all we have are the JPEG formatted images the first step is going to be uncompressing them back into a raw form. The image quality isn't actually that big of an issue - in the application I'm looking at 320x240 monochrome is going to be sufficient. As a really simple example consider finding the center of a white ball on a black background - the (very simple for example) image processing pipeline (steps needed to process the image) would be:

1. Collect the image from the camera (if the image is compressed it needs to be uncompressed)

2. If the image is in color, convert it to black and white

3. Threshold the image (force each pixel to 0 or 255 based on a preset level)

4. Scan the image for white pixels and record the coordinates of each

5. Average the coordinates of the white pixels (could combine with step 4)

6. Return the average of the x and y coordinates for the white pixels

In this particular case you are looking at the raw values of each pixel in the image and are more interested in their location than the actual value, or image quality. In the end the only thing that is getting saved out is two integers.

Granted this is a rather simple and contrived example, HOWEVER it is a very fundamental machine vision operation that is taught in machine vision classes, and forms the foundation knowledge for more complex operations such as face tracking, object tracking, object detection, navigation and many of the other operations.


Worth giving some background. The RAW image from the camera sensor is exactly that, very RAW as it comes off the sensor. It's not DNG raw or similar - it's as the raw data comes from the camera. No black level compensation, no lens shading correctly, none of the 15 or so stages that the ISP does in real time. It's not the sort of raw you would get from a DSLR for example, which will have had some processing applied. A lot of this processing you would then need to do yourself to get a sensible image. Since the ISP does this processing in HW in real time you cannot do the same work on the Arm, it's just not fast enough. It's actually quicker to take the JPG and decompress it. That said, we can get the ISP to output uncompressed (but processed) data, YUV420 for example, so if you really don't want to decompress you can use that.

Internally, the ISP converts to YUV422 or 420 from RAW very early on, before it applies a lot of it's image processing (gain, white balance, face tracking, smile detection etc).

So I think we are talking about different sorts of RAW...
Soon to be employed engineer - Hurrah! Volunteer at the Raspberry Pi Foundation, helper at PiAcademy September 2014.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11926
Joined: Sat Jul 30, 2011 7:41 pm
by LloydMoore » Thu May 03, 2012 3:34 pm
Agree - if the uncompressed image is available that should cover the majority of the applications that I can think of. At least from my point of view getting the compression out of the way and being able to access the RGB, YUV, or HSV data gets you there.
Posts: 8
Joined: Mon Jan 16, 2012 9:24 pm
by gsh » Thu May 03, 2012 6:46 pm
Why not just get a USB webcam?

Then you can just use the standard interface for accessing the data
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 789
Joined: Sat Sep 10, 2011 11:43 am
by LloydMoore » Thu May 03, 2012 7:38 pm
Ya that would work for many applications as well. I'm not sure about the USB overhead to the system going that route as I haven't tried it. There are also some lower level control items that you cannot do unless you have direct control of the camera (depends on the camera being used).
Posts: 8
Joined: Mon Jan 16, 2012 9:24 pm
by AlienStCNX » Thu May 03, 2012 9:01 pm
@JamesH: I'd like to second the request for uncompressed data, also because of computer vision. Any idea when this module might be available?
Posts: 6
Joined: Thu Nov 10, 2011 7:44 pm
by jamesh » Thu May 03, 2012 9:54 pm
I couldn't possibly comment on when. Suffice to say I've had more than volunteer from the Broadcom camera/apps team in Cambridge to help out when the HW is ready....

But I really do wonder why people are so desperate for uncompressed. JPG at high quality is pretty much indistinguishable from uncompressed, and much smaller. Only reason I can is speed of decompress on the Arm, and that slightly offset by the reduced transfer time of half the data from the GPU (it has HW JPG encode). Half the time in image processing you subsample the image anyway so the JPG artefacts get even less noticeable.
Soon to be employed engineer - Hurrah! Volunteer at the Raspberry Pi Foundation, helper at PiAcademy September 2014.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11926
Joined: Sat Jul 30, 2011 7:41 pm
by Paul Jurczak » Fri May 04, 2012 6:48 am
JamesH said:


...

However, I'm always surprised at the persistent desire for RAW data. It's pretty unnecessary with high qualilty JPG's, and the tunings done on the camera at Broadcom are very good indeed.

...


For media/presentation purposes, there is rarely a need to tinker with well tuned image pipe.

For scientific or metrology applications, there are situations where RAW data (Bayer pattern) is needed. The biggest culprit is usually the debayering algorithm, which always introduces color interpolation artifacts.

JPEG images are often unsuitable for machine vision due to compression artifacts: 8x8 blocks, which wreak havoc with edge detection.
Posts: 30
Joined: Wed Feb 22, 2012 11:46 pm
by Paul Jurczak » Fri May 04, 2012 7:08 am
hzrnbgy said:


If you are willing to compromise with a different solution other than the RPi, you can use this board

http://www.mouser.com/ProductD.....N%252b9qEw

and this camera

http://www.ebay.com/itm/VGA-OV.....687wt_1163

plus some code and you can have the RGB565-encoded image data in memory, free to do whatever processing you have in mind. You also have access to a rich set of peripherals (timers, PWM, ethernet) that may or may not be useful for your project.

I can share you some of my code if you ask…


This is exactly what I was looking for to make a rock bottom cheap smartcam. I'm interested to see the details of your solution, do you have a website?

Having found about RPi, I think it is now in general a better solution than STM32F4 + camera module:

1. RPi wins in processing power and memory category – very important for image processing;

2. Cost is about equal, if you compare production STM32F4 board (you can use Discovery board only for prototyping, any commercial applications are prohibited) + camera module with RPi + cheap USB webcam (or future camera module in $10 – $15 range I hope);

3. STM32F4 solution can be much cheaper, if you are willing to design your own PCB and have a prospect to sell enough of them to recover development costs and get into economy of scale zone;

4. RPi wins in software development friendliness;

5. In some cases, STM32F4 will be better suited due to its richness of I/O;
Posts: 30
Joined: Wed Feb 22, 2012 11:46 pm
by hzrnbgy » Fri May 04, 2012 8:53 am
I would eventually incorporate the Ethernet peripheral so I can stream some realtime low fps video, much like a cheap DIY IP cam. Programming is the easy part, it's the hardware that I'm struggling with. I can probably whipped out a basic PCB using Eagle but its not going to be nowhere near commercial grade. But I do have plans on building a basic PCB for some robotics imaging and security applications I have in mind.

Having said that, its true that the RPi is faster and has more memory than a 192KB RAM CortexM4 chip, but I'd like to be able to make/do something now instead of relying on someone/some company to do the programming/driver development for me. And I'm not really into Linux programming as it imposes too much limitation on what you can do on a hardware level. I have a Pandaboard but I haven't dig deep enough in Linux to know how to toggle one of the onboard LED.
Posts: 106
Joined: Mon Dec 26, 2011 10:55 pm
by LloydMoore » Fri May 04, 2012 3:02 pm
I looked into doing a smart IP camera using the STM32 and an inexpensive camera. At least in my case there really wasn't enough markup that could be had for that type of a product. I was able to find existing cameras (mostly WiFi) on the market for about $80. There is some potential there but you would need to sell them in a good quantity. This isn't actually the project that I'm working on currently as a result of that.

I don't actually have any information up on my web site about this project yet. When I get closer to having it ready I'll start putting information there. My primary business is consulting in the industrial automation, machine vision and robotics spaces. I work on my own products when I get time between projects, and they also support my consulting work.

I think you already have the basic idea of what I was exploring for the IP smart camera though. Take the STM32F4 processor, strap a camera and an Ethernet PHY to it and party down. The STM32F4 has lots of I/O that would be very helpful there as well.

My company site is http://www.CyberData-Robotics.com if you are still interested. Do you have a site for your company?
Posts: 8
Joined: Mon Jan 16, 2012 9:24 pm
by hzrnbgy » Fri May 04, 2012 4:26 pm
In case the question is directed to me, my answer would be some company that not related to anything embedded programming. I'm more of a hobbyist/enthusiast programmer.
Posts: 106
Joined: Mon Dec 26, 2011 10:55 pm
by Paul Jurczak » Fri May 04, 2012 8:15 pm
LloydMoore said:


...

My company site is http://www.CyberData-Robotics.com if you are still interested. Do you have a site for your company?

...


I'm a computer vision consultant, so far doing OK without the website. I was looking at STM32F4 + CMOS image sensor as a hardware base for more capable version of CMUCam http://cmucam.org/ (CMUCam5?). I will send you an email directly, since this is already far off RPi topic.
Posts: 30
Joined: Wed Feb 22, 2012 11:46 pm