Can I use this sensor module with RPI

[mans0987]

I want to use Aptina MT9J003 (http://www.1stvision.com/cameras/sensor ... T9J003.pdf ) with Rasbery PI. I can use it in half resolution and I need to be able to do live feed.

Can I do this? What do I need?

If I can not do this with R-PI, is there any other board that I buy to do this? for example beagel board?

[joan]

I'd say not. A quick look at the specs suggest it'll use more bandwidth than the Pi can supply.

[mans0987]

Thanks for quick reply.
I can reduce the bandwidth by reducing resolution. I need to develop a test system to test the sensors and hence I don't need to have the image in full resolution and live. I can do it by reducing the resolution and also reducing frame rate.

[joan]

Search the forum for krydab. The chap did a somewhat similar job with a camera.

[mans0987]

Thanks. I did search the forum but nothing similar to this was found. Can you please give me more information?

[krydab]

Hi!

I was working with Micron MT9D111. It is only 2-magapixel camera. The main task that I had to do was to receive snapshot image from cam and decode QRcode. I don't need to receive live video from it. My camera has PLL mechanism and I am able to downgrade paralell output speed from 6MHz to 6MHz/16=375kHz. And with 375kHz I can read jpeg data when interrupts are disabled. 6MHz is the master clock, that means clock which I generate and connect to camera. I didn't check what is the highest frequency of paralell output that also works well.

[mans0987]

Thanks,
How did you connect your camera to r-pi? \is there any need for a specific interface?

Can you share your code?

[krydab]

You should rather look into your datasheet how to connect cam. As I said my cam is probably different. I have to connect I2C for cam setup and 8 gpio pins from which I have to read data in particular moments. My cam gives me many possible ways of reading data. I will give you my code if I see that it will be helpful for you. Now I'm sure that if you plug your camera and use my code, it won't be working properly. It is much more complicated. You need to write kernel module also, because you have to disable interrupts. Don't get me wrong, I can help you, but I won't do all your job.

It helped me, so probably will be valuable for you too:
http://www.raspberrypi.org/phpBB3/viewt ... 63&t=28325

[consulting]

Short interjection:
It would be useful an fully enough to get 720p/25 fps video up to 10 mbps for PAL
or the equivalence for NTSC.

These formats are easily to upconvert to pretty FullHD.

[sjoerd]

Hi!

I was working with Micron MT9D111. It is only 2-magapixel camera. The main task that I had to do was to receive snapshot image from cam and decode QRcode. I don't need to receive live video from it. My camera has PLL mechanism and I am able to downgrade paralell output speed from 6MHz to 6MHz/16=375kHz. And with 375kHz I can read jpeg data when interrupts are disabled. 6MHz is the master clock, that means clock which I generate and connect to camera. I didn't check what is the highest frequency of paralell output that also works well.

Hi,
(sorry for hyjacking this thread a little bit..)
I'm trying to getting the MT9D111 work with an mbed platform and also got a Raspberry PI board to try if things aren't working with the mbed. Would mind sharing the code? I'm kinda stuck in slowing down the pixel clock with JPEG output in snapshot mode
It would be great to see how you solved those things and I can test the camera on my raspberry pi board.

thanks in advance!

Best regards,
Sjoerd

[mrameshk]

Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress

[jamesh]

Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress

Without access to the source code on the GPU which houses the camera driver (there is a custom camera driver for each sensor), you will not make any progress.

[6by9]

Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress

Without access to the source code on the GPU which houses the camera driver (there is a custom camera driver for each sensor), you will not make any progress.

Not quite true. If you want to use a parallel camera, then there are probably sufficient GPIOs available to connect it in and do everything on the ARM. Whether you can clock it in the right way and at sufficient speed to receive the data is another matter.....

[jamesh]

Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress

Without access to the source code on the GPU which houses the camera driver (there is a custom camera driver for each sensor), you will not make any progress.

Not quite true. If you want to use a parallel camera, then there are probably sufficient GPIOs available to connect it in and do everything on the ARM. Whether you can clock it in the right way and at sufficient speed to receive the data is another matter.....

Although not quite true, true enough for most purposes!!

[fastmapper]

Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress

Without access to the source code on the GPU which houses the camera driver (there is a custom camera driver for each sensor), you will not make any progress.

Not quite true. If you want to use a parallel camera, then there are probably sufficient GPIOs available to connect it in and do everything on the ARM. Whether you can clock it in the right way and at sufficient speed to receive the data is another matter.....

You are right. I have worked on a project doing that very thing. I was somewhat successful, but I moved to the BeagleBone Black after I experienced the following issues:
1. Pixel read rates were limited to about 6 megapixels per second
2. Interrupt interference from Linux suggested strict interrupt control or moving to bare metal
3. High speed pixel clocking was difficult to keep clean in a prototyping environment

Clearly the CSI-2 interface, GPU, and related software are all designed to support a camera by avoiding all of the above issues.

I have no doubt that it is possible to use GPIO pins, bare metal software, and careful handling of high frequency signals to acquire images at 6 megapixels per second on the Raspberry Pi.

[ebutera]

Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress

Without access to the source code on the GPU which houses the camera driver (there is a custom camera driver for each sensor), you will not make any progress.

With the new compute module coming ("..this time aimed at business and industrial users..") that is a big limit on camera use.
I understand the technical reasons behind that, but is there something that could be done to make a "general" driver available?

For example a generic driver to just configure basic parameters like: sensor pixel format, data lanes, pixel clock...

But basically the question is: is there interest in finding a way to support more image sensors?

[6by9]

With the new compute module coming ("..this time aimed at business and industrial users..") that is a big limit on camera use.
I understand the technical reasons behind that, but is there something that could be done to make a "general" driver available?

For example a generic driver to just configure basic parameters like: sensor pixel format, data lanes, pixel clock...

But basically the question is: is there interest in finding a way to support more image sensors?

I can't say too much, and I don't know what direction the Foundation are going, but there may be hope. Please don't bank on anything though as the development I'm thinking of is not directly related to RPi and may or may not be applicable. It's also still in the relatively early stages anyway.

[jamesh]

Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress

Without access to the source code on the GPU which houses the camera driver (there is a custom camera driver for each sensor), you will not make any progress.

With the new compute module coming ("..this time aimed at business and industrial users..") that is a big limit on camera use.
I understand the technical reasons behind that, but is there something that could be done to make a "general" driver available?

For example a generic driver to just configure basic parameters like: sensor pixel format, data lanes, pixel clock...

But basically the question is: is there interest in finding a way to support more image sensors?

There certainly is interest, but the question is how much. Most industrial/embedded applications have no need of cameras so the demand may actually be quite low. Of course there are quite a few machine vision applications, and I wonder whether it would be worthwhile producing drivers for a second camera aimed in that direction (B/W, good low light performance, high speed etc).

There is a problem with generic drivers that it really difficult if not impossible to cope with the myriad of combinations and idiosyncrasies of different sensors from different manufacturers. It may be possible to do something for a subset of cameras, perhaps those from a single manufacturer, but that is some months/years away.

However, Brcm do have a number of tuned drivers for quite a few cameras - I do wonder whether the Foundation would be able to release those. It would increase the number of possibilities, but they may not want to wreck sales of their own camera which is what would happen. The market for cameras's isnt huge and diluting it like that would make their income disappear. It's really not as simple as just providing new camera drivers...

[ebutera]

With the new compute module coming ("..this time aimed at business and industrial users..") that is a big limit on camera use.
I understand the technical reasons behind that, but is there something that could be done to make a "general" driver available?

For example a generic driver to just configure basic parameters like: sensor pixel format, data lanes, pixel clock...

But basically the question is: is there interest in finding a way to support more image sensors?

There certainly is interest, but the question is how much. Most industrial/embedded applications have no need of cameras so the demand may actually be quite low. Of course there are quite a few machine vision applications, and I wonder whether it would be worthwhile producing drivers for a second camera aimed in that direction (B/W, good low light performance, high speed etc).

There is a problem with generic drivers that it really difficult if not impossible to cope with the myriad of combinations and idiosyncrasies of different sensors from different manufacturers. It may be possible to do something for a subset of cameras, perhaps those from a single manufacturer, but that is some months/years away.

However, Brcm do have a number of tuned drivers for quite a few cameras - I do wonder whether the Foundation would be able to release those. It would increase the number of possibilities, but they may not want to wreck sales of their own camera which is what would happen. The market for cameras's isnt huge and diluting it like that would make their income disappear. It's really not as simple as just providing new camera drivers...

Can someone from the Foundation say an official word about that? I mean, if they are willing to support other image sensors.

How to technically do that it's just "details". We can open another topic about that, but only if there is some interest from the Foundation.

[gsh]

Hi,

James asked me to make a more official response about this:

Basically we're unlikely to ever support lots of random types of camera with the full ISP. It just takes too long and a half arsed job is just not worth doing

_But_ we should be able to open up the CSI interface to be controlled directly by the ARM. The way it works is you talk to the camera through I2C, get it running and outputting pictures. Then you allocate some memory from the GPU (you need to do this if you want to encode the video with the hardware encoder but can skip if you're just going to be hacking it with the ARM) then set up the CSI interface to output a frame to a specified buffer (the CSI hardware will flip flop between two buffers), then wait for an interrupt.

That's what we're aiming for with the compute module...

Gordon

[ebutera]

Hi,

James asked me to make a more official response about this:

Basically we're unlikely to ever support lots of random types of camera with the full ISP. It just takes too long and a half arsed job is just not worth doing

_But_ we should be able to open up the CSI interface to be controlled directly by the ARM. The way it works is you talk to the camera through I2C, get it running and outputting pictures. Then you allocate some memory from the GPU (you need to do this if you want to encode the video with the hardware encoder but can skip if you're just going to be hacking it with the ARM) then set up the CSI interface to output a frame to a specified buffer (the CSI hardware will flip flop between two buffers), then wait for an interrupt.

That's what we're aiming for with the compute module...

Gordon

Thanks, that's exaclty what i was hoping when speaking about a generic driver.

It would be great to be able to use the current bcm2835 v4l2 driver as i'm interested in encoding the video. It's just awesome to skip all the openmax details and get the h264 encoded data from the v4l2 interface.

If i need all the isp functions then i should buy a supported camera (or pay to get it supported?), so everybody should be happy.

The big question: is there a time schedule? For example that you want to support it when the compute kits will be sold in June, a week later, a month later....

[6by9]

It would be great to be able to use the current bcm2835 v4l2 driver as i'm interested in encoding the video. It's just awesome to skip all the openmax details and get the h264 encoded data from the v4l2 interface.

To get sensible performance bcm2835-v4l2 pretty much requires that the camera data (ie CSI-2 receiver peripheral) is controlled by the GPU so that the ISP can process it efficiently into the format that the video encoder needs. If the CSI-2 peripheral is controlled by the ARM, then that close link is broken, so 1080P (and possibly even 720P) video encode is likely to be a tough challenge, if not impossible. It certainly wouldn't fit in with the assumptions that are currently made within most of the code supporting bcm2835-v4l2.

If i need all the isp functions then i should buy a supported camera (or pay to get it supported?), so everybody should be happy.

More like if you require almost any ISP function you need to buy a supported camera, or you need to be looking at YUV sensors.
Bayer sensors generally (someone's bound to find an exception) don't have things like AGC or AWB control loops on board, so those are run on GPU based on the statistics produced from the ISP. Getting those settings right is the sensor tuning that you may have heard being referred to, and can take an incredible amount of time and in-depth algorithm knowledge to get right.

[fastmapper]

More like if you require almost any ISP function you need to buy a supported camera, or you need to be looking at YUV sensors.
Bayer sensors generally (someone's bound to find an exception) don't have things like AGC or AWB control loops on board, so those are run on GPU based on the statistics produced from the ISP. Getting those settings right is the sensor tuning that you may have heard being referred to, and can take an incredible amount of time and in-depth algorithm knowledge to get right.

I thought that the majority of Bayer image sensors supported both automatic gain control (AGC) and automatic white balance (AWB). Looking at Omnivision's current 5 megapixel offerings I find that the OV5640, OV5645, OV5647, OV5648, OV5650, and the OV5653 all support both AGC and AWB. The OV5656, OV5658, OV5680, and the OV5690 apparently do not. So of 10 parts (all Bayer image sensors), 6 (the majority) support those functions.

Wait a minute! The OV5647 is used in the Raspberry Pi camera. That means it has a Bayer image sensor including built-in support for both AGC and AWB. Using those functions on the image sensor would offload the processing required by both the GPU and the ARM processor on the Raspberry Pi.

While there are numerous sensors for imaging and many provide color conversion to YUV, I'm unaware of any sensor that directly measures YUV. There are certainly billions of devices (e.g. cell phones, digital cameras, laptops, webcams, security cameras) that use CMOS image sensors with Bayer arrangements similar to the OV5647.

[6by9]

I thought that the majority of Bayer image sensors supported both automatic gain control (AGC) and automatic white balance (AWB). Looking at Omnivision's current 5 megapixel offerings I find that the OV5640, OV5645, OV5647, OV5648, OV5650, and the OV5653 all support both AGC and AWB. The OV5656, OV5658, OV5680, and the OV5690 apparently do not. So of 10 parts (all Bayer image sensors), 6 (the majority) support those functions.

By YUV sensor I mean one that has the capability to read out YUV (or RGB). To do so you need to have a whole load of processing blocks.
Therefore of your list, OV5640 and OV5645 to me are both YUV sensors. Of your remaining sensors you have 4 with and 4 without - no overall majority.

Original request was for AptinaMT9J003 - no AGC or AWB (although you do get auto black level).
I have access to datasheets for lot of sensors at the office, but most are under NDA so I can't disclose them or details. Certainly most of Sony's and Toshiba Bayer sensors that I'm aware of don't support AGC or AWB.
Further Googling:
Aptina MT9P031 http://www.aptina.com/assets/downloadDocument.do?id=76 - nope.
Aptina MT9P011 http://www.aptina.com/assets/downloadDocument.do?id=177 - nope
Aptina MTP9012 http://www.aptina.com/assets/downloadDocument.do?id=178 - AGC(along with a scaler and other processing steps), but no AWB .

AGC and AWB both require the image to be processed by something to collect stats and then complete the control loop. It's up to the manufacturer as to whether they want to add those silicon blocks into their sensor - sometimes they do, sometimes they don't.

Wait a minute! The OV5647 is used in the Raspberry Pi camera. That means it has a Bayer image sensor including built-in support for both AGC and AWB. Using those functions on the image sensor would offload the processing required by both the GPU and the ARM processor on the Raspberry Pi.[/qute]
Yes you could simplify the control algos if you choose an appropriate sensor and trust the module vendor's algo.
There's also the little issue of making the image usable that is required by most people - they don't want the Bayer pattern, so you've demosaicing and all the other image processing steps (lens shading, distortion correction, denoising, resizing, etc) to do somewhere. Where are you doing those steps? I doubt the hardware ISP is going to be opened up (it's too bl**dy complicated apart from anything else!).

While there are numerous sensors for imaging and many provide color conversion to YUV, I'm unaware of any sensor that directly measures YUV. There are certainly billions of devices (e.g. cell phones, digital cameras, laptops, webcams, security cameras) that use CMOS image sensors with Bayer arrangements similar to the OV5647.

As above, by YUV sensor I mean one that can readout YUV. They all tend to sense as Bayer, but some have an inbuilt ISP so they actually produce YUV, or sometimes even JPEG. In doing so they will have AGC and AWB as it is much easier to collect the stat in the Bayer domain. They tend to max out around 5MPix as customers start getting fussy about image quality around there.
https://www.toshiba.com/taec/adinfo/cmos/ TCM3211PB, TCM3212GBA
Aptina MT9M131 http://www.1stvision.com/cameras/IDS/da ... 131_PB.pdf YCbCr, RGB565, RGB555, RGB444, raw Bayer, or processed Bayer.
Or your OV5640 or OV5645 that you quoted above.

I'm dropping off this thread now - Gordon has stated the Foundation's intent, and I guess people will encounter the intricacies of dealing with sensors as and when that happens. For me, I just turn up at the office and hit them every other day.

[jamesh]

6x9 has pretty much covered it, but a couple of things are worth adding.

Running the ISP on the GPU doesn't really cost anything in GPU processor time, to offloading the ISP to the sensor (if it has one) doesn't really change overall processing capability. There is some impact on RAM accesses but pretty unnoticeable at the usual resolutions.

Generally, the tuning available in camera is worse that that from a (well) tuned GPU ISP.

The OV5647 can be used in Bayer or YUV mode - we use Bayer mode and do all the tuning in the 2835's ISP.

I am not familiar with any sensors over 5MP that have a YUV output mode - it seems to be a shrinking market. Most are even smaller at 2MP or so.

We are not likely to see a generic camera handling system for many months, if not over a year.