mans0987
Posts: 4
Joined: Tue Feb 05, 2013 11:01 am

Can I use this sensor module with RPI

Tue Feb 05, 2013 11:08 am

I want to use Aptina MT9J003 (http://www.1stvision.com/cameras/sensor ... T9J003.pdf ) with Rasbery PI. I can use it in half resolution and I need to be able to do live feed.

Can I do this? What do I need?

If I can not do this with R-PI, is there any other board that I buy to do this? for example beagel board?

User avatar
joan
Posts: 14472
Joined: Thu Jul 05, 2012 5:09 pm
Location: UK

Re: Can I use this sensor module with RPI

Tue Feb 05, 2013 11:22 am

I'd say not. A quick look at the specs suggest it'll use more bandwidth than the Pi can supply.

mans0987
Posts: 4
Joined: Tue Feb 05, 2013 11:01 am

Re: Can I use this sensor module with RPI

Tue Feb 05, 2013 11:26 am

Thanks for quick reply.
I can reduce the bandwidth by reducing resolution. I need to develop a test system to test the sensors and hence I don't need to have the image in full resolution and live. I can do it by reducing the resolution and also reducing frame rate.

User avatar
joan
Posts: 14472
Joined: Thu Jul 05, 2012 5:09 pm
Location: UK

Re: Can I use this sensor module with RPI

Tue Feb 05, 2013 11:41 am

Search the forum for krydab. The chap did a somewhat similar job with a camera.

mans0987
Posts: 4
Joined: Tue Feb 05, 2013 11:01 am

Re: Can I use this sensor module with RPI

Tue Feb 05, 2013 12:10 pm

Thanks. I did search the forum but nothing similar to this was found. Can you please give me more information?

krydab
Posts: 19
Joined: Fri Jan 04, 2013 8:33 pm

Re: Can I use this sensor module with RPI

Tue Feb 05, 2013 5:09 pm

Hi!

I was working with Micron MT9D111. It is only 2-magapixel camera. The main task that I had to do was to receive snapshot image from cam and decode QRcode. I don't need to receive live video from it. My camera has PLL mechanism and I am able to downgrade paralell output speed from 6MHz to 6MHz/16=375kHz. And with 375kHz I can read jpeg data when interrupts are disabled. 6MHz is the master clock, that means clock which I generate and connect to camera. I didn't check what is the highest frequency of paralell output that also works well.

mans0987
Posts: 4
Joined: Tue Feb 05, 2013 11:01 am

Re: Can I use this sensor module with RPI

Tue Feb 05, 2013 7:34 pm

Thanks,
How did you connect your camera to r-pi? \is there any need for a specific interface?

Can you share your code?

krydab
Posts: 19
Joined: Fri Jan 04, 2013 8:33 pm

Re: Can I use this sensor module with RPI

Wed Feb 06, 2013 9:27 am

You should rather look into your datasheet how to connect cam. As I said my cam is probably different. I have to connect I2C for cam setup and 8 gpio pins from which I have to read data in particular moments. My cam gives me many possible ways of reading data. I will give you my code if I see that it will be helpful for you. Now I'm sure that if you plug your camera and use my code, it won't be working properly. It is much more complicated. You need to write kernel module also, because you have to disable interrupts. Don't get me wrong, I can help you, but I won't do all your job.

It helped me, so probably will be valuable for you too:
http://www.raspberrypi.org/phpBB3/viewt ... 63&t=28325

consulting
Posts: 3
Joined: Thu Feb 07, 2013 6:18 pm

sensor module with RPI

Thu Feb 07, 2013 6:30 pm

Short interjection:
It would be useful an fully enough to get 720p/25 fps video up to 10 mbps for PAL
or the equivalence for NTSC.

These formats are easily to upconvert to pretty FullHD.

sjoerd
Posts: 2
Joined: Mon Feb 25, 2013 8:08 pm

Re: Can I use this sensor module with RPI

Mon Feb 25, 2013 8:15 pm

krydab wrote:Hi!

I was working with Micron MT9D111. It is only 2-magapixel camera. The main task that I had to do was to receive snapshot image from cam and decode QRcode. I don't need to receive live video from it. My camera has PLL mechanism and I am able to downgrade paralell output speed from 6MHz to 6MHz/16=375kHz. And with 375kHz I can read jpeg data when interrupts are disabled. 6MHz is the master clock, that means clock which I generate and connect to camera. I didn't check what is the highest frequency of paralell output that also works well.
Hi,
(sorry for hyjacking this thread a little bit..)
I'm trying to getting the MT9D111 work with an mbed platform and also got a Raspberry PI board to try if things aren't working with the mbed. Would mind sharing the code? I'm kinda stuck in slowing down the pixel clock with JPEG output in snapshot mode :(
It would be great to see how you solved those things and I can test the camera on my raspberry pi board.

thanks in advance!

Best regards,
Sjoerd

mrameshk
Posts: 4
Joined: Wed Mar 05, 2014 7:23 pm

Re: Can I use this sensor module with RPI

Mon Mar 24, 2014 9:32 pm

Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24133
Joined: Sat Jul 30, 2011 7:41 pm

Re: Can I use this sensor module with RPI

Tue Mar 25, 2014 9:47 am

mrameshk wrote:Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress
Without access to the source code on the GPU which houses the camera driver (there is a custom camera driver for each sensor), you will not make any progress.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7532
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Can I use this sensor module with RPI

Tue Mar 25, 2014 10:19 am

jamesh wrote:
mrameshk wrote:Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress
Without access to the source code on the GPU which houses the camera driver (there is a custom camera driver for each sensor), you will not make any progress.
Not quite true. If you want to use a parallel camera, then there are probably sufficient GPIOs available to connect it in and do everything on the ARM. Whether you can clock it in the right way and at sufficient speed to receive the data is another matter.....
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24133
Joined: Sat Jul 30, 2011 7:41 pm

Re: Can I use this sensor module with RPI

Tue Mar 25, 2014 10:48 am

6by9 wrote:
jamesh wrote:
mrameshk wrote:Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress
Without access to the source code on the GPU which houses the camera driver (there is a custom camera driver for each sensor), you will not make any progress.
Not quite true. If you want to use a parallel camera, then there are probably sufficient GPIOs available to connect it in and do everything on the ARM. Whether you can clock it in the right way and at sufficient speed to receive the data is another matter.....
Although not quite true, true enough for most purposes!!
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

fastmapper
Posts: 27
Joined: Wed Mar 27, 2013 3:54 am

Re: Can I use this sensor module with RPI

Tue Mar 25, 2014 8:04 pm

6by9 wrote:
jamesh wrote:
mrameshk wrote:Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress
Without access to the source code on the GPU which houses the camera driver (there is a custom camera driver for each sensor), you will not make any progress.
Not quite true. If you want to use a parallel camera, then there are probably sufficient GPIOs available to connect it in and do everything on the ARM. Whether you can clock it in the right way and at sufficient speed to receive the data is another matter.....
You are right. I have worked on a project doing that very thing. I was somewhat successful, but I moved to the BeagleBone Black after I experienced the following issues:
1. Pixel read rates were limited to about 6 megapixels per second
2. Interrupt interference from Linux suggested strict interrupt control or moving to bare metal
3. High speed pixel clocking was difficult to keep clean in a prototyping environment

Clearly the CSI-2 interface, GPU, and related software are all designed to support a camera by avoiding all of the above issues.

I have no doubt that it is possible to use GPIO pins, bare metal software, and careful handling of high frequency signals to acquire images at 6 megapixels per second on the Raspberry Pi.

ebutera
Posts: 5
Joined: Fri Apr 11, 2014 3:31 pm

Re: Can I use this sensor module with RPI

Fri Apr 11, 2014 4:10 pm

jamesh wrote:
mrameshk wrote:Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress
Without access to the source code on the GPU which houses the camera driver (there is a custom camera driver for each sensor), you will not make any progress.
With the new compute module coming ("..this time aimed at business and industrial users..") that is a big limit on camera use.
I understand the technical reasons behind that, but is there something that could be done to make a "general" driver available?

For example a generic driver to just configure basic parameters like: sensor pixel format, data lanes, pixel clock...

But basically the question is: is there interest in finding a way to support more image sensors?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7532
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Can I use this sensor module with RPI

Fri Apr 11, 2014 7:31 pm

ebutera wrote:With the new compute module coming ("..this time aimed at business and industrial users..") that is a big limit on camera use.
I understand the technical reasons behind that, but is there something that could be done to make a "general" driver available?

For example a generic driver to just configure basic parameters like: sensor pixel format, data lanes, pixel clock...

But basically the question is: is there interest in finding a way to support more image sensors?
I can't say too much, and I don't know what direction the Foundation are going, but there may be hope. Please don't bank on anything though as the development I'm thinking of is not directly related to RPi and may or may not be applicable. It's also still in the relatively early stages anyway.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24133
Joined: Sat Jul 30, 2011 7:41 pm

Re: Can I use this sensor module with RPI

Sat Apr 12, 2014 10:04 am

ebutera wrote:
jamesh wrote:
mrameshk wrote:Hi All,

I am also working on interfacing an Aptina Mt9v011 with the Rpi. Haven't made much progress
Without access to the source code on the GPU which houses the camera driver (there is a custom camera driver for each sensor), you will not make any progress.
With the new compute module coming ("..this time aimed at business and industrial users..") that is a big limit on camera use.
I understand the technical reasons behind that, but is there something that could be done to make a "general" driver available?

For example a generic driver to just configure basic parameters like: sensor pixel format, data lanes, pixel clock...

But basically the question is: is there interest in finding a way to support more image sensors?
There certainly is interest, but the question is how much. Most industrial/embedded applications have no need of cameras so the demand may actually be quite low. Of course there are quite a few machine vision applications, and I wonder whether it would be worthwhile producing drivers for a second camera aimed in that direction (B/W, good low light performance, high speed etc).

There is a problem with generic drivers that it really difficult if not impossible to cope with the myriad of combinations and idiosyncrasies of different sensors from different manufacturers. It may be possible to do something for a subset of cameras, perhaps those from a single manufacturer, but that is some months/years away.

However, Brcm do have a number of tuned drivers for quite a few cameras - I do wonder whether the Foundation would be able to release those. It would increase the number of possibilities, but they may not want to wreck sales of their own camera which is what would happen. The market for cameras's isnt huge and diluting it like that would make their income disappear. It's really not as simple as just providing new camera drivers...
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

ebutera
Posts: 5
Joined: Fri Apr 11, 2014 3:31 pm

Re: Can I use this sensor module with RPI

Mon Apr 14, 2014 1:54 pm

jamesh wrote:
ebutera wrote: With the new compute module coming ("..this time aimed at business and industrial users..") that is a big limit on camera use.
I understand the technical reasons behind that, but is there something that could be done to make a "general" driver available?

For example a generic driver to just configure basic parameters like: sensor pixel format, data lanes, pixel clock...

But basically the question is: is there interest in finding a way to support more image sensors?
There certainly is interest, but the question is how much. Most industrial/embedded applications have no need of cameras so the demand may actually be quite low. Of course there are quite a few machine vision applications, and I wonder whether it would be worthwhile producing drivers for a second camera aimed in that direction (B/W, good low light performance, high speed etc).

There is a problem with generic drivers that it really difficult if not impossible to cope with the myriad of combinations and idiosyncrasies of different sensors from different manufacturers. It may be possible to do something for a subset of cameras, perhaps those from a single manufacturer, but that is some months/years away.

However, Brcm do have a number of tuned drivers for quite a few cameras - I do wonder whether the Foundation would be able to release those. It would increase the number of possibilities, but they may not want to wreck sales of their own camera which is what would happen. The market for cameras's isnt huge and diluting it like that would make their income disappear. It's really not as simple as just providing new camera drivers...
Can someone from the Foundation say an official word about that? I mean, if they are willing to support other image sensors.

How to technically do that it's just "details". We can open another topic about that, but only if there is some interest from the Foundation.

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1445
Joined: Sat Sep 10, 2011 11:43 am

Re: Can I use this sensor module with RPI

Mon Apr 14, 2014 3:46 pm

Hi,

James asked me to make a more official response about this:

Basically we're unlikely to ever support lots of random types of camera with the full ISP. It just takes too long and a half arsed job is just not worth doing

_But_ we should be able to open up the CSI interface to be controlled directly by the ARM. The way it works is you talk to the camera through I2C, get it running and outputting pictures. Then you allocate some memory from the GPU (you need to do this if you want to encode the video with the hardware encoder but can skip if you're just going to be hacking it with the ARM) then set up the CSI interface to output a frame to a specified buffer (the CSI hardware will flip flop between two buffers), then wait for an interrupt.

That's what we're aiming for with the compute module...

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

ebutera
Posts: 5
Joined: Fri Apr 11, 2014 3:31 pm

Re: Can I use this sensor module with RPI

Mon Apr 14, 2014 4:37 pm

gsh wrote:Hi,

James asked me to make a more official response about this:

Basically we're unlikely to ever support lots of random types of camera with the full ISP. It just takes too long and a half arsed job is just not worth doing

_But_ we should be able to open up the CSI interface to be controlled directly by the ARM. The way it works is you talk to the camera through I2C, get it running and outputting pictures. Then you allocate some memory from the GPU (you need to do this if you want to encode the video with the hardware encoder but can skip if you're just going to be hacking it with the ARM) then set up the CSI interface to output a frame to a specified buffer (the CSI hardware will flip flop between two buffers), then wait for an interrupt.

That's what we're aiming for with the compute module...

Gordon
Thanks, that's exaclty what i was hoping when speaking about a generic driver.

It would be great to be able to use the current bcm2835 v4l2 driver as i'm interested in encoding the video. It's just awesome to skip all the openmax details and get the h264 encoded data from the v4l2 interface.

If i need all the isp functions then i should buy a supported camera (or pay to get it supported?), so everybody should be happy.

The big question: is there a time schedule? For example that you want to support it when the compute kits will be sold in June, a week later, a month later....

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7532
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Can I use this sensor module with RPI

Mon Apr 14, 2014 5:15 pm

ebutera wrote:It would be great to be able to use the current bcm2835 v4l2 driver as i'm interested in encoding the video. It's just awesome to skip all the openmax details and get the h264 encoded data from the v4l2 interface.
To get sensible performance bcm2835-v4l2 pretty much requires that the camera data (ie CSI-2 receiver peripheral) is controlled by the GPU so that the ISP can process it efficiently into the format that the video encoder needs. If the CSI-2 peripheral is controlled by the ARM, then that close link is broken, so 1080P (and possibly even 720P) video encode is likely to be a tough challenge, if not impossible. It certainly wouldn't fit in with the assumptions that are currently made within most of the code supporting bcm2835-v4l2.
ebutera wrote:If i need all the isp functions then i should buy a supported camera (or pay to get it supported?), so everybody should be happy.
More like if you require almost any ISP function you need to buy a supported camera, or you need to be looking at YUV sensors.
Bayer sensors generally (someone's bound to find an exception) don't have things like AGC or AWB control loops on board, so those are run on GPU based on the statistics produced from the ISP. Getting those settings right is the sensor tuning that you may have heard being referred to, and can take an incredible amount of time and in-depth algorithm knowledge to get right.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

fastmapper
Posts: 27
Joined: Wed Mar 27, 2013 3:54 am

Re: Can I use this sensor module with RPI

Tue Apr 15, 2014 3:11 am

6by9 wrote:More like if you require almost any ISP function you need to buy a supported camera, or you need to be looking at YUV sensors.
Bayer sensors generally (someone's bound to find an exception) don't have things like AGC or AWB control loops on board, so those are run on GPU based on the statistics produced from the ISP. Getting those settings right is the sensor tuning that you may have heard being referred to, and can take an incredible amount of time and in-depth algorithm knowledge to get right.
I thought that the majority of Bayer image sensors supported both automatic gain control (AGC) and automatic white balance (AWB). Looking at Omnivision's current 5 megapixel offerings I find that the OV5640, OV5645, OV5647, OV5648, OV5650, and the OV5653 all support both AGC and AWB. The OV5656, OV5658, OV5680, and the OV5690 apparently do not. So of 10 parts (all Bayer image sensors), 6 (the majority) support those functions.

Wait a minute! The OV5647 is used in the Raspberry Pi camera. That means it has a Bayer image sensor including built-in support for both AGC and AWB. Using those functions on the image sensor would offload the processing required by both the GPU and the ARM processor on the Raspberry Pi.

While there are numerous sensors for imaging and many provide color conversion to YUV, I'm unaware of any sensor that directly measures YUV. There are certainly billions of devices (e.g. cell phones, digital cameras, laptops, webcams, security cameras) that use CMOS image sensors with Bayer arrangements similar to the OV5647.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7532
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Can I use this sensor module with RPI

Tue Apr 15, 2014 7:56 am

fastmapper wrote:I thought that the majority of Bayer image sensors supported both automatic gain control (AGC) and automatic white balance (AWB). Looking at Omnivision's current 5 megapixel offerings I find that the OV5640, OV5645, OV5647, OV5648, OV5650, and the OV5653 all support both AGC and AWB. The OV5656, OV5658, OV5680, and the OV5690 apparently do not. So of 10 parts (all Bayer image sensors), 6 (the majority) support those functions.
By YUV sensor I mean one that has the capability to read out YUV (or RGB). To do so you need to have a whole load of processing blocks.
Therefore of your list, OV5640 and OV5645 to me are both YUV sensors. Of your remaining sensors you have 4 with and 4 without - no overall majority.

Original request was for AptinaMT9J003 - no AGC or AWB (although you do get auto black level).
I have access to datasheets for lot of sensors at the office, but most are under NDA so I can't disclose them or details. Certainly most of Sony's and Toshiba Bayer sensors that I'm aware of don't support AGC or AWB.
Further Googling:
Aptina MT9P031 http://www.aptina.com/assets/downloadDocument.do?id=76 - nope.
Aptina MT9P011 http://www.aptina.com/assets/downloadDocument.do?id=177 - nope
Aptina MTP9012 http://www.aptina.com/assets/downloadDocument.do?id=178 - AGC(along with a scaler and other processing steps), but no AWB .

AGC and AWB both require the image to be processed by something to collect stats and then complete the control loop. It's up to the manufacturer as to whether they want to add those silicon blocks into their sensor - sometimes they do, sometimes they don't.
fastmapper wrote:Wait a minute! The OV5647 is used in the Raspberry Pi camera. That means it has a Bayer image sensor including built-in support for both AGC and AWB. Using those functions on the image sensor would offload the processing required by both the GPU and the ARM processor on the Raspberry Pi.[/qute]
Yes you could simplify the control algos if you choose an appropriate sensor and trust the module vendor's algo.
There's also the little issue of making the image usable that is required by most people - they don't want the Bayer pattern, so you've demosaicing and all the other image processing steps (lens shading, distortion correction, denoising, resizing, etc) to do somewhere. Where are you doing those steps? I doubt the hardware ISP is going to be opened up (it's too bl**dy complicated apart from anything else!).
fastmapper wrote:While there are numerous sensors for imaging and many provide color conversion to YUV, I'm unaware of any sensor that directly measures YUV. There are certainly billions of devices (e.g. cell phones, digital cameras, laptops, webcams, security cameras) that use CMOS image sensors with Bayer arrangements similar to the OV5647.
As above, by YUV sensor I mean one that can readout YUV. They all tend to sense as Bayer, but some have an inbuilt ISP so they actually produce YUV, or sometimes even JPEG. In doing so they will have AGC and AWB as it is much easier to collect the stat in the Bayer domain. They tend to max out around 5MPix as customers start getting fussy about image quality around there.
https://www.toshiba.com/taec/adinfo/cmos/ TCM3211PB, TCM3212GBA
Aptina MT9M131 http://www.1stvision.com/cameras/IDS/da ... 131_PB.pdf YCbCr, RGB565, RGB555, RGB444, raw Bayer, or processed Bayer.
Or your OV5640 or OV5645 that you quoted above.

I'm dropping off this thread now - Gordon has stated the Foundation's intent, and I guess people will encounter the intricacies of dealing with sensors as and when that happens. For me, I just turn up at the office and hit them every other day.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24133
Joined: Sat Jul 30, 2011 7:41 pm

Re: Can I use this sensor module with RPI

Tue Apr 15, 2014 9:12 am

6x9 has pretty much covered it, but a couple of things are worth adding.

Running the ISP on the GPU doesn't really cost anything in GPU processor time, to offloading the ISP to the sensor (if it has one) doesn't really change overall processing capability. There is some impact on RAM accesses but pretty unnoticeable at the usual resolutions.

Generally, the tuning available in camera is worse that that from a (well) tuned GPU ISP.

The OV5647 can be used in Bayer or YUV mode - we use Bayer mode and do all the tuning in the 2835's ISP.

I am not familiar with any sensors over 5MP that have a YUV output mode - it seems to be a shrinking market. Most are even smaller at 2MP or so.

We are not likely to see a generic camera handling system for many months, if not over a year.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

Return to “Camera board”