Patrick.Pi
Posts: 1
Joined: Thu Mar 30, 2017 2:28 pm

CSI - Camera interface

Thu Mar 30, 2017 2:38 pm

Hi,
We are building a camera for the Raspberry Pi 3.
We want to connect a 2M image sensor to the MIPI interface.
Did anyone connect image sensor (other than the Raspberry Pi Camera) and manage to get it work?
How complicate is it (if possible at all) to write a driver for the camera?

Thanks.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5806
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: CSI - Camera interface

Thu Mar 30, 2017 4:20 pm

viewtopic.php?f=43&t=109137
viewtopic.php?f=38&t=120702

I'm writing the V4L2 kernel driver for the CSI peripheral at the moment. If you have a V4L2 subdevice driver for your sensor then it should be a relatively trivial to get it working.

Note that that is only reading the data in from the sensor in whatever the native format happens to be. For YUV sensors that tends to be YUYV (or variant) and that is not consumable by many of the multimedia blocks on the SoC. Bayer even less so.
There is a MMAL component "vc.ril.isp" that will take both YUYV (and variants), YUV planar, RGB, and Bayer. It can then resize (if desired) and convert to YUV planar (that is used by most blocks), RGB (less so), or back to YUYV (if you want to do that). The Bayer side is still a work in progress and hasn't got any of the useful controls you need yet. Also be aware that it will NOT include any control loops for AE/AGC/AWB, it is just the image processing pipe.
There's also an optimisation required in order to pass the images from V4L2 to MMAL efficiently, but that will follow on later.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

bahadirg
Posts: 1
Joined: Thu Nov 26, 2015 9:47 am

Re: CSI - Camera interface

Thu Apr 27, 2017 2:27 pm

Hi 6by9,

Thanks for your work on raw CSI access. My aim is to test different sensors and hdmi input also. I have some experience on OpenMAX IL but just wanted to understand whether I must learn mmal for raw access (even if v4l2 side is implemented)

I have also read relevant topics and would like to clarify my understanding since I am not very familiar with v4l2 side.

I will try to list my understandings below and hope for your feedback :)

1- If I understood correctly, even if v4l2 driver is done, it will still be only for reading raw data from the sensor. (no GPU ISP involved ?)

2- And one would still need OpenMAX or MMAL components to do additional stuff (i.e feeding it to encoder or to egl_render component etc.)

3- Current way to do it is to use raspiraw (which is only available on MMAL not on OpenMax IL), but that code will not be fully supported once v4l2 support is released.

4- In a nutshell what would be the correct approach for basing my future development for raw input + gpu accelerated encoding?

4.1- v4l2 for reading raw frames -> mmal.isp -> mmal.encode

4.2- v4l2 for reading raw frames -> mmal.encode (i.e no need for vc.isp component you have added)

4.3- v4l2 for reading raw frames-> OpenMAX IL for encoding, egl_rendering etc.. (as long as the sensor spits out compatible format)

5. If 4.2 and 4.3 both will be possible one shall use mmal if needs zero_copy (or easy usage) ; and shall use OpenMAX IL if one needs more documentation.

Sorry for the long post

Regards

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5806
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: CSI - Camera interface

Tue May 23, 2017 2:42 pm

bahadirg wrote:Hi 6by9,

Thanks for your work on raw CSI access. My aim is to test different sensors and hdmi input also. I have some experience on OpenMAX IL but just wanted to understand whether I must learn mmal for raw access (even if v4l2 side is implemented)

I have also read relevant topics and would like to clarify my understanding since I am not very familiar with v4l2 side.

I will try to list my understandings below and hope for your feedback :)

1- If I understood correctly, even if v4l2 driver is done, it will still be only for reading raw data from the sensor. (no GPU ISP involved ?)

2- And one would still need OpenMAX or MMAL components to do additional stuff (i.e feeding it to encoder or to egl_render component etc.)

3- Current way to do it is to use raspiraw (which is only available on MMAL not on OpenMax IL), but that code will not be fully supported once v4l2 support is released.

4- In a nutshell what would be the correct approach for basing my future development for raw input + gpu accelerated encoding?

4.1- v4l2 for reading raw frames -> mmal.isp -> mmal.encode

4.2- v4l2 for reading raw frames -> mmal.encode (i.e no need for vc.isp component you have added)

4.3- v4l2 for reading raw frames-> OpenMAX IL for encoding, egl_rendering etc.. (as long as the sensor spits out compatible format)

5. If 4.2 and 4.3 both will be possible one shall use mmal if needs zero_copy (or easy usage) ; and shall use OpenMAX IL if one needs more documentation.
Sorry, this reply escaped my attention.

1. The initial V4L2 subdevice driver is solely capturing the frames. I may look at making a V4L2 memory-to-memory device wrapping the ISP at a later date, but that is certainly not a priority.
2. Correct unless we go nuts on creating V4L2 memory-to-memory devices and wrap the encoder as well.
3. rawcam will remain supported as it is a useful tool in avoiding having to push everything into the kernel. (Fixing the double-free issue is on my list too).
4. If your source produces something useful as a format then 4.2 is the easiest, with 4.1 next. IL has many nasty edges and memory bandwidth inefficiencies compared to MMAL. I have had a request to get vc.ril.isp working under IL, so it may happen but will still be less efficient than MMAL zero copy.

I have the V4L2 driver working now (see viewtopic.php?f=38&t=120702&start=250#p1164774 if you want to help beta test) and I'm now working on integrating it with MMAL.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Interfacing (DSI, CSI, I2C, etc.)”