loopy83
Posts: 5
Joined: Fri May 10, 2019 6:34 am

raspiraw and other sensors (IMX332)

Fri May 10, 2019 7:00 am

Hello guys!

while trying to create a good platform to connect different types of MIPI sensors to a SOC platform (Jetson TX1, iMX6, ...) I found the raspiraw project for the RPI. I already had laying around a RPI B Rev.1.2, ordered a PI camera module and will give it a try.

The camera will arrive today, so I will spend the weekend to figure out if everything is fine.

After that my goal is to get a IMX332 running on the RPI.
https://www.sony-semicon.co.jp/products ... 171109.pdf

I have access to the datasheet and I already got the sensormodule running on another "framegrabber". So I already have a very basic I2C regsiter set that should work.
It only need 1.xV Core, 2.xV AVDD and a clock and thats it. So I will create a small board with two LDO to convert 3.3V PI voltage to 1.x/2.xV and I will use a normal Oscillator with 12MHz or sth like this to stimulate the sensor.
After that I expect the MIPI to be active and the rest (hopefully only configuration) is on SW side.

I already checked the source code of raspiraw and found two .h files including all the I2C communication and the basic sensor configuration.

I found this thread which is pretty much the same task - with another sensor.
viewtopic.php?f=43&t=225272

Did some of you guys already implemented other sensors by modifying the raspiraw code?
Was it almost straight forward, or were there many workarounds needed to get it running?

After I got this sensor running I'm trying to test with another chip and then I will try to implement the same on different on bigger platforms with 4-MIPI lanes. raspiraw will not work anymore for other platforms, but hopefully iMX6 or NV or Qualcomm have similar interfaces to the pipeline to bypass the processing and get raw data.
Do you think this will be possible, or did you already fail by trying the same?

Thank you very much!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6872
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: raspiraw and other sensors (IMX332)

Fri May 10, 2019 12:07 pm

Simply put yes you should be able to get other sensors up and running.

You've actually got register sets in there for 3 devices already:
- Omnivision OV5647 (the V1 Pi camera module)
- Sony IMX219 (the V2 Pi camera module)
- Analog Devices ADV7282-M (analogue video to CSI2 bridge chip).
There's also a related app using the same pieces but driving the Toshiba TC358743 HDMI to CSI2 bridge chip at at https://github.com/6by9/raspi_tc358743.

We've also used it for a couple of other sensor tests as well.
Amusingly it's also used internally by one of the big sensor manufacturers for some of their development work :D

Most sensors do have PLLs on the front end to adapt to many oscillator frequencies, but 9.6MHz, 19.2MHz, 24MHz, and 27MHz are the standard ones you tend to find. It's be worth checking with Sony for their recommendation.

The Raspberry Pi Compute Module (CM) exposes all 4 CSI2 data lanes on the CAM1 interface if you need to be running at those rates. Using the CM3L+ means you've got pretty much a Pi3 but you'll need to find a USB hub and USB LAN adapter.

raspiraw is useful for quick development, however as you say it is not cross platform.
The cross platform API should be V4L2, however various manufacturers use it in slightly different (incompatible) ways. IIRC imx6 is pretty much standard. NVidea is close to standard. Qualcomm do all sorts of odd things.

For V4L2 you would implement a subdevice driver for your sensor along the lines of https://github.com/raspberrypi/linux/bl ... c/imx258.c, and then tie it together to the bcm2835-unicam driver for the CSI2 receiver using device-tree, eg https://github.com/raspberrypi/linux/bl ... verlay.dts.
Please note that V4L2 will only be delivering the raw frames. Should you want any processing then you can look at https://github.com/6by9/yavta which takes the frames from V4L2 and passes them back into MMAL (the Pi multimedia API) for processing through the ISP. There is also ongoing work with http://libcamera.org/ to produce cross-platform complex camera pipelines.

The biggest niggle with kernel drivers is that they have to be licenced as GPLv2, and therefore you have an obligation to release the source code for it. Some sensor manufacturers can be awkward over releasing register sets, so that can present a problem. Sony generally seem to be OK over it, but speak to them early if you want to go this route.
On the Pi you do have the option of sticking with raspiraw which is licenced under a BSD 3-clause licence, therefore you can keep it closed. I don't know how feasible that would be on other platforms.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

loopy83
Posts: 5
Joined: Fri May 10, 2019 6:34 am

Re: raspiraw and other sensors (IMX332)

Mon May 13, 2019 7:36 am

Thank you very much Dave for your very interessting post!

The GPL license is a question I started to discuss internally a few weeks ago and it could be indeed a showstopper for this kind of concept since we're not interested in breaking our NDAs with the sensor manufacturer.

My PI Cam didnt arrive in time, so I was not able to give raspiraw a try over the weekend.
I will try to find all diffs between raspiraw and your tc358743 app. Maybe this will bring some light into my question, how many efforts it took to implement another sensor beside the officially supported ones.

For the compute model:
I already checked this one, but RPI does not have native 1G LAN or USB3.0 support, so bandwidth will always be limited by these devices. I just picked the RPI because of your ready to use tool as a kind of proof of concept.
The configuration on Jetson or iMX of the pipeline is a completely different story then, but it should work like your raspiraw work like a charm on the RPI platform - but maybe on V4L2.

RAW data is fine... I'm OK with it to debayer and processing all pixels on my host PC.

Thank you very much!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6872
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: raspiraw and other sensors (IMX332)

Mon May 13, 2019 9:46 am

loopy83 wrote:
Mon May 13, 2019 7:36 am
Thank you very much Dave for your very interessting post!

The GPL license is a question I started to discuss internally a few weeks ago and it could be indeed a showstopper for this kind of concept since we're not interested in breaking our NDAs with the sensor manufacturer.
Talk to the sensor manufacturer - some have no issue, or are happy as long as they have sanity checked the settings.
loopy83 wrote:My PI Cam didnt arrive in time, so I was not able to give raspiraw a try over the weekend.
I will try to find all diffs between raspiraw and your tc358743 app. Maybe this will bring some light into my question, how many efforts it took to implement another sensor beside the officially supported ones.
tc358743 is not the best for comparison as it is a fairly involved device (download EDID, detect source resolution, handle reconfiguration, etc). Better to just compare between the tables within raspiraw.
There are a couple of issues open with regard the exposure and gain setup functions - trying to make a parameterisable function for programming bits into a register is actually quite tricky!
loopy83 wrote:For the compute model:
I already checked this one, but RPI does not have native 1G LAN or USB3.0 support, so bandwidth will always be limited by these devices. I just picked the RPI because of your ready to use tool as a kind of proof of concept.
Fair enough if you're trying to shuffle data off elsewhere for processing.
loopy83 wrote:The configuration on Jetson or iMX of the pipeline is a completely different story then, but it should work like your raspiraw work like a charm on the RPI platform - but maybe on V4L2.
You'll have to do your own research on other platforms as I have no direct experience of them.
I believe imx.6 is ONLY V4L2 kernel drivers for sensors. They then use a (modified?) GStreamer v4l2src plugin to access it. Keeping register sets in userland is going to be messy.
The Jetson I have no information on. A quick search implies that they have gone down a proprietary path too - http://petermoran.org/csi-cameras-on-tx2/
In Nvidia’s “Get Started with the JetPack Camera API” they explain that the best way to interface with the Jetson’s multimedia hardware (including the ports for CSI cameras) is via their libargus C++ library or through gstreamer. Nvidia does not support the V4L2 video protocol for CSI cameras.
I thought I'd seen this all configured via device tree, in which case the sensor drivers may again have to be in the kernel.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Lee Jackson
Posts: 5
Joined: Mon May 06, 2019 12:52 pm

Re: raspiraw and other sensors (IMX332)

Tue May 14, 2019 2:55 pm

loopy83 wrote:
Mon May 13, 2019 7:36 am
The GPL license is a question I started to discuss internally a few weeks ago and it could be indeed a showstopper for this kind of concept since we're not interested in breaking our NDAs with the sensor manufacturer.
We have the same issues with GPL license ago if we keep using the V4L2 the kernel driver for the camera. But now we moved to the userland solution which is BSD 3-Clause license based on the Raspiraw.
loopy83 wrote:
Mon May 13, 2019 7:36 am
RAW data is fine... I'm OK with it to debayer and processing all pixels on my host PC.
If you just want to connect the mipi camera to PC via USB3 interface, you can have a look at our USB camera shield, which helps you connect any cameras via USB interface, and only need to wirte a configuration file for the sensor (almost the sensor register settings).
You can find more detail from: http://www.arducam.com/docs/usb-cameras/introduction/
Hope it is another alternate solution.

loopy83
Posts: 5
Joined: Fri May 10, 2019 6:34 am

Re: raspiraw and other sensors (IMX332)

Wed May 15, 2019 6:32 am

Hi Lee,
If you just want to connect the mipi camera to PC via USB3 interface, you can have a look at our USB camera shield, which helps you connect any cameras via USB interface, and only need to wirte a configuration file for the sensor (almost the sensor register settings).
You can find more detail from: http://www.arducam.com/docs/usb-cameras/introduction/
Hope it is another alternate solution.
This is defnitly worth a closer look! Thank you for sharing - I didnt found this one during my extensive google search!
Thank you very much! I will contact a distributor near my location...

loopy83
Posts: 5
Joined: Fri May 10, 2019 6:34 am

Re: raspiraw and other sensors (IMX332)

Wed May 15, 2019 11:46 am

@Lee:

Do you have some more documentation beside the git and website docs?

I try to understand the configuration of the board itself and the API.

But what is the meaning of these parts?
Is there a detailed description? Because I guess this is the main task when adding new sensors?

{
"camera_parameter" : {
"CFG_MODE":1, => RAW = 0. RGB = 1, YUV = 2, JGG = 3, MON = 4, RAW_D = 5, MON_D = 6
"TYPE":"MT9N001", => OK
"SIZE":[3488,2616], => OK
"BIT_WIDTH":8, => 10bit for RAW10?
"FORMAT":[0,3], => Pixel Pattern?
"I2C_MODE":3, => I2C 8/8 16/16 OK
"I2C_ADDR":"0x20", => OK
"TRANS_LVL":64
},
"board_parameter" : [], => ???
"board_parameter_dev2" : [], => ???
"board_parameter_dev3_inf2" : [], => ???
"board_parameter_dev3_inf3" : [], => ???
"register_parameter" : [], => normale I2C sensor config
"register_parameter_dev3_inf2" : [], => ???
"register_parameter_dev3_inf3" : [] => ???
}

Could you please help out?

Thank you very much!
Last edited by loopy83 on Wed May 15, 2019 11:50 am, edited 1 time in total.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6872
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: raspiraw and other sensors (IMX332)

Wed May 15, 2019 11:50 am

loopy83 wrote:
Wed May 15, 2019 11:46 am
@Lee:

Do you have some more documentation beside the git and website docs?

I try to understand the configuration of the board itself and the API.
Please take support requests for 3rd party products running on x86 hardware to a more appropriate location.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

loopy83
Posts: 5
Joined: Fri May 10, 2019 6:34 am

Re: raspiraw and other sensors (IMX332)

Wed May 15, 2019 11:51 am

Ok! Sorry!

Lee Jackson
Posts: 5
Joined: Mon May 06, 2019 12:52 pm

Re: raspiraw and other sensors (IMX332)

Wed May 15, 2019 1:54 pm

6by9 wrote:
Wed May 15, 2019 11:50 am
loopy83 wrote:
Wed May 15, 2019 11:46 am
@Lee:

Do you have some more documentation beside the git and website docs?

I try to understand the configuration of the board itself and the API.
Please take support requests for 3rd party products running on x86 hardware to a more appropriate location.
Yes, it is not a good place here to talk about the support for other platforms rather than Raspberry pi.
But please note that the USB camera shield is also supporting Raspberry pi platform, please check this one:
https://github.com/ArduCAM/ArduCAM_USB_ ... aspberryPi

Return to “Camera board”