6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9927
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Fri Jan 24, 2020 12:47 pm

No, we don't program the NVM on the IMX219.
Note too that that is a programmed DPC approach. The ISP has an automatic DPC block that looks for significant discrepancies of a pixel compared to those of the same colour surrounding it, and compensates for it should it be above a threshold. That's the sort of approach that can't be done on the sensor as it requires multiple lines of context.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

devmonkey
Posts: 16
Joined: Tue Jul 05, 2016 7:38 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jan 28, 2020 10:06 am

I've just moved my project onto a PI zero and run into lack of CPU. I only use the red pixels and currently have to copy them out of the bayer array which accounts for about 25% of my workload. On the OV5467 do you think I could use TIMING_X/Y_INC registers to just get it to read out red pixels? The sensor CFA is arranged:

Code: Select all

  	0	1	2	3
0:	B	G	B	G
1:	G	R	G	R
2:	B	G	B	G
3:	G	R	G	R
Red pixels are on odd rows in odd columns. The TIMING_X/Y_INC use upper nibble for odd inc, lower nibble for even inc. So if I ask for a sensor window that starts on an odd row/col, e.g. [1,1] the first pixel will be a red pixel. To sample odd columns and rows from this point on what do you think of setting TIMING_Y_INC=TIMING_X_INC=0x21?

Lower nibble (0x1) being irrelevant since we never land on an even row/column. If I dropped the sensor down to RAW8 and made the window width a multiple of 32 (to avoid padding) then I would have a ready made RED pixel array.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9927
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jan 28, 2020 10:42 am

devmonkey wrote:
Tue Jan 28, 2020 10:06 am
I've just moved my project onto a PI zero and run into lack of CPU. I only use the red pixels and currently have to copy them out of the bayer array which accounts for about 25% of my workload. On the OV5467 do you think I could use TIMING_X/Y_INC registers to just get it to read out red pixels? The sensor CFA is arranged:

Code: Select all

  	0	1	2	3
0:	B	G	B	G
1:	G	R	G	R
2:	B	G	B	G
3:	G	R	G	R
Red pixels are on odd rows in odd columns. The TIMING_X/Y_INC use upper nibble for odd inc, lower nibble for even inc. So if I ask for a sensor window that starts on an odd row/col, e.g. [1,1] the first pixel will be a red pixel. To sample odd columns and rows from this point on what do you think of setting TIMING_Y_INC=TIMING_X_INC=0x21?

Lower nibble (0x1) being irrelevant since we never land on an even row/column. If I dropped the sensor down to RAW8 and made the window width a multiple of 32 (to avoid padding) then I would have a ready made RED pixel array.
Sorry. I have to restate it again. https://github.com/6by9/raspiraw/blob/m ... odes.h#L30
// These register settings were as logged off the line
// by jbeale. There is a datasheet for OV5647 floating
// about on the internet, but the Pi Foundation/Trading have
// information from Omnivision under NDA, therefore
// we can not offer support on this.
// There is some information/discussion on the Freescale
// i.MX6 forums about supporting OV5647 on that board.
// There may be information available there that is of use.
//
// REQUESTS FOR SUPPORT ABOUT THESE REGISTER VALUES WILL
// BE IGNORED.
One thing you could do is to use the unpacking/repacking options of the receiver to convert to 16bit packing (data in the bottom 10 bits) rather than the weird SMIA 4 pixels in 5 bytes arrangement. That should save you a load of masking and bitshifting.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

devmonkey
Posts: 16
Joined: Tue Jul 05, 2016 7:38 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jan 28, 2020 3:20 pm

Ok no worries. I did a few tests and couldn't get the sensor to just send a single pixel per bayer quad, I don't think my (guessed) understanding TIMING_X/Y_INC is correct.

Anyway, good news I've solved my perf problem. My code rips out the red pixels from a 2592x480 rawcam bayer buffer into 1296x240 array, does a bunch of processing and then draws it and a load of stats onto the framebuffer 320x240, then fwrites that to /dev/fb1. It requests frames from rawcam at 50 fps.

I had a noticeable drop in the FPS I could process at after moving the code to a pi zero w. To write this code I did as others have and cloned raspiraw then changed it to my needs. I didn't change any of the options in the build script and having just looked GCC optimisations are off. Anyway bumping up the GCC optimisation fixed my performance problem, so anyone doing a lot of looping through pixel arrays using raspiraw based code and having an issue with speed check the -OX option in ./buildme.

I got 16 FPS with the default setting -O0 up to 48 FPS with -O2 or -Ofast, quite a massive improvement! This is on a PZW with wifi on and an ILI9341 SPI TFT 320x240 using the default raspbian fbtft device on fb1.

Here are the results of my particular app for different optimisation levels:

Code: Select all

-O0
FPS: 16.66
FPS: 16.67
FPS: 16.68
FPS: 16.68

-O1
FPS: 29.29
FPS: 33.76
FPS: 34.14
FPS: 29.97

-O2
FPS: 48.43
FPS: 47.85
FPS: 47.63

-O3
FPS: 45.84
FPS: 42.65
FPS: 45.45

-Ofast
FPS: 47.22
FPS: 48.44

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9927
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jan 28, 2020 3:42 pm

devmonkey wrote:
Tue Jan 28, 2020 3:20 pm
Ok no worries. I did a few tests and couldn't get the sensor to just send a single pixel per bayer quad, I don't think my (guessed) understanding TIMING_X/Y_INC is correct.
Whilst I can't directly help, I will say that you need to alter other registers to define the readout size, not just set the _INC registers.
Modes 6 & 7 do use _INC to implement pixel skipping, although with odd values rather than you even ones. As a test you could alter those and see if the sensor will actually stream normally before looking at amending the full res mode.
devmonkey wrote:I had a noticeable drop in the FPS I could process at after moving the code to a pi zero w. To write this code I did as others have and cloned raspiraw then changed it to my needs. I didn't change any of the options in the build script and having just looked GCC optimisations are off. Anyway bumping up the GCC optimisation fixed my performance problem, so anyone doing a lot of looping through pixel arrays using raspiraw based code and having an issue with speed check the -OX option in ./buildme.

I got 16 FPS with the default setting -O0 up to 48 FPS with -O2 or -Ofast, quite a massive improvement! This is on a PZW with wifi on and an ILI9341 SPI TFT 320x240 using the default raspbian fbtft device on fb1.
I use raspiraw for lower level debugging, therefore disabling optimisations means gdb can actually do something useful instead of reporting "optimised out" for all the variables you're interested in! Yes, enabling them will gain some performance (a fair amount from your report).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

rego21
Posts: 30
Joined: Fri Feb 16, 2018 4:09 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Sun Feb 09, 2020 11:34 am

lak4cyut wrote:
Wed Aug 21, 2019 9:30 am
6by9 wrote:
Thu May 30, 2019 9:46 am
The main ones:
- Licence. Any kernel driver almost has to be GPLv2 and therefore open source. Check with your sensor supplier that they are happy for the register set to be released. Copying raspiraw means that the register set can be hidden in your userspace app which can be under any licence you fancy (including closed). It's all a little silly as it is trivial to put an I2C analyser on the lines to the camera module, run your app, and capture the relevant commands.
- Framework. V4L2 is the Linux standard APIs and is therefore portable to other platforms. MMAL is specific to the Pi.
- Support. I'm less inclined to fix any issues discovered in rawcam than in V4L2. V4L2 is seen as the way forward.
rcasiodu wrote:In the first method, can you give me an example of drive file? Could I just modify the ov5647.c(/drivers/media/i2c/ov5647.c) to other sensor?
/drivers/media/i2c/ov5647.c is a relatively basic driver, but would work. imx258.c is a little more comprehensive, but lacks the dt/fwnode configuration and regulator control (not essential if you set the power control GPIO some other way). I believe ov5640.c is a reasonable example of that.
rcasiodu wrote:How to use the v4l2 driver to get raw data from sensor? Is it possible to transfer raw10/raw12 to yuv data?(use arm core?)
You use the V4L2 API to get raw frames out.
"v4l2-ctl --stream--map=3 --stream-to=foo.raw --stream-count=100" would be a simple existing tool.
https://linuxtv.org/downloads/v4l-dvb-a ... ure.c.html is the standard example for grabbing frames. Do what you want in process_image(). You want to be using the MMAP method, not read (userptr isn't supported).
Hi 6by9,
Thanks for your information.
I built a custom kernel to enable bcm2835-unicam and ov5647 kernel modules and add ov5647 dt overlay, try to capture camera image via V4L2 official interface. (Of course I bought a Raspberry Pi official camera module - OV5647 version.)
And use v4l2-ctl to capture image.

Code: Select all

v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=frame.raw --stream-count=1
But the captured image is abnormal (overlay & distortion) (I put the image below).
Would you like to provide me some hint about how to debug this problem? Or do you know what caused it.
It drive me crazy...

Thanks for the help.

frame.jpg
frame2.jpg

Detail info:
Kernel base version: 4.19.64
Raspberry Pi: 3 model B v1.2
Camera Module: Raspbery Pi (OV5647) Rev 1.3
Hii,

Im also trying to enable bcm2835-unicam, can you explain me what are the required steps? I read there is only needed to change the DT.

Thank you!

rcasiodu
Posts: 17
Joined: Tue May 28, 2019 2:28 am

Re: Raw sensor access / CSI-2 receiver peripheral

Wed Apr 08, 2020 12:45 am

Hi 6By9,

I want to connect mono sensor, is it correct to add the following codes

Code: Select all

	/*isp module*/
	status = mmal_port_parameter_set_uint32(isp->input[0], MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE, ~(1<<11));
    if (status != MMAL_SUCCESS)
    {
       vcos_log_error("Could not set block override - update your firmware : error %d", status);
    }	
after isp cpmponent is created in raspiraw.c?

Code: Select all

	status = mmal_component_create("vc.ril.isp", &isp);
	if (status != MMAL_SUCCESS)
	{
		vcos_log_error("Failed to create isp");
		goto component_destroy;
	}
	status = mmal_port_parameter_set_boolean(isp->input[0], MMAL_PARAMETER_ZERO_COPY, MMAL_TRUE);
	if (status != MMAL_SUCCESS)
	{
		vcos_log_error("Failed to set zero copy");
		goto component_disable;
	}
	status = mmal_port_parameter_set_boolean(isp->output[0], MMAL_PARAMETER_ZERO_COPY, MMAL_TRUE);
	if (status != MMAL_SUCCESS)
	{
		vcos_log_error("Failed to set zero copy");
		goto component_disable;
	}
How to get the output data after isp process, if i use -processing option? What is the data format of isp output?

I think when i use the -processing_yuv option to output yuv image, the isp is also work? Am i correct? Can i use this option with mono sensor?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9927
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Wed Apr 08, 2020 10:52 am

rcasiodu wrote:
Wed Apr 08, 2020 12:45 am
I want to connect mono sensor, is it correct to add the following codes

Code: Select all

	/*isp module*/
	status = mmal_port_parameter_set_uint32(isp->input[0], MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE, ~(1<<11));
    if (status != MMAL_SUCCESS)
    {
       vcos_log_error("Could not set block override - update your firmware : error %d", status);
    }	
after isp cpmponent is created in raspiraw.c?
Yes, bit 11 should disable demosaicing. See https://www.raspberrypi.org/forums/view ... 5#p1388502
rcasiodu wrote:

Code: Select all

	status = mmal_component_create("vc.ril.isp", &isp);
	if (status != MMAL_SUCCESS)
	{
		vcos_log_error("Failed to create isp");
		goto component_destroy;
	}
	status = mmal_port_parameter_set_boolean(isp->input[0], MMAL_PARAMETER_ZERO_COPY, MMAL_TRUE);
	if (status != MMAL_SUCCESS)
	{
		vcos_log_error("Failed to set zero copy");
		goto component_disable;
	}
	status = mmal_port_parameter_set_boolean(isp->output[0], MMAL_PARAMETER_ZERO_COPY, MMAL_TRUE);
	if (status != MMAL_SUCCESS)
	{
		vcos_log_error("Failed to set zero copy");
		goto component_disable;
	}
How to get the output data after isp process, if i use -processing option? What is the data format of isp output?

I think when i use the -processing_yuv option to output yuv image, the isp is also work? Am i correct? Can i use this option with mono sensor?
The pipeline (if I've got it right) is:

Code: Select all

rawcam -+-> save to file (-o filename) 
        +-> awb thread (-awb)
        +-> processing thread
        +-> isp -+-> save to file (-oY filename)
                 +-> processing_yuv thread
                 +-> video_render
Yes, you can use the ISP with a mono sensor. Even with demosaicing on you won't notice much difference in the image (unless you apply some AWB gains). You do need to program all channels equally in things like lens shading and black level to avoid odd effects.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

rcasiodu
Posts: 17
Joined: Tue May 28, 2019 2:28 am

Re: Raw sensor access / CSI-2 receiver peripheral

Thu Apr 09, 2020 3:57 am

Thanks 6by9!

What's the dev->isp_ip means in the callback function?

Code: Select all

		if (dev->isp_ip)
		{
			MMAL_BUFFER_HEADER_T *out = mmal_queue_get(dev->isp_ip_pool->queue);
			if (out)
			{
				//vcos_log_error("replicate buffer %p for isp", buffer);
				mmal_buffer_header_replicate(out, buffer);
				out->data = buffer->data;
				out->alloc_size = buffer->alloc_size;
				status = mmal_port_send_buffer(dev->isp_ip, out);
				if ( status != MMAL_SUCCESS)
					vcos_log_error("Failed to send buffer %p to isp - %d", buffer, status);
			}
		}
Last edited by rcasiodu on Thu Apr 09, 2020 7:09 am, edited 1 time in total.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9927
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Thu Apr 09, 2020 6:38 am

rcasiodu wrote:
Thu Apr 09, 2020 3:57 am
Thanks 6by9!

The isp is enabled by default in the callback function?

Code: Select all

		if (dev->isp_ip)
		{
			MMAL_BUFFER_HEADER_T *out = mmal_queue_get(dev->isp_ip_pool->queue);
			if (out)
			{
				//vcos_log_error("replicate buffer %p for isp", buffer);
				mmal_buffer_header_replicate(out, buffer);
				out->data = buffer->data;
				out->alloc_size = buffer->alloc_size;
				status = mmal_port_send_buffer(dev->isp_ip, out);
				if ( status != MMAL_SUCCESS)
					vcos_log_error("Failed to send buffer %p to isp - %d", buffer, status);
			}
		}
The ISP is enabled via the calls to mmal_port_enabled at https://github.com/6by9/raspiraw/blob/m ... aw.c#L2071 and https://github.com/6by9/raspiraw/blob/m ... aw.c#L2115, plus the mmal_component_enable at https://github.com/6by9/raspiraw/blob/m ... aw.c#L1847
The section you've quoted is the passing of the buffer from rawcam to the ISP if it is enabled.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9927
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Thu Apr 09, 2020 6:41 am

rego21 wrote:
Sun Feb 09, 2020 11:34 am
Hii,

Im also trying to enable bcm2835-unicam, can you explain me what are the required steps? I read there is only needed to change the DT.

Thank you!
Sorry, I'd missed this one. This isn't really the right thread for discussing the V4L2 driver. Have a look at the existing ov5647 or imx219 DT overlays for examples. It only requires DT changes if there is already a suitable sensor driver for your chosen sensor.

I have been given the job of writing up some documentation for the CSI receiver, so I'll get on with that instead.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

rcasiodu
Posts: 17
Joined: Tue May 28, 2019 2:28 am

Re: Raw sensor access / CSI-2 receiver peripheral

Thu Apr 09, 2020 7:16 am

Thanks! Your answer is really helpful. Looking forwards to the document.
6by9 wrote:
Thu Apr 09, 2020 6:41 am
rego21 wrote:
Sun Feb 09, 2020 11:34 am
Hii,

Im also trying to enable bcm2835-unicam, can you explain me what are the required steps? I read there is only needed to change the DT.

Thank you!
Sorry, I'd missed this one. This isn't really the right thread for discussing the V4L2 driver. Have a look at the existing ov5647 or imx219 DT overlays for examples. It only requires DT changes if there is already a suitable sensor driver for your chosen sensor.

I have been given the job of writing up some documentation for the CSI receiver, so I'll get on with that instead.

User avatar
HermannSW
Posts: 3198
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Raw sensor access / CSI-2 receiver peripheral

Tue May 05, 2020 3:12 pm

Good news from 6by9 in another thread, there will be HQ camera (imx477) support in raspiraw:
viewtopic.php?f=43&t=272926&p=1655519#p1655498
6by9 wrote:
Tue May 05, 2020 12:38 pm
HermannSW wrote:
Tue May 05, 2020 12:08 pm
...
Not true. We have permission from Sony to release an imx477 kernel driver and associated register sets, and that can be transferred to raspiraw (I have it somewhere already).
What I am not at liberty to do is to discuss how the register set "works" nor how it can be adapted.
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/working_with_FPGAs

User avatar
HermannSW
Posts: 3198
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Raw sensor access / CSI-2 receiver peripheral

Mon May 11, 2020 9:45 am

HermannSW wrote:
Tue May 05, 2020 3:12 pm
6by9 wrote:
Tue May 05, 2020 12:38 pm
Not true. We have permission from Sony to release an imx477 kernel driver and associated register sets, and that can be transferred to raspiraw (I have it somewhere already).
Is there an ETA for "release an imx477 kernel driver and associated register sets"?
Is that a precondition for updating raspiraw to support imx477 ("I have it somewhere already")?
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/working_with_FPGAs

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9927
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Mon May 11, 2020 10:04 am

HermannSW wrote:
Mon May 11, 2020 9:45 am
HermannSW wrote:
Tue May 05, 2020 3:12 pm
6by9 wrote:
Tue May 05, 2020 12:38 pm
Not true. We have permission from Sony to release an imx477 kernel driver and associated register sets, and that can be transferred to raspiraw (I have it somewhere already).
Is there an ETA for "release an imx477 kernel driver and associated register sets"?
Is that a precondition for updating raspiraw to support imx477 ("I have it somewhere already")?
https://github.com/raspberrypi/linux/pull/3605
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
HermannSW
Posts: 3198
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Raw sensor access / CSI-2 receiver peripheral

Mon May 11, 2020 1:52 pm

Cool -- when will imx477 be supported here?
https://github.com/6by9/raspiraw
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/working_with_FPGAs

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9927
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Mon May 11, 2020 3:18 pm

HermannSW wrote:
Mon May 11, 2020 1:52 pm
Cool -- when will imx477 be supported here?
https://github.com/6by9/raspiraw
When I get around to it! I've had a look for the header I did have, and I can't find it quickly. It'll probably be quicker to recreate.

Or someone else can take the kernel driver, create their own version of the header, and make a PR.

Oh, blow it, it'll be quicker to write than to review a PR.....
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9927
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Mon May 11, 2020 3:31 pm

Done - https://github.com/6by9/raspiraw/commit ... a5bccd171a
Untested though, particularly on the exposure/gain/flip/vts setup.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mcuciuc
Posts: 3
Joined: Mon May 11, 2020 9:53 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Mon May 11, 2020 10:02 pm

6by9 wrote:
Mon May 11, 2020 3:31 pm
Done - https://github.com/6by9/raspiraw/commit ... a5bccd171a
Untested though, particularly on the exposure/gain/flip/vts setup.
Awesome, thanks for the quick update!
I've had to change i2c_ident_value in imx477_modes.h to 0x7704, but in the callbacks I get all buffers returning with "filled 0".

naushir
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 69
Joined: Mon Apr 25, 2016 10:21 am

Re: Raw sensor access / CSI-2 receiver peripheral

Tue May 12, 2020 3:03 pm

mcuciuc wrote:
Mon May 11, 2020 10:02 pm
6by9 wrote:
Mon May 11, 2020 3:31 pm
Done - https://github.com/6by9/raspiraw/commit ... a5bccd171a
Untested though, particularly on the exposure/gain/flip/vts setup.
Awesome, thanks for the quick update!
I've had to change i2c_ident_value in imx477_modes.h to 0x7704, but in the callbacks I get all buffers returning with "filled 0".
If you are feeling adventurous, you could try getting RAW (.dng) images from the HQ camera through the new libcamera stack. You get a full viewfinder with AWB/AGC running (if you wish) as well.

Pull and compile the 5.4 kernel from https://github.com/raspberrypi/linux/tree/rpi-5.4.y (this now includes the HQ cam driver).
Follow the build instructions at https://github.com/raspberrypi/document ... /README.md.

Once libcamera is built, run:

Code: Select all

build/src/qcam/qcam -s role=viewfinder -s role=stillraw 
A viewfinder should pop up and you have a button to save a dng file.

mcuciuc
Posts: 3
Joined: Mon May 11, 2020 9:53 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Sat May 16, 2020 12:19 pm

naushir wrote:
Tue May 12, 2020 3:03 pm
Once libcamera is built, run:

Code: Select all

build/src/qcam/qcam -s role=viewfinder -s role=stillraw 
A viewfinder should pop up and you have a button to save a dng file.
Thanks for the suggestion, this looks like a much better approach to what I need it for :D (grabbing as quickly as possible long exposure raw files with user-specified analog gain). I ran into some issues though and I could only get it working on a Raspberry Pi 3 B+ with the IMX219 and no HDMI connected. In that case

Code: Select all

./libcamera/build/src/cam/cam -c 1 -s role=viewfinder -s role=stillraw -C -F 
happily produces two streams.

However, if I connect the HDMI at boot time I get

Code: Select all

[0:06:54.086648216] [1147]  INFO RPI raspberrypi.cpp:617 Sensor: imx219 - Selected mode: 3280x2464-pBAA
[0:06:54.090729461] [1147]  INFO RPI_S_W staggered_ctrl.h:58 Init ctrl 0x00980911 with delay 2
[0:06:54.090895398] [1147]  INFO RPI_S_W staggered_ctrl.h:58 Init ctrl 0x009e0903 with delay 1
[0:06:54.258244098] [1147] ERROR V4L2 v4l2_videodevice.cpp:1098 /dev/video15[cap]: Not enough buffers provided by V4L2VideoDevice
[0:06:54.262420499] [1147] ERROR RPI raspberrypi.cpp:761 Failed to allocate buffers
Failed to start capture

Switching to the IMX477 I get a different error

Code: Select all

[0:00:47.077956856] [565]  INFO RPI raspberrypi.cpp:617 Sensor: imx477 - Selected mode: 4056x3040-pBCC
[0:00:47.081586596] [565]  INFO RPI_S_W staggered_ctrl.h:58 Init ctrl 0x00980911 with delay 2
[0:00:47.081735137] [565]  INFO RPI_S_W staggered_ctrl.h:58 Init ctrl 0x009e0903 with delay 1
[0:00:47.253111960] [565] ERROR V4L2 v4l2_videodevice.cpp:1091 /dev/video0[cap]: Unable to request 4 buffers: Cannot allocate memory
[0:00:47.253620450] [565] ERROR RPI raspberrypi.cpp:761 Failed to allocate buffers
Failed to start capture

On the Raspberry Pi 4 B I couldn't get any of the sensors to show up on "cam -l"

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9927
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Sat May 16, 2020 12:45 pm

This really ought to be on a libcamera thread.
mcuciuc wrote:
Sat May 16, 2020 12:19 pm
Thanks for the suggestion, this looks like a much better approach to what I need it for :D (grabbing as quickly as possible long exposure raw files with user-specified analog gain). I ran into some issues though and I could only get it working on a Raspberry Pi 3 B+ with the IMX219 and no HDMI connected. In that case

Code: Select all

./libcamera/build/src/cam/cam -c 1 -s role=viewfinder -s role=stillraw -C -F 
happily produces two streams.

However, if I connect the HDMI at boot time I get

Code: Select all

[0:06:54.086648216] [1147]  INFO RPI raspberrypi.cpp:617 Sensor: imx219 - Selected mode: 3280x2464-pBAA
[0:06:54.090729461] [1147]  INFO RPI_S_W staggered_ctrl.h:58 Init ctrl 0x00980911 with delay 2
[0:06:54.090895398] [1147]  INFO RPI_S_W staggered_ctrl.h:58 Init ctrl 0x009e0903 with delay 1
[0:06:54.258244098] [1147] ERROR V4L2 v4l2_videodevice.cpp:1098 /dev/video15[cap]: Not enough buffers provided by V4L2VideoDevice
[0:06:54.262420499] [1147] ERROR RPI raspberrypi.cpp:761 Failed to allocate buffers
Failed to start capture

Switching to the IMX477 I get a different error

Code: Select all

[0:00:47.077956856] [565]  INFO RPI raspberrypi.cpp:617 Sensor: imx477 - Selected mode: 4056x3040-pBCC
[0:00:47.081586596] [565]  INFO RPI_S_W staggered_ctrl.h:58 Init ctrl 0x00980911 with delay 2
[0:00:47.081735137] [565]  INFO RPI_S_W staggered_ctrl.h:58 Init ctrl 0x009e0903 with delay 1
[0:00:47.253111960] [565] ERROR V4L2 v4l2_videodevice.cpp:1091 /dev/video0[cap]: Unable to request 4 buffers: Cannot allocate memory
[0:00:47.253620450] [565] ERROR RPI raspberrypi.cpp:761 Failed to allocate buffers
Failed to start capture

On the Raspberry Pi 4 B I couldn't get any of the sensors to show up on "cam -l"
It sounds like you have insufficient memory assigned to the CMA heap. Default is only 64MB and image buffers are frequently large (10MB for 8MPix off IMX219, or 18MB for 12MPix from IMX477).
Two ways to increase it.
1) Add "dtoverlay=vc4-fkms-v3d" to /boot/config.txt which enables the updated display stack, and also increases the heap to 256MB by default
2) Add "cma=256M" to /boot/cmdline.txt (do NOT add any carriage returns). This has the downside of renaming the heap from linux,cma to reserved, but that shouldn't matter for your current use case.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mcuciuc
Posts: 3
Joined: Mon May 11, 2020 9:53 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Sat May 16, 2020 2:56 pm

6by9 wrote:
Sat May 16, 2020 12:45 pm
This really ought to be on a libcamera thread.

It sounds like you have insufficient memory assigned to the CMA heap. Default is only 64MB and image buffers are frequently large (10MB for 8MPix off IMX219, or 18MB for 12MPix from IMX477).
Two ways to increase it.
1) Add "dtoverlay=vc4-fkms-v3d" to /boot/config.txt which enables the updated display stack, and also increases the heap to 256MB by default
2) Add "cma=256M" to /boot/cmdline.txt (do NOT add any carriage returns). This has the downside of renaming the heap from linux,cma to reserved, but that shouldn't matter for your current use case.
Thanks! Adding "dtoverlay=vc4-fkms-v3d" to /boot/config.txt made libcamera work with the IMX477, irrespective of HDMI being connected or not.

I'll fumble with libcamera for a while and will post any further questions on a libcamera thread.

ayilm1
Posts: 4
Joined: Wed May 20, 2020 2:33 am

Re: Raw sensor access / CSI-2 receiver peripheral

Wed May 20, 2020 2:58 am

Hi 6by9

Thanks so much for publishing this work. Like a few other posters here I'm wanting to use this to facilitate some basic verification tests on an FPGA CSI-2 source. My source has no I2C interface and as such has one static configuration. It's also configured for YUV422 8-bit format, however the data is just internally generated in the FPGA (I have two test patterns that can be selected with a DIP switch), and as such I'm really just interested in verifying that the correct bytes are being received. I will also be initially testing with an OV7251 camera, which is due to arrive soon, so I have a known good source to verify that the software works. The OV7251 has a lot of similarities to my FPGA source; almost identical resolution (FPGA is slightly taller) and single MIPI lane. The only difference is the default data type (without I2C configuration) is RAW8 instead of YUV422 8b; both are still 8b payloads. However since I'm only interested in validating that I get the correct data in the output file, I don't think this matters - I just need to make sure the image_id is set correctly (0x2A/0x1E respectively) so the incoming data is matched to the correct buffer.

I'm therefore wondering, what is the minimum scope of changes I would need to make to your code to facilitate outputting this byte data to a file? I want to ignore the ISP and rendering functionality your code seems to implement. I've gone through raspiraw.c and have attempted to make some changes but as I venture toward the MMAL-specific callbacks I'm starting to lose track of how the buffers are being passed around. For the initial rx config, I've hamfistedly just hard-coded all the parameters I care about for no reason other than it seemed less effort compared to defining a modes.h for a generic I2C-less device (basically everything using modReg/update_regs has consequently been commented out).

Code: Select all

rx_cfg.unpack = MMAL_CAMERA_RX_CONFIG_UNPACK_NONE;
rx_cfg.pack = MMAL_CAMERA_RX_CONFIG_PACK_NONE;
	
// only ever want to use 1 lane for our sensor
rx_cfg.data_lanes = 1;
	
// Just hard-coding RAW8 for the Arducam. This will be YUV422-8 for FPGA
rx_cfg.image_id = 0x2A;	
The output port config is butchered in a similar fashion.

Code: Select all

#STATIC_WIDTH	640
#STATIC_HEIGHT	480
	
output->format->es->video.crop.width = STATIC_WIDTH;
output->format->es->video.crop.height = STATIC_HEIGHT;
output->format->es->video.width = VCOS_ALIGN_UP(STATIC_WIDTH, 16);
output->format->es->video.height = VCOS_ALIGN_UP(STATIC_HEIGHT, 16);
output->format->encoding = 0;
The more modifications I make, the more I get the feeling that this is going to become a nightmare to debug when (rather than if) I get nothing in the output file. This is all quite far out of my domain and I'm just looking for the simplest way of verifying the data that Pi is receiving from the FPGA, so any insight and guidance you can provide in how to achieve this is very much appreciated.

Thanks in advance.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9927
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Wed May 20, 2020 11:02 am

ayilm1 wrote:
Wed May 20, 2020 2:58 am
Hi 6by9

Thanks so much for publishing this work. Like a few other posters here I'm wanting to use this to facilitate some basic verification tests on an FPGA CSI-2 source. My source has no I2C interface and as such has one static configuration. It's also configured for YUV422 8-bit format, however the data is just internally generated in the FPGA (I have two test patterns that can be selected with a DIP switch), and as such I'm really just interested in verifying that the correct bytes are being received. I will also be initially testing with an OV7251 camera, which is due to arrive soon, so I have a known good source to verify that the software works. The OV7251 has a lot of similarities to my FPGA source; almost identical resolution (FPGA is slightly taller) and single MIPI lane. The only difference is the default data type (without I2C configuration) is RAW8 instead of YUV422 8b; both are still 8b payloads. However since I'm only interested in validating that I get the correct data in the output file, I don't think this matters - I just need to make sure the image_id is set correctly (0x2A/0x1E respectively) so the incoming data is matched to the correct buffer.

I'm therefore wondering, what is the minimum scope of changes I would need to make to your code to facilitate outputting this byte data to a file? I want to ignore the ISP and rendering functionality your code seems to implement. I've gone through raspiraw.c and have attempted to make some changes but as I venture toward the MMAL-specific callbacks I'm starting to lose track of how the buffers are being passed around. For the initial rx config, I've hamfistedly just hard-coded all the parameters I care about for no reason other than it seemed less effort compared to defining a modes.h for a generic I2C-less device (basically everything using modReg/update_regs has consequently been commented out).
<snip>
The more modifications I make, the more I get the feeling that this is going to become a nightmare to debug when (rather than if) I get nothing in the output file. This is all quite far out of my domain and I'm just looking for the simplest way of verifying the data that Pi is receiving from the FPGA, so any insight and guidance you can provide in how to achieve this is very much appreciated.

Thanks in advance.
My recommendation is to use the bcm2835-unicam driver instead of MMAL rawcam as at least then you can add additional logging into the driver.
There's already a driver for ov7251 from mainline in the 5.4 release, so you in theory only need to build it and create a device tree overlay for it.
(There is the potential for the clock configuration to differ between what the driver is expecting vs that implemented on your board - life can get a little tricky then if you don't have a full datasheet for the sensor)

Having just discussed it with a colleague, it should be possible to create a dummy sensor driver that provides the relevant hooks expected by the bcm2835-unicam driver, and reads a width, height, and pixel format from device tree. That would be usable by anyone adding an uncontrolled CSI2 source to the Pi. I have other things to look at today, but will see where I get to over implementing it.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”