therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 49
Joined: Tue Jan 07, 2020 9:15 am

Re: STICKY: libcamera - open source camera stack

Wed May 06, 2020 7:35 pm

I mean I want to re-use the algorithms supplied by RPi within a custom code since libcamera cannot replay captured files. I wonder if it is easy to implement and if the raw data file is easy to format so it can be processed. I guess I can find the response by myself in the source code but it is still unavailable for me on github :?
Hopefully you've got the code now. But note that whilst you can re-use the algorithms supplied by Raspberry Pi, these are algorithms that control the hardware ISP (for instance calculating parameters for white balance or lens shading correction), not algorithms that actually perform these functions (including also de-Bayering), which I'm getting the impression is what you might be wanting.
Sorry for this question but where can I speak to them directly?
Here, it would seem, if Laurent answers your question! But there are also some contact options on this page: https://libcamera.org/contributing.html

had
Posts: 13
Joined: Wed Mar 11, 2020 2:34 pm

Re: STICKY: libcamera - open source camera stack

Wed May 06, 2020 7:37 pm

A reprocessing API is on our roadmap, and will allow pushing captured raw Bayer images through the ISP. I'm afraid I can't provide an exact date for when this feature will be available.
Great! I totally understand you cannot provide an exact date, but maybe you have an approximate date in mind? :D

By the way, thanks everybody for all your responses and amazing work!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8675
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: STICKY: libcamera - open source camera stack

Wed May 06, 2020 7:42 pm

Apologies for the duff URL - I'd missed that our github repo was private. All posts referencing it have been corrected.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

pinchartl
Posts: 5
Joined: Wed May 06, 2020 6:56 pm

Re: STICKY: libcamera - open source camera stack

Wed May 06, 2020 7:42 pm

had wrote:
Wed May 06, 2020 7:37 pm
A reprocessing API is on our roadmap, and will allow pushing captured raw Bayer images through the ISP. I'm afraid I can't provide an exact date for when this feature will be available.
Great! I totally understand you cannot provide an exact date, but maybe you have an approximate date in mind? :D
Ballpark estimate, I would say between 3 and 6 months. We have lots of work on our plate, which is both exciting and scary at the same time :-)
By the way, thanks everybody for all your responses and amazing work!
You're welcome. We've worked hard to make this happen, your thanks are very appreciated.

User avatar
schoolpost
Posts: 32
Joined: Sun Feb 19, 2017 10:47 am
Location: Canada
Contact: Website

Re: STICKY: libcamera - open source camera stack

Fri May 08, 2020 9:17 pm

There was a mention of python bindings? When would these be available or are they already?

I'm slowly working myself into the world of C/C++ programming, but in the mean time would love to see a way to use this with python in some way! :)

trejan
Posts: 1828
Joined: Tue Jul 02, 2019 2:28 pm

Re: STICKY: libcamera - open source camera stack

Fri May 08, 2020 9:56 pm

schoolpost wrote:
Fri May 08, 2020 9:17 pm
There was a mention of python bindings? When would these be available or are they already?
viewtopic.php?f=43&t=273018&start=25#p1656319

dan s
Posts: 73
Joined: Tue Feb 18, 2020 10:42 pm

Re: STICKY: libcamera - open source camera stack

Sat May 09, 2020 6:32 am

pinchartl wrote:
Wed May 06, 2020 7:16 pm
Just to clarify, libcamera doesn't have Python bindings *yet*. This is something we're considering, but whether to develop them in the near future or wait for the libcamera public API to stabilize first hasn't been decided.
My vote would be sooner rather than later. In my opinion, you will get a broad spectrum of people using it faster if bindings are available for a high-level language such as Python.

had
Posts: 13
Joined: Wed Mar 11, 2020 2:34 pm

Re: STICKY: libcamera - open source camera stack

Mon May 11, 2020 12:16 pm

Hello,

According to https://github.com/raspberrypi/document ... /README.md, the i2c interface must be enabled, I guess i2c-1 is in use by libcamera, but is it still possible to communicate with other devices through i2c-1 while using libcamera ? What is the IMX477 i2c address ?

Thanks

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8675
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: STICKY: libcamera - open source camera stack

Mon May 11, 2020 12:20 pm

had wrote:
Mon May 11, 2020 12:16 pm
Hello,

According to https://github.com/raspberrypi/document ... /README.md, the i2c interface must be enabled, I guess i2c-1 is in use by libcamera, but is it still possible to communicate with other devices through i2c-1 while using libcamera ? What is the IMX477 i2c address ?

Thanks
No, i2c-10 is BSC0 pinmuxed to the camera and display connectors. (i2c-0 is BSC0 muxed to GPIOs 0&1)
Enabling the ov5647/imx219/imx477 overlay will also enable that.
Userspace doesn't directly access I2C for libcamera, therefore i2c-dev is not required either.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

kbingham
Posts: 6
Joined: Wed Mar 06, 2019 11:20 am

Re: STICKY: libcamera - open source camera stack

Mon May 11, 2020 12:25 pm

had wrote:
Mon May 11, 2020 12:16 pm
Hello,

According to https://github.com/raspberrypi/document ... /README.md, the i2c interface must be enabled, I guess i2c-1 is in use by libcamera, but is it still possible to communicate with other devices through i2c-1 while using libcamera ? What is the IMX477 i2c address ?

Thanks
libcamera does not use the i2c bus itself, it just talks to the kernel drivers using the V4L2 interfaces. The Linux kernel will have a driver (I don't think it's yet released) for the IMX477 HQ Camera, which will use the I2C address of the camera. Once that module is loaded, then the kernel will be controlling the device and you will not (directly) be able to access it without going through the driver.

If other devices are on the I2C-1 bus, then they should be accessible as with any other I2C device following the Linux kernel I2C framework.

fooforever
Posts: 26
Joined: Wed Aug 15, 2012 11:21 pm

Re: STICKY: libcamera - open source camera stack

Wed May 13, 2020 3:04 pm

This might be anecdotal at the moment but I thought I'd ask before I started writing more test applications: Is using libcamera likely to be higher performance for reading raw data off the sensor?
Currently I can get around 5fps using the modified raspiraw as a library in a C++ application.

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 49
Joined: Tue Jan 07, 2020 9:15 am

Re: STICKY: libcamera - open source camera stack

Wed May 13, 2020 5:06 pm

This might be anecdotal at the moment but I thought I'd ask before I started writing more test applications: Is using libcamera likely to be higher performance for reading raw data off the sensor?
Sorry, but no. The bottleneck is to do with getting the pixels out of the sensor and across the CSI-2 bus.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8675
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: STICKY: libcamera - open source camera stack

Wed May 13, 2020 5:18 pm

therealdavidp wrote:
Wed May 13, 2020 5:06 pm
This might be anecdotal at the moment but I thought I'd ask before I started writing more test applications: Is using libcamera likely to be higher performance for reading raw data off the sensor?
Sorry, but no. The bottleneck is to do with getting the pixels out of the sensor and across the CSI-2 bus.
Although you don't say which sensor or mode you are referring to only giving 5fps.
IMX477 can deliver 12MPix @ 10fps. IMX219 can do 8MPix @ 20fps (datasheet says 21fps on 2 lanes). You're hitting the ISP bandwidth limits with these though.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

fooforever
Posts: 26
Joined: Wed Aug 15, 2012 11:21 pm

Re: STICKY: libcamera - open source camera stack

Thu May 14, 2020 10:27 am

6by9 wrote:
Wed May 13, 2020 5:18 pm
therealdavidp wrote:
Wed May 13, 2020 5:06 pm
This might be anecdotal at the moment but I thought I'd ask before I started writing more test applications: Is using libcamera likely to be higher performance for reading raw data off the sensor?
Sorry, but no. The bottleneck is to do with getting the pixels out of the sensor and across the CSI-2 bus.
Although you don't say which sensor or mode you are referring to only giving 5fps.
IMX477 can deliver 12MPix @ 10fps. IMX219 can do 8MPix @ 20fps (datasheet says 21fps on 2 lanes). You're hitting the ISP bandwidth limits with these though.
I'm using mode 2 on the V1 camera. The docs seem to imply I should be able to get up to 15fps out of it , but I've never been able to when needing the raw bayer data.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8675
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: STICKY: libcamera - open source camera stack

Thu May 14, 2020 10:42 am

fooforever wrote:
Thu May 14, 2020 10:27 am
I'm using mode 2 on the V1 camera. The docs seem to imply I should be able to get up to 15fps out of it , but I've never been able to when needing the raw bayer data.
So how have you been asking for it? raspiraw, V4L2, or raspistill -r? The first two will do 15fps happily. The last won't as it's doing the full image processing and producing a JPEG for each frame as well.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

fooforever
Posts: 26
Joined: Wed Aug 15, 2012 11:21 pm

Re: STICKY: libcamera - open source camera stack

Thu May 14, 2020 10:53 am

6by9 wrote:
Thu May 14, 2020 10:42 am
So how have you been asking for it? raspiraw, V4L2, or raspistill -r? The first two will do 15fps happily. The last won't as it's doing the full image processing and producing a JPEG for each frame as well.
I'm using the raspiraw fork from here as a library in a C++ app written with QT. Frames are stored in memory before they're sent out over the network, but the 5fps is measured locally on the Pi (3) before anything is done with them. Maybe I need to do some more profiling.
https://github.com/massimiliano-mantion ... tor-as-lib

User avatar
HermannSW
Posts: 2534
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: STICKY: libcamera - open source camera stack

Thu May 14, 2020 1:36 pm

2592x1944_s tool in tools directory of that branch is from me:
https://github.com/massimiliano-mantion ... 592x1944_s

It captures every other line and results in 2592x972 frame with full FoV, which when run through double tool gives you a full 2592x1944 frame at 30fps. I am pretty sure that 2592x1944 will come out at 15fps with raspiraw from v1 camera.
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

DanR
Posts: 77
Joined: Fri Jan 18, 2013 1:20 pm

Re: STICKY: libcamera - open source camera stack

Mon May 18, 2020 11:17 am

This is a question for 6by9 as he'd most likely know; does libcamera work with the TC358743 and ADV72880 overlay drivers?

TIA,

Dan

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8675
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: STICKY: libcamera - open source camera stack

Mon May 18, 2020 11:31 am

DanR wrote:
Mon May 18, 2020 11:17 am
This is a question for 6by9 as he'd most likely know; does libcamera work with the TC358743 and ADV72880 overlay drivers?
Not really. libcamera is mainly about running AE/AWB/AF algorithms, which aren't applicable for those devices.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

DanR
Posts: 77
Joined: Fri Jan 18, 2013 1:20 pm

Re: STICKY: libcamera - open source camera stack

Mon May 18, 2020 11:49 am

I thought that might be the case, although it would be nice if it did. I just now need to find a way to get the liveMedia library to add audio to my rtsp stream, fun times ahead ;)

I must get one of the new HD cams and lenses to play with so I better start saving...

Thanks for the quick reply!!

Dan

had
Posts: 13
Joined: Wed Mar 11, 2020 2:34 pm

Re: STICKY: libcamera - open source camera stack

Fri May 29, 2020 11:56 am

Hi,

Is it possible to load and unload the camera overlay (dtoverlay=imx219 or dtoverlay=imx477) dynamically, so that it can be manipulated without a reboot?

Thanks

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8675
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: STICKY: libcamera - open source camera stack

Fri May 29, 2020 2:14 pm

had wrote:
Fri May 29, 2020 11:56 am
Hi,

Is it possible to load and unload the camera overlay (dtoverlay=imx219 or dtoverlay=imx477) dynamically, so that it can be manipulated without a reboot?

Thanks
Do NOT add them at to config.txt, and then load with
sudo dtoverlay imx219
and unload with
sudo dtoverlay -r imx219

You will probably need to blacklist bcm2835-v4l2, otherwise the firmware will detect the sensor and load the firmware based V4L2 driver.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

razvanphp
Posts: 1
Joined: Fri May 29, 2020 4:21 pm

Re: STICKY: libcamera - open source camera stack

Fri May 29, 2020 7:09 pm

I've compiled libcamera on jetson nano. After

Code: Select all

LD_LIBRARY_PATH=/usr/local/lib/aarch64-linux-gnu
I can run cam bin, but I get this error:

Code: Select all

jetson@jetson:~/libcamera$ cam -l
[3:00:16.007996780] [22095]  INFO Camera camera_manager.cpp:277 libcamera v0.0.0+1404-16c5f44c
[3:00:16.102847294] [22096] ERROR V4L2 v4l2_videodevice.cpp:518 /dev/video1[cap]: V4L2 API v4.9.140 too old, v5.0.0 or later is required
[3:00:16.103527312] [22096] ERROR V4L2 v4l2_videodevice.cpp:1091 /dev/video1[cap]: Unable to request 0 buffers: Invalid argument
Available cameras:
jetson@jetson:~/tomasi$ uname -a
Linux jetson 4.9.140-tegra #1 SMP PREEMPT Wed Apr 8 18:10:49 PDT 2020 aarch64 aarch64 aarch64 GNU/Linux
I'm trying to access the new HQ camera there, should it work via libcamera or we need kernel support for it? It's also not available in v4l2-ctl and the following error appears in syslog:

Code: Select all

[Fri May 29 16:11:44 2020] tegra-vii2c 546c0000.i2c: no acknowledge from address 0x10
[Fri May 29 16:11:44 2020] imx219 8-0010: imx219_board_setup: error during i2c read probe (-121)
[Fri May 29 16:11:44 2020] imx219 8-0010: board setup failed
[Fri May 29 16:11:44 2020] imx219: probe of 8-0010 failed with error -121
Thank you for your great work!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8675
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: STICKY: libcamera - open source camera stack

Fri May 29, 2020 9:04 pm

razvanphp wrote:
Fri May 29, 2020 7:09 pm
I've compiled libcamera on jetson nano. After

Code: Select all

LD_LIBRARY_PATH=/usr/local/lib/aarch64-linux-gnu
I can run cam bin, but I get this error:

Code: Select all

jetson@jetson:~/libcamera$ cam -l
[3:00:16.007996780] [22095]  INFO Camera camera_manager.cpp:277 libcamera v0.0.0+1404-16c5f44c
[3:00:16.102847294] [22096] ERROR V4L2 v4l2_videodevice.cpp:518 /dev/video1[cap]: V4L2 API v4.9.140 too old, v5.0.0 or later is required
[3:00:16.103527312] [22096] ERROR V4L2 v4l2_videodevice.cpp:1091 /dev/video1[cap]: Unable to request 0 buffers: Invalid argument
Available cameras:
jetson@jetson:~/tomasi$ uname -a
Linux jetson 4.9.140-tegra #1 SMP PREEMPT Wed Apr 8 18:10:49 PDT 2020 aarch64 aarch64 aarch64 GNU/Linux
I'm trying to access the new HQ camera there, should it work via libcamera or we need kernel support for it? It's also not available in v4l2-ctl and the following error appears in syslog:

Code: Select all

[Fri May 29 16:11:44 2020] tegra-vii2c 546c0000.i2c: no acknowledge from address 0x10
[Fri May 29 16:11:44 2020] imx219 8-0010: imx219_board_setup: error during i2c read probe (-121)
[Fri May 29 16:11:44 2020] imx219 8-0010: board setup failed
[Fri May 29 16:11:44 2020] imx219: probe of 8-0010 failed with error -121
Thank you for your great work!
Sorry, I suspect you're not going to get much success using libcamera on a jetson, and you won't get any support from a raspberry pi forum.

Libcamera has a pipeline handler that is specific to the platform to coordinate the sensor, receiver, and isp. I don't believe such a pipeline handler is even started for jetson, and generally it requires fairly detailed information on the hardware.
The libcamera project also have a rule that it must run on mainline Linux if you want to merge it into the main project repo, or at least patches submitted to the mailing lists with the intent to get them merged. I don't believe that is possible for most Nvidia platforms. Last time I looked Nvidia didn't use standard v4l2 sensor subdevice drivers either whichakes the job even tougher.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

CanPi
Posts: 47
Joined: Wed May 20, 2020 12:21 pm

IMX477 CCM Non Monotonic with CCT

Sun Jun 14, 2020 7:47 am

therealdavidp wrote:
Sat Jun 13, 2020 9:59 pm
Any reason why values don't move monotonically, sometimes dramatically so, from one extreme of the cct range to the other? For instance look at how the very first value of the matrices varies with cct. If because of the type of illuminant (e.g. D50 sunlight vs LED vs fluorescent) is there a way to know what cct goes with which?
The CCM measurements are calculated directly from a set of calibration images. Indeed they do move about but generally we go with whatever the fitting procedure gives us. Anyone interested can find out all about the procedure here https://www.raspberrypi.org/documentati ... ra_1p0.pdf (see chapter 6).
Thanks for that David, I agree that using RGB fits is less than ideal and may be part of the problem.

To clarify why I point this out pretend that, all else equal, the power of halogen lights illuminating the scene are increased just a little, resulting in cct going from 2900 to 3000K. The red white balance multipliers change monotonically as expected - but look at the erratic behavior of the red channel primary coefficient in the ccm. If you had a red led in the image its intensity would suddenly jump to 20% more than expected compared to a previous capture, that's not good. Nor is that +/-10% zig zagging up to 4700K.

May I ask what algorithm is being used for the fit? The normal equation is a pretty weak performer as measured in dE. And others tend to be multi-modal, so some regularization may be helpful to get past local minima. Unfortunately I am not an expert or a programmer, so I can only point out things that look suspicious without being able to fix them.

Return to “Camera board”