v4l2 on a CM - where is /dev/video1?

[nickds1]

Hi,

I'm trying to do some fairly complex streaming with gstreamer using a CM with cam0 & cam1 (both official RPi cameras) - both work fine individually using raspivid and -cs 0 & -cs 1.

I would really really prefer to have both /dev/video0 and /dev/video1 available for this, but loading bcm2835_v4l2 only creates /dev/video0 - I'd like both as then I could use v4l2src device=/dev/video1 etc.

is there any way to get the second camera recognised as /dev/video1 ?- raspivid can see them but I need each camera separately to use videomixer2 and other gstreamer elements (e.g. glshader)., i.e. I'd rather not use raspivid, but if there is a way of using it to get two streams (one from each camera) into gstreamer videiomixer2, then that'd be ok.

Any pointers/experience with this welcomed - I've seen discussions from a year ago implying that dual v4l2 camera support was being considered, but where did this end up - I've not been able to find anything concrete on this...

PS. timeframe does not allow me to wait for the CM3

Many thanks

Rob

[6by9]

Any pointers/experience with this welcomed - I've seen discussions from a year ago implying that dual v4l2 camera support was being considered, but where did this end up - I've not been able to find anything concrete on this...

It never got there - too many ways to skin the cat, and I got distracted by other things.
h
It is still on the list of things to look into. The annoyance is the static structure at https://github.com/raspberrypi/linux/bl ... mera.c#L77, but hopefully that only needs to become an array of pointers that we can iterate over. It's then lots of find/replace to amend the references to gdev.
Perhaps that isn't as big a job as it first appeared. I'll have another look into it.

As another option, the closed source UV4L2 driver does allow you to load it twice with the different camera numbers specified.

[nickds1]

Hi,

Thanks for the prompt reply. I thought of UV4L2 but was concerned that there might be a performance hit vs. the kernel drivers - the CM is not exactly blessed with excess CPU

Any thoughts would be appreciated...

[6by9]

It seems to be coming together pretty easily. If it's going to work at all, I'd hope to get it sorted in the next day or so.

If you're up to building a kernel image for your own testing, I can push something that has a hope of working but hard-coded as wanting 2 cameras and lacking on error checking/cleanup.

[6by9]

Well that did go better than I'd hoped.
Change, including checking how many cameras are attached, just pushed to https://github.com/6by9/linux/tree/rpi-4.4.y
It wants a bit more testing yet (need to check with only one camera detected), but it enumerates both devices, and 'v4l2-ctl -d 0 --overlay=1' does bring preview up for one, and "-d 1" for the other.
It should also give the option of specifying the /dev/videoN node numbers in case you need to avoid USB webcams or similar.

[nickds1]

That is hugely impressive - I'm really grateful for your work on this - as it happens, I don't currently have facilities to build a kernel - is this just a library change or is it part of the kernel proper? If so, is there any way to get a kernel build to test?

Many thanks

[6by9]

That is hugely impressive - I'm really grateful for your work on this - as it happens, I don't currently have facilities to build a kernel - is this just a library change or is it part of the kernel proper? If so, is there any way to get a kernel build to test?

It's a kernel module.
I've just pushed the version I built for the 4.4.6 kernel to https://github.com/6by9/RPiTest/tree/ma ... le_cameras
Please note that these are dependent on the revision of kernel you're running. 4.4.6 is the "next" branch for Pi, 4.1 is the stable branch.

If you have a device that you can test with (I won't guarantee stability), then

Code: Select all

sudo BRANCH=next rpi-update
sudo reboot
wget https://github.com/6by9/RPiTest/raw/master/v4l2_multiple_cameras/bcm2835-v4l2.ko
sudo cp bcm2835-v4l2.ko /lib/modules/4.4.6+/kernel/drivers/media/platform/bcm2835/bcm2835-v4l2.ko
sudo modprobe bcm2835-v4l2

should get and install the 4.4.6 kernel and the new module. Hopefully you'll get /dev/video0 and /dev/video1.
I've still got more testing to do, so do shout if you find any issues.

The fact it fell into place so easily is more down to another year of experience with V4l2 and how Linux wants to configure things. I think I'd got myself too tied up in knots before on the options, and looking at USB drivers as an example was a bad move as they enumerate differently to platform devices.

[nickds1]

Hi - this is great - what is the risk of taking this update? Should I wait a few more days or are you done with this for now?

I'm fairly new to RPis, especially the CM as I'm helping someone out here - is there a risk of bricking the unit?

I need a reasonably stable environment for the next week or so - I apprecaite thoughts on this...

Many thanks

[RpiName]

I thought of UV4L2 but was concerned that there might be a performance hit vs. the kernel drivers - the CM is not exactly blessed with excess CPU

there are no perceptible differences on "performance". however, as opposite to the kernel module, if uv4l crashes, the system will not be compromised. if, as uv4l has no known bugs.

[6by9]

Hi - this is great - what is the risk of taking this update? Should I wait a few more days or are you done with this for now?

I'm fairly new to RPis, especially the CM as I'm helping someone out here - is there a risk of bricking the unit?

You should never be able to totally brick a Pi, though you can corrupt the filesystem sufficiently to require reimaging the SD card or EMMC.

I wouldn't expect this change by itself to be a risk, but switching to 4.4 rather than 4.1 may be a slightly larger risk. Generally kernel bumps are fairly stable as all patches are reviewed and tested before merging.
Whichever way around, you're going to need to do a "sudo rpi-update" to get the change until the next Raspbian release, and that does have a modest inherent risk.

I've noticed one theoretical issue, but as the GPU will never say it has more than 2 cameras it isn't in reality. It needs a slightly more thorough code review and more testing, but I'm not anticipating any great changes.

[6by9]

I thought of UV4L2 but was concerned that there might be a performance hit vs. the kernel drivers - the CM is not exactly blessed with excess CPU

there are no perceptible differences on "performance". however, as opposite to the kernel module, if uv4l crashes, the system will not be compromised. if, as uv4l has no known bugs.

Swings and roundabouts.
If you crash uv4l then it's a black box that can't be easily debugged. At least with the kernel you'll get a back trace and have the source to look at.
I haven't encountered a kernel crash for long time, other than due to a dodgy DM9601 ethernet adapter on a Pi Zero (my own fault for being a cheapskate).

"No known bugs": that just makes me want to misquote Donald Rumsfeld - https://en.wikipedia.org/wiki/There_are_known_knowns

[nickds1]

I've noticed one theoretical issue, but as the GPU will never say it has more than 2 cameras it isn't in reality. It needs a slightly more thorough code review and more testing, but I'm not anticipating any great changes.

Yep - I saw that you'd hard-coded literal "2" as the maximum number of cameras that are supported and that arrays of pointers etc. are dimensioned off of that.

A theoretical rather than practical limitation (in this application at least!)

Cheers

[6by9]

I've noticed one theoretical issue, but as the GPU will never say it has more than 2 cameras it isn't in reality. It needs a slightly more thorough code review and more testing, but I'm not anticipating any great changes.

Yep - I saw that you'd hard-coded literal "2" as the maximum number of cameras that are supported and that arrays of pointers etc. are dimensioned off of that.

A theoretical rather than practical limitation (in this application at least!)

Not quite the issue.
At https://github.com/6by9/linux/blob/rpi- ... ra.c#L1808 I get the value back from the GPU of how many cameras are connected, and then

Code: Select all

	for (camera=0; camera<num_cameras; camera++)
	{
		dev = kzalloc(sizeof(struct bm2835_mmal_dev), GFP_KERNEL);
		if (!dev)
			return -ENOMEM;
...
		gdev[camera] = dev;
	}

If the GPU returned num_cameras=3, I've just corrupted memory as I've overrun my array. In reality the GPU can't reply with a value >2 as it too has an array of size 2 Always better safe than sorry though.

[nickds1]

Well - all upgraded and running 4.4.6+ #869 with your modified driver...

I have both cameras running, but now have my gstreamer command just stopping...

Code: Select all

FPS=10                          # Frames per second
WIDTH=320                       # Image width
HEIGHT=240                      # Image height
UPLINK_HOST=192.168.1.73        # Receiving host
PORT=5200                       # UDP port
#
# Start streaming data
#
function start_streaming
{
  gst-launch-1.0 -v videomixer name=mixer sink_1::xpos=0 sink_2::xpos=$WIDTH \
  ! x264enc tune=zerolatency \
  ! h264parse \
  ! rtph264pay config-interval=10 pt=96 \
  ! queue \
  ! udpsink host=$UPLINK_HOST port=$PORT \
  v4l2src device=/dev/video0 \
  ! video/x-h264,width=$WIDTH,height=$HEIGHT,framerate=$FPS/1 \
  ! h264parse \
  ! decodebin \
  ! queue \
  ! mixer.sink_1 \
  v4l2src device=/dev/video1 \
  ! video/x-h264,width=$WIDTH,height=$HEIGHT,framerate=$FPS/1 \
  ! h264parse \
  ! decodebin \
  ! queue \
  ! mixer.sink_2
}
start_streamer

Zero CPU - just doesn't stream... thoughts would be appreciated!

Thanks

[piglet]

I haven't encountered a kernel crash for long time, other than due to a dodgy DM9601 ethernet adapter on a Pi Zero (my own fault for being a cheapskate).

Small aside - that's been "fixed": viewtopic.php?f=28&t=135671&start=25#p916438

[6by9]

Well - all upgraded and running 4.4.6+ #869 with your modified driver...

I have both cameras running, but now have my gstreamer command just stopping...

I did say I had more testing to do. Sadly the weekend became busy and I didn't have a chance to do too much. My initial tests had all been with enabling the overlay of each camera at the same time, and that was all OK (I've also now pushed extra commits that allow the overlay location to be moved and resized, although I haven't rebuilt the test module).

Can you stream each camera individually? ie is it both simultaneously that is the issue?
I'll try to have another play tonight.

If I've read your command line correctly, you're asking V4L2 for H264, then immediately decoding the streams using decodebin, mixing them with videomixer, and then re-encoding. Why?! The V4L2 driver and videomixer both handle I420, so you'll save a bundle of processing (and potential image quality loss). Something like:

Code: Select all

  gst-launch-1.0 -v videomixer name=mixer sink_1::xpos=0 sink_2::xpos=$WIDTH \
  ! x264enc tune=zerolatency \
  ! h264parse \
  ! rtph264pay config-interval=10 pt=96 \
  ! queue \
  ! udpsink host=$UPLINK_HOST port=$PORT \
  v4l2src device=/dev/video0 \
  ! video/x-raw,format=I420,width=$WIDTH,height=$HEIGHT,framerate=$FPS/1 \
  ! queue \
  ! mixer.sink_1 \
  v4l2src device=/dev/video1 \
  ! video/x-raw,format=I420,width=$WIDTH,height=$HEIGHT,framerate=$FPS/1 \
  ! queue \
  ! mixer.sink_2

should work more efficiently.

Small aside - that's been "fixed": viewtopic.php?f=28&t=135671&start=25#p916438

I know. https://github.com/raspberrypi/linux/issues/1045 includes a contributor using the handle "6by9" - very dubious character It's more a sticky tape fix than anything though.

[gregeric]

It's more a sticky tape fix than anything though.

I used to like you, that was my finest hour. (campag at github)

[6by9]

It's more a sticky tape fix than anything though.

I used to like you, that was my finest hour. (campag at github)

I won't knock the debugging skills involved in getting there - it was a very nice find. I'm only following through on the comments that the DM9601 driver is at fault for advertising the wrong buffer size. I must admit I then looked at the network stack drivers and got lost in the maze of twisty passages, all alike, so stopped and accepted caution within the drivers that were relying on the value as a good enough solution!
It's the mantra that all developers should follow - always expect to be given <expletive deleted>. Always, always, always code defensively if there are any outside parties responsible for giving you data.

[6by9]

Well - all upgraded and running 4.4.6+ #869 with your modified driver...

I have both cameras running, but now have my gstreamer command just stopping...

I did say I had more testing to do. Sadly the weekend became busy and I didn't have a chance to do too much. My initial tests had all been with enabling the overlay of each camera at the same time, and that was all OK (I've also now pushed extra commits that allow the overlay location to be moved and resized, although I haven't rebuilt the test module).

Can you stream each camera individually? ie is it both simultaneously that is the issue?
I'll try to have another play tonight.

I've just tried it with two instances of qv4l2 to view 640x480 I420 or RGBX8888 from the two devices. Works fine, other than a chunk of latency which is inherent in the way qv4l2 renders the images.

Giving GStreamer a quick whirl:

Code: Select all

pi@raspberrypi ~ $ GST_DEBUG=fakesink:7 gst-launch-1.0 -v videomixer name=mixer sink_1::xpos=0 sink_2::xpos=320 \
> ! queue \
> ! fakesink \
> v4l2src device=/dev/video0 \
> ! video/x-raw,format=I420,width=320,height=240,framerate=10/1 \
> ! queue \
> ! mixer.sink_1 \
> v4l2src device=/dev/video1 \
> ! video/x-raw,format=I420,width=320,height=240,framerate=10/1 \
> ! queue \
> ! mixer.sink_2
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2865): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming task paused, reason not-negotiated (-4)
Execution ended after 0:00:36.096388088
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

I am on an old Wheezy install. But that did remind me of the bug in GStreamer which is even referenced in the V4L2 driver source:

Code: Select all

/* Gstreamer bug https://bugzilla.gnome.org/show_bug.cgi?id=726521
 * v4l2src does bad (and actually wrong) things when the vidioc_enum_framesizes
 * function says type V4L2_FRMSIZE_TYPE_STEPWISE, which we do by default.
 * It's happier if we just don't say anything at all, when it then
 * sets up a load of defaults that it thinks might work.
 * If gst_v4l2src_is_broken is non-zero, then we remove the function from
 * our function table list (actually switch to an alternate set, but same
 * result).
 */

So load the driver with

Code: Select all

sudo modprobe bcm2835-v4l2 gst_v4l2src_is_broken=1

and I can then get stuff streaming with:

Code: Select all

GST_DEBUG=v4l2src:7 gst-launch-1.0 -v videomixer name=mixer sink_1::xpos=0 sink_2::xpos=320 \
! queue \
! fakesink \
v4l2src device=/dev/video0 \
! video/x-raw,format=I420,width=320,height=240,framerate=10/1 \
! queue \
! mixer.sink_1 \
v4l2src device=/dev/video1 \
! video/x-raw,format=I420,width=320,height=240,framerate=10/1 \
! queue \
! mixer.sink_2

I think I'm happy with those patches based on the testing I've done, so I'll create a pull request to get it merged properly.

[nickds1]

Thanks again - interesting - The gstreamer use of fakesink indeed does work well, but by adding in each other of other elements in turn I found that the CPU killer is x264enc - as soon as that element is there, CPU goes to 98% and stays there. I'd be interested if you see the same.

Not sure if there is anything else to be done at this point - its classed as a "bad" plugin, and I can see why...

[6by9]

Thanks again - interesting - The gstreamer use of fakesink indeed does work well, but by adding in each other of other elements in turn I found that the CPU killer is x264enc - as soon as that element is there, CPU goes to 98% and stays there. I'd be interested if you see the same.

Not sure if there is anything else to be done at this point - its classed as a "bad" plugin, and I can see why...

I didn't have x264enc available even though I thought I had plugins-bad, hence ditching it and using fakesink.
Probably worth looking at omxh264enc which will use the GPU to do the H264 encode. The ARM core on Pi1 and the CM is not really powerful enough to try to do H264 encoding (or decoding).

[jamesh]

+1 on omxh264enc - I've used that in Pi gstreamer pipelines and it works fine.