jjbpi
Posts: 6
Joined: Wed Jun 13, 2018 6:46 pm

camera v2.1 variable fps?

Wed Jun 13, 2018 7:11 pm

Hi everyone,

I've been working on building a home CCTV system. Early stages still but so far I've got Zoneminder installed and setup on a virtual machine, with a Logitech Webcam Pro HD C920 USB camera attached. Zoneminder can fetch the video stream directly from this and all seems good.

Now, I've got a spare RPi 3 Model B for which I have bought a RPi Camera V2.1 (not the NOIR version). I'm running Arch Linux ARMv7 on this Pi (I prefer to use Arch Linux, but will consider alternatives for this project if necessary). I'm using the following command line on the Pi to stream the camera image ...

Code: Select all

/opt/vc/bin/raspivid -o - -t 0 -n -vf -hf -ISO 400 -ex auto -mm average -md 6 -b 20000000 -if | cvlc -vv stream:///dev/stdin --sout "#standard{access=http,mux=ts,dst=localhost:55000}" :demux=h264
I've configured Zoneminder to fetch the stream from http://picam1:55000, and this is working well. So now I have two camera feeds configured in Zoneminder, both of which are 1280x720 resolution. During the day time both camera images look great and I'm getting around 30FPS from the HD C920 and around 36FPS from the Pi Camera. However, at night the HD C920 automatically reduces its FPS to around 15FPS, while the Pi Camera does not. It stays at around 36FPS and this results in a very dark image in which nearly nothing is visible. I know that if I modify the raspivid command line on the Pi to set the FPS to 15 then I get a lighter image (but still not as clear as the HD C920). So my question here is - is there any way to get the Pi Camera to drop it's FPS automatically in low light, as the HD C920 seems to do? Can anybody recommend some better options for the raspivid command line above?

Thanks in advance.
JB

HermannSW
Posts: 701
Joined: Fri Jul 22, 2016 9:09 pm

Re: camera v2.1 variable fps?

Thu Jun 14, 2018 7:09 am

You should try this solution only if there is no other (like killing raspivid when it becomes darker based on a sensor value, and start newly with changed fps setting, by some cron job).

I did a fork of userland raspivid, that does changes to camera settings after the GPU is done with initializing the camera.
These are the changes, no documentation yet:
https://github.com/Hermann-SW/userland/commits/master

First I used it to change v2 camera framerate to >120fps before 6by9 did raise the limit to 200fps for 640x480 mode in this March Raspbian Stretch (to demonstrate that v2 camera can take good 640x480 video at 180fps).
There are some other options as well.
All work by injecting I2C commands at some frame where you can be sure that GPU is done with initializing (frame 5, or better 25).
This technique allows to change shutter time for every single frame of a video taken with raspivid(!), "--shutter2" option:
viewtopic.php?f=43&t=212694&p=1313054#p1313054

So for your usecase you could check darkness measured by a sensor in callback function executed for every single frame, and inject I2C commands for adjusting framerate as needed. You can see how to change framerate for v2 camera starting row 1501 in this commit:
https://github.com/Hermann-SW/userland/ ... a52b3R1501

Or on latest how to deal with both cameras starting line 1625:
https://github.com/Hermann-SW/userland/ ... id.c#L1625
--> Raspberry camera / gstreamer / raspivid / raspiraw (bookmark list):
https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/fork-raspiraw      https://github.com/Hermann-SW/userland      https://twitter.com/HermannSW

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5228
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: camera v2.1 variable fps?

Thu Jun 14, 2018 9:11 am

I've not tested if raspivid behaves itself, but generally setting a frame rate of 0 on the camera component allows the AE algorithm to vary the framerate between the absolute max limits defined by MMAL_PARAMETER_FPS_RANGE (defaults to 5 to 30fps).

The range that AE will vary the exposure time over depends on the exposure mode set with -ex. "auto" allows up to 66ms, whilst "night" allows up to 500ms (and cranks the exposure time up in preference to higher gain). "nightpreview" allows up to 125ms.
Add "-set" to your command line and raspivid will print out the values for all the gains and exposure time as the control loops vary them.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jjbpi
Posts: 6
Joined: Wed Jun 13, 2018 6:46 pm

Re: camera v2.1 variable fps?

Thu Jun 14, 2018 10:08 pm

6by9 wrote:
Thu Jun 14, 2018 9:11 am
I've not tested if raspivid behaves itself, but generally setting a frame rate of 0 on the camera component allows the AE algorithm to vary the framerate between the absolute max limits defined by MMAL_PARAMETER_FPS_RANGE (defaults to 5 to 30fps).

The range that AE will vary the exposure time over depends on the exposure mode set with -ex. "auto" allows up to 66ms, whilst "night" allows up to 500ms (and cranks the exposure time up in preference to higher gain). "nightpreview" allows up to 125ms.
Add "-set" to your command line and raspivid will print out the values for all the gains and exposure time as the control loops vary them.
Thanks for this 6by9 - i changed my command line to

Code: Select all

/opt/vc/bin/raspivid -set -t 0 -n -vf -hf -w 1280 -h 720 -ISO 800 -fps 0 -ex night -mm average -awb auto -o - | cvlc -vv stream:///dev/stdin --sout "#standard{access=http,mux=ts,dst=localhost:55000}" :demux=h264
As you suggested, I set the FPS to 0. This does now give a variable FPS. Also by settings ISO to 800 and ex to night the exposure looks very much like the HD C920 camera, which is good. There are still a few issues though:
  • The FPS on the Pi Camera is quite eratic ... right now, at night time with only street lights outside, it's swinging between 9.6fps and 15.2fps.
  • With ISO 800 it's quite grainy.
  • The image appears to be freezing for a second of two every 10 seconds or so.
    In the terminal I'm seeing this outout from CVLC ....

    Code: Select all

    [72c00868] main stream output debug: adding a new sout input for `h264` (sout_input: 0x71100bb8)
    [72c03078] main mux debug: adding a new input
    [72c03078] mux_ts mux debug: adding input codec=h264 pid=100
    [72c03078] mux_ts mux debug: new PCR PID is 100
    [72e00668] main input debug: Buffering 40%
    [72e00668] main input debug: Buffering 53%
    [72e00668] main input debug: Buffering 66%
    [72e00668] main input debug: Buffering 80%
    [72e00668] main input debug: Buffering 93%
    [72e00668] main input debug: Stream buffering done (320 ms in 524 ms)
    [72e00668] main input debug: Decoder wait done in 0 ms
    [72c03078] main mux warning: late buffer for mux input (8766)
    [72c03078] main mux warning: late buffer for mux input (17686)
    [72c03078] main mux warning: late buffer for mux input (9910)
    [72c03078] main mux warning: late buffer for mux input (36417)
    [72c03078] main mux warning: late buffer for mux input (29933)
    [72c03078] main mux warning: late buffer for mux input (24402)
    [72c03078] main mux warning: late buffer for mux input (3189)
    [72c03078] main mux warning: late buffer for mux input (29169)
    [72c03078] main mux warning: late buffer for mux input (8380)
    [72c03078] main mux warning: late buffer for mux input (35119)
    [72c03078] main mux warning: late buffer for mux input (17270)
    [72c03078] main mux warning: late buffer for mux input (44816)
    [72c03078] main mux warning: late buffer for mux input (70736)
    [72c03078] main mux warning: late buffer for mux input (20553)
    [72c03078] main mux warning: late buffer for mux input (49552)
    [72c03078] main mux warning: late buffer for mux input (74004)
    [72c03078] main mux warning: late buffer for mux input (28457)
    [72c03078] main mux warning: late buffer for mux input (56290)
    [72c03078] main mux warning: late buffer for mux input (82293)
    [72c03078] main mux warning: late buffer for mux input (41807)
    [72c03078] main mux warning: late buffer for mux input (66521)
    [72c03078] main mux warning: late buffer for mux input (93490)
    [72c03078] main mux warning: late buffer for mux input (120295)
    [72c03078] main mux warning: late buffer for mux input (77520)
    [72c03078] main mux warning: late buffer for mux input (103280)
    [72c03078] main mux warning: late buffer for mux input (128699)
    [72c03078] main mux warning: late buffer for mux input (155586)
    [72c03078] main mux warning: late buffer for mux input (109756)
    [72c03078] main mux warning: late buffer for mux input (139756)
    [72e00668] main input error: ES_OUT_SET_(GROUP_)PCR  is called too late (pts_delay increased to 300 ms)
    [72e00668] main input debug: ES_OUT_RESET_PCR called
    [72c03078] main mux warning: late buffer for mux input (166210)
    [72e00668] main input debug: Buffering 0%
    [72e00668] main input debug: Buffering 13%
    [72e00668] main input debug: Buffering 26%
    [72e00668] main input debug: Buffering 40%
    [72e00668] main input debug: Buffering 53%
    [72e00668] main input debug: Buffering 66%
    [72e00668] main input debug: Buffering 80%
    [72e00668] main input debug: Buffering 93%
    [72e00668] main input debug: Stream buffering done (320 ms in 533 ms)
    [72e00668] main input debug: Decoder wait done in 0 ms
    [72c03078] main mux warning: late buffer for mux input (70053)
    [72c03078] main mux warning: late buffer for mux input (30724)
    
    Any ideas what could be causing this?
Thanks!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5228
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: camera v2.1 variable fps?

Fri Jun 15, 2018 9:13 am

You don't say how you're telling vlc to use the Logitech camera, but at a guess you're pointing it at V4L2 (/dev/video0). In which case it is being given timestamps with each incoming buffer.
When you pipe from raspivid to vlc there are no timestamps, just the raw data. vlc therefore guesses at a framerate and makes up the timestamps. When the camera alters the framerate these guesses become wrong, and it reckons that frames have been dropped or are otherwise "late".

The Pi camera does have a V4L2 driver, but it's not quite as flexible as using raspivid, but will give vlc accurate timestamps.

Code: Select all

sudo modprobe bcm2835-v4l2
You may need to configure extra options, the main one I can think of is to run

Code: Select all

v4l2-ctl --set-ctrl=exposure_dynamic_framerate=1
to allow variable frame rate (default is always fixed). Otherwise most stuff should be under the control of vlc.
You can select scene mode night (which engages night exposure mode) by running

Code: Select all

v4l2-ctl --set-ctrl=scene_mode=8
Setting ISO800 means that the analogue gain of the sensor is up at around x8.0. Gain always increases the noise level too, whereas increasing the exposure time doesn't. Unless you really need it, leave the gain under the control of the AE algorithm (ISO 0 = auto).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jjbpi
Posts: 6
Joined: Wed Jun 13, 2018 6:46 pm

Re: camera v2.1 variable fps?

Fri Jun 15, 2018 11:10 am

Just a quick reply for now - thank for the tips, I'll try these v4l2-ctl commands. I have already configured the Pi to load the bcm2835-v4l2 driver on boot. The Logitech camera is not connected to the Pi, it's connected to my server which is running Zoneminder. I've configured Zoneminder to access the local camera at /dev/video0. So I admit I'm not entirely sure what streaming method / parameters are being used there.
Will let you know how my tests go.

Thanks

jjbpi
Posts: 6
Joined: Wed Jun 13, 2018 6:46 pm

Re: camera v2.1 variable fps?

Sat Jun 16, 2018 7:23 pm

6by9,
After your last message, in which you suggested I load the v4l2 driver, I realized a few things. When I first connected the pi camera and booted up the pi, I first tried running cvlc to stream the camera (without raspivid piping into it). This had worked on another machine before, so I just expected it to work with the pi ... but it did not. cvlc was not able to access any camera devices. So I looked for /dev/video0, and it was not there. At this point I was not aware of raspivid. I googled how to get the camera to be detected and how to get /dev/video0 to show up and came across a forum explaining how to load the bcm2835-v4l2 driver. This obviously got /dev/video0 to show up, but for some reason cvlc still cannot access it. So more googling later I discovered raspivid ... and how to pipe raspivid into cvlc .... this has been working, but with the issues I've already explained in previous posts.

Your previous post got me thinking that perhaps the v4l2 was only an option that you were suggesting ... and there was another driver / way of accessing the camera that should work without loading the bcm2835-v4l2. So, I unpicked what I did to get the bcm2835-v4l2 driver to load at boot, and rebooted .... then raspivid could not access the camera ....
More googling later I discovered that in Rasbian you have to run the "raspi-config" utility to enable the camera! Raspi-config does not exist in Arch Linux ARM.... even more googling happened and I found the raspi-config package in the Arch AUR ... so I installed that and enabled the camera module. Now Raspivid works, without the bcm2835-v4l2 driver. So now I'm waiting for night time to see if I can fine tune the raspivid settings to get the best exposure and fps ... I'm spending WAAYYY too much time tinkering here.

I have a question though, maybe you can answer - If I'm now not using the bcm2835-v4l2 driver then what driver is in use? I can't see anything in "lsmod" about the camera (nothing obvious). Raspivid is obviously able to access / drive the camera somehow, but how?

jjbpi
Posts: 6
Joined: Wed Jun 13, 2018 6:46 pm

Re: camera v2.1 variable fps?

Tue Jun 19, 2018 11:52 am

This is very informative, but should be easier to find ...

http://picamera.readthedocs.io/en/lates ... -operation

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 19854
Joined: Sat Jul 30, 2011 7:41 pm

Re: camera v2.1 variable fps?

Tue Jun 19, 2018 1:19 pm

jjbpi wrote:
Sat Jun 16, 2018 7:23 pm
6by9,
After your last message, in which you suggested I load the v4l2 driver, I realized a few things. When I first connected the pi camera and booted up the pi, I first tried running cvlc to stream the camera (without raspivid piping into it). This had worked on another machine before, so I just expected it to work with the pi ... but it did not. cvlc was not able to access any camera devices. So I looked for /dev/video0, and it was not there. At this point I was not aware of raspivid. I googled how to get the camera to be detected and how to get /dev/video0 to show up and came across a forum explaining how to load the bcm2835-v4l2 driver. This obviously got /dev/video0 to show up, but for some reason cvlc still cannot access it. So more googling later I discovered raspivid ... and how to pipe raspivid into cvlc .... this has been working, but with the issues I've already explained in previous posts.

Your previous post got me thinking that perhaps the v4l2 was only an option that you were suggesting ... and there was another driver / way of accessing the camera that should work without loading the bcm2835-v4l2. So, I unpicked what I did to get the bcm2835-v4l2 driver to load at boot, and rebooted .... then raspivid could not access the camera ....
More googling later I discovered that in Rasbian you have to run the "raspi-config" utility to enable the camera! Raspi-config does not exist in Arch Linux ARM.... even more googling happened and I found the raspi-config package in the Arch AUR ... so I installed that and enabled the camera module. Now Raspivid works, without the bcm2835-v4l2 driver. So now I'm waiting for night time to see if I can fine tune the raspivid settings to get the best exposure and fps ... I'm spending WAAYYY too much time tinkering here.

I have a question though, maybe you can answer - If I'm now not using the bcm2835-v4l2 driver then what driver is in use? I can't see anything in "lsmod" about the camera (nothing obvious). Raspivid is obviously able to access / drive the camera somehow, but how?
Raspivid access the camera software on the GPU via the MMAL API, not through any Linux devices.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

Return to “Camera board”

Who is online

Users browsing this forum: No registered users and 6 guests