lak4cyut
Posts: 6
Joined: Wed Aug 21, 2019 6:27 am

Very low FPS when V4L2 Streaming (Unicam + custom v4l2-subdev driver)

Thu Nov 07, 2019 8:46 am

HI Everyone,
I try to write a custom v4l2-subdev driver to control my own camera module (with bcm2835-unicam v4l2 driver) and finally, I finished it.
I can get the correct image and set/get parameters via v4l2 framework.

But I encounter a strange behavior when I try to play the video.
(I play the video by 'ffplay' tool)
When I use below command to play video, the fps is almost 0.5 ~ 1, very low, even I assigned the fps (30) in sub-dev driver.

Code: Select all

ffplay -f v4l2 -framerate 30 -video_size 1920x1088 -i /dev/video0
or

Code: Select all

ffplay -f v4l2 -video_size 1920x1088 -i /dev/video0
but when I use another format - 'rawvideo', it can play very smooth.

Code: Select all

ffplay /dev/video0 -f rawvideo -pixel_format bayer_rggb8  -framerate 30 -video_size 1920x1088
I actually don't know what different between those 2 way to play video, but it seems 'rawvideo' not rely on v4l2 camera parameters (get_fmt/set_fmt).

My questions are
1. Why? Why the fps so low when play video via v4l2 but 'rawvideo' doesn't? What caused this behavior?
2. Is my sub-dev driver or dts wrong?


This behavior is stuck me a few day... I hope there is someone like to help me, I will very grateful.
Thanks.

Additional,
My device info.

Code: Select all

pi@raspberrypi:~ $ v4l2-ctl --all
Driver Info:
	Driver name      : unicam
	Card type        : unicam
	Bus info         : platform:unicam 3f801000.csi
	Driver version   : 4.19.80
	Capabilities     : 0x85200001
		Video Capture
		Read/Write
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x05200001
		Video Capture
		Read/Write
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : unicam
	Model            : unicam
	Serial           : 
	Bus info         : platform:3f801000.csi
	Media version    : 4.19.80
	Hardware revision: 0x00000001 (1)
	Driver version   : 4.19.80
Interface Info:
	ID               : 0x03000005
	Type             : V4L Video
Entity Info:
	ID               : 0x00000003 (3)
	Name             : unicam
	Function         : V4L2 I/O
	Flags         : default
	Pad 0x01000004   : 0: 
	  Link 0x02000009: from remote pad 0x1000002 of entity 'pns310 0-0048': Data, Enabled, Immutable
Priority: 2
Video input : 0 (Camera 0: ok)
Format Video Capture:
	Width/Height      : 1920/1088
	Pixel Format      : 'RGGB' (8-bit Bayer RGRG/GBGB)
	Field             : None
	Bytes per Line    : 1920
	Size Image        : 2088960
	Colorspace        : Default
	Transfer Function : Default (maps to Rec. 709)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Full Range)
	Flags             : 
Streaming Parameters Video Capture:
	Capabilities     : timeperframe
	Frames per second: 30.000 (30/1)
	Read buffers     : 2

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9850
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Very low FPS when V4L2 Streaming (Unicam + custom v4l2-subdev driver)

Thu Nov 07, 2019 11:07 am

ffplay will likely be trying to convert your image data to some format that it can display. If that conversion takes longer than a frame time (quite likely with Bayer) then it will drop frames.
Test with v4l2-ctl first:

Code: Select all

v4l2-ctl --stream-mmap=3 --stream-count=1000 --stream-to=/dev/null
If you want to save a few frames then change /dev/null to a filename, but do remember that the file system is likely to be significantly slower than the incoming data. You can use --stream-skip to skip a number of frames at the start.

For basic viewing you can count 8 bit Bayer as 8bpp greyscale. I tend to use Vooya on x86 Linux for image viewing.
If you want a quick test app then https://github.com/6by9/yavta should be able to use the Pi's ISP to convert from Bayer and put the resulting image on the screen as an overlay. Download, make, and then run with "./yavta -m --capture=1000 /dev/video0".
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lak4cyut
Posts: 6
Joined: Wed Aug 21, 2019 6:27 am

Re: Very low FPS when V4L2 Streaming (Unicam + custom v4l2-subdev driver)

Fri Nov 08, 2019 6:34 am

Hi 6by9,
Thanks for your help again.

After test with v4l2-ctl, I guess my subdev driver and DT-overlay are correct because its framerate is stable on 30 fps. ( :D :D )

Code: Select all

pi@raspberrypi:~ $ v4l2-ctl --stream-mmap=3 --stream-count=1000 --stream-to=/dev/null
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
And I also test with your tool, it also play a smooth video but its color is incorrect. (I try to use the -f option but it seems not any effect..)
I am going to study the source code of the tool, which should be very helpful for me.

So, my eye is back to the reason why ffplay with '-f v4l2' cannot play the video well.
As you said,
ffplay will likely be trying to convert your image data to some format that it can display.
But
ffplay -f v4l2 -video_size 1920x1088 -i /dev/video0
and
ffplay -f rawvideo /dev/video0 -pixel_format bayer_rggb8 -framerate 30 -video_size 1920x1088
are both convert the bayer frame to a rgb (or yuv?) color space and show on the screen, why 'rawvideo' is so smooth but 'v4l2' doesn't?
Is it because 'rawvideo' decode it with HW ISP?

Mrlbppy
Posts: 19
Joined: Fri Aug 30, 2019 12:05 pm

Re: Very low FPS when V4L2 Streaming (Unicam + custom v4l2-subdev driver)

Thu Mar 26, 2020 1:01 pm

Hi,
Great to see you have used unicam successfully. Would you like to share your development's detail ? I want to do the same thing as you.
but I am not very clear about the v4l.

lak4cyut
Posts: 6
Joined: Wed Aug 21, 2019 6:27 am

Re: Very low FPS when V4L2 Streaming (Unicam + custom v4l2-subdev driver)

Wed Apr 15, 2020 8:06 am

Mrlbppy wrote:
Thu Mar 26, 2020 1:01 pm
Hi,
Great to see you have used unicam successfully. Would you like to share your development's detail ? I want to do the same thing as you.
but I am not very clear about the v4l.
Sorry for later...
Sure, what you want to know?
I am very like to share anything that I known.

Return to “Camera board”