How to correctly set up v4l2 (m2m) under 64-bit kernel / 64-bit userland?

[sakaki]

Hello,

I'm finalizing a release of my gentoo-on-rpi3-64bit image, which given recent stabilizations I've decided will now be shipping with a 4.19.y kernel (rather than 4.14.y, which was the original plan).

Amongst other things, this allows the use of the pi camera module v2 in a full 64-bit system via /dev/video0, which is nice. Some v4l2 m2m codecs also appear (/dev/video10 etc.). But to be honest I am stumbling around with some of this, so I would really appreciate a few pointers on the following:

• To get the camera interfaces to appear, I need to set "start_x=1" in /boot/config.txt, and this is true even for the m2m interfaces; is my understanding on that correct? I.e. if I just want m2m and have no camera module attached, I still need to use start_x=1?
• I am current specifying "dtoverlay=vc4-kms-v3d,cma-256" and "gpu_mem=16" in /boot/config.txt; are those sensible settings? When the v4l2 camera or v4l2 m2m codecs interfaces are in use, does gpu_mem need to be increased or is this irrelevant?
• Would you expect the v4l2 m2m endpoints to operate correctly with a 64-bit userland? I have ffmpeg 4.1.1 compiled with the v4l extensions, but am getting segfaults every time I try to use h264_v4l2m2m. Do I have to set up some other init of the /dev/video1x interfaces to make them usable? Is there a simple ffmpeg incantation on a known (e.g. h264) source somewhere I can try to verify if the m2m stuff is working properly?

The camera interface itself appears to function fine, I can do things like:

Code: Select all

demouser@pi64 ~ $ ffmpeg -f video4linux2 -input_format h264 -video_size 1280x720 -framerate 30 -i /dev/video0 -vcodec copy -an test.h264

to grab video and e.g.

Code: Select all

demouser@pi64 ~ $ dd if=/dev/video0 bs=1M count=1 of=test.jpg

to grab a snapshot. I'd really like to be able to say that the v4l2 codecs are working however (and be able to provide some simple examples of use). Any assistance would be greatly appreciated!

Here's what I'm seeing with the camera plugged in for /dev/video*:

Code: Select all

pi64 ~ # uname -a
Linux pi64 4.19.25-v8-78eb13b25d5e-bis+ #2 SMP PREEMPT Tue Feb 26 13:27:47 GMT 2019 aarch64 GNU/Linux
pi64 ~ # v4l2-ctl --all -d /dev/video10
Driver Info (not using libv4l2):
	Driver name   : bcm2835-codec
	Card type     : bcm2835-codec
	Bus info      : platform:bcm2835-codec
	Driver version: 4.19.25
	Capabilities  : 0x84208000
		Video Memory-to-Memory
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04208000
		Video Memory-to-Memory
		Streaming
		Extended Pix Format
Priority: 2
Format Video Capture:
	Width/Height      : 640/480
	Pixel Format      : 'YU12'
	Field             : None
	Bytes per Line    : 640
	Size Image        : 460800
	Colorspace        : Rec. 709
	Transfer Function : Default (maps to Rec. 709)
	YCbCr/HSV Encoding: Default (maps to Rec. 709)
	Quantization      : Default (maps to Limited Range)
	Flags             : 
Format Video Output:
	Width/Height      : 640/480
	Pixel Format      : 'MPG4'
	Field             : None
	Bytes per Line    : 0
	Size Image        : 524288
	Colorspace        : Rec. 709
	Transfer Function : Default (maps to Rec. 709)
	YCbCr/HSV Encoding: Default (maps to Rec. 709)
	Quantization      : Default (maps to Full Range)
	Flags             : 
Selection: compose, Left 0, Top 0, Width 640, Height 480
Selection: compose_default, Left 0, Top 0, Width 640, Height 480
Selection: compose_bounds, Left 0, Top 0, Width 640, Height 480
pi64 ~ # v4l2-ctl --all -d /dev/video11
Driver Info (not using libv4l2):
	Driver name   : bcm2835-codec
	Card type     : bcm2835-codec
	Bus info      : platform:bcm2835-codec
	Driver version: 4.19.25
	Capabilities  : 0x84208000
		Video Memory-to-Memory
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04208000
		Video Memory-to-Memory
		Streaming
		Extended Pix Format
Priority: 2
Format Video Capture:
	Width/Height      : 640/480
	Pixel Format      : 'H264'
	Field             : None
	Bytes per Line    : 0
	Size Image        : 524288
	Colorspace        : Rec. 709
	Transfer Function : Default (maps to Rec. 709)
	YCbCr/HSV Encoding: Default (maps to Rec. 709)
	Quantization      : Default (maps to Full Range)
	Flags             : 
Format Video Output:
	Width/Height      : 640/480
	Pixel Format      : 'YU12'
	Field             : None
	Bytes per Line    : 640
	Size Image        : 460800
	Colorspace        : Rec. 709
	Transfer Function : Default (maps to Rec. 709)
	YCbCr/HSV Encoding: Default (maps to Rec. 709)
	Quantization      : Default (maps to Limited Range)
	Flags             : 
Selection: crop, Left 0, Top 0, Width 640, Height 480
Selection: crop_default, Left 0, Top 0, Width 640, Height 480
Selection: crop_bounds, Left 0, Top 0, Width 640, Height 480

Codec Controls

             video_bitrate_mode 0x009909ce (menu)   : min=0 max=1 default=0 value=0 flags=update
                  video_bitrate 0x009909cf (int)    : min=25000 max=25000000 step=25000 default=10000000 value=10000000
         repeat_sequence_header 0x009909e2 (bool)   : default=0 value=0
            h264_i_frame_period 0x00990a66 (int)    : min=0 max=2147483647 step=1 default=60 value=60
                     h264_level 0x00990a67 (menu)   : min=0 max=13 default=11 value=11
                   h264_profile 0x00990a6b (menu)   : min=0 max=4 default=4 value=4
pi64 ~ # v4l2-ctl --all -d /dev/video12
Driver Info (not using libv4l2):
	Driver name   : bcm2835-codec
	Card type     : bcm2835-codec
	Bus info      : platform:bcm2835-codec
	Driver version: 4.19.25
	Capabilities  : 0x84208000
		Video Memory-to-Memory
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04208000
		Video Memory-to-Memory
		Streaming
		Extended Pix Format
Priority: 2
Format Video Capture:
	Width/Height      : 0/0
	Pixel Format      : 'YUYV'
	Field             : None
	Bytes per Line    : 0
	Size Image        : 0
	Colorspace        : Rec. 709
	Transfer Function : Default (maps to Rec. 709)
	YCbCr/HSV Encoding: Default (maps to Rec. 709)
	Quantization      : Default (maps to Limited Range)
	Flags             : 
Format Video Output:
	Width/Height      : 0/0
	Pixel Format      : 'BA81'
	Field             : None
	Bytes per Line    : 0
	Size Image        : 0
	Colorspace        : Rec. 709
	Transfer Function : Default (maps to Rec. 709)
	YCbCr/HSV Encoding: Default (maps to Rec. 709)
	Quantization      : Default (maps to Full Range)
	Flags             : 
Crop Capability Video Capture:
	Bounds      : Left 0, Top 0, Width 0, Height 0
	Default     : Left 0, Top 0, Width 0, Height 0
	Pixel Aspect: 1/1
Crop Capability Video Output:
	Bounds      : Left 0, Top 0, Width 0, Height 0
	Default     : Left 0, Top 0, Width 0, Height 0
	Pixel Aspect: 1/1
Crop: Left 0, Top 0, Width 0, Height 0
Selection: crop, Left 0, Top 0, Width 0, Height 0
Selection: crop_default, Left 0, Top 0, Width 0, Height 0
Selection: crop_bounds, Left 0, Top 0, Width 0, Height 0
Selection: compose, Left 0, Top 0, Width 0, Height 0
Selection: compose_default, Left 0, Top 0, Width 0, Height 0
Selection: compose_bounds, Left 0, Top 0, Width 0, Height 0
Selection: compose_padded, Left 0, Top 0, Width 0, Height 0
Selection: native_size, Left 0, Top 0, Width 0, Height 0
Selection: crop, Left 0, Top 0, Width 0, Height 0
Selection: crop_default, Left 0, Top 0, Width 0, Height 0
Selection: crop_bounds, Left 0, Top 0, Width 0, Height 0
Selection: compose, Left 0, Top 0, Width 0, Height 0
Selection: compose_default, Left 0, Top 0, Width 0, Height 0
Selection: compose_bounds, Left 0, Top 0, Width 0, Height 0
Selection: compose_padded, Left 0, Top 0, Width 0, Height 0
Selection: native_size, Left 0, Top 0, Width 0, Height 0

ffmpeg setup:

Code: Select all

pi64 ~ # ffmpeg -codecs | grep -i m2m
ffmpeg version 4.1.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8.2.0 (Gentoo 8.2.0-r6 p1.7)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --docdir=/usr/share/doc/ffmpeg-4.1.1/html --mandir=/usr/share/man --enable-shared --cc=aarch64-unknown-linux-gnu-gcc --cxx=aarch64-unknown-linux-gnu-g++ --ar=aarch64-unknown-linux-gnu-ar --optflags='-march=armv8-a+crc -mtune=cortex-a53 -O2 -pipe' --disable-static --enable-avfilter --enable-avresample --disable-stripping --disable-optimizations --disable-libcelt --enable-nonfree --disable-indev=oss --disable-indev=jack --disable-outdev=oss --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-gcrypt --disable-gnutls --disable-gmp --enable-gpl --enable-hardcoded-tables --enable-iconv --disable-libtls --disable-lzma --enable-network --disable-opencl --enable-openssl --enable-postproc --disable-libsmbclient --enable-ffplay --enable-sdl2 --disable-vaapi --disable-vdpau --enable-xlib --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-zlib --disable-libcdio --disable-libiec61883 --disable-libdc1394 --disable-libcaca --disable-openal --enable-opengl --enable-libv4l2 --disable-libpulse --disable-libdrm --disable-libjack --disable-libopencore-amrwb --disable-libopencore-amrnb --disable-libcodec2 --disable-libfdk-aac --disable-libopenjpeg --disable-libbluray --disable-libgme --disable-libgsm --disable-mmal --disable-libmodplug --disable-libopus --disable-libilbc --disable-librtmp --disable-libssh --disable-libspeex --disable-libsrt --enable-librsvg --disable-ffnvcodec --enable-libvorbis --disable-libvpx --disable-libzvbi --disable-appkit --disable-libbs2b --disable-chromaprint --disable-libflite --disable-frei0r --disable-libfribidi --disable-fontconfig --disable-ladspa --disable-libass --disable-lv2 --enable-libfreetype --disable-librubberband --disable-libzmq --disable-libzimg --disable-libsoxr --enable-pthreads --disable-libvo-amrwbenc --enable-libmp3lame --disable-libkvazaar --disable-libaom --disable-libopenh264 --disable-libsnappy --enable-libtheora --disable-libtwolame --disable-libwavpack --disable-libwebp --enable-libx264 --disable-libx265 --enable-libxvid --disable-gnutls --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-altivec --disable-amd3dnow --disable-amd3dnowext --disable-aesni --disable-avx --disable-avx2 --disable-fma3 --disable-fma4 --disable-mmx --disable-mmxext --disable-sse --disable-sse2 --disable-sse3 --disable-ssse3 --disable-sse4 --disable-sse42 --disable-xop --cpu=armv8-a+crc --disable-doc --disable-htmlpages --enable-manpages
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
 DEV.L. h263                 H.263 / H.263-1996, H.263+ / H.263-1998 / H.263 version 2 (decoders: h263 h263_v4l2m2m ) (encoders: h263 h263_v4l2m2m )
 DEV.LS h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 h264_v4l2m2m ) (encoders: libx264 libx264rgb h264_v4l2m2m )
 DEV.L. hevc                 H.265 / HEVC (High Efficiency Video Coding) (decoders: hevc hevc_v4l2m2m ) (encoders: hevc_v4l2m2m )
 DEV.L. mpeg1video           MPEG-1 video (decoders: mpeg1video mpeg1_v4l2m2m )
 DEV.L. mpeg2video           MPEG-2 video (decoders: mpeg2video mpegvideo mpeg2_v4l2m2m )
 DEV.L. mpeg4                MPEG-4 part 2 (decoders: mpeg4 mpeg4_v4l2m2m ) (encoders: mpeg4 libxvid mpeg4_v4l2m2m )
 D.V.L. vc1                  SMPTE VC-1 (decoders: vc1 vc1_v4l2m2m )
 DEV.L. vp8                  On2 VP8 (decoders: vp8 vp8_v4l2m2m ) (encoders: vp8_v4l2m2m )
 D.V.L. vp9                  Google VP9 (decoders: vp9 vp9_v4l2m2m )

thanks!
sakaki

[sakaki]

OK well I have got somewhere with this, but still there must be something obvious I'm missing.
Download the following small sample file (<1MB):

Code: Select all

demouser@pi64 ~ $ wget -c http://mirrors.standaloneinstaller.com/video-sample/grb_2.webm

Transcode it to h264 in software:

Code: Select all

demouser@pi64 ~ $ time ffmpeg -i grb_2.webm -vcodec h264 grb_2.mov
ffmpeg version 4.1.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8.2.0 (Gentoo 8.2.0-r6 p1.7)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --docdir=/usr/share/doc/ffmpeg-4.1.1/html --mandir=/usr/share/man --enable-shared --cc=aarch64-unknown-linux-gnu-gcc --cxx=aarch64-unknown-linux-gnu-g++ --ar=aarch64-unknown-linux-gnu-ar --optflags='-march=armv8-a+crc -mtune=cortex-a53 -O2 -pipe' --disable-static --enable-avfilter --enable-avresample --disable-stripping --disable-optimizations --disable-libcelt --enable-nonfree --disable-indev=oss --disable-indev=jack --disable-outdev=oss --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-gcrypt --disable-gnutls --disable-gmp --enable-gpl --enable-hardcoded-tables --enable-iconv --disable-libtls --disable-lzma --enable-network --disable-opencl --enable-openssl --enable-postproc --disable-libsmbclient --enable-ffplay --enable-sdl2 --disable-vaapi --disable-vdpau --enable-xlib --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-zlib --disable-libcdio --disable-libiec61883 --disable-libdc1394 --disable-libcaca --disable-openal --enable-opengl --enable-libv4l2 --disable-libpulse --disable-libdrm --disable-libjack --disable-libopencore-amrwb --disable-libopencore-amrnb --disable-libcodec2 --disable-libfdk-aac --disable-libopenjpeg --disable-libbluray --disable-libgme --disable-libgsm --disable-mmal --disable-libmodplug --disable-libopus --disable-libilbc --disable-librtmp --disable-libssh --disable-libspeex --disable-libsrt --enable-librsvg --disable-ffnvcodec --enable-libvorbis --disable-libvpx --disable-libzvbi --disable-appkit --disable-libbs2b --disable-chromaprint --disable-libflite --disable-frei0r --disable-libfribidi --disable-fontconfig --disable-ladspa --disable-libass --disable-lv2 --enable-libfreetype --disable-librubberband --disable-libzmq --disable-libzimg --disable-libsoxr --enable-pthreads --disable-libvo-amrwbenc --enable-libmp3lame --disable-libkvazaar --disable-libaom --disable-libopenh264 --disable-libsnappy --enable-libtheora --disable-libtwolame --disable-libwavpack --disable-libwebp --enable-libx264 --disable-libx265 --enable-libxvid --disable-gnutls --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-altivec --disable-amd3dnow --disable-amd3dnowext --disable-aesni --disable-avx --disable-avx2 --disable-fma3 --disable-fma4 --disable-mmx --disable-mmxext --disable-sse --disable-sse2 --disable-sse3 --disable-ssse3 --disable-sse4 --disable-sse42 --disable-xop --cpu=armv8-a+crc --disable-doc --disable-htmlpages --enable-manpages
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, matroska,webm, from 'grb_2.webm':
  Metadata:
    encoder         : Lavf57.66.105
  Duration: 00:00:27.89, start: 0.033000, bitrate: 157 kb/s
    Stream #0:0: Video: vp9 (Profile 0), yuv420p(tv), 720x480, SAR 8:9 DAR 4:3, 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc (default)
Stream mapping:
  Stream #0:0 -> #0:0 (vp9 (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x5599d11fe0] using SAR=8/9
[libx264 @ 0x5599d11fe0] using cpu capabilities: ARMv8 NEON
[libx264 @ 0x5599d11fe0] profile High, level 3.0, 4:2:0, 8-bit
[libx264 @ 0x5599d11fe0] 264 - core 157 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mov, to 'grb_2.mov':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 720x480 [SAR 8:9 DAR 4:3], q=-1--1, 29.97 fps, 30k tbn, 29.97 tbc (default)
    Metadata:
      encoder         : Lavc58.35.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame=  835 fps= 45 q=-1.0 Lsize=     392kB time=00:00:27.76 bitrate= 115.8kbits/s speed=1.51x    
video:382kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.705321%
[libx264 @ 0x5599d11fe0] frame I:7     Avg QP:16.33  size:  4788
[libx264 @ 0x5599d11fe0] frame P:267   Avg QP:17.82  size:   811
[libx264 @ 0x5599d11fe0] frame B:561   Avg QP:18.63  size:   251
[libx264 @ 0x5599d11fe0] consecutive B-frames:  6.6%  9.6%  5.7% 78.1%
[libx264 @ 0x5599d11fe0] mb I  I16..4: 13.3% 77.7%  9.0%
[libx264 @ 0x5599d11fe0] mb P  I16..4:  1.3%  3.3%  0.2%  P16..4:  4.4%  1.3%  0.6%  0.0%  0.0%    skip:89.0%
[libx264 @ 0x5599d11fe0] mb B  I16..4:  0.2%  0.2%  0.0%  B16..8:  5.0%  0.3%  0.1%  direct: 0.4%  skip:93.7%  L0:51.9% L1:45.8% BI: 2.3%
[libx264 @ 0x5599d11fe0] 8x8 transform intra:69.2% inter:66.1%
[libx264 @ 0x5599d11fe0] coded y,uvDC,uvAC intra: 20.1% 25.0% 7.6% inter: 1.0% 1.2% 0.0%
[libx264 @ 0x5599d11fe0] i16 v,h,dc,p: 45% 28%  8% 20%
[libx264 @ 0x5599d11fe0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 48% 12% 28%  2%  2%  2%  2%  3%  2%
[libx264 @ 0x5599d11fe0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 42% 16% 22%  3%  5%  4%  4%  3%  3%
[libx264 @ 0x5599d11fe0] i8c dc,h,v,p: 70% 10% 14%  6%
[libx264 @ 0x5599d11fe0] Weighted P-Frames: Y:3.0% UV:1.1%
[libx264 @ 0x5599d11fe0] ref P L0: 69.5% 14.1% 12.0%  4.3%  0.1%
[libx264 @ 0x5599d11fe0] ref B L0: 70.0% 27.5%  2.5%
[libx264 @ 0x5599d11fe0] ref B L1: 97.3%  2.7%
[libx264 @ 0x5599d11fe0] kb/s:112.12

real	0m18.512s
user	1m5.972s
sys	0m1.471s

Resulting file plays fine. Now try forcing the v4l2m2m codec:

Code: Select all

demouser@pi64 ~ $ time ffmpeg -i grb_2.webm -vcodec h264_v4l2m2m grb_2-v4l2.mov
ffmpeg version 4.1.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8.2.0 (Gentoo 8.2.0-r6 p1.7)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --docdir=/usr/share/doc/ffmpeg-4.1.1/html --mandir=/usr/share/man --enable-shared --cc=aarch64-unknown-linux-gnu-gcc --cxx=aarch64-unknown-linux-gnu-g++ --ar=aarch64-unknown-linux-gnu-ar --optflags='-march=armv8-a+crc -mtune=cortex-a53 -O2 -pipe' --disable-static --enable-avfilter --enable-avresample --disable-stripping --disable-optimizations --disable-libcelt --enable-nonfree --disable-indev=oss --disable-indev=jack --disable-outdev=oss --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-gcrypt --disable-gnutls --disable-gmp --enable-gpl --enable-hardcoded-tables --enable-iconv --disable-libtls --disable-lzma --enable-network --disable-opencl --enable-openssl --enable-postproc --disable-libsmbclient --enable-ffplay --enable-sdl2 --disable-vaapi --disable-vdpau --enable-xlib --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-zlib --disable-libcdio --disable-libiec61883 --disable-libdc1394 --disable-libcaca --disable-openal --enable-opengl --enable-libv4l2 --disable-libpulse --disable-libdrm --disable-libjack --disable-libopencore-amrwb --disable-libopencore-amrnb --disable-libcodec2 --disable-libfdk-aac --disable-libopenjpeg --disable-libbluray --disable-libgme --disable-libgsm --disable-mmal --disable-libmodplug --disable-libopus --disable-libilbc --disable-librtmp --disable-libssh --disable-libspeex --disable-libsrt --enable-librsvg --disable-ffnvcodec --enable-libvorbis --disable-libvpx --disable-libzvbi --disable-appkit --disable-libbs2b --disable-chromaprint --disable-libflite --disable-frei0r --disable-libfribidi --disable-fontconfig --disable-ladspa --disable-libass --disable-lv2 --enable-libfreetype --disable-librubberband --disable-libzmq --disable-libzimg --disable-libsoxr --enable-pthreads --disable-libvo-amrwbenc --enable-libmp3lame --disable-libkvazaar --disable-libaom --disable-libopenh264 --disable-libsnappy --enable-libtheora --disable-libtwolame --disable-libwavpack --disable-libwebp --enable-libx264 --disable-libx265 --enable-libxvid --disable-gnutls --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-altivec --disable-amd3dnow --disable-amd3dnowext --disable-aesni --disable-avx --disable-avx2 --disable-fma3 --disable-fma4 --disable-mmx --disable-mmxext --disable-sse --disable-sse2 --disable-sse3 --disable-ssse3 --disable-sse4 --disable-sse42 --disable-xop --cpu=armv8-a+crc --disable-doc --disable-htmlpages --enable-manpages
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, matroska,webm, from 'grb_2.webm':
  Metadata:
    encoder         : Lavf57.66.105
  Duration: 00:00:27.89, start: 0.033000, bitrate: 157 kb/s
    Stream #0:0: Video: vp9 (Profile 0), yuv420p(tv), 720x480, SAR 8:9 DAR 4:3, 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc (default)
Stream mapping:
  Stream #0:0 -> #0:0 (vp9 (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0x5578982fe0] driver 'bm2835 mmal' on card 'mmal service 16.1'
[h264_v4l2m2m @ 0x5578982fe0] driver 'bcm2835-codec' on card 'bcm2835-codec'
    Last message repeated 1 times
[h264_v4l2m2m @ 0x5578982fe0] Using device /dev/video11
[h264_v4l2m2m @ 0x5578982fe0] driver 'bcm2835-codec' on card 'bcm2835-codec'
[h264_v4l2m2m @ 0x5578982fe0] Failed to set number of B-frames
    Last message repeated 1 times
[h264_v4l2m2m @ 0x5578982fe0] Failed to set timeperframeFailed to set header mode
[h264_v4l2m2m @ 0x5578982fe0] Failed to set gop size
[h264_v4l2m2m @ 0x5578982fe0] h264 profile not found
[h264_v4l2m2m @ 0x5578982fe0] Encoder adjusted: qmin (0), qmax (51)
[h264_v4l2m2m @ 0x5578982fe0] Failed to set minimum video quantizer scale
[h264_v4l2m2m @ 0x5578982fe0] Failed to set maximum video quantizer scale
Output #0, mov, to 'grb_2-v4l2.mov':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: h264 (h264_v4l2m2m) (avc1 / 0x31637661), yuv420p, 720x480 [SAR 8:9 DAR 4:3], q=2-31, 200 kb/s, 29.97 fps, 30k tbn, 29.97 tbc (default)
    Metadata:
      encoder         : Lavc58.35.100 h264_v4l2m2m
[mov @ 0x5578981e90] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
[mov @ 0x5578981e90] Non-monotonous DTS in output stream 0:0; previous: 834834, current: 0; changing to 834835. This may result in incorrect timestamps in the output file.
frame=  835 fps=205 q=-0.0 Lsize=     780kB time=00:00:27.82 bitrate= 229.8kbits/s speed=6.83x    
video:776kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.533368%

real	0m4.168s
user	0m10.225s
sys	0m0.723s

A lot quicker; however while the resulting video is playable, it has a weird green cast.

This is close enough to working it makes me think I just have some parameter that needs setting somewhere - any hints please?

Thanks!

sakaki

[6by9]

Hello,

I'm finalizing a release of my gentoo-on-rpi3-64bit image, which given recent stabilizations I've decided will now be shipping with a 4.19.y kernel (rather than 4.14.y, which was the original plan).

Amongst other things, this allows the use of the pi camera module v2 in a full 64-bit system via /dev/video0, which is nice.

It'll also run the V1 camera module quite happily.

Some v4l2 m2m codecs also appear (/dev/video10 etc.). But to be honest I am stumbling around with some of this, so I would really appreciate a few pointers on the following:

• To get the camera interfaces to appear, I need to set "start_x=1" in /boot/config.txt, and this is true even for the m2m interfaces; is my understanding on that correct? I.e. if I just want m2m and have no camera module attached, I still need to use start_x=1?

The codecs are built into the standard start.elf and start_x.elf. They are NOT built into start_cd.elf (cutdown).

[*]I am current specifying "dtoverlay=vc4-kms-v3d,cma-256" and "gpu_mem=16" in /boot/config.txt; are those sensible settings? When the v4l2 camera or v4l2 m2m codecs interfaces are in use, does gpu_mem need to be increased or is this irrelevant?

"gpu_mem=16" will trigger the bootloader to load start_cd.elf, so no codecs.
The codecs need some gpu_mem to operate. For decode typically you'll be looking at a half dozen YUV420 frames, so at 1080P that's around 20MB. Running with 32 or 48M would be sensible.
To use the camera we normally suggest 128MB of gpu_mem, however as you have vc4-kms-v3d you should be able to run most use cases in 64MB.

[*]Would you expect the v4l2 m2m endpoints to operate correctly with a 64-bit userland? I have ffmpeg 4.1.1 compiled with the v4l extensions, but am getting segfaults every time I try to use h264_v4l2m2m. Do I have to set up some other init of the /dev/video1x interfaces to make them usable? Is there a simple ffmpeg incantation on a known (e.g. h264) source somewhere I can try to verify if the m2m stuff is working properly?

I haven't tried a 64bit userland, but I would expect it to work. I've tended to use GStreamer instead of ffmpeg. IIRC I had to make a couple of tweaks to ffmpeg to make it do sane things over selection regions (cropping the overall image, which is almost mandatory on codecs as they encode in 16x16 macroblocks and therefore 1080P isn't a nice multiple of macroblocks high). I'd have to dig out what I was doing, although it may not be required still.

GStreamer 1.14 should work without issues. If not running X, then I tend to run

Code: Select all

gst-launch-1.0 -e -vvv filesrc location=foo.mp4 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 ! kmssink

Under X you should be able to use glimagesink instead of kmssink, hwoever that involves a few extra conversions to get it rendered and I haven't checked overall performance.

[6by9]

OK, 3 ffmpeg patches that still appear to apply to top of tree.
Worth trying without them first, but I don't know how well that will work.

Code: Select all

commit 0550486542f4c4628abf9e5cf1090c8f328f215f
Author: Dave Stevenson <dave.stevenson@raspberrypi.org>
Date:   Thu Mar 22 16:01:35 2018 +0000

    v4l2_buffers: Add handling for NV21 and YUV420P

diff --git a/libavcodec/v4l2_buffers.c b/libavcodec/v4l2_buffers.c
index aef911f..0f4dbb2 100644
--- a/libavcodec/v4l2_buffers.c
+++ b/libavcodec/v4l2_buffers.c
@@ -321,11 +321,21 @@ int ff_v4l2_buffer_buf_to_avframe(AVFrame *frame, V4L2Buffer *avbuf)
     /* 1.1 fixup special cases */
     switch (avbuf->context->av_pix_fmt) {
     case AV_PIX_FMT_NV12:
+    case AV_PIX_FMT_NV21:
         if (avbuf->num_planes > 1)
             break;
         frame->linesize[1] = avbuf->plane_info[0].bytesperline;
         frame->data[1] = frame->buf[0]->data + avbuf->plane_info[0].bytesperline * avbuf->context->format.fmt.pix_mp.height;
         break;
+    case AV_PIX_FMT_YUV420P:
+    /* No YV12? support? */
+        if (avbuf->num_planes > 1)
+            break;
+        frame->linesize[1] = avbuf->plane_info[0].bytesperline >> 1;
+        frame->linesize[2] = avbuf->plane_info[0].bytesperline >> 1;
+        frame->data[1] = frame->buf[0]->data + avbuf->plane_info[0].bytesperline * avbuf->context->format.fmt.pix_mp.height;
+        frame->data[2] = frame->data[1] + ((avbuf->plane_info[0].bytesperline * avbuf->context->format.fmt.pix_mp.height) >> 2);
+        break;
     default:
         break;
     }
@@ -468,6 +478,7 @@ int ff_v4l2_buffer_enqueue(V4L2Buffer* avbuf)
 
     avbuf->buf.flags = avbuf->flags;
 
+    av_log(NULL, AV_LOG_ERROR, "ff_v4l2_buffer_enqueue: VIDIOC_QBUF : fd %d, index %u type %u\n", buf_to_m2mctx(avbuf)->fd, avbuf->buf.index, avbuf->buf.type);
     ret = ioctl(buf_to_m2mctx(avbuf)->fd, VIDIOC_QBUF, &avbuf->buf);
     if (ret < 0)
         return AVERROR(errno);

Code: Select all

commit 6d23e1aa79337d4f9468b8dadcebab824c01669b
Author: Dave Stevenson <dave.stevenson@raspberrypi.org>
Date:   Thu Mar 22 16:02:33 2018 +0000

    v4l2: Allow larger buffers than expected (but trim them down)

diff --git a/libavdevice/v4l2.c b/libavdevice/v4l2.c
index 1b9c6e7..062ef23 100644
--- a/libavdevice/v4l2.c
+++ b/libavdevice/v4l2.c
@@ -537,13 +537,14 @@ static int mmap_read_frame(AVFormatContext *ctx, AVPacket *pkt)
         if (ctx->video_codec_id == AV_CODEC_ID_CPIA)
             s->frame_size = buf.bytesused;
 
-        if (s->frame_size > 0 && buf.bytesused != s->frame_size) {
+        if (s->frame_size > 0 && buf.bytesused < s->frame_size) {
             av_log(ctx, AV_LOG_ERROR,
                    "Dequeued v4l2 buffer contains %d bytes, but %d were expected. Flags: 0x%08X.\n",
                    buf.bytesused, s->frame_size, buf.flags);
             enqueue_buffer(s, &buf);
             return AVERROR_INVALIDDATA;
         }
+        buf.bytesused = s->frame_size;
     }
 
     /* Image is at s->buff_start[buf.index] */

Code: Select all

commit b366d8435ba3ad133829abf24d07c83852bfeab9
Author: Dave Stevenson <dave.stevenson@raspberrypi.org>
Date:   Thu Apr 19 09:33:30 2018 +0100

    v4l2_m2m: Request selection correctly at appropriate times.
    
    On format changed the selection denoting the image crop
    may well change. Reread the selection on format changed and
    update width/height appropriately.
    
    When setting the selection specify the target.

diff --git a/libavcodec/v4l2_context.c b/libavcodec/v4l2_context.c
index efcb042..aec8ce3 100644
--- a/libavcodec/v4l2_context.c
+++ b/libavcodec/v4l2_context.c
@@ -146,6 +146,7 @@ static int v4l2_handle_event(V4L2Context *ctx)
     struct v4l2_format out_fmt = s->output.format;
     struct v4l2_event evt = { 0 };
     int full_reinit, reinit, ret;
+    struct v4l2_selection selection = {0};
 
     ret = ioctl(s->fd, VIDIOC_DQEVENT, &evt);
     if (ret < 0) {
@@ -178,6 +179,14 @@ static int v4l2_handle_event(V4L2Context *ctx)
     if (reinit) {
         s->capture.height = v4l2_get_height(&cap_fmt);
         s->capture.width = v4l2_get_width(&cap_fmt);
+
+        selection.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+        selection.target = V4L2_SEL_TGT_COMPOSE;
+        ret = ioctl(s->fd, VIDIOC_G_SELECTION, &selection);
+        if (ret) {
+            av_log(logger(ctx), AV_LOG_DEBUG, "crop output %dx%d\n", selection.r.width, selection.r.height);
+            /* update the size of the resulting frame */
+        }
     }
 
     if (full_reinit || reinit)
@@ -197,6 +206,11 @@ static int v4l2_handle_event(V4L2Context *ctx)
         if (ret < 0)
             av_log(logger(ctx), AV_LOG_WARNING, "update avcodec height and width\n");
 
+        if (selection.r.width && selection.r.height) {
+            s->avctx->width  = selection.r.width;
+            s->avctx->height = selection.r.height;
+        }
+
         ret = ff_v4l2_m2m_codec_reinit(s);
         if (ret) {
             av_log(logger(ctx), AV_LOG_ERROR, "v4l2_m2m_codec_reinit\n");
diff --git a/libavcodec/v4l2_m2m_dec.c b/libavcodec/v4l2_m2m_dec.c
index 710e40e..81f727a 100644
--- a/libavcodec/v4l2_m2m_dec.c
+++ b/libavcodec/v4l2_m2m_dec.c
@@ -38,7 +38,7 @@ static int v4l2_try_start(AVCodecContext *avctx)
     V4L2m2mContext *s = ((V4L2m2mPriv*)avctx->priv_data)->context;
     V4L2Context *const capture = &s->capture;
     V4L2Context *const output = &s->output;
-    struct v4l2_selection selection;
+    struct v4l2_selection selection = {0};
     int ret;
 
     /* 1. start the output process */
@@ -67,8 +67,11 @@ static int v4l2_try_start(AVCodecContext *avctx)
 
     /* 3. set the crop parameters */
     selection.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
-    selection.r.height = avctx->coded_height;
-    selection.r.width = avctx->coded_width;
+    selection.r.height = avctx->height;
+    selection.r.width = avctx->width;
+    selection.r.left = 0;
+    selection.r.top = 0;
+    selection.target = V4L2_SEL_TGT_COMPOSE;
     ret = ioctl(s->fd, VIDIOC_S_SELECTION, &selection);
     if (!ret) {
         ret = ioctl(s->fd, VIDIOC_G_SELECTION, &selection);

[sakaki]

Thanks for the detailed responses 6by9, very useful, thanks!

Well, I can confirm that:

Code: Select all

demouser@pi64 ~ $ wget -c http://mirrors.standaloneinstaller.com/video-sample/grb_2.mp4
demouser@pi64 ~ $ gst-launch-1.0 -e -vvv filesrc location=grb_2.mp4 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 ! glimagesink

works on my Gentoo test system, so hardware-assisted video codecs have finally arrived on 64-bit userland for the RPi3! \^-^/

I'll try applying your patches now and see if they improve things for ffmpeg.

Best, sakaki

[sakaki]

OK, 3 ffmpeg patches that still appear to apply to top of tree.
Worth trying without them first, but I don't know how well that will work.

OK, tried these, they apply fine to 4.1.1 ffmpeg.

The good news is that these deltas make m2m codecs work! You can now do e.g.:

Code: Select all

demouser@pi64 ~ $ wget -c http://jell.yfish.us/media/jellyfish-10-mbps-hd-h264.mkv
demouser@pi64 ~ $ ffplay -vcodec h264_v4l2m2m -i jellyfish-10-mbps-hd-h264.mkv

And it displays smoothly! Playing this file using s/w decoding is laggy and takes up all the cores, so the improvement is really marked. Also tried with mpeg4 on a different file; that works too. Haven't tested mpeg2 yet as I don't have an RPi3 with the license applied, but I assume that would also be fine.

However, the bad news is that capturing from the camera now seems to hang during the probe phase, if e.g. you do:

Code: Select all

demouser@pi64 ~ $ ffmpeg -f video4linux2 -input_format h264 -video_size 1280x720 -framerate 30 -i /dev/video0 -vcodec copy -an test.h264

it just stops; if you ctrl-c out of it, it says:

Code: Select all

[video4linux2,v4l2 @ 0x55b683e820] Could not find codec parameters for stream 0 (Video: h264, none, 1280x720): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264, none, 1280x720, 30 fps, 30 tbr, 1000k tbn, 2000k tbc
Output #0, h264, to 'test.h264':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: h264, none, 1280x720, q=2-31, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame=    0 fps=0.0 q=-1.0 Lsize=       0kB time=00:00:00.00 bitrate=N/A speed=   0x    
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
Exiting normally, received signal 2.

The effect is repeatable: if I apply the patches, I get the video decode working but no camera capture; if I do not apply them, I get probing (and hence camera capture) working but no video decode.

Here's the output from the same camera capture command as above, run under 4.1.1 with no patches:

Code: Select all

Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 2549.477482, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720, 30 fps, 30 tbr, 1000k tbn, 2000k tbc
Output #0, h264, to 'test.h264':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720, q=2-31, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame=  124 fps= 32 q=-1.0 Lsize=    3064kB time=00:00:04.06 bitrate=6172.9kbits/s speed=1.04x    
video:3064kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%

Is there something obvious that might need tweaking here? It's really very close and it'd be nice to have both codecs and camera functioning ^-^

Thanks, sakaki

[sakaki]

OK a bit of bisecting later... the problem appears to be with the second patch, 6d23e1aa79337d4f9468b8dadcebab824c01669b.

Dropping it, but retaining the two others, seems to allow both video capture and codec use.

So, going to go with the two-patch variant for now.

hth, sakaki

[6by9]

Interesting. I seem to recall the second patch was for the camera where for eg 1080p yu12 we need the buffer to be allocated as 1920x1088 and it then compresses the chroma planes down. Anything with height not being a multiple 16.

I really ought to upstream the patches for nv21 and yu12. Not sure about the selection ones - it was more trying to get a sensible from it when bringing the driver up. I have to assume that it works against some driver somewhere.

[sakaki]

[*]I am current specifying "dtoverlay=vc4-kms-v3d,cma-256" and "gpu_mem=16" in /boot/config.txt; are those sensible settings? When the v4l2 camera or v4l2 m2m codecs interfaces are in use, does gpu_mem need to be increased or is this irrelevant?

"gpu_mem=16" will trigger the bootloader to load start_cd.elf, so no codecs.
The codecs need some gpu_mem to operate. For decode typically you'll be looking at a half dozen YUV420 frames, so at 1080P that's around 20MB. Running with 32 or 48M would be sensible.
To use the camera we normally suggest 128MB of gpu_mem, however as you have vc4-kms-v3d you should be able to run most use cases in 64MB.

So interestingly, things seemed to work fine (both camera and v4l2 m2m) with gpu_mem=16 in /boot/config.txt; also worked for gpu_mem >=128, but would fail for e.g. gpu_mem=64.

Looking at the actual memory split with vcgencmd in each case, it seems that 128MiB is automatically getting allocated when the camera is used (start_x=1) but the minimal gpu_mem=16 is requested (i.e. something is overriding this 'obviously too small' value). At 'sensible' settings of gpu_mem (like 64) however, the setting is respected, although this proves to be insufficient to run e.g. the camera and an m2m decoder together.

Just in case this catches anyone else out in the future. gpu_mem=128 appears to be a sensible default.

Best, sakaki

[kirgene]

I'm getting a bunch of errors in dmesg (20-22 lines) while starting playback and a few such errors during playback:

Code: Select all

...
[ 6600.658148] vc_sm_cma_import_dmabuf_internal: Expecting an uncached alias for dma_addr 0x4ab00000
[ 6600.680280] vc_sm_cma_import_dmabuf_internal: Expecting an uncached alias for dma_addr 0x4ae00000
[ 6600.729710] vc_sm_cma_import_dmabuf_internal: Expecting an uncached alias for dma_addr 0x4b100000
...

I'm using https://github.com/6by9/FFmpeg (branch v4l2-drmprime)
The playback seems to be running fine despite the errors.

[6by9]

I'm getting a bunch of errors in dmesg (20-22 lines) while starting playback and a few such errors during playback:

Code: Select all

...
[ 6600.658148] vc_sm_cma_import_dmabuf_internal: Expecting an uncached alias for dma_addr 0x4ab00000
[ 6600.680280] vc_sm_cma_import_dmabuf_internal: Expecting an uncached alias for dma_addr 0x4ae00000
[ 6600.729710] vc_sm_cma_import_dmabuf_internal: Expecting an uncached alias for dma_addr 0x4b100000
...

I'm using https://github.com/6by9/FFmpeg (branch v4l2-drmprime)
The playback seems to be running fine despite the errors.

It's really a warning than an error because vcsm-cma corrects for this incorrect address. I made it shouty because it does mean something is badly wrong in the configuration.
I thought that had been fixed for Pi0-3 with https://github.com/raspberrypi/linux/co ... b9b5892681. Pi4 needs https://github.com/raspberrypi/linux/co ... b9b5892681

[warped]

Hi,
I'm trying to get v4l2 m2m video decode working on rpi3 with 64bit 5.2 mainline kernel in my distro (minimyth2).
I backported https://github.com/raspberrypi/linux/tr ... 4_services to 5.2 mainline.
So far I'm getting following modules loaded by 5.2 mainline kernel:

Code: Select all

Module                  Size  Used by
bcm2835_codec          36864  0
v4l2_mem2mem           24576  1 bcm2835_codec
videobuf2_dma_contig    20480  1 bcm2835_codec
gpu_sched              36864  0
rpcsec_gss_krb5        32768  0
evdev                  24576  2
ir_rc6_decoder         16384  0
rc_rc6_mce             16384  0
mceusb                 20480  0
rc_core                40960  4 mceusb,ir_rc6_decoder,rc_rc6_mce
vc4                   155648  0
cec                    45056  1 vc4
drm_kms_helper        180224  1 vc4
fb_sys_fops            16384  1 drm_kms_helper
sysimgblt              16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
syscopyarea            16384  1 drm_kms_helper
drm                   409600  3 vc4,gpu_sched,drm_kms_helper
drm_panel_orientation_quirks    16384  1 drm
snd_soc_core          184320  1 vc4
dwc2                  155648  0
snd_pcm_dmaengine      16384  1 snd_soc_core
snd_compress           20480  1 snd_soc_core
snd_pcm                90112  3 vc4,snd_pcm_dmaengine,snd_soc_core
udc_core               57344  1 dwc2
pwm_bcm2835            16384  0
snd_timer              32768  1 snd_pcm
bcm2835_v4l2           45056  0
snd                    69632  4 snd_compress,snd_timer,snd_soc_core,snd_pcm
videobuf2_v4l2         24576  3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
bcm2835_mmal_vchiq     28672  2 bcm2835_codec,bcm2835_v4l2
v4l2_common            16384  1 bcm2835_v4l2
raspberrypi_hwmon      16384  0
videobuf2_common       49152  4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
hwmon                  28672  1 raspberrypi_hwmon
videodev              188416  6 bcm2835_codec,v4l2_common,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
bcm2835_rng            16384  0
rng_core               16384  1 bcm2835_rng
i2c_bcm2835            16384  0
media                  36864  4 videobuf2_common,videodev,v4l2_mem2mem,videobuf2_v4l2
videobuf2_vmalloc      16384  1 bcm2835_v4l2
videobuf2_memops       16384  2 videobuf2_dma_contig,videobuf2_vmalloc
uio_pdrv_genirq        16384  0
uio                    20480  1 uio_pdrv_genirq
sg                     28672  0
uinput                 20480  1
ipv6                  458752  16

list of loaded modules looks reasonable for me....
Unfortunately there is no /dev/videoX

May somebody hint me with pointers how to get v4l2 m2m hw decode working on mainline 5.2 kernel?

[kilograham]

Code: Select all

demouser@pi64 ~ $ wget -c http://jell.yfish.us/media/jellyfish-10-mbps-hd-h264.mkv
demouser@pi64 ~ $ ffplay -vcodec h264_v4l2m2m -i jellyfish-10-mbps-hd-h264.mkv

And it displays smoothly! Playing this file using s/w decoding is laggy and takes up all the cores, so the improvement is really marked. Also tried with mpeg4 on a different file; that works too. Haven't tested mpeg2 yet as I don't have an RPi3 with the license applied, but I assume that would also be fine.

I am trying this under 32 bit Raspbian on Pi 4, and ffplay is dropping frames with v4l2m2m (software decode looks better)... was wondering if you had tried this with your Gentoo under Pi 4?

[tmm1]

A lot quicker; however while the resulting video is playable, it has a weird green cast.

I'm seeing the same green tint on stock raspbian with rpi3+, using ffmpeg 4.2 and even with the yuv420p patch above applied.

Is there an ffmpeg fork with more up to date patches? Do you mind if I submit these patches to ffmpeg-devel for review and potential merge?

[satadru]

Is there a working 5.x series arm64 kernel and working ffmpeg binary that allows hardware decoding of h264 to work?

This is what I'm seeing using current ffmpeg with the patches 1 & 3 mentioned above on the file mentioned above:

Code: Select all

DISPLAY=:0 ffplay -vcodec h264_v4l2m2m -i jellyfish-10-mbps-hd-h264.mkv
ffplay version 4.2.2-1 Copyright (c) 2003-2019 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.2.1-9ubuntu2)
  configuration: --prefix=/usr --extra-version=1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-indev=oss --disable-indev=jack --disable-outdev=oss --enable-bzlib --enable-gpl --enable-postproc --enable-ffplay --enable-xlib --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-zlib --enable-opengl --enable-libv4l2 --enable-libfreetype --enable-pthreads --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-libxvid --cpu=armv8-a+crc --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, matroska,webm, from 'jellyfish-10-mbps-hd-h264.mkv':0/0   
  Metadata:
    encoder         : libebml v1.2.0 + libmatroska v1.1.0
    creation_time   : 2016-02-06T04:00:51.000000Z
  Duration: 00:00:30.03, start: 0.000000, bitrate: 9955 kb/s
    Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1k tbn, 59.94 tbc (default)
[h264_v4l2m2m @ 0x7f7c004f70] driver 'bcm2835-codec' on card 'bcm2835-codec-isp'
[h264_v4l2m2m @ 0x7f7c004f70] driver 'bcm2835-codec' on card 'bcm2835-codec-encode'
[h264_v4l2m2m @ 0x7f7c004f70] driver 'bcm2835-codec' on card 'bcm2835-codec-decode'
[h264_v4l2m2m @ 0x7f7c004f70] Using device /dev/video10
[h264_v4l2m2m @ 0x7f7c004f70] driver 'bcm2835-codec' on card 'bcm2835-codec-decode'
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0

Weirdly it wants to use /dev/video10 when it appears that it should be using /dev/video11

Code: Select all

v4l2-ctl --list-formats -d /dev/video10
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

	[0]: 'YU12' (Planar YUV 4:2:0)
	[1]: 'YV12' (Planar YVU 4:2:0)
	[2]: 'NV12' (Y/CbCr 4:2:0)
	[3]: 'NV21' (Y/CrCb 4:2:0)
	[4]: 'RGBP' (16-bit RGB 5-6-5)
v4l2-ctl --list-formats -d /dev/video11
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

	[0]: 'H264' (H.264, compressed)

System info:

Code: Select all

cat /proc/device-tree/model 
Raspberry Pi 3 Model B Plus Rev 1.3
uname -a
Linux bridge 5.5.2-g6a3905568-raspi2 #1 SMP Fri Feb 14 15:56:18 EST 2020 aarch64 aarch64 aarch64 GNU/Linux

I don't actually have a camera attached. I just want hardware video decoding.

My config.txt has the following relevant lines:

Code: Select all

dtoverlay=vc4-kms-v3d,cma-256
gpu_mem=128
start_x=1

[6by9]

V4L2 M2M devices have a CAPTURE and an OUTPUT queue. These were named based on webcam/tuner devices being CAPTURE, and screen rendering devices being OUTPUT. So transferring that to M2M the input queue consumes the data, same as the OUTPUT queue, and the output side producing the data is the CAPTURE queue. Yes, the OUTPUT queue is the input to the codec. Bad naming or what!

"v4l2-ctl --list-formats-out -d 10" will list the input (OUTPUT) formats to /dev/video10, whilst "v4l2-ctl --list-formats -d 10" lists the output (CAPTURE) formats.

Code: Select all

pi@raspberrypi:~ $ v4l2-ctl -d 10 --list-formats-out
ioctl: VIDIOC_ENUM_FMT
	Type: Video Output Multiplanar

	[0]: 'H264' (H.264, compressed)
	[1]: 'MJPG' (Motion-JPEG, compressed)
pi@raspberrypi:~ $ v4l2-ctl -d 10 --list-formats
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

	[0]: 'YU12' (Planar YUV 4:2:0)
	[1]: 'YV12' (Planar YVU 4:2:0)
	[2]: 'NV12' (Y/CbCr 4:2:0)
	[3]: 'NV21' (Y/CrCb 4:2:0)
	[4]: 'RGBP' (16-bit RGB 5-6-5)

So it takes in H264 or MJPEG, and produces YU12, YV12, NV12, NV21, or RGB565.

/dev/video11 is the encoder

Code: Select all

pi@raspberrypi:~ $ v4l2-ctl -d 11 --list-formats-out
ioctl: VIDIOC_ENUM_FMT
	Type: Video Output Multiplanar

	[0]: 'YU12' (Planar YUV 4:2:0)
	[1]: 'YV12' (Planar YVU 4:2:0)
	[2]: 'NV12' (Y/CbCr 4:2:0)
	[3]: 'NV21' (Y/CrCb 4:2:0)
	[4]: 'RGBP' (16-bit RGB 5-6-5)
	[5]: 'RGB3' (24-bit RGB 8-8-8)
	[6]: 'BGR3' (24-bit BGR 8-8-8)
	[7]: 'BGR4' (32-bit BGRA/X 8-8-8-8)
	[8]: 'YUYV' (YUYV 4:2:2)
	[9]: 'YVYU' (YVYU 4:2:2)
	[10]: 'UYVY' (UYVY 4:2:2)
	[11]: 'VYUY' (VYUY 4:2:2)
pi@raspberrypi:~ $ v4l2-ctl -d 11 --list-formats
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

	[0]: 'H264' (H.264, compressed)

All those formats in, and H264 out. (Hmm, MJPEG should be listed there too).

The original patches were written when we were using the single planar API. We swapped to the multi planar API for compatibility with Chromium, and I suspect FFmpeg isn't handling the single planar formats via the multi planar API. Kodi almost certainly has patches for this if they haven't hit mainline FFmpeg. It was the equivalent of v4l2_buffer_buf_to_swframe that was missing previously.

[satadru]

Is there any chance of these patches making it into the raspbian FFmpeg? Trying to use stock FFmpeg on raspbian I get this:

Trying

Code: Select all

DISPLAY=:0 ffplay -vcodec h264_v4l2m2m -i jellyfish-10-mbps-hd-h264.mkv

on

Code: Select all

vcgencmd version
Apr 20 2020 15:42:40 
Copyright (c) 2012 Broadcom
version ed60c22eaee6ee7216b132591919025cba0f68e5 (clean) (release) (start_x)
uname -a
Linux raspberrypi 4.19.115-v7+ #1305 SMP Fri Apr 17 11:53:39 BST 2020 armv7l GNU/Linux
cat /etc/issue
Raspbian GNU/Linux 10 \n \l

And Im still seeing this:

Code: Select all

[h264_v4l2m2m @ 0x65f07f00] driver 'bcm2835-codec' on card 'bcm2835-codec-isp'
[h264_v4l2m2m @ 0x65f07f00] driver 'bcm2835-codec' on card 'bcm2835-codec-encode'
[h264_v4l2m2m @ 0x65f07f00] driver 'bcm2835-codec' on card 'bcm2835-codec-decode'
[h264_v4l2m2m @ 0x65f07f00] Using device /dev/video10
[h264_v4l2m2m @ 0x65f07f00] driver 'bcm2835-codec' on card 'bcm2835-codec-decode'
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0   
Mixed negative and positive linesizes are not supported. 0B f=0/0 

Code: Select all

v4l2-ctl -d 11 --list-formats
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'H264' (H.264, compressed)
        [1]: 'MJPG' (Motion-JPEG, compressed)
v4l2-ctl -d 10 --list-formats
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'YU12' (Planar YUV 4:2:0)
        [1]: 'YV12' (Planar YVU 4:2:0)
        [2]: 'NV12' (Y/CbCr 4:2:0)
        [3]: 'NV21' (Y/CrCb 4:2:0)
        [4]: 'RGBP' (16-bit RGB 5-6-5)

(No difference using the 32 bit or 64 bit 4.19 or 5.4 kernels out of hexxeh/rpi-firmware - master or next.)