Convert color format using GPU in Gstreamer (alternative to videoconvert).

[Lich]

Hello!

I use OpenCV with Gstreamer. Color format conversion is required to transfer video frames from Gstreamer to OpenCV and back.

The Gstreamer "omxh264dec" decoder has the RGBA output color format, "glimagesink" has the RGBA input color format. OpenCV uses the RGB/BGR/GRAY formats (without alpha channel) and cannot work with the RGBA.

Currently, I use the "videoconvert" method to convert the color format. Piplines for VideoCapture() and VideoWriter():

Code: Select all

pipline_in ='rtspsrc location=rtsp://web_camera_ip latency=400 ! rtph264depay ! h264parse ! omxh264dec ! videoconvert ! appsink'
pipline_out ='appsrc ! videoconvert ! video/x-raw, format=RGBA ! glimagesink sync=false'

Unfortunately, videoconvert does not use the GPU and loads the CPU significantly - up to 60% when working with HD video (1280x720, 20 fps).
I also tried the glcolorconvert instead videoconvert, it's also slow

Code: Select all

... glupload ! glcolorconvert ! video/x-raw(memory:GLMemory), format=BGR ! appsink

1. How do I use the GPU to convert the color format?
2. Is it possible to change the output color format of the omxh264dec to BGR?
3. Will v4l2h264dec and v4l2convert help to reduce the CPU load?

Code: Select all

... ! v4l2h264dec ! v4l2convert ! video/x-raw, format=BGR ! appsink

Now I use the standard Raspbian Stretch gst-plugins-good version 1.10.4 and it does not contain v4l2h264dec and v4l2convert methods, I'm gonna need to compile a new version of this package.

Thanks in advance for answers.

[6by9]

The GStreamer plugin may support the mapping between Gstreamer and IL for RGBA, but that doesn't mean the underlying component does. The video_decode component supports I420/YU12, YV12, NV12, NV21, and RGB565. Conversion from YUV to RGB is a non-trivial operation.

v4l2h264dec and v4l2convert are supported with the 4.19 kernel and GStreamer 1.14 or later. Raspbian Buster has GStreamer 1.14, so no rebuilding required.

There are a couple of patches I'll recommend to the Raspbian maintainers to backport to make them more useful
https://github.com/GStreamer/gst-plugin ... 4c7b3d7994
https://github.com/GStreamer/gst-plugin ... 5ed8588abd

If you must stick with Stretch, then someone has previously posted a script for downloading and building the whole of Gstreamer - https://gist.githubusercontent.com/spha ... r-build.sh
Please ensure you understand the script before running it blindly (you should never run random scripts from the internet without some understanding of what they do, particularly if they use sudo).

I haven't worked out if GStreamer automatically uses dmabufs or not, but they do reduce the amount of buffer copying that is required.

Code: Select all

v4l2h264dec capture-io-mode=4 ! v4l2convert output-io-mode=5 ! appsink

saves a copy of the data between the decoder and conversion step.

[Lich]

6by9, thank you for the answer!

I plan to switch to Raspbian Buster and Raspberry Pi 4 (at Raspbian Stretch, I updated the kernel to 4.19). I hope that in Raspbian Buster with Gstreamer 1.14 I will be able to configure the gst-plugin-bad to output video through the GLES layer. In Raspbian Stretch I have compiled this plugin manually: https://www.raspberrypi.org/forums/view ... c&start=50. Also in Raspbian Stretch I used your version of gst-omx (gstreamer-1.0-omx-rpi_1.10.5-2+rp+patches_armhf.deb) to make omxh264dec and omxh264enc work.

About conversion from YUV (this format is used in h264 video). As I understand it, conversion from YUV to RGBA is performed by "omxh264dec" method using the GPU, no additional slow "videoconvert" method is required.

The problem is converting the RGBA format to RGB/BGR and vice versa. For example, the first pipline loads the CPU by 5%, the second pipline loads the CPU by 20%:

Code: Select all

gst-launch-1.0 rtspsrc location='web_camera_ip' latency=400 ! rtph264depay ! h264parse ! omxh264dec ! glimagesink sync=false

Code: Select all

gst-launch-1.0 rtspsrc location='web_camera_ip' latency=400 ! queue ! rtph264depay ! h264parse ! omxh264dec ! videoconvert ! video/x-raw, format=BGR ! glimagesink sync=false

(The second line gives out a black screen because glimagesink requires RGBA color format for the input image)

I will try to use v4l2h264dec and v4l2convert in my project. Оne question: what do "capture-io-mode=4" and "output-io-mode=5 mean"? Do I need to add " ! video/x-raw, format=BGR ! "?


The new installation of Raspbian Buster and OpenCV will take some time, but I will definitely post the results in this thread. I really hope that I can use Raspberry to recognize images in real time.

[6by9]

6by9, thank you for the answer!

I plan to switch to Raspbian Buster and Raspberry Pi 4 (at Raspbian Stretch, I updated the kernel to 4.19). I hope that in Raspbian Buster with Gstreamer 1.14 I will be able to configure the gst-plugin-bad to output video through the GLES layer. In Raspbian Stretch I have compiled this plugin manually: https://www.raspberrypi.org/forums/view ... 6&start=50. Also in Raspbian Stretch I used your version of gst-omx (gstreamer-1.0-omx-rpi_1.10.5-2+rp+patches_armhf.deb) to make omxh264dec and omxh264enc work.

About conversion from YUV (this format is used in h264 video). As I understand it, conversion from YUV to RGBA is performed by "omxh264dec" method using the GPU, no additional slow "videoconvert" method is required.

There was some odd code in one version of gst-omx that connected together video decode and resize or egl_render (IIRC) to produce GL (RGBA) buffers, but I thought that was only in Jessie and not Stretch or Buster.

I can state categorically that the underlying video_decode component does NOT support RGBA, so something else is doing the conversion. If you're after RGB/BGR, then that conversion is a waste of time anyway and you're best off going direct from YUV to RGB/BGR.

The problem is converting the RGBA format to RGB/BGR and vice versa. For example, the first pipline loads the CPU by 5%, the second pipline loads the CPU by 20%:

Code: Select all

gst-launch-1.0 rtspsrc location='web_camera_ip' latency=400 ! rtph264depay ! h264parse ! omxh264dec ! glimagesink sync=false

Code: Select all

gst-launch-1.0 rtspsrc location='web_camera_ip' latency=400 ! queue ! rtph264depay ! h264parse ! omxh264dec ! videoconvert ! video/x-raw, format=BGR ! glimagesink sync=false

(The second line gives out a black screen because glimagesink requires RGBA color format for the input image)

I will try to use v4l2h264dec and v4l2convert in my project. Оne question: what do "capture-io-mode=4" and "output-io-mode=5 mean"? Do I need to add " ! video/x-raw, format=BGR ! "?

Code: Select all

pi@raspberrypi:~ $ gst-inspect-1.0 v4l2video12convert
Factory Details:
  Rank                     none (0)
  Long-name                V4L2 Video Converter
  Klass                    Filter/Converter/Video/Scaler
  Description              Transform streams via V4L2 API
  Author                   Nicolas Dufresne <nicolas.dufresne@collabora.com>

<snip>
Element Properties:
<snip>
  output-io-mode      : Output side I/O mode (matches sink pad)
                        flags: readable, writable
                        Enum "GstV4l2IOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (1): rw               - GST_V4L2_IO_RW
                           (2): mmap             - GST_V4L2_IO_MMAP
                           (3): userptr          - GST_V4L2_IO_USERPTR
                           (4): dmabuf           - GST_V4L2_IO_DMABUF
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT
  capture-io-mode     : Capture I/O mode (matches src pad)
                        flags: readable, writable
                        Enum "GstV4l2IOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (1): rw               - GST_V4L2_IO_RW
                           (2): mmap             - GST_V4L2_IO_MMAP
                           (3): userptr          - GST_V4L2_IO_USERPTR
                           (4): dmabuf           - GST_V4L2_IO_DMABUF
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT

So they are explcitly requesting dmabuf export from the codec, and dmabuf import for the converter.
V4L2 is odd in the queue naming. It was riginally written for capture devices such as webcams (data source), and output devices for video overlays (data sink). When it comes to memory to memory devices, the input is the data sink so is known as "output", and the output is the data source so is known as "capture". Lots of room for confusion there.

[Lich]

I installed Debian Buster. Unfortunately, gst-plugins-goo in version 1.14 does not contain v4l2convert. The following v24l components are available (list is obtained by gst-inspect-1.0):

video4linux2: v4l2h264enc: V4L2 H.264 Encoder
video4linux2: v4l2h264dec: V4L2 H264 Decoder
video4linux2: v4l2h263dec: V4L2 H263 Decoder
video4linux2: v4l2mpeg4dec: V4L2 MPEG4 Decoder
video4linux2: v4l2jpegdec: V4L2 JPEG Decoder
video4linux2: v4l2deviceprovider (GstDeviceProviderFactory)
video4linux2: v4l2radio: Radio (video4linux2) Tuner
video4linux2: v4l2sink: Video (video4linux2) Sink
video4linux2: v4l2src: Video (video4linux2) Source

What other packages do I need to install?

[6by9]

Note the two patches I'd linked to (they've been picked into our internal repos for testing, so should be available when I get a chance to do that)

One includes devices that support bayer formats as compatible with v4l2convert. We support bayer on the input side of our v4l2 device, however there is a kernel module parameter on bcm2835-codec to disable this to allow it to work with this older gstreamer. Add "bcm2835-codec.disable_bayer=1" to /boot/cmdline.txt to enable that.

The second patch changes the names so that the first instance shows up as v4l2convert and not v4l2video12convert.

[Lich]

I enabled v4l2video12convert with "bcm2835-codec.disable_bayer=1", thank you very much!

The first experiment:

Code: Select all

gst-launch-1.0 --gst-debug-level=3 filesrc location=/path/test.mp4 ! qtdemux ! h264parse ! omxh264dec ! videoconvert ! video/x-raw, format=BGR ! fakesink

CPU load is 25%

Code: Select all

gst-launch-1.0 --gst-debug-level=3 filesrc location=/path/test.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 ! v4l2video12convert output-io-mode=5 capture-io-mode=4 ! video/x-raw, format=BGR ! fakesink

CPU load is 6%

But I don't know how it's gonna work with "appsink". I haven't installed OpenCV on my new Raspbian Buster yet.

In parallel, I am trying different "sink" methods for output because Buster supports the OpenGL driver. For example, "kmssink" does not work with the OpenGL driver turned on or off. Now I've compiled a gst-plugins-bad with OpenGL support, and "glimagesink" plays video with hardware acceleration in the window (GLES only works on a full screen layer).

In theory, OpenCV can also be compiled with OpenGL support, but this may not work with Raspberry.

[6by9]

But I don't know how it's gonna work with "appsink". I haven't installed OpenCV on my new Raspbian Buster yet.

You stick appsink on the end and it should just work. It is the buffers being passed between the decoder and converter that are dmabufs.

In parallel, I am trying different "sink" methods for output because Buster supports the OpenGL driver. For example, "kmssink" does not work with the OpenGL driver turned on or off. Now I've compiled a gst-plugins-bad with OpenGL support, and "glimagesink" plays video with hardware acceleration in the window (GLES only works on a full screen layer).

In theory, OpenCV can also be compiled with OpenGL support, but this may not work with Raspberry.

kmssink talks to KMS directly, therefore X can not be running. (KMS supports a single master at a time, and X will have taken that role).

[Lich]

I started testing OpenCV with Gstreamer. I use v4l2 methods to decode video stream and convert color format. Video is displayed using the OpenGL renderer.

Test video streaming - H264, 1280x720, 20 fps

Transferring a frame from Gstreamer to OpenCV (display disabled):

Code: Select all

pipline_r = 'rtspsrc location=rtsp://web_camera_ip latency=100 ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 ! v4l2video12convert output-io-mode=5 capture-io-mode=4 ! appsink sync=false'
vs = cv2.VideoCapture(pipline_r)

while True:
    ret, frame = vs.read()
    if ret:
      cv2.waitKey(1)

CPU load is 13%

Transferring a frame from OpenCV to Gstreamer and displaying it through OpenGL rendering:

Code: Select all

pipline_r = 'rtspsrc location=rtsp://web_camera_ip latency=100 ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 ! v4l2video12convert output-io-mode=5 capture-io-mode=4 ! appsink sync=false'
pipeline_s = "appsrc ! glimagesink sync=false"
vs = cv2.VideoCapture(pipline_r)
out = cv2.VideoWriter(pipeline_s, cv2.CAP_GSTREAMER, 0, 20.0, (1280,720))
 
while True:
    ret, frame = vs.read()
    if ret:
      out.write(frame)
      cv2.waitKey(1)

CPU load is 38%, delay is minimal (real-time video).

The best result shows the use of the imshow method to display:

Code: Select all

pipline_r = rtspsrc location=rtsp://web_camera_ip latency=100 ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 ! v4l2video12convert output-io-mode=5 capture-io-mode=4 ! appsink sync=false
vs = cv2.VideoCapture(pipline_r)
 
while True:
    ret, frame = vs.read()
    if ret:
      cv2.imshow('',frame)
      cv2.waitKey(1)

CPU load is 32%, delay is minimal (real-time video).

Next, I will try to use the display via GLES. I want to minimize the CPU load on I/O tasks because image recognition requires a lot of computing resources. Are there other ways to optimize the I/O for my examples? Maybe changing appsink parameters, such as max-buffers, can help?

[kskill]

I have prepared a sample code to pull images from file src.
Pull images contains lines on the top
Can any one guide me into this.

Code: Select all

pipeline = gst_parse_launch
("filesrc location=/bunny-1080.mp4 !"
 "qtdemux !"
 "h264parse !"
 "v4l2h264dec capture-io-mode=4 !"
 "v4l2convert output-io-mode=5 capture-io-mode=4 !"
 "capsfilter caps=\"video/x-raw, format=BGR\" !"
 "appsink name=appsink",
 NULL); 

Extra lines only generated if i add output-io-mode=5 in v4l2convert plugin.
Input : bunny.mp4 https://streamable.com/o09pls
Output :

Same problem arises when run using gst-launch-1.0

Code: Select all

gst-launch-1.0 filesrc location=/bunny-1080.mp4 ! qtdemux !  h264parse ! v4l2h264dec capture-io-mode=4 !  v4l2convert output-io-mode=5 capture-io-mode=4 ! capsfilter caps=video/x-raw,format=BGR ! jpegenc ! multifilesink location=img%d.jpg

[6by9]

1080p is an awkward one from the codec.
Many (most) video codecs work in macroblocks of 16x16, and that includes H264. 1080 is not a multiple of 16, therefore the output image is 1920x1088, and is passed with cropping information to say only 1920x1080 pixels are active. The image you've provided would imply to me that that sizing information has not been passed across and the converter has been told that the image is 1920x1080, and therefore it computes the chroma planes (which come after the luma) to be 8 lines too early and it picks up the rubbish.

It'd need a little bit of checking to see whether the cropping information gets passed across between the GStreamer components or not. Adding -vvv to your gst-launch-1.0 command line will normally dump out all the props of each link.

Without the use of dmabufs GStreamer will be copying all the pixels between the decoder output buffer and the converter input buffer, and it'll be treating each plane independently and therefore getting it "right".

[kskill]

Hi thanks for your reply,

I have tried debugging the pipeline :

Code: Select all

gst-launch-1.0 -vvv filesrc location=/bunny-1080.mp4 ! qtdemux !  h264parse ! v4l2h264dec capture-io-mode=4 ! v4l2convert output-io-mode=5 capture-io-mode=4 ! capsfilter caps=video/x-raw,format=BGR ! jpegenc ! multifilesink location=img%d.jpg

I got following output :

Code: Select all

/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4, profile=(string)high, codec_data=(buffer)01640028ffe1001b67640028acd940780227e5c044000003000400000300c03c60c65801000668ebe3cb22c0, width=(int)1920, height=(int)1080, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1

/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)1920, height=(int)1080, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true

/GstPipeline:pipeline0/v4l2h264dec:v4l2h264dec0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)1920, height=(int)1080, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true

/GstPipeline:pipeline0/v4l2h264dec:v4l2h264dec0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)24/1

/GstPipeline:pipeline0/v4l2convert:v4l2convert0.GstPad:src: caps = video/x-raw, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)24/1, format=(string)BGR, width=(int)1920, height=(int)1080

/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)24/1, format=(string)BGR, width=(int)1920, height=(int)1080

/GstPipeline:pipeline0/GstJpegEnc:jpegenc0.GstPad:sink: caps = video/x-raw, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)24/1, format=(string)BGR, width=(int)1920, height=(int)1080

/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)24/1, format=(string)BGR, width=(int)1920, height=(int)1080

/GstPipeline:pipeline0/v4l2convert:v4l2convert0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)24/1

/GstPipeline:pipeline0/GstJpegEnc:jpegenc0.GstPad:src: caps = image/jpeg, sof-marker=(int)4, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)24/1, interlace-mode=(string)progressive, colorimetry=(string)sRGB, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono

/GstPipeline:pipeline0/GstMultiFileSink:multifilesink0.GstPad:sink: caps = image/jpeg, sof-marker=(int)4, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)24/1, interlace-mode=(string)progressive, colorimetry=(string)sRGB, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono

Can you guide me into this ?
Where should i pass the size parameter in above pipeline to get a correct image 1920x1080

[kskill]

Hi @6by9

Can you please help me out

[6by9]

Hi @6by9

Can you please help me out

Remove the capture-io-mode=4 and output-io-mode=5. Anything else is going to be a fairly deep dive into the inner workings of GStreamer.
On a CM4 it appears to be about 8% of 1 ARM CPU core, although admittedly it will hit memory bandwidth moderately hard.

I'm not sure I've implemented full cropping in the kernel driver, but it's not something I'm going to look at now.