Lich
Posts: 20
Joined: Wed Apr 11, 2018 8:41 pm
Location: Russia

Convert color format using GPU in Gstreamer (alternative to videoconvert).

Mon Jul 15, 2019 10:19 pm

Hello!

I use OpenCV with Gstreamer. Color format conversion is required to transfer video frames from Gstreamer to OpenCV and back.

The Gstreamer "omxh264dec" decoder has the RGBA output color format, "glimagesink" has the RGBA input color format. OpenCV uses the RGB/BGR/GRAY formats (without alpha channel) and cannot work with the RGBA.

Currently, I use the "videoconvert" method to convert the color format. Piplines for VideoCapture() and VideoWriter():

Code: Select all

pipline_in ='rtspsrc location=rtsp://web_camera_ip latency=400 ! rtph264depay ! h264parse ! omxh264dec ! videoconvert ! appsink'
pipline_out ='appsrc ! videoconvert ! video/x-raw, format=RGBA ! glimagesink sync=false'
Unfortunately, videoconvert does not use the GPU and loads the CPU significantly - up to 60% when working with HD video (1280x720, 20 fps).
I also tried the glcolorconvert instead videoconvert, it's also slow

Code: Select all

... glupload ! glcolorconvert ! video/x-raw(memory:GLMemory), format=BGR ! appsink

1. How do I use the GPU to convert the color format?
2. Is it possible to change the output color format of the omxh264dec to BGR?
3. Will v4l2h264dec and v4l2convert help to reduce the CPU load?

Code: Select all

... ! v4l2h264dec ! v4l2convert ! video/x-raw, format=BGR ! appsink
Now I use the standard Raspbian Stretch gst-plugins-good version 1.10.4 and it does not contain v4l2h264dec and v4l2convert methods, I'm gonna need to compile a new version of this package.

Thanks in advance for answers.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7279
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Convert color format using GPU in Gstreamer (alternative to videoconvert).

Tue Jul 16, 2019 9:23 am

The GStreamer plugin may support the mapping between Gstreamer and IL for RGBA, but that doesn't mean the underlying component does. The video_decode component supports I420/YU12, YV12, NV12, NV21, and RGB565. Conversion from YUV to RGB is a non-trivial operation.

v4l2h264dec and v4l2convert are supported with the 4.19 kernel and GStreamer 1.14 or later. Raspbian Buster has GStreamer 1.14, so no rebuilding required.

There are a couple of patches I'll recommend to the Raspbian maintainers to backport to make them more useful
https://github.com/GStreamer/gst-plugin ... 4c7b3d7994
https://github.com/GStreamer/gst-plugin ... 5ed8588abd

If you must stick with Stretch, then someone has previously posted a script for downloading and building the whole of Gstreamer - https://gist.githubusercontent.com/spha ... r-build.sh
Please ensure you understand the script before running it blindly (you should never run random scripts from the internet without some understanding of what they do, particularly if they use sudo).

I haven't worked out if GStreamer automatically uses dmabufs or not, but they do reduce the amount of buffer copying that is required.

Code: Select all

v4l2h264dec capture-io-mode=4 ! v4l2convert output-io-mode=5 ! appsink
saves a copy of the data between the decoder and conversion step.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Lich
Posts: 20
Joined: Wed Apr 11, 2018 8:41 pm
Location: Russia

Re: Convert color format using GPU in Gstreamer (alternative to videoconvert).

Tue Jul 16, 2019 9:43 pm

6by9, thank you for the answer!

I plan to switch to Raspbian Buster and Raspberry Pi 4 (at Raspbian Stretch, I updated the kernel to 4.19). I hope that in Raspbian Buster with Gstreamer 1.14 I will be able to configure the gst-plugin-bad to output video through the GLES layer. In Raspbian Stretch I have compiled this plugin manually: https://www.raspberrypi.org/forums/view ... c&start=50. Also in Raspbian Stretch I used your version of gst-omx (gstreamer-1.0-omx-rpi_1.10.5-2+rp+patches_armhf.deb) to make omxh264dec and omxh264enc work.

About conversion from YUV (this format is used in h264 video). As I understand it, conversion from YUV to RGBA is performed by "omxh264dec" method using the GPU, no additional slow "videoconvert" method is required.

The problem is converting the RGBA format to RGB/BGR and vice versa. For example, the first pipline loads the CPU by 5%, the second pipline loads the CPU by 20%:

Code: Select all

gst-launch-1.0 rtspsrc location='web_camera_ip' latency=400 ! rtph264depay ! h264parse ! omxh264dec ! glimagesink sync=false

Code: Select all

gst-launch-1.0 rtspsrc location='web_camera_ip' latency=400 ! queue ! rtph264depay ! h264parse ! omxh264dec ! videoconvert ! video/x-raw, format=BGR ! glimagesink sync=false
(The second line gives out a black screen because glimagesink requires RGBA color format for the input image)

I will try to use v4l2h264dec and v4l2convert in my project. Оne question: what do "capture-io-mode=4" and "output-io-mode=5 mean"? Do I need to add " ! video/x-raw, format=BGR ! "?


The new installation of Raspbian Buster and OpenCV will take some time, but I will definitely post the results in this thread. I really hope that I can use Raspberry to recognize images in real time.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7279
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Convert color format using GPU in Gstreamer (alternative to videoconvert).

Wed Jul 17, 2019 9:01 am

Lich wrote:
Tue Jul 16, 2019 9:43 pm
6by9, thank you for the answer!

I plan to switch to Raspbian Buster and Raspberry Pi 4 (at Raspbian Stretch, I updated the kernel to 4.19). I hope that in Raspbian Buster with Gstreamer 1.14 I will be able to configure the gst-plugin-bad to output video through the GLES layer. In Raspbian Stretch I have compiled this plugin manually: https://www.raspberrypi.org/forums/view ... 6&start=50. Also in Raspbian Stretch I used your version of gst-omx (gstreamer-1.0-omx-rpi_1.10.5-2+rp+patches_armhf.deb) to make omxh264dec and omxh264enc work.

About conversion from YUV (this format is used in h264 video). As I understand it, conversion from YUV to RGBA is performed by "omxh264dec" method using the GPU, no additional slow "videoconvert" method is required.
There was some odd code in one version of gst-omx that connected together video decode and resize or egl_render (IIRC) to produce GL (RGBA) buffers, but I thought that was only in Jessie and not Stretch or Buster.

I can state categorically that the underlying video_decode component does NOT support RGBA, so something else is doing the conversion. If you're after RGB/BGR, then that conversion is a waste of time anyway and you're best off going direct from YUV to RGB/BGR.
Lich wrote:The problem is converting the RGBA format to RGB/BGR and vice versa. For example, the first pipline loads the CPU by 5%, the second pipline loads the CPU by 20%:

Code: Select all

gst-launch-1.0 rtspsrc location='web_camera_ip' latency=400 ! rtph264depay ! h264parse ! omxh264dec ! glimagesink sync=false

Code: Select all

gst-launch-1.0 rtspsrc location='web_camera_ip' latency=400 ! queue ! rtph264depay ! h264parse ! omxh264dec ! videoconvert ! video/x-raw, format=BGR ! glimagesink sync=false
(The second line gives out a black screen because glimagesink requires RGBA color format for the input image)

I will try to use v4l2h264dec and v4l2convert in my project. Оne question: what do "capture-io-mode=4" and "output-io-mode=5 mean"? Do I need to add " ! video/x-raw, format=BGR ! "?

Code: Select all

[email protected]:~ $ gst-inspect-1.0 v4l2video12convert
Factory Details:
  Rank                     none (0)
  Long-name                V4L2 Video Converter
  Klass                    Filter/Converter/Video/Scaler
  Description              Transform streams via V4L2 API
  Author                   Nicolas Dufresne <[email protected]>

<snip>
Element Properties:
<snip>
  output-io-mode      : Output side I/O mode (matches sink pad)
                        flags: readable, writable
                        Enum "GstV4l2IOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (1): rw               - GST_V4L2_IO_RW
                           (2): mmap             - GST_V4L2_IO_MMAP
                           (3): userptr          - GST_V4L2_IO_USERPTR
                           (4): dmabuf           - GST_V4L2_IO_DMABUF
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT
  capture-io-mode     : Capture I/O mode (matches src pad)
                        flags: readable, writable
                        Enum "GstV4l2IOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (1): rw               - GST_V4L2_IO_RW
                           (2): mmap             - GST_V4L2_IO_MMAP
                           (3): userptr          - GST_V4L2_IO_USERPTR
                           (4): dmabuf           - GST_V4L2_IO_DMABUF
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT
So they are explcitly requesting dmabuf export from the codec, and dmabuf import for the converter.
V4L2 is odd in the queue naming. It was riginally written for capture devices such as webcams (data source), and output devices for video overlays (data sink). When it comes to memory to memory devices, the input is the data sink so is known as "output", and the output is the data source so is known as "capture". Lots of room for confusion there.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Lich
Posts: 20
Joined: Wed Apr 11, 2018 8:41 pm
Location: Russia

Re: Convert color format using GPU in Gstreamer (alternative to videoconvert).

Wed Jul 17, 2019 10:23 pm

I installed Debian Buster. Unfortunately, gst-plugins-goo in version 1.14 does not contain v4l2convert. The following v24l components are available (list is obtained by gst-inspect-1.0):

video4linux2: v4l2h264enc: V4L2 H.264 Encoder
video4linux2: v4l2h264dec: V4L2 H264 Decoder
video4linux2: v4l2h263dec: V4L2 H263 Decoder
video4linux2: v4l2mpeg4dec: V4L2 MPEG4 Decoder
video4linux2: v4l2jpegdec: V4L2 JPEG Decoder
video4linux2: v4l2deviceprovider (GstDeviceProviderFactory)
video4linux2: v4l2radio: Radio (video4linux2) Tuner
video4linux2: v4l2sink: Video (video4linux2) Sink
video4linux2: v4l2src: Video (video4linux2) Source

What other packages do I need to install?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7279
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Convert color format using GPU in Gstreamer (alternative to videoconvert).

Thu Jul 18, 2019 6:33 am

Note the two patches I'd linked to (they've been picked into our internal repos for testing, so should be available when I get a chance to do that)

One includes devices that support bayer formats as compatible with v4l2convert. We support bayer on the input side of our v4l2 device, however there is a kernel module parameter on bcm2835-codec to disable this to allow it to work with this older gstreamer. Add "bcm2835-codec.disable_bayer=1" to /boot/cmdline.txt to enable that.

The second patch changes the names so that the first instance shows up as v4l2convert and not v4l2video12convert.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Lich
Posts: 20
Joined: Wed Apr 11, 2018 8:41 pm
Location: Russia

Re: Convert color format using GPU in Gstreamer (alternative to videoconvert).

Thu Jul 18, 2019 10:08 pm

I enabled v4l2video12convert with "bcm2835-codec.disable_bayer=1", thank you very much!

The first experiment:

Code: Select all

gst-launch-1.0 --gst-debug-level=3 filesrc location=/path/test.mp4 ! qtdemux ! h264parse ! omxh264dec ! videoconvert ! video/x-raw, format=BGR ! fakesink
CPU load is 25%

Code: Select all

gst-launch-1.0 --gst-debug-level=3 filesrc location=/path/test.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 ! v4l2video12convert output-io-mode=5 capture-io-mode=4 ! video/x-raw, format=BGR ! fakesink
CPU load is 6%

But I don't know how it's gonna work with "appsink". I haven't installed OpenCV on my new Raspbian Buster yet.

In parallel, I am trying different "sink" methods for output because Buster supports the OpenGL driver. For example, "kmssink" does not work with the OpenGL driver turned on or off. Now I've compiled a gst-plugins-bad with OpenGL support, and "glimagesink" plays video with hardware acceleration in the window (GLES only works on a full screen layer).

In theory, OpenCV can also be compiled with OpenGL support, but this may not work with Raspberry.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7279
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Convert color format using GPU in Gstreamer (alternative to videoconvert).

Fri Jul 19, 2019 10:46 am

Lich wrote:
Thu Jul 18, 2019 10:08 pm
But I don't know how it's gonna work with "appsink". I haven't installed OpenCV on my new Raspbian Buster yet.
You stick appsink on the end and it should just work. It is the buffers being passed between the decoder and converter that are dmabufs.
Lich wrote:In parallel, I am trying different "sink" methods for output because Buster supports the OpenGL driver. For example, "kmssink" does not work with the OpenGL driver turned on or off. Now I've compiled a gst-plugins-bad with OpenGL support, and "glimagesink" plays video with hardware acceleration in the window (GLES only works on a full screen layer).

In theory, OpenCV can also be compiled with OpenGL support, but this may not work with Raspberry.
kmssink talks to KMS directly, therefore X can not be running. (KMS supports a single master at a time, and X will have taken that role).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Lich
Posts: 20
Joined: Wed Apr 11, 2018 8:41 pm
Location: Russia

Re: Convert color format using GPU in Gstreamer (alternative to videoconvert).

Sun Jul 21, 2019 10:52 pm

I started testing OpenCV with Gstreamer. I use v4l2 methods to decode video stream and convert color format. Video is displayed using the OpenGL renderer.

Test video streaming - H264, 1280x720, 20 fps

Transferring a frame from Gstreamer to OpenCV (display disabled):

Code: Select all

pipline_r = 'rtspsrc location=rtsp://web_camera_ip latency=100 ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 ! v4l2video12convert output-io-mode=5 capture-io-mode=4 ! appsink sync=false'
vs = cv2.VideoCapture(pipline_r)

while True:
    ret, frame = vs.read()
    if ret:
      cv2.waitKey(1)
CPU load is 13%

Transferring a frame from OpenCV to Gstreamer and displaying it through OpenGL rendering:

Code: Select all

pipline_r = 'rtspsrc location=rtsp://web_camera_ip latency=100 ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 ! v4l2video12convert output-io-mode=5 capture-io-mode=4 ! appsink sync=false'
pipeline_s = "appsrc ! glimagesink sync=false"
vs = cv2.VideoCapture(pipline_r)
out = cv2.VideoWriter(pipeline_s, cv2.CAP_GSTREAMER, 0, 20.0, (1280,720))
 
while True:
    ret, frame = vs.read()
    if ret:
      out.write(frame)
      cv2.waitKey(1)
CPU load is 38%, delay is minimal (real-time video).

The best result shows the use of the imshow method to display:

Code: Select all

pipline_r = rtspsrc location=rtsp://web_camera_ip latency=100 ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 ! v4l2video12convert output-io-mode=5 capture-io-mode=4 ! appsink sync=false
vs = cv2.VideoCapture(pipline_r)
 
while True:
    ret, frame = vs.read()
    if ret:
      cv2.imshow('',frame)
      cv2.waitKey(1)
CPU load is 32%, delay is minimal (real-time video).

Next, I will try to use the display via GLES. I want to minimize the CPU load on I/O tasks because image recognition requires a lot of computing resources. Are there other ways to optimize the I/O for my examples? Maybe changing appsink parameters, such as max-buffers, can help?

Return to “OpenMAX”