Video artefacts in RGB/BGR modes

[mattday]

Background: I'm using the MMAL API with some C++ code based on Chris Cumming's work. I'm capturing video in the cropped 1920x1080 mode. The preview port is set to use MMAL_ENCODING_I420 and feeds a vc.ril.resize component which outputs 480x270 images. My application takes these as grayscale to do most of the processing before looking back at the corresponding higher resolution colour video port frames. When the video port is set to use MMAL_ENCODING_I420 then I have not noticed any issues.

The problem occurs when I try to use MMAL_ENCODING_RGB24 or MMAL_ENCODING_BGR24 for the video port. I see a horizontal line across some frames (not all of them and not always in the same position). I have attached an example to illustrate. I have tried various things including different Pis, power supplies and v1/v2 camera modules, but this happens consistently.

HLineExample2.jpg (234.41 KiB) Viewed 1167 times

[6by9]

I'm almost certain know what's going on, but tracking down the exact circumstances that you've used to manage to create them could be tricky.

YUV can use multiple colour conversion matrices, typically JFIF (JPEG), BT601 (SD video), BT709 (HD video), or BT2020 (4k video). It can also use either the full luma range of 0-255 range (JPEG), or a limited range of 16-235 (video).
In the absence of other info the image conversion routines take <=720x576 to be BT601, and >720x576 as BT709.

The camera component converts the image to RGB as it gets the data in. The first stripe was <= 576 lines, so was guessed to be BT601, whilst the second stripe was > 576 lines so has used the BT709 matrix.
You can force the decision using the color_space value in the port format (https://github.com/raspberrypi/userland ... rmat.h#L62) to one of the colour space encodings in (https://github.com/raspberrypi/userland ... ngs.h#L247).
(The intent was that the camera component would adapt the conversion matrices based on the chosen colour space, but that plumbing never happened. BT601 is your safest bet).

If you can give me a simple test case that shows the issue, then I can hook the debugger up and probably track down the issue in the underlying firmware. Alternatively I might just change the default to BT601 which should also solve the issue.

[mattday]

That's spot on! Many thanks.

I can confirm that setting the video port's color_space value to MMAL_COLOR_SPACE_ITUR_BT601 eliminates this issue.

I'll have a go at creating a test case for you. If I can get something reasonably minimal, then I'll share it via a PM.

[6by9]

That's spot on! Many thanks.

I can confirm that setting the video port's color_space value to MMAL_COLOR_SPACE_ITUR_BT601 eliminates this issue.

I'll have a go at creating a test case for you. If I can get something reasonably minimal, then I'll share it via a PM.

Glad to hear my guess was correct. I could have sworn that was fixed ages ago though.
Don't stress if a test case doesn't fall out easily. I suspect raspividyuv storing RGB at an appropriate resolution will hit the same issue, but I'm in the middle of something else at the moment so can't test it now.

[mattday]

Regarding a previous fix to the issue, I found that my firmware was a little out of date (3rd July 2017) because I had not updated the Pi to Raspbian 9. So I set up a fresh system using the latest Raspbian image (firmware was then 24th Oct 2017). The issue persisted when the colour space was not set explicitly.

[6by9]

I'm now a little confused, as the camera component is set to take unknown as the same as JFIF, so that should be consistent for the whole frame.
Annoyingly my build seems to not want to run raspividyuv at the moment, so I can't check the full reason why things are going wrong.

Can I just confirm that you are taking images from the camera video port direct to your client app? Something like:

Code: Select all

       [0] I420 -> resize -> app
camera [1] RGB -> app
       [2] (disabled)

That is almost identical to raspividyuv, which makes my test case valid.

[mattday]

I'm now a little confused, as the camera component is set to take unknown as the same as JFIF, so that should be consistent for the whole frame.
Annoyingly my build seems to not want to run raspividyuv at the moment, so I can't check the full reason why things are going wrong.

Can I just confirm that you are taking images from the camera video port direct to your client app? Something like:

Code: Select all

       [0] I420 -> resize -> app
camera [1] RGB -> app
       [2] (disabled)

That is almost identical to raspividyuv, which makes my test case valid.

I've just checked my code and confirm this setup.