Convert YUYV 4:2:2, what's wrong?

[dast]

I have captured a raw YUYV 4:2:2 image on my Raspberry Pi 2 using v4l2:

Code: Select all

v4l2-ctl -d /dev/video0 -i 0 -s NTSC --set-fmt-video=width=720,height=480,pixelformat=0
v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=frame.raw --stream-count=1

Now I try to convert the raw YUYV 4:2:2 image with the following Python script:

Code: Select all

import Image
import sys

image_name = sys.argv[1]
width = int(sys.argv[2])
height = int(sys.argv[3])

f_yuyv = open(image_name, "rb")
image_out = Image.new("RGB", (width, height))
pix = image_out.load()

print "width=", width, "height=", height

for i in range(0,height):
    for j in range(0, width/2):

        y1 = ord(f_yuyv.read(1));
        u  = ord(f_yuyv.read(1));
        y2 = ord(f_yuyv.read(1));
        v  = ord(f_yuyv.read(1));

        B = 1.164 * (y1-16) + 2.018 * (u - 128)
        G = 1.164 * (y1-16) - 0.813 * (v - 128) - 0.391 * (u - 128)
        R = 1.164 * (y1-16) + 1.596 * (v - 128)
        pix[j*2, i] = int(R), int(G), int(B)

        B = 1.164 * (y2-16) + 2.018 * (u - 128)
        G = 1.164 * (y2-16) - 0.813 * (v - 128) - 0.391 * (u - 128)
        R = 1.164 * (y2-16) + 1.596 * (v - 128)
        pix[j*2+1, i] = int(R), int(G), int(B)

image_out.save("out.bmp")
image_out.show()

The result I get looks a little bit strange:

Anyone who has an idea where's the problem?

Furthermore, here the properties of the video device I am using to capture the raw YUYV 4:2:2 image:

Code: Select all

pi@raspberrypi:~ $ v4l2-ctl --all
Driver Info (not using libv4l2):
        Driver name   : usbtv
        Card type     : usbtv
        Bus info      : usb-3f980000.usb-1.3
        Driver version: 4.4.11
        Capabilities  : 0x85200001
                Video Capture
                Read/Write
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps   : 0x05200001
                Video Capture
                Read/Write
                Streaming
                Extended Pix Format
Priority: 2
Video input : 0 (Composite: ok)
Video Standard = 0x0000f900
        PAL-M/60
        NTSC-M/M-JP/443/M-KR
Format Video Capture:
        Width/Height  : 720/480
        Pixel Format  : 'YUYV'
        Field         : Interlaced
        Bytes per Line: 1440
        Size Image    : 691200
        Colorspace    : Broadcast NTSC/PAL (SMPTE170M/ITU601)
        Flags         :
Streaming Parameters Video Capture:
        Frames per second: 29.970 (30000/1001)
        Read buffers     : 2

pi@raspberrypi:~ $ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'YUYV'
        Name        : YUYV 4:2:2

Regards,
Daniel.

[6by9]

Code: Select all

v4l2-ctl -d /dev/video0 -i 0 -s NTSC --set-fmt-video=width=720,height=480,pixelformat=0

I'd recommend selecting by 4CC, so "pixelformat=YUYV". It removes any ambiguity and makes it transferable.

Code: Select all

Format Video Capture:
        Width/Height  : 720/480
        Pixel Format  : 'YUYV'
        Field         : Interlaced

That's a problem. You've got an interlaced source and you're only displaying one field. Every other line is memset to 0, which in YUV space is green. You really need to deinterlace the two fields properly. If you look between the green lines I think your image is correct.

[dast]

Thanks!

Can you give me a hint how the deinterlace can be done?

[6by9]

Can you give me a hint how the deinterlace can be done?

Several techniques. https://en.wikipedia.org/wiki/Deinterla ... ng_methods
Line doubling is going to be the simplest to implement.

Your bigger issue is going to be that having used v4l2-ctl to capture a single field to a file, you've lost the signalling on the buffer as to whether it was the top or bottom field. The deinterlacer needs that to know which line to work on.
In your example you could find the lines which are all zero, but that may not be guaranteed if the buffers have been filled once and then returned to the driver for filling again. It'd be better to work directly with V4L2 if you can to preserve the signalling (https://launchpad.net/python-v4l2-capture looks like it might give you the required functionality).

[dast]

Thanks for the link to the python-v4l2-capture project!

I was able to run it on my Raspberry Pi and made some snaps using: ./capture_picture.py

But again one strange behaviour: Some images seems to be ok, but others are again have these green lines!






It seems that some images contain both fields, and some only one of the fields (top, bottom) ...
Any idea how I get the full images?
Maybe it's also possible that I didn't understand it in detail.
If so, please give me some hint

Thanks!

[6by9]

Please read the V4L2 spec to understand better what your capture card is telling you. http://hverkuil.home.xs4all.nl/spec/media.html is the easiest version I know of to read, as the whole document is in one page.

Interlacing field order. Your driver says it is delivering "Interlaced" frames.
http://hverkuil.home.xs4all.nl/spec/med ... ield-order

V4L2_FIELD_INTERLACED 4 Images contain both fields, interleaved line by line. The temporal order of the fields (whether the top or bottom field is first transmitted) depends on the current video standard. M/NTSC transmits the bottom field first, all other standards the top field first.

Look at the diagrams. Each buffer delivered to you should contain two fields, line interleaved. However for the very first buffer the device doesn't have two fields to combine, so every other line will be green. It's not ideal deinterleaving, but it'll work for many situations.
Each buffer should also be flagged appropriately by V4L2 in the field member of struct v4l2_buffer to signal which field has just been updated. Check in the Python library as to how that is signalled to you as the application.

[dast]

Thanks again!

So you mean I should check the
__u32 field
in the
struct v4l2_buffer
returned by the
ioctl(self->fd, VIDIOC_DQBUF, &buffer))
?

And depending on that result deque a new buffer?

[6by9]

Thanks again!

So you mean I should check the
__u32 field
in the
struct v4l2_buffer
returned by the
ioctl(self->fd, VIDIOC_DQBUF, &buffer))
?

And depending on that result deque a new buffer?

I don't have your hardware so can't tell.
Have a look at both the field and flags values in there. Does the driver give you any different values between frames?
If it works the way I expect, then the simplest approach is possibly to just drop the first frame always. With v4l2-ctl you can add the --stream-skip=N parameter to skip a number of frames at the start.

[dast]

So, now I wrote a simple c-program directly using the ioctl-functions
of the v4l2 driver and took several snaps using 4 buffers in a loop.

I also logged the properties of the returned VIDIOC_DQBUF functions:

Code: Select all

pi@raspberrypi:~ $ ./a.out 
NUM_BUFFERS = 4
req.count = 4
------------------------------------------------------------
 index: 0
 type: V4L2_BUF_TYPE_VIDEO_CAPTURE
 bytesused: 691200
 flags: 0x2005
   V4L2_BUF_FLAG_MAPPED
   V4L2_BUF_FLAG_DONE
   V4L2_BUF_FLAG_TIMESTAMP_MASK
   V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC
 field: V4L2_FIELD_INTERLACED
 sequence: 0
------------------------------------------------------------
 index: 1
 type: V4L2_BUF_TYPE_VIDEO_CAPTURE
 bytesused: 691200
 flags: 0x2005
   V4L2_BUF_FLAG_MAPPED
   V4L2_BUF_FLAG_DONE
   V4L2_BUF_FLAG_TIMESTAMP_MASK
   V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC
 field: V4L2_FIELD_INTERLACED
 sequence: 1
------------------------------------------------------------
 index: 2
 type: V4L2_BUF_TYPE_VIDEO_CAPTURE
 bytesused: 691200
 flags: 0x2005
   V4L2_BUF_FLAG_MAPPED
   V4L2_BUF_FLAG_DONE
   V4L2_BUF_FLAG_TIMESTAMP_MASK
   V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC
 field: V4L2_FIELD_INTERLACED
 sequence: 2
------------------------------------------------------------
 index: 3
 type: V4L2_BUF_TYPE_VIDEO_CAPTURE
 bytesused: 691200
 flags: 0x2005
   V4L2_BUF_FLAG_MAPPED
   V4L2_BUF_FLAG_DONE
   V4L2_BUF_FLAG_TIMESTAMP_MASK
   V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC
 field: V4L2_FIELD_INTERLACED
 sequence: 3
------------------------------------------------------------
 index: 0
 type: V4L2_BUF_TYPE_VIDEO_CAPTURE
 bytesused: 691200
 flags: 0x2045
   V4L2_BUF_FLAG_MAPPED
   V4L2_BUF_FLAG_DONE
   V4L2_BUF_FLAG_ERROR
   V4L2_BUF_FLAG_TIMESTAMP_MASK
   V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC
 field: V4L2_FIELD_INTERLACED
 sequence: 4
------------------------------------------------------------
 index: 1
 type: V4L2_BUF_TYPE_VIDEO_CAPTURE
 bytesused: 691200
 flags: 0x2045
   V4L2_BUF_FLAG_MAPPED
   V4L2_BUF_FLAG_DONE
   V4L2_BUF_FLAG_ERROR
   V4L2_BUF_FLAG_TIMESTAMP_MASK
   V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC
 field: V4L2_FIELD_INTERLACED
 sequence: 5
------------------------------------------------------------
 index: 2
 type: V4L2_BUF_TYPE_VIDEO_CAPTURE
 bytesused: 691200
 flags: 0x2005
   V4L2_BUF_FLAG_MAPPED
   V4L2_BUF_FLAG_DONE
   V4L2_BUF_FLAG_TIMESTAMP_MASK
   V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC
 field: V4L2_FIELD_INTERLACED
 sequence: 6
------------------------------------------------------------
 index: 3
 type: V4L2_BUF_TYPE_VIDEO_CAPTURE
 bytesused: 691200
 flags: 0x2005
   V4L2_BUF_FLAG_MAPPED
   V4L2_BUF_FLAG_DONE
   V4L2_BUF_FLAG_TIMESTAMP_MASK
   V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC
 field: V4L2_FIELD_INTERLACED
 sequence: 7
------------------------------------------------------------
 index: 0
 type: V4L2_BUF_TYPE_VIDEO_CAPTURE
 bytesused: 691200
 flags: 0x2005
   V4L2_BUF_FLAG_MAPPED
   V4L2_BUF_FLAG_DONE
   V4L2_BUF_FLAG_TIMESTAMP_MASK
   V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC
 field: V4L2_FIELD_INTERLACED
 sequence: 8
------------------------------------------------------------
 index: 1
 type: V4L2_BUF_TYPE_VIDEO_CAPTURE
 bytesused: 691200
 flags: 0x2045
   V4L2_BUF_FLAG_MAPPED
   V4L2_BUF_FLAG_DONE
   V4L2_BUF_FLAG_ERROR
   V4L2_BUF_FLAG_TIMESTAMP_MASK
   V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC
 field: V4L2_FIELD_INTERLACED
 sequence: 9

Here are the converted corresponding images:











As you see, there is no difference e.g. between the first and the second
frame in the properties of the VIDIOC_DQBUF functions (in the v4l2_buffer
structure) but the first image has many of the "green" lines, the second
one lesser ...

So I have now idea how to detect if the returned frame is complete or not .

Maybe only the first frames (1-4) are distorted and the other are ok? So,
one way would be to skip the first ones (e.g. 5) an take the 6th frame?!
What do you think?

Regards,
Daniel.

[6by9]

Frames with V4L2_BUF_FLAG_ERROR set should be discarded. I see a couple of those.

You can look at the sequence number and if field == V4L2_FIELD_INTERLACED && sequence == 0 to determine first frame in a generic enough manner. Based on the V4L2 spec, I can see no way for such a frame to be valid.
In theory you ought to allow for the wrap around of the sequence number, but 2^32 frames should keep you going for a fair while.
I would say that your device seems to contradict the spec, which reads:

__u32 sequence Set by the driver, counting the frames (not fields!) in sequence. This field is set for both input and output devices.

Yours would seem to be counting fields, even though it is doing a cheap deinterlace on them.

My only other query would be over anything odd in the kernel logs. Some forms of USB transfer are known to have had some issues, and USB TV capture cards tend to use lots of bandwidth. The odd coloured stripes would be indicative of some missed transfers within the frame.