How do you convert a MMAL_BUFFER_HEADER_T buffer to a v4l2_buffer buffer?

[KWLee]

Hi,

I'm currently combining the functionality of uvc-gadget (https://github.com/jdonald/uvc-gadget) and 6by9's yavta to allow a TC358743 output to be converted from UYVY/RGB24 to YUYV using a pi's internal ISP. The aim of this is to stream an HDMI source as a UVC webcam.
The issue I'm having is in my isp callback function, when I try to queue the converted data back to the uvc v4l2 device VIDIOC_QBUF gives me a bad address error (error 14). I presume I'm incorrectly parsing the MMAL_BUFFER_HEADER_T * buffer into the v4l2_buffer.

Here's my callback function code

Code: Select all

static void
isp_output_callback (MMAL_PORT_T * port, MMAL_BUFFER_HEADER_T * buffer)
{
  static int i = 0;
  struct v4l2_buffer ubuf;
  int ret =0;
  MMAL_STATUS_T  status;
  print ("isp_output_callback Buffer %p from isp, filled %d, timestamp %llu, flags %04X\n",
    buffer, buffer->length, buffer->pts, buffer->flags);
      struct v4l2_device *dev = (struct v4l2_device *) port->userdata;
      print("setup ubuf details %p, %d %d %d %d\n",buffer->data,buffer->data[0],buffer->alloc_size, dev->vbuf.index, buffer->length);

  {
    CLEAR(ubuf);
    ubuf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
    ubuf.memory = V4L2_MEMORY_USERPTR;
    ubuf.m.userptr = (unsigned long )(buffer->data) ;
    ubuf.length = buffer->alloc_size;
    ubuf.index = dev->vbuf.index;;//vbuf.index;
    ubuf.bytesused = buffer->length;
  }

  ret = ioctl (dev->udev->uvc_fd, VIDIOC_QBUF, &ubuf);
  if (ret < 0) {
    printf ("UVC: Unable to queue buffer %d: %s (%d) %d.\n",
            ubuf.index, strerror (errno), errno, dev->udev->uvc_fd);
    // Check for a USB disconnect/shutdown event. 
    if (errno == ENODEV) {
      dev->udev->uvc_shutdown_requested = 1;
      printf ("UVC: Possible USB shutdown requested from "
              "Host, seen during VIDIOC_QBUF\n");
       }
    status = mmal_port_send_buffer (dev->isp->output[0], buffer);
  i++
  }

    

I could just be doing it completely wrong so any help would be great.

background

• UVC gadget requires YUYV pixelformat, and TC358743 only outputs RGB24 or UYVY. If it's not converted the colours are incorrect.
• I've tried gstreamer to do the conversion to a v4l2loopback but the framerates are low about 5fps on my pizero with the cpu at 100%.
• I've modified yavta to do the format conversion and then simply write to the v4l2loopback device. This works and the framerates are about x2 faster than gstreamer but I think removing the loopback device will help enormously. So this leads me to this dive into MMAL/V4L2 programming.

[KWLee]

In the mmal.h header it says:

Queues (MMAL_QUEUE_T) are a facility that allows thread-safe processing of buffer headers from the client. Callbacks triggered by a MMAL component when it sends a buffer header to the client can simply put the buffer in a queue and let the main processing thread of the client get its data from the queue.

To put it simply and in the callback I need to use mmal_queue_put to queuethe buffer.A block of code is then needed to deal with the queue. In my mmal setup I use vcos_thread_create to make a thread to process the the buffer. In the thread the queue is retrieved at which point it can be converted to a v4l2_buffer. Now VIDIOC_QBUF nolonger fails.
If you look at 6by9's yavta this same process is used to save streamed data to a file.

[6by9]

Does that mean you've fully solved the issue?

I do recall having some "fun" when trying to map vcsm buffers back into DispmanX as the kernel had some extra magic flags on the allocation which caused some memory mapping operations to fail, but I don't think that's the problem here.
Your other option would have been to use dmabufs to zero copy from MMAL into V4L2, but I won't go into that if you've solved the problem.

[KWLee]

Thanks for the reply, and all your support to the community.

Does that mean you've fully solved the issue?

Well it's solved my VIDIOC_QBUF bad address error issue. Data appears to be 'queued' for UVC but nothing is appearing on VLC or Facetime. (As mentioned previously unmodified uvc-gadget gives me incorrect colours). I still have more work to do.

Your other option would have been to use dmabufs to zero copy from MMAL into V4L2, but I won't go into that if you've solved the problem.

Any links to examples?

[6by9]

Thanks for the reply, and all your support to the community.

Does that mean you've fully solved the issue?

Well it's solved my VIDIOC_QBUF bad address error issue. Data appears to be 'queued' for UVC but nothing is appearing on VLC or Facetime. (As mentioned previously unmodified uvc-gadget gives me incorrect colours). I still have more work to do.

I haven't dug into what that project is trying to do. Is it emulating a UVC device from userspace, or are we building a kernel module? I think the latter, so it will show up as a /dev/videoN V4L2_BUF_TYPE_VIDEO_OUTPUT device?

Your easiest route is probably to use GStreamer instead of yavta.
v4l2src will allow you to integrate to bcm2835-unicam and tc358743
v4l2convert will allow you to do format conversions and resizing using hardware.
v4l2sink will allow you to pass data into an OUTPUT V4L2 device.

I see you referenced using GStreamer before using a v4l2loopback - I don't see how that comes into the setup.

My guess at the required command line would be

Code: Select all

gst-launch-1.0 -v \
        v4l2src io-mode=4 num-buffers=-1 device=/dev/video0 \
        ! "video/x-raw,framerate=50/1,format=UYVY" \
        ! v4l2convert output-io-mode=5 \
        ! "video/x-raw,width=1280,height=720,framerate=50/1,format=YUYV" \
        ! v4l2sink device=/dev/video1

Note that the /dev/videoN nodes for v4l2src and sink may need to be altered to match how things enumerate.
The uvc-gadget also seems to have a hard requirement of 640x360 or 1280x720, so you can't randomly throw it any resolution you like.

Your other option would have been to use dmabufs to zero copy from MMAL into V4L2, but I won't go into that if you've solved the problem.

Any links to examples?

Looking at the uvc-gadget source code, it doesn't support V4L2_MEMORY_DMABUF, so that's not an option.

[KWLee]

Thanks for the gstreamer option.

I haven't dug into what that project is trying to do. Is it emulating a UVC device from userspace, or are we building a kernel module? I think the latter, so it will show up as a /dev/videoN V4L2_BUF_TYPE_VIDEO_OUTPUT device?

I'm pretty sure uvc-gadget is a userspace program, /dev/video0 (bcm2835-unicam) and /dev/video1 (g_webcam) need to be loaded first. It doesn't generate any /dev/videoN devices and it also handles some usb events. The version I'm using (https://github.com/jdonald/uvc-gadget) is a fork from Laurent Pinchart (ideasonboard.org/uvc-gadget.git). However Laurent's version seems to have progress alot since the fork.

I've gone through the gstreamer route before and found that I needed to output (v4l2sink) to a loopback device (in my case /dev/video2) and then have uvc-gadget use it as the v4l2 input device.

Here's my recipe using a 1280x720 laptop as a source. gstreamer seems to automatically configure the output to YUY2.

Code: Select all

gst-launch-1.0 -v v4l2src device=/dev/video0  !  video/x-raw,format=UYVY,colorimetry=bt601,framerate=5/1 !  v4l2convert  !  v4l2sink device=/dev/video2 & 
./uvc-gadget -v /dev/video2 -u /dev/video1 -n3 -r1 #assuming 1280x720 source
#gst-launch-1.0 version 1.16.2
#Linux raspi-b 4.19.122+ #5 Mon Jun 1 10:10:40 HKT 2020 armv6l GNU/Linux

On my Pi Zero the frames rates, unfortunately, terrible but I did notice on your pipeline you use output-io-mode (MMAP and USERPTR) modes, which I've never used.

After the gstreamer route, I modified yavta.c to write directly to the v4l2loopback device. After striping out the encoders and making the ISP spit out YUY2 instead of I420 it was straightforward to write to the loopback device.

Code: Select all

write (dev->fd2, buffer->data, buffer->length);

The framerates were about twice as fast. With the new code I hope by cutting out the middleman (v4l2loopback) framerates will be even better.

[KWLee]

Here's my final method for converting a MMAL_BUFFER_HEADER_T to a v4l2_buffer.
My solution (is it the right or wrong method?) was to create a dummy memory block and to memcpy the MMAL_BUFFER_HEADER_T buffer over to it. But memcpy is 'bad' isn't it, since it could slow down performance. Fortunately I found it was only necessary to do this memcpy once. Once the dummy buffer was written to the buffer->data pointer seem to be useable.

Code: Select all

  MMAL_BUFFER_HEADER_T *buffer;
  MMAL_STATUS_T status;
  struct v4l2_buffer ubuf;
  unsigned char *bmap; // my dummy buffer
  
  while(in thread)
   buffer = mmal_queue_timedwait (dev->save_queue, 10); 
    if(FIRST_RUN_ONLY){ //first iteration of save thread loop
      bmap = (unsigned char *) malloc (buffer->length);
      memcpy (bmap, (unsigned char *) buffer->data, buffer->length); 
    }    
    ubuf.m.userptr = (unsigned long) (buffer->data); // point to MMAL buffer not dummy 
    etc...
    ioctl (dev->udev->uvc_fd, VIDIOC_QBUF, &ubuf); //queue to UVC video output Works!
    etc...

My save thread function

Code: Select all

save_thread (void *arg)
{
  struct v4l2_device *dev = (struct v4l2_device *) arg; 
  MMAL_BUFFER_HEADER_T *buffer;
  MMAL_STATUS_T status;
  struct v4l2_buffer ubuf;
  static int i = 0; 
  int ret = 0; 
  unsigned char *bmap;
  while (!dev->thread_quit) {
    buffer = mmal_queue_timedwait (dev->save_queue, 10); 
    if (!buffer){
      continue;
    }    
    if(i==0){ //first iteration of save thread loop
      bmap = (unsigned char *) malloc (buffer->length);
      CLEAR (ubuf);
      memcpy (bmap, (unsigned char *) buffer->data, buffer->length); // first and only copy using memcpy but dummy buffer never used
    }    
    print ("Buffer %p saving, %p, filled %d, timestamp %llu, flags %04X\n, index %d, counter %d\n", buffer, buffer->data,  buffer->length, buffer->pts, buffer->flags, dev->vbuf.index, dev->counter);
    //CLEAR (ubuf);
    ubuf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
    //point ubuf to the new data locations buffers and then q buf
    ubuf.memory = V4L2_MEMORY_USERPTR;
    ubuf.m.userptr = (unsigned long) (buffer->data); //point to MMAL buffer
    ubuf.length = buffer->alloc_size;
    ubuf.index = dev->vbuf.index;;      
    ubuf.bytesused = buffer->length;

    ret = ioctl (dev->udev->uvc_fd, VIDIOC_QBUF, &ubuf); //queue to UVC video output
    if (ret < 0) { 
      if (errno == ENODEV) {
        dev->udev->uvc_shutdown_requested = 1; 
        printf ("UVC: Possible USB shutdown requested from "
                "Host, seen during VIDIOC_QBUF\n");
      }    
          }
    else
    {
      dev->udev->qbuf_count++;
      print ("UVC: queueued buffer %d ok q %lld dq %lld, vbuf %lld %lld\n",
        ubuf.index, dev->udev->qbuf_count, dev->udev->dqbuf_count,
        dev->qbuf_count, dev->dqbuf_count);

      if (!dev->udev->first_buffer_queued) {
        print ("udev first buffers start\n");
        uvc_video_stream (dev->udev, 1);
        dev->udev->first_buffer_queued = 1;
        dev->udev->is_streaming = 1;
      }
    }
    //buffer->length = 0;
    status = mmal_port_send_buffer (dev->isp->output[0], buffer);
    if (status != MMAL_SUCCESS) {
      print ("mmal_port_send_buffer failed on buffer %p, status %d", buffer, status);
    }                           
    i++;
    dev->dq_ubuf_ok = true;
  }
  free (bmap);
  return NULL;
}

So my program that modifies uvc-gadget to do MMAL ISP to convert UYVY or RBG24 to YUY2 now works but there are some issue

• CPU usage low 30-40% of CPU, but the frame rate is only 8.5fps (tested using output from quicktime). The frame rate seems to be a USB/UVC related, since changing the streaming_maxpacket size from 1024 to 2048 changes the fps from 4.3 to 8.5fps. I need to investigate this further. My previous framerate tests using gstreamer, uvc-gadget and a modified yavta+v4l2loopback are most likely incorrect.
• The video output suffers from video tearing, it could be due to the low frame rate. When an object moves on the screen, it becomes segmented into several rows.

  

  Untitled.jpg (21.28 KiB) Viewed 138 times

[KWLee]

The reason I was unable to access the buffer data with

Code: Select all

ubuf.m.userptr = (unsigned long )(buffer->data) ;

and that queueing the v4l2_buffer would fail when using

Code: Select all

ret = ioctl (dev->udev->uvc_fd, VIDIOC_QBUF, &ubuf);

was due to MMAL_PARAMETER_ZERO_COPY being set on the isp output port.

[6by9]

The reason I was unable to access the buffer data with

Code: Select all

ubuf.m.userptr = (unsigned long )(buffer->data) ;

and that queueing the v4l2_buffer would fail when using

Code: Select all

ret = ioctl (dev->udev->uvc_fd, VIDIOC_QBUF, &ubuf);

was due to MMAL_PARAMETER_ZERO_COPY being set on the isp output port.

That's going to be the same thing as is behind https://github.com/raspberrypi/userland/issues/386. The allocations get tagged in the kernel in a particular way with VMA flags, and then it won't map back again. It's all rather messy and not an area I fully understand.

You may be able to use vcsm_init_ex(1,-1); before your first MMAL call to use the newer version of VCSM for the allocations when using ZERO_COPY.