xultz
Posts: 6
Joined: Thu Apr 26, 2018 5:24 pm

Best way to show a YUV frame on dispmanx

Mon Apr 30, 2018 6:15 pm

Hi! I'm developing a device which will need to show YUV420 frames on the screen. I'm trying to use dispmanx to show the frame, so I don't need to make any type of conversion.

In my project, I'm using the CVBS video output, and I set the screen to rotate, by setting display_rotate=1 on /boot/config.txt, which worked fine. This way, the screen have a resolution of 480 pixels for the width, and 720 pixels for the height. The image is 480x640 pixels. Since it is in YUV420, the file have 480x640x1.5 pixels, which means exactly 460,800 bytes. If I upload this file to rawpixels.net, set the desired width and heght, and image format to IYUV, it shows the image perfectly.

In order to show the image on the screen, I tried to modify the hello_dispmanx.c example at /opv/vc/src/hello_pi/hello_dispmanx folder. I am able to show the image on the screen with this code:
https://pastebin.com/wiauyT2f

If anyone is curious, the frame file can be found here:
https://filebin.net/8fb0jrwnbjsihcrp

The code is a bit weird, since I had to create two buffers. One buffer, with 460,800 bytes to hold the data from the file (that's OK), and another one with 614,400 bytes (thats 640 * 480 * 2), where I need to copy the data from the frames into it. This is the buffer that goes to dispmanx, with the command vc_dispmanx_resource_write_data. I kinda found this size by trial and error (lots of errors, by the way). Apart the weirdness, the code works and shows perfectly the image on the screen.

What is bottering me is the output of the command /opt/vc/bin/vcdbg reloc. Before calling my program, that puts the image on the screen, here is the output of vcdbg, which looks fine:

Code: Select all

Relocatable heap version 4 found at 0x3b400000
total space allocated is 56M, with 56M relocatable, 0 legacy and 0 offline
0 legacy blocks of size 2359296

free list at 0x3e99dce0
54M free memory in 1 free block(s)
largest free block is 54M bytes

0x3b400000: free 54M
[   5] 0x3e99dd00: used  576 (refcount 1 lock count 0, size      512, align    4, data 0x3e99dd20, d0rual) 'ILCS VC buffer pool'
[   4] 0x3e99df40: used 1.0M (refcount 1 lock count 8, size  1091584, align 4096, data 0x3e99e000, d1rual) 'ARM FB'
[   3] 0x3eaa9760: used 1.3M (refcount 2 lock count 8, size  1382400, align   32, data 0x3eaa9780, d1rual) 'transpose buffer0'
[   2] 0x3ebfafa0: used  16K (refcount 1 lock count 0, size    16384, align   32, data 0x3ebfafc0, d0ruAl) 'audioplus_tmp_buf'
[   1] 0x3ebfefe0: used 4.0K (refcount 1 lock count 0, size        0, align 4096, data 0x3ebff000, d1rual) 'camera fast alloc arena'
small allocs not requested
Right after running my program, here is the output:

Code: Select all

Relocatable heap version 4 found at 0x3b400000
total space allocated is 56M, with 56M relocatable, 0 legacy and 0 offline
0 legacy blocks of size 2359296

free list at 0x3e99dce0
54M free memory in 1 free block(s)
largest free block is 54M bytes

0x3b400000: free 54M
0x3e99dd00: corrupt entry (space 0x0)
could not resync
heap corruption detected
small allocs not requested
It doesn't look that good...

My question here is:
- Do I have to be concerned about that output and the messages of heap corruption (I'm quite sure that yes, I must be concerned about it)?
- What am I doing wrong in my code?
- Do anyone have any example code that is able to show a YUV image on the screen of the Raspberry?
- Should I use any other way to show the image on the screen (like EGL or something else)? If yes, do anyone have any documentation that explains how can I do it?

Thank you in advance for any help!

With best regards
Christian Schultz

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5065
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Best way to show a YUV frame on dispmanx

Tue May 01, 2018 9:23 am

xultz wrote:
Mon Apr 30, 2018 6:15 pm
My question here is:
- Do I have to be concerned about that output and the messages of heap corruption (I'm quite sure that yes, I must be concerned about it)?
The VideoCore blocks have an independent L1 cache (and L2 except on Pi2 & 3 IIRC), and the malloc list is in that cached region. "vcgencmd cache_flush" will flush the caches and shoulld result in the correct data being read back. If it still reads back as corrupt then you have corrupted memory
xultz wrote:- What am I doing wrong in my code?
- Do anyone have any example code that is able to show a YUV image on the screen of the Raspberry?
I've modified your code slightly as I only have a QVGA image lying around rather than VGA.
It looks like you're passing an invalid pitch into vc_dispmanx_resource_write_data. YUV420 is only 12bits/pixel, not 16. It's a little ugly as the strict pitch (number of bytes to get down exactly one line) is 8bits per pixel and the interface code doesn't handle the extra planes.

Code: Select all

    ret = vc_dispmanx_resource_write_data(resource, VC_IMAGE_YUV420, (pitch * 3)>>1, img_yuv, &dst_rect );
works happily with no corruption reported (using pitch*2 did report corruption).
xultz wrote:- Should I use any other way to show the image on the screen (like EGL or something else)? If yes, do anyone have any documentation that explains how can I do it?
Personally I'd head for MMAL and create a MMAL_COMPONENT_DEFAULT_VIDEO_RENDERER component (normally "vc.ril.video_render"). That handles almost all the formats going, and can avoid the memcpy associated with vc_dispmanx_resource_write_data if setting MMAL_PARAMETER_ZERO_COPY. If you were using Python then I'd point you at the PiCamera library as that exposes overlays by doing exactly that.
The headers include lots of Doxygen markup.
https://github.com/raspberrypi/userland ... _basic_1.c is a reasonably readable example which decodes video. Replace the decode component with a renderer, remove anything to do with the output port (as render doesn't have one), set the port format to your parameters and you should be good to go. Look at the raspicam apps for an example of how to set up the display rectangle, layer, alpha, etc via MMAL.

Your code amended to set the correct pitch and use defines for width/height throughout.
You don't actually need img_yuv anymore, you can just pass imgs to vc_dispmanx_resource_write_data.

Code: Select all

#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <assert.h>
#include <unistd.h>
#include <sys/time.h>

#include "bcm_host.h"

#define WIDTH   320
#define HEIGHT  240

#ifndef ALIGN_UP
#define ALIGN_UP(x,y)  ((x + (y)-1) & ~((y)-1))
#endif

int main(void)
{    
    uint32_t screen = 0;
    int ret;

    DISPMANX_DISPLAY_HANDLE_T display;		// Esse display será aberto por vc_dispmanx_display_open()
    DISPMANX_MODEINFO_T info;				// Esta estrutura será preenchida pelo comando vc_dispmanx_display_get_info()
    DISPMANX_RESOURCE_HANDLE_T resource;	// Este handle é obtido pelo comando vc_dispmanx_resource_create()
    DISPMANX_UPDATE_HANDLE_T update;		// Este handle é obtido pelo comando vc_dispmanx_update_start()
    DISPMANX_ELEMENT_HANDLE_T element;		// Este handle é obtido pelo comando vc_dispmanx_element_add
    VC_RECT_T src_rect;
    VC_RECT_T dst_rect;

    FILE * fp;

    uint8_t  * img_yuv;
    uint8_t * imgs;
    		
    uint32_t vc_image_ptr;				// Este valor é preenchido (provavelmente) pela função vc_dispmanx_resource_create()
    int width = WIDTH;					// Tamanho do retângulo, definido em #define acima
    int	height = HEIGHT;				// Tamanho do retângulo, definido em #define acima
    int pitch = ALIGN_UP(width, 16);

    imgs = calloc((WIDTH*HEIGHT*3)>>1, 1);
    img_yuv = calloc((WIDTH*HEIGHT*3)>>1, 1);

    VC_DISPMANX_ALPHA_T alpha = { DISPMANX_FLAGS_ALPHA_FROM_SOURCE | DISPMANX_FLAGS_ALPHA_FIXED_ALL_PIXELS, 255, 0 };

    bcm_host_init();

    fp = fopen("frame.yuv", "rb");
    assert(fp);

    ret = fread(imgs, 1, (WIDTH*HEIGHT*3)>>1, fp);
    assert(ret);

    fclose(fp);

    display = vc_dispmanx_display_open( screen );			
    ret = vc_dispmanx_display_get_info( display, &info);	
    assert(ret == 0);

    resource = vc_dispmanx_resource_create( VC_IMAGE_YUV420, width, height, &vc_image_ptr );
    assert( resource );

    memcpy(img_yuv, imgs, (WIDTH*HEIGHT*3)>>1);	

    vc_dispmanx_rect_set( &dst_rect, 0, 0, width, height);
    ret = vc_dispmanx_resource_write_data(resource, VC_IMAGE_YUV420, (pitch * 3)>>1, img_yuv, &dst_rect );
    assert( ret == 0 );

    update = vc_dispmanx_update_start( 10 );
    assert( update );

    vc_dispmanx_rect_set( &src_rect, 0, 0, width << 16, height << 16 );
    vc_dispmanx_rect_set( &dst_rect, 0, 0, width, height );

    element = vc_dispmanx_element_add( update, display, 2000, &dst_rect, resource, &src_rect, DISPMANX_PROTECTION_NONE, &alpha, NULL, VC_IMAGE_ROT0 );

    ret = vc_dispmanx_update_submit_sync(update);
    assert( ret == 0 );

    sleep(5);

    update = vc_dispmanx_update_start( 10 );
    assert(update);

    ret = vc_dispmanx_element_remove(update, element);
    assert( ret == 0 );

    ret = vc_dispmanx_update_submit_sync(update);
    assert( ret == 0 );

    ret = vc_dispmanx_resource_delete(resource);
    assert( ret == 0 );

    ret = vc_dispmanx_display_close(display);
    assert( ret == 0 );

    return 0;
}
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5065
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Best way to show a YUV frame on dispmanx

Tue May 01, 2018 9:25 am

And now I notice that you linked to your rotated VGA image - doh!
Resetting the resolution to 480x640 works fine with that file with no corruption reported.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

tvjon
Posts: 611
Joined: Mon Jan 07, 2013 9:11 am

Re: Best way to show a YUV frame on dispmanx

Wed May 02, 2018 2:17 pm

Thank you both for the useful details.

6by9 version is also important to avoid overwriting of the (top of screen) menu bar. Interestingly that occurs on display 0 (hdmi) even when I output the picture just to my attached vga666 display.

I've attached the huge vcdbg output for the 6by9 first run. Subsequent runs had a normal few bytes.

Next I'll look at offsets other than 0, 0 for displaying the picture.
vcdbg.zip
(41.09 KiB) Downloaded 6 times

xultz
Posts: 6
Joined: Thu Apr 26, 2018 5:24 pm

Re: Best way to show a YUV frame on dispmanx

Thu May 03, 2018 6:18 pm

Thank you so much for your help, 6by9!

I took a look for the mmal lib, it was the first time I heard about it. I read this page: http://www.jvcref.com/files/PI/documentation/html/, the informatin there made sense to me, but it look like a bit fuzzy. I'm not experienced enough to read just the API docs from the headers and use it (or I'm not smart enough, maybe...).
Anyway, I would like to make you another question, please apologize me if I'm abusing your good will :)
A test I made was changing the code to load some more frames to memory, and write them to displamnx with vc_dispmanx_resource_write_data and a vc_dispmanx_update_start right after, and it worked great. I got an insanely good frame rate with it.
What I am trying to develop with the CM3, is getting a chunk of video using the SPI port via a weird protocol that I developed (which means that I cannot complain about it). I need to decode this video, and since it is encoded in VP9, I'm made it with libvpx. After twisting an example code, I was able to decode the chunk in a series of frames (the libvpx give me the frames in YUV420), the next step is to show them on the screen, that's the reason I was trying to make it with dispmanx.
My doubt now is: how can I schedule the frame switching? The normal way of thinking is: having a queue of frame buffers, I need to make loop where I show the frame, wait the time for the next frame (based on the video frame rate), show the next one, and so one. But I looks like to me that doing that by a simple busy waiting is not the best way in terms of precision (of the delays), and computer power usage. What I would like to know is: is there a better way to do it? Where should I start studying about it?

Thank you again for the provided help, and thank you in advance for any more help

With best regards
Christian Schultz

Return to “Graphics programming”

Who is online

Users browsing this forum: No registered users and 0 guests