terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Getting direct access to a dispmanx resource

Sun Feb 24, 2019 7:06 pm

Hi,

So I've been experimenting with some software-rendering code. I've been using Dispmanx directly for some time and it has been quite efficient so I thought I would carry on in the same style.

For the actual drawing code I migrated some C++ functions into NEON-Simd/Asm and they benchmarked quite favourably for the sort of software rendering I'd be doing. Given that plus some extra threads I think we could achieve acceptable performance with some s/w only stuff rendering into a regular memory buffer.

The raw buffer is then sent to the Dispmanx resource using:

Code: Select all

int result = vc_dispmanx_resource_write_data(this->front_resource, VC_IMAGE_RGBA32, this->pitch, pBuffer, &dst_rect);
However this call seems to be the route of the performance problem. For 1920x1080x32 (as above) this call alone takes about 40ms!
Obviously changing surface formats (IE: RGB565) halves that to 20ms. (This is on an RPi 3b+). But I really want to stick to 32bit ARGB for rendering and to have the alpha blending features of the Dispmanx layers.

On the Dispmanx side I found that switching from

Code: Select all

vc_dispmanx_update_submit_sync(update);
to

Code: Select all

vc_dispmanx_update_submit(update, NULL, 0);
Gave a slight bump in performance.

I also found that changing the config.txt gpu_mem setting from 128mb to 256mb made a huge difference. This reduced the 40ms down to about 25ms. This is strange as i'm running from shell on boot and there is nothing using the GPU, so a single Dispmanx surface with update shouldn't be getting anywhere close any sort of memory limit, even 128mb. Can't explain this one, but it was repeatable over different test runs.

Given that the core performance hit seems to be the resource_write operation I've started looking to see if there wasn't a sneakier way to transfer the image data. I don't think even if I had a pointer to the raw image in GPU shared memory that I'd want to write directly to it, as it will most likely be uncached from the ARM's point of view, so rendering to a buffer and then transferring (possibly via NEON streaming store) etc would be the way to go.

I found the hello_fft work, which includes an interface to the mailbox as well as some functions for locking and mapping GPU memory to an ARM pointer, combined with notes I found here:
http://blog.eihis.com/2015/07/03/raspbe ... gpu-hacks/

I had hoped it might be possible to do something like this:

Code: Select all

	GPU_HOST gpuHost;

		gpu_get_host_info(&gpuHost);

		int fd = mbox_open();
		uint32_t pb = vc_dispmanx_resource_get_image_handle(this->front_resource);

		mem_lock(fd, pb);
		VC_IMAGE_T* pImage = (VC_IMAGE_T*)mapmem(BUS_TO_PHYS(pb), sizeof(VC_IMAGE_T));
		memset32(pImage->image_data, 0xff, this->size);
		mem_unlock(fd, pb);
		unmapmem(pImage, sizeof(VC_IMAGE_T));
		mbox_close(fd);
I've tried various different options which each result in bus_error or do nothing so I'm obviously missing something, but I'm sure something like this should be possible. I noted from another topic from HermanHermitage about adding a marker to the image data and doing a full scan of /dev/mem to find it but the above approach sounds like it would be less hacky (and probably quicker) if it can be made to work.

Does anyone have any ideas ?

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: Getting direct access to a dispmanx resource

Sun Feb 24, 2019 7:58 pm

Update:

I found the following mailbox message:

Get Dispmanx Resource mem handle
Tag: 0x00030014
Request:
Length: 4
Value:
u32: dispmanx resource handle
Response:
Length: 8
Value:
u32: 0 is successful
u32: mem handle for resource
Gets the mem_handle associated with a created dispmanx resource. This can be locked and the memory directly written from the arm to avoid having to copy the image data to GPU.

Which I've put in as:

Code: Select all

unsigned get_dispmanx_resource_mem(int file_desc, unsigned handle)
{
	int i = 0;
	unsigned p[32];
	p[i++] = 0;			 // buffer size in bytes (including header values, end tag and padding)
	p[i++] = 0x00000000; // process request
	p[i++] = 0x30014;	 // (the tag id = get dispmanx resource mem handle)
	p[i++] = 4;			 // (size of the value buffer)
	p[i++] = 4; // (size of the data)
	p[i++] = handle;

	p[i++] = 0x00000000; // end tag
	p[0] = i * sizeof *p; // actual size

	mbox_property(file_desc, p);
	return p[6];
}
This call returns success 0x80000000 in p[], but it doesn't seem to return a valid handle to lock.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7443
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 11:13 am

DispmanX dates back to the original multimedia coprocessor days, so generally views the world as being on a physically separate chip to the client and has to copy everything around. There were a couple of extensions tacked on the side, but I couldn't say as to how functional they are.

If you're just trying to display some s/w rendered frames then you're probably better off using the MMAL video_render component with zero copy. It looks like I even wrote a simple example app in the past - https://github.com/6by9/mmal_render_example.
MMAL zero copy defaults to cached on the ARM, uncached on the VPU. The required cache flush/invalidates are done on the buffer as it is passed back and forth.

About the only thing I think you miss out on is vsync callbacks, although you could register with dispmanx just for those if you really need them. video_render will return the buffers automatically once they have been removed from the display.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 11:41 am

Hi,

Thanks for the info!

I grabbed your sample and modified it to do a fullscreen 1920x1080 and tried a few different encoding modes, RGBA and RGB32.
RGB32 seems to generate the following errors:
mmal: mmal_vc_port_info_set: failed to set port info (2:0): EINVAL
mmal: mmal_vc_port_set_format: mmal_vc_port_info_set failed 0x80e9a0 (EINVAL)
However it does go on to work, but instead of full screen it looks letter-boxed.

The performance of this approach is pretty great I have to say. I already have a dispmanx vsync callback setup, so I can definitely rely on that for sync'ing anything up.

I think I'd definitely go with this approach in future, however for my own personal curiosity I'd love to get the other dispmanx approach working.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7443
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 11:54 am

terraspace wrote:
Mon Feb 25, 2019 11:41 am
I grabbed your sample and modified it to do a fullscreen 1920x1080 and tried a few different encoding modes, RGBA and RGB32.
RGB32 seems to generate the following errors:
mmal: mmal_vc_port_info_set: failed to set port info (2:0): EINVAL
mmal: mmal_vc_port_set_format: mmal_vc_port_info_set failed 0x80e9a0 (EINVAL)
Check the firmware version you are running with "vcgencmd version".
RGB32 was only added December 14th 2018. Prior to that you needed to use MMAL_ENCODING_RGBA or MMAL_ENCODING_BGRA.
Prior to Oct 15th 2018 the per-pixel alpha was previously ignored anyway. That update also added the ALPHA_FLAGS_PREMULT and ALPHA_FLAGS_MIX flags so that you could combine per pixel and per plane alpha.
terraspace wrote:However it does go on to work, but instead of full screen it looks letter-boxed.

The performance of this approach is pretty great I have to say. I already have a dispmanx vsync callback setup, so I can definitely rely on that for sync'ing anything up.

I think I'd definitely go with this approach in future, however for my own personal curiosity I'd love to get the other dispmanx approach working.
If you provide a complete test app that I can simply build and run, then I'll take a quick look as to why it doesn't do what you want. Sorry, I'm not prepared to invest the time to write the app myself as I'm likely to diverge from what you've done in subtle ways.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7443
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 12:47 pm

Tying this to your duplicate post - https://www.raspberrypi.org/forums/view ... &p=1434133
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 1:03 pm

Here's a quick test,

I took the std. hello_dispmanx sample and added the mailbox related files, added a block of code in that should get the resource into a raw ptr..

https://github.com/john-terraspace/dispmanx.git

Let me know if that makes sense.
Thanks!
John

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7443
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 2:02 pm

I have run it up, and actually I'm not going to debug that in detail. Having to run as sudo in order to mmap random chunks of /dev/mem is too nasty.

vc_dispmanx_resource_get_image_handle does return a bus address pointer to the VC_IMAGE_T *.

get_dispmanx_resource_mem gives you the mem_handle from that structure directly. You don't actually need the rest of the VC_IMAGE_T *.
mem_lock gives you a bus address for the image data buffer (NOT the VC_IMAGE_T*).
mapmem should be able to map that and you can write to it.

My end result is

Code: Select all

/*
Copyright (c) 2012, Broadcom Europe Ltd
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
    * Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.
    * Neither the name of the copyright holder nor the
      names of its contributors may be used to endorse or promote products
      derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

// A simple demo using dispmanx to display an overlay


#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <assert.h>
#include <unistd.h>
#include <sys/time.h>

#include "bcm_host.h"

// New includes and defines
#include "mailbox.h"
#include "interface/vctypes/vc_image_structs.h"

typedef VC_IMAGE_T VC_IMAGE_BUF_T;
#define BUS_TO_PHYS(x) ((x)&~0xC0000000)

#define WIDTH   200
#define HEIGHT  200

#ifndef ALIGN_UP
#define ALIGN_UP(x,y)  ((x + (y)-1) & ~((y)-1))
#endif

typedef struct
{
    DISPMANX_DISPLAY_HANDLE_T   display;
    DISPMANX_MODEINFO_T         info;
    void                       *image;
    DISPMANX_UPDATE_HANDLE_T    update;
    DISPMANX_RESOURCE_HANDLE_T  resource;
    DISPMANX_ELEMENT_HANDLE_T   element;
    uint32_t                    vc_image_ptr;

} RECT_VARS_T;

static RECT_VARS_T  gRectVars;

static void FillRect( VC_IMAGE_TYPE_T type, void *image, int pitch, int aligned_height, int x, int y, int w, int h, int val )
{
    int         row;
    int         col;

    uint16_t *line = (uint16_t *)image + y * (pitch>>1) + x;

    for ( row = 0; row < h; row++ )
    {
        for ( col = 0; col < w; col++ )
        {
            line[col] = val;
        }
        line += (pitch>>1);
    }
}

int main(void)
{
    RECT_VARS_T    *vars;
    uint32_t        screen = 0;   
    int             ret;
    VC_RECT_T       src_rect;
    VC_RECT_T       dst_rect;
    VC_IMAGE_TYPE_T type = VC_IMAGE_RGB565;
    
    // New variables ...
    int             fd;
    VC_IMAGE_T* pImage;
    uint32_t memHandle;
    uint32_t addr;
    
    int width = WIDTH, height = HEIGHT;
    int pitch = ALIGN_UP(width*2, 32);
    int aligned_height = ALIGN_UP(height, 16);
    VC_DISPMANX_ALPHA_T alpha = { DISPMANX_FLAGS_ALPHA_FROM_SOURCE | DISPMANX_FLAGS_ALPHA_FIXED_ALL_PIXELS, 
                             120, /*alpha 0->255*/
                             0 };

    vars = &gRectVars;

    bcm_host_init();

    printf("Open display[%i]...\n", screen );
    vars->display = vc_dispmanx_display_open( screen );

    ret = vc_dispmanx_display_get_info( vars->display, &vars->info);
    assert(ret == 0);
    printf( "Display is %d x %d\n", vars->info.width, vars->info.height );

    vars->image = calloc( 1, pitch * height );
    assert(vars->image);

    memset(vars->image, 0xFF, pitch*height);
    vars->resource = vc_dispmanx_resource_create( type,
                                                  width,
                                                  height,
                                                  &vars->vc_image_ptr );
    assert( vars->resource );
    vc_dispmanx_rect_set( &dst_rect, 0, 0, width, height);
    ret = vc_dispmanx_resource_write_data(  vars->resource,
                                            type,
                                            pitch,
                                            vars->image,
                                            &dst_rect );
    
    assert( ret == 0 );
    vars->update = vc_dispmanx_update_start( 10 );
    assert( vars->update );

    vc_dispmanx_rect_set( &src_rect, 0, 0, width << 16, height << 16 );

    vc_dispmanx_rect_set( &dst_rect, ( vars->info.width - width ) / 2,
                                     ( vars->info.height - height ) / 2,
                                     width,
                                     height );

    vars->element = vc_dispmanx_element_add(    vars->update,
                                                vars->display,
                                                2000,               // layer
                                                &dst_rect,
                                                vars->resource,
                                                &src_rect,
                                                DISPMANX_PROTECTION_NONE,
                                                &alpha,
                                                NULL,             // clamp
                                                VC_IMAGE_ROT0 );

    ret = vc_dispmanx_update_submit_sync( vars->update );
    assert( ret == 0 );

    // New code to try get raw pointer to resource pixel data
    // Based on some notes from Herman Hermitage and this link:
    // http://blog.eihis.com/2015/07/03/raspberry-pi-dispmanx-resources-and-videocore-gpu-hacks/
    fd = mbox_open();
    
    //pImage = vc_dispmanx_resource_get_image_handle( vars->resource ); // This call is supposed to return a ptr to a VC_IMAGE_T struct in GPU mem.. 
    // In theory, it has a mem_handle field, but given this returned address you cannot directly access it, not sure if it should be mem mapped first?
    // with something like:
    // VC_IMAGE_T* pImage2 = (VC_IMAGE_T*)mapmem(BUS_TO_PHYS(pImage), sizeof(VC_IMAGE_T));
    
//    printf("VC_IMAGE_T %ux%u format %u, memhandle %u\n", pImage->width, pImage->height, pImage->type, pImage->mem_handle );
    memHandle = get_dispmanx_resource_mem(fd, vars->resource); // Use the mailbox to get a pointer to the resource as per:  https://github.com/raspberrypi/firmware/wiki/Mailbox-property-interface
    printf("memhandle %u\n", memHandle );
	addr = mem_lock(fd, memHandle); // Lock the gpu handle, should return a pointer. NB Physical/bus address
    printf("addr %x\n", addr );
	
	unsigned char* pImageData = (unsigned char*)mapmem(BUS_TO_PHYS(addr), pitch*height);

    if (pImageData)
    {
        sleep(4);
        FillRect( type, pImageData, pitch, aligned_height,  0,  0, width,      height,      0xFFFF );
        sleep(1);
        FillRect( type, pImageData, pitch, aligned_height,  0,  0, width,      height,      0xF800 );
        sleep(1);
        FillRect( type, pImageData, pitch, aligned_height, 20, 20, width - 40, height - 40, 0x07E0 );
        sleep(1);
        FillRect( type, pImageData, pitch, aligned_height, 40, 40, width - 80, height - 80, 0x001F );
        sleep(1);
    //	memset(pImageData, 0xff, 100); // write something into the pixel data

        unmapmem(pImageData, pitch*height);
    }
	
	mem_unlock(fd, memHandle); // unlock gpu mem handle
    mbox_close(fd); // close mailbox
    
    printf( "Sleeping for 4 seconds...\n" );
    sleep( 4 );

    vars->update = vc_dispmanx_update_start( 10 );
    assert( vars->update );
    ret = vc_dispmanx_element_remove( vars->update, vars->element );
    assert( ret == 0 );
    ret = vc_dispmanx_update_submit_sync( vars->update );
    assert( ret == 0 );
    ret = vc_dispmanx_resource_delete( vars->resource );
    assert( ret == 0 );
    ret = vc_dispmanx_display_close( vars->display );
    assert( ret == 0 );

    return 0;
}

I've rearranged a couple of blocks so the resource is put on the screen before mmaping. You can then see that it is directly writing the live buffer as the writes for red, green, and blue are done. Yes, that could tear.
Pass as to the caching options in this path. As you are unmapping almost immediately I would expect that to flush but can't guarantee it.

I'm not pushing that back to github as others may then take that as a good example, and it really isn't.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 2:18 pm

Got it :)

I thought the logic was almost right, just wasn't quite sure which steps were needed or not needed and in what order exactly.

Randomly mapping from /dev/mem isn't exactly a good idea... but it's interesting to see "how" this could work, but as I said before I think your mmal method is much better and probably a lot faster too so I'd stick with that.

Thanks for the help!

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 2:25 pm

Just put this updated code back into my original experiment, I will say one thing.. It is pretty fast.

1920x1080x32bit argb buffer, complete fill with pattern data via neon simd with all the dispmanx update and this code to copy the data from the normal cached memory buffer to the vc resource ptr runs in 8ms (+- 120fps).

tvjon
Posts: 710
Joined: Mon Jan 07, 2013 9:11 am

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 3:36 pm

6by9 wrote:
Mon Feb 25, 2019 2:02 pm
...
I'm not pushing that back to github as others may then take that as a good example, and it really isn't.
Can you provide a link to

helpers/debug_utils/

please, as I can't find such in any of the userlands or /opt/vc ?

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 4:18 pm

I ignored all of the helpers stuff as I couldn't find in the vc folders in the pi or firmare on git. I assumed it was more for internal dev and building of the firmware and user-space libs. If you go bare-bones against what I did there you shouldn't need it ?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7443
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 4:50 pm

Yup, I'm not quite sure why vc_image_structs.h includes "helpers/debug_utils/debug_writer.h". I tried including it so that I could look inside the VC_IMAGE_T, but it's not actually required.

Just remove the include and you should be fine.
Or hack vc_image_structs.h to remove

Code: Select all

/******************************************************************************
 Debugging rules (defined in camera_debug.c)
 *****************************************************************************/
extern DEBUG_WRITE_ENUM_LOOKUP_T vc_image_type_lookup[];
extern DEBUG_WRITE_ENUM_LOOKUP_T vc_image_bayer_order_lookup[];
extern DEBUG_WRITE_ENUM_LOOKUP_T vc_image_bayer_format_lookup[];
extern DEBUG_WRITE_RULE_T vc_image_info_rule[];
extern DEBUG_WRITE_RULE_T vc_image_rule[];
from almost the end of vc_image_structs.h and again you should be OK.

At some point I'll try to do an official cleanup of that as there really shouldn't be that dependency.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

tvjon
Posts: 710
Joined: Mon Jan 07, 2013 9:11 am

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 5:05 pm

Just seen your replies *after* I'd edited out the lines in vc_image_structs.h

I don't normally like to do that because then I end up with diiferent versions of official headers.

There are a few oddities in some of the includes, yes.

Over the weekend I was looking for a MMAL audio decoder but in fact it turns out to be "NONE".

Anyway 6by9's listing in this thread now builds & runs ok, thank you.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7443
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 5:22 pm

tvjon wrote:
Mon Feb 25, 2019 5:05 pm
Over the weekend I was looking for a MMAL audio decoder but in fact it turns out to be "NONE".
There's no default component listed, but I would have expected "vc.ril.audio_decode" to work. I thought OMXPlayer used it, but on closer look I think it uses ffmpeg.

Again there's some history behind it. Audio is significantly lighter weight than video, so on almost all systems there was no need to offload it from the ARM. If there's no need to offload it then there is no need for the codec code to be present in the VPU firmware.

There are MMAL components that wrap avcodec (interface/mmal/components/avcode_audio_decoder) which should be part of libmmal_components. I haven't checked them, but again they should work and be fairly easy to drop into a pipeline.
Sorry, not one I'm going to play with at the moment. I haven't tried building mmal_play recently, but amend the default audio_decode component and I suspect it may just work (host_applications/linux/apps/mmal-play, although the source is in host_applications/vmcs/test_apps/mmalplay). The niggle may be the changes between avcodec and ffmpeg between various versions of Debian. I couldn't say which was actually being used but recall a load of pain when I got it wrong trying to send the output of raspivid into container writers.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7443
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 5:26 pm

6by9 wrote:
Mon Feb 25, 2019 5:22 pm
Sorry, not one I'm going to play with at the moment. I haven't tried building mmal_play recently, but amend the default audio_decode component and I suspect it may just work (host_applications/linux/apps/mmal-play, although the source is in host_applications/vmcs/test_apps/mmalplay).
And this is where I notice that host_applications/linux/apps/mmal-play isn't in the userland repo.
I don't think I'll get shot for posting it here, and I guess I ought to look at getting it merged in (assuming the app works).

Code: Select all

#******************************************************************************
#*
#*    Description:
#*        mmal-play app Makefile
#*
#******************************************************************************

include $(LINUX_VOB)/makerules/linux_env.mk

TARGET := mmal-play
OBJSUBDIR := $(TARGET)

MMAL_PLAY_SRC := $(BRCM_VIDEOCORE_FIRMWARE_DIR)/host_applications/vmcs/test_apps/mmalplay
SRCS := $(MMAL_PLAY_SRC)/mmalplay.c \
        $(MMAL_PLAY_SRC)/playback.c

CFLAGS += -DLINUX_FRAMEWORK_EXCLUDE_MAIN -D_HAVE_TIMER_T -DENABLE_CONTAINERS_STANDALONE

INCDIR := \
        $(MMAL_PLAY_SRC) \
        $(BRCM_VIDEOCORE_FIRMWARE_DIR) \
        $(BRCM_VIDEOCORE_FIRMWARE_DIR)/host_applications/linux/libs/sm \
        $(BRCM_VIDEOCORE_FIRMWARE_DIR)/interface/vcos \
        $(BRCM_VIDEOCORE_FIRMWARE_DIR)/interface/vcos/pthreads \
        $(BRCM_VIDEOCORE_FIRMWARE_DIR)/interface/vcos/generic \
        $(BRCM_VIDEOCORE_FIRMWARE_DIR)/interface/mmal \
        $(BRCM_VIDEOCORE_FIRMWARE_DIR)/interface/mmal/vc \
        $(BRCM_VIDEOCORE_FIRMWARE_DIR)/containers

WARNINGS_AS_ERRORS := 1

LIBS += pthread rt dl m
LIBS += mmal mmal_vc mmal_components containers vcos vchiq
LIBS += vcsm
#LIBS += tvservice

#
# Include default build rules
#
include $(LINUX_VOB)/makerules/user_mode_rules.mk

all: $(PROG)
install: install-prog
#clean: clean-prog
edit: No, doesn't help much as there are includes in there further makefile rules that are included.
This app was originally run under Android, so things were a little different.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

tvjon
Posts: 710
Joined: Mon Jan 07, 2013 9:11 am

Re: Getting direct access to a dispmanx resource

Mon Feb 25, 2019 6:12 pm

"There's no default component listed, but I would have expected "vc.ril.audio_decode" to work. I thought OMXPlayer used it, but on closer look I think it uses ffmpeg."

Along with demux etc..

"Again there's some history behind it.
Audio is significantly lighter weight than video, so on almost all systems there was no need to offload it from the ARM. If there's no need to offload it then there is no need for the codec code to be present in the VPU firmware."

The history aspect I find very interesting actually.

" The niggle may be the changes between avcodec and ffmpeg between various versions of Debian. "

Exactly, & amongst iterations of their libraries!

"I couldn't say which was actually being used but recall a load of pain when I got it wrong trying to send the output of raspivid into container writers."

I'm not surprised. In some ways I actually prefer to use GPU based components, even though it's harder to understand. The software works over several YEARS & mostly works fine on any RPi SBC, without scores of #ifdef's.

"Sorry, not one I'm going to play with at the moment."

I understand.

"... and I guess I ought to look at getting it merged in (assuming the app works). "

I have it here. I got it to build & (apart obviously from no sound) it works just fine. It's a good alternative to hello_video.bin, playing more formats, & less intimidating (to me) than omxplayer, if only because it's not written in C++ :)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7443
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Getting direct access to a dispmanx resource

Tue Feb 26, 2019 10:39 am

If you've got makefiles or other changes that will get mmalplay running with a standard userland, then please feel free to create a PR to get them merged.
There's mmalcamas well which probably uses an almost identical makefile. That was part of the regression tests that were set up, and should be simpler than raspistill/vid.

I see that the MMAL avcodec_audio_decoder component isn't built - that's a bit annoying, but probably down to the libavcodec/ffmpeg API changes. I wonder if it can be ressurected easily. It's probably part of the reason that Broadcom wrote their own containers parsers.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

tvjon
Posts: 710
Joined: Mon Jan 07, 2013 9:11 am

Re: Getting direct access to a dispmanx resource

Tue Feb 26, 2019 12:19 pm

"There's mmalcamas well which probably uses an almost identical makefile."

Yes, it also works fine, & apart from file names, yes, same makefile.

"It's probably part of the reason that Broadcom wrote their own containers parsers."

I think that was a smart decision really. It's very handy that mmalplay plays mp4's directly eg., unlike hello_video.bin & without the relative complexity of omxplayer.


I don't find it much fun to even try to keep track of all the libavcodec/ffmpeg changes. For example I modified a VLC version (224 I think) to use the DPI interface to display an IP camera project.
I didn't attempt to go through that again for the next n versions, so just use the old modified one. Life is so short...

Oh, just remembered, most mp4 files play too fast. There is a clock component, but I've not had a chance to investigate. Actually - just occurred to me, I wonder if too fast is because it was supposed to be locked to audio clock, but there's no audio?

Step works (using the same technique I used - getchar() :-)

Seek works for MP4, but strangely, stripping out the audio from some MP4 files, thus leaving just the h264 track, doesn't seek.

It'd be nice to sidestep libav etc. & just do audio via the GPU where I imagine it could remain stable for a very long time...

A rudimentary (but not using libavcodec/ffmpeg) would be very nice to give basic audio output using mmalplay.

Jpeg's also display ok, but only up to about 1 megabyte. > & the display goes black, maybe returning several tens of seconds later, but more often my attached LG HDMI display reports it can't cope with a change to 87 KHz line rate.
Presumably this is related to the HVS in some way?

I find that particularly interesting, as a couple of years or so ago (tempus fugit) I managed to get the MJPEG built in decoder working for arbitrary jpeg's with EXACTLY the same results as mmalplay gives now for my test jpegs!

I never "announced" this with a forum post, as ISTR you hadn't rejoined the Foundation then, & frankly, I wasn't confident of any useful outcome at the time.

"If you've got makefiles or other changes that will get mmalplay running with a standard userland, then please feel free to create a PR to get them merged."

I confess I've never done that so far, so I'll need to read about how to go about it. I'll try to do it later this evening.


Just going back a moment to terraspace's dispmanx (in case the thread-deviation police intervene :) ) I wanted to get the source in this thread built so I could look at it later when I have time.
Just running your amended code I notice it takes several seconds to get the first square showing on screen - much slower than say dom's de facto hello_dispmanx.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7443
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Getting direct access to a dispmanx resource

Tue Feb 26, 2019 12:54 pm

tvjon wrote:
Tue Feb 26, 2019 12:19 pm
"There's mmalcamas well which probably uses an almost identical makefile."

Yes, it also works fine, & apart from file names, yes, same makefile.

"It's probably part of the reason that Broadcom wrote their own containers parsers."

I think that was a smart decision really. It's very handy that mmalplay plays mp4's directly eg., unlike hello_video.bin & without the relative complexity of omxplayer.


I don't find it much fun to even try to keep track of all the libavcodec/ffmpeg changes. For example I modified a VLC version (224 I think) to use the DPI interface to display an IP camera project.
I didn't attempt to go through that again for the next n versions, so just use the old modified one. Life is so short...

Oh, just remembered, most mp4 files play too fast. There is a clock component, but I've not had a chance to investigate. Actually - just occurred to me, I wonder if too fast is because it was supposed to be locked to audio clock, but there's no audio?
Clocking was always a funny one with MMAL, and I never fully understood how they implemented it.
tvjon wrote:Step works (using the same technique I used - getchar() :-)

Seek works for MP4, but strangely, stripping out the audio from some MP4 files, thus leaving just the h264 track, doesn't seek.

It'd be nice to sidestep libav etc. & just do audio via the GPU where I imagine it could remain stable for a very long time...
Please note this decoding is all happening on the ARM, not the VPU. Whilst those libraries are more likely to be stable, there is no guarantee on that, and they may get deprecated.
tvjon wrote:A rudimentary (but not using libavcodec/ffmpeg) would be very nice to give basic audio output using mmalplay.

Jpeg's also display ok, but only up to about 1 megabyte. > & the display goes black, maybe returning several tens of seconds later, but more often my attached LG HDMI display reports it can't cope with a change to 87 KHz line rate.
Presumably this is related to the HVS in some way?

I find that particularly interesting, as a couple of years or so ago (tempus fugit) I managed to get the MJPEG built in decoder working for arbitrary jpeg's with EXACTLY the same results as mmalplay gives now for my test jpegs!

I never "announced" this with a forum post, as ISTR you hadn't rejoined the Foundation then, & frankly, I wasn't confident of any useful outcome at the time.
Most likely the HVS is overloaded. It mainly depends on the amount of scaling, particularly vertically. More than around 2500x1200 will cause problems, so it's more likely to be a problem of resolution than file size.
tvjon wrote:"If you've got makefiles or other changes that will get mmalplay running with a standard userland, then please feel free to create a PR to get them merged."

I confess I've never done that so far, so I'll need to read about how to go about it. I'll try to do it later this evening.
Git really isn't that bad, although https://xkcd.com/1597/ is frighteningly close to the truth sometimes.
Create an account on github.com if you haven't already got one.
Go to https://github.com/raspberrypi/userland/ and almost top right is the word "Fork". Click it and it'll ask you where you want to fork to - you want it under your account.
On your Pi, "git clone [email protected]:<your_name>/userland.git" will retrieve your fork of userland.
Make you changes. "git add <filenames>". "git commit" and it'll open an editor (probably nano) to give your commit a description. Quit the editor and you've made your commit. "git status" at any time tells you what files have changed, and whether they've been added (staged) for commiting. "git diff" shows you the diffs from the top of tree.
"git push origin HEAD:master" pushes back to github. You'll need to sort out pushing your SSH keys to github first - https://help.github.com/en/articles/con ... b-with-ssh
Go to your repo and it'll probably tell you "recent updates to branch master - create pull request". Click the button and it'll go through the form to create a PR back into the upstream userland repo.
tvjon wrote:Just going back a moment to terraspace's dispmanx (in case the thread-deviation police intervene :) ) I wanted to get the source in this thread built so I could look at it later when I have time.
Just running your amended code I notice it takes several seconds to get the first square showing on screen - much slower than say dom's de facto hello_dispmanx.
I'd foolishly trashed my dev directory for this as I hoped it was done and dusted. Fortunately it's not too bad to recreate.
I had deliberately inserted a load of sleeps in there to show that we were modifying the displayed resource with no intermediate dispmanx calls.

Running it myself I get a grey box up almost immediately (the result of the memset(0xFF) and vc_dispmanx_resource_write_data), we the wait 4 seconds, fill with red, wait a second, fill with green, wait a second, fill with blue, wait 4 seconds, and teardown.
Remove the sleeps and the rgb box is up almost instantly.
The vc_dispmanx_resource_write_data step could be skipped totally as we're directly overwriting the resource's image buffer, but it demonstrated the point.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: Getting direct access to a dispmanx resource

Tue Feb 26, 2019 2:32 pm

I can confirm those results, the updates are instant once you remove the sleeps.

So as a follow on to this, I started animated the contents I was drawing into the buffer, which resulted in a lot of flicker (not tearing). Not to worry me thinks, something like that would be expected if we're directly modifying the resource data during a present interval. So I created two resources (to double buffer).
A back one and a front one, lock the back one, render there then use

Code: Select all

result = vc_dispmanx_element_change_source(update, this->element, this->back_resource);
To swap the front and back buffers.

That's all great, but it still flickers like a pig, then I realised, the code is still trying to run at 120'ish fps, which the vc is probably not keeping up with so you could land up with half frames, no frame .. who knows.. So I thought no worries, I will vsync it.. I really wouldn't want over 60fps anyway.

No for the life of me I can't figure the vsync .. It's definitely triggering:

Code: Select all


	vc_dispmanx_vsync_callback(gRectVars.display, vsync_ui_callback, NULL);
}

void vsync_ui_callback(DISPMANX_UPDATE_HANDLE_T update, void *arg)
{
	std::cout << "frame" << std::endl;
}

The vsync gets an update handle, but if I try call anything in the vsync that uses that update handle, like the above swap buffer nothing happens.. any ideas?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7443
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Getting direct access to a dispmanx resource

Tue Feb 26, 2019 3:34 pm

Calling back into libraries from callbacks can be very dodgy.

I'm expecting you to need to have a dispmanx_update_start before the dispmanx_element_change_source, and then a vc_dispmanx_update_submit at the end. Use vc_dispmanx_update_submit_sync and I believe it should synchronise to the vsync automagically.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: Getting direct access to a dispmanx resource

Tue Feb 26, 2019 3:40 pm

That's what I thought about the vc_dispmanx_update_submit_sync, however when you do it that there are two issues:

1. You get occasional stutter, which seems odd if the swap is vsync'ed and we know the rest of the code is well over 60fps it should be smooth.
2. The submit_sync takes the time up from 8ms to between 30-40ms (not 16.6ms like you'd expect if it were syncing).

Code: Select all

DISPMANX_UPDATE_HANDLE_T update = vc_dispmanx_update_start(0);
renderSurface->Present(update);
int result = vc_dispmanx_update_submit_sync(update);

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: Getting direct access to a dispmanx resource

Tue Feb 26, 2019 4:03 pm

Oddly now running it again it's smooth and <=16ms .. might have been due to having breakpoints set and attached to it via Visual Studio..

It's a pity you can't cleanly trigger the update directly from the vsync, although I guess one could always just have the update running on it's own thread and use the vsync to wake it..

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7443
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Getting direct access to a dispmanx resource

Tue Feb 26, 2019 4:31 pm

terraspace wrote:
Tue Feb 26, 2019 4:03 pm
It's a pity you can't cleanly trigger the update directly from the vsync, although I guess one could always just have the update running on it's own thread and use the vsync to wake it..
I'm not an expert on DispmanX and what you can and can't do from the callbacks.
Generally it is safer to signal your own thread to do the actual work as you don't know what else you may be blocking if you take any significant time in the callback.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Graphics programming”