Videocore/Firmware linux API


33 posts   Page 1 of 2   1, 2
by cheery » Tue May 08, 2012 8:46 pm
I looked into firmware/opt/vc/ today, and read through the examples. The example is using vc_dispmanx -prefixed commands.

Then I peeked inside the videocore library itself. It's a little bit of a messed up with all sorts of commands, of which I'm not sure what of those commands have some use.

Since I like to try my hand on this kind of things, I wonder where to get documentation for the firmware linux API, or something other such?

Whatever was the case, I'd like to experiment with desktop environments of different kinds. Wondering how to control/access surfaces from other processes, so that compositing window manager could be buildt.

Also wondering.. shouldn't this thing be able to detect display changes? After all those display ports ought be hotpluggable shouldn't they?
User avatar
Posts: 219
Joined: Wed Jan 25, 2012 9:39 pm
by jamesh » Tue May 08, 2012 9:11 pm
The library to access the VC has been around since VC1 days (10 years?), and has been built on since then, so there is quite a menagerie of commands I'm afraid. dispmanx is a VC library for handling bitmaps, but I'm not sure there is a document available (i.e. to the public)  describing all the commands/api's. IIRC dispmanx is being deprecated in favour of a different library, one of the Khronos ones I think. Can't remember the name offhand. AT least that one will be documented as its a standard API.
Soon to be unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11583
Joined: Sat Jul 30, 2011 7:41 pm
by cheery » Tue May 08, 2012 9:24 pm
Why are the hello_pi -examples using that API if it's deprecated in favor of a some made by khronos?

BTW. Is it this? openwf
User avatar
Posts: 219
Joined: Wed Jan 25, 2012 9:39 pm
by jamesh » Wed May 09, 2012 6:42 am
Because dispmanx x is in the process of being deprecated. Hasn't been fully dispatched yet. Yes, I think OpenWF is what will be used, but don't quote me on it.
Soon to be unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11583
Joined: Sat Jul 30, 2011 7:41 pm
by cheery » Mon May 14, 2012 7:27 am
If you are going to spend months on providing the new API, could you provide some docs for the existing one even if it's deprecated?

I'd really like to get this going. And I'd need to be able to:
  • provide, retrieve and access video surfaces of processes
  • detect displays and initialize them for rendering (appears there's already example for this)
It's enough if I could do it somehow for now!
User avatar
Posts: 219
Joined: Wed Jan 25, 2012 9:39 pm
by jamesh » Mon May 14, 2012 8:38 am
cheery wrote:If you are going to spend months on providing the new API, could you provide some docs for the existing one even if it's deprecated?

I'd really like to get this going. And I'd need to be able to:
  • provide, retrieve and access video surfaces of processes
  • detect displays and initialize them for rendering (appears there's already example for this)
It's enough if I could do it somehow for now!


All the docs I have access to are Broadcom confidential I am afraid, and I have no idea who to talk to to release then, or the amount of work required to get them in a state where release is possible.
Soon to be unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11583
Joined: Sat Jul 30, 2011 7:41 pm
by cheery » Mon May 14, 2012 11:07 am
Three days ago there was added OpenMAX ilcomponents documentation into the firmware repository: https://github.com/raspberrypi/firmware ... umentation

There's some functions that look like interesting, but is this the library you mentioned earlier? It didn't look like easy to grasp. And didn't find anything else but video encoding/decoding in it for now.

I'd be glad if you could provide something. But don't let my interests go over more important tasks. Although if someone was doing xorg support recently, he might know how to do what I need. I'd probably get all right with an additional example in hello_pi.
User avatar
Posts: 219
Joined: Wed Jan 25, 2012 9:39 pm
by jamesh » Mon May 14, 2012 11:20 am
Not the same stuff. I think documentation on dispmanx was what was requested.

OpenMAX is a complete and utter nightmare to use btw.
Soon to be unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11583
Joined: Sat Jul 30, 2011 7:41 pm
by cheery » Mon May 14, 2012 10:20 pm
I went through the include file firmware/opt/vc/include/interface/vmcs_host/vc_dispmanx.h, it may not be so tricky to use without any docs.

It looks a lot like it'd allow multiple GLE-framebuffers at the same time, but not compositing on them. Am I correct?

Code: Select all
int vc_dispman_init( void );
void vc_dispmanx_stop( void );
int vc_dispmanx_rect_set( VC_RECT_T *rect, uint32_t x_offset, uint32_t y_offset, uint32_t width, uint32_t height );

These look quite clear. Apparently dispmanx provides a rectangle API.. seems retarded really.

Code: Select all
DISPMANX_RESOURCE_HANDLE_T vc_dispmanx_resource_create( VC_IMAGE_TYPE_T type, uint32_t width, uint32_t height, uint32_t *native_image_handle );
int vc_dispmanx_resource_write_data( DISPMANX_RESOURCE_HANDLE_T res, VC_IMAGE_TYPE_T src_type, int src_pitch, void * src_address, const VC_RECT_T * rect );
int vc_dispmanx_resource_write_data_handle( DISPMANX_RESOURCE_HANDLE_T res, VC_IMAGE_TYPE_T src_type, int src_pitch, VCHI_MEM_HANDLE_T handle, uint32_t offset, const VC_RECT_T * rect );
int vc_dispmanx_resource_read_data( DISPMANX_RESOURCE_HANDLE_T handle, const VC_RECT_T* p_rect, void *dst_address, uint32_t dst_pitch );
int vc_dispmanx_resource_delete( DISPMANX_RESOURCE_HANDLE_T res );

This also seems quite very much self-explanatory. These ones are for GPU resource buffers.

Code: Select all
// Opens a display on the given device
DISPMANX_DISPLAY_HANDLE_T vc_dispmanx_display_open( uint32_t device );
// Opens a display on the given device in the request mode
DISPMANX_DISPLAY_HANDLE_T vc_dispmanx_display_open_mode( uint32_t device, uint32_t mode );
// Open an offscreen display
DISPMANX_DISPLAY_HANDLE_T vc_dispmanx_display_open_offscreen( DISPMANX_RESOURCE_HANDLE_T dest, VC_IMAGE_TRANSFORM_T orientation );
// Change the mode of a display
int vc_dispmanx_display_reconfigure( DISPMANX_DISPLAY_HANDLE_T display, uint32_t mode );
// Sets the desstination of the display to be the given resource
int vc_dispmanx_display_set_destination( DISPMANX_DISPLAY_HANDLE_T display, DISPMANX_RESOURCE_HANDLE_T dest );
// Set the background colour of the display
int vc_dispmanx_display_set_background( DISPMANX_UPDATE_HANDLE_T update, DISPMANX_DISPLAY_HANDLE_T display,                                                                        uint8_t red, uint8_t green, uint8_t blue );
// get the width, height, frame rate and aspect ratio of the display
int vc_dispmanx_display_get_info( DISPMANX_DISPLAY_HANDLE_T display, DISPMANX_MODEINFO_T * pinfo );
// Closes a display
int vc_dispmanx_display_close( DISPMANX_DISPLAY_HANDLE_T display );

These are almost self-explanatory. Apparently opens a display handle, which can be attached to a resource buffer above.

Code: Select all
// Updates
// Start a new update, DISPMANX_NO_HANDLE on error
DISPMANX_UPDATE_HANDLE_T vc_dispmanx_update_start( int32_t priority );
// Add an elment to a display as part of an update
DISPMANX_ELEMENT_HANDLE_T vc_dispmanx_element_add ( DISPMANX_UPDATE_HANDLE_T update, DISPMANX_DISPLAY_HANDLE_T display,
                                                                     int32_t layer, const VC_RECT_T *dest_rect, DISPMANX_RESOURCE_HANDLE_T src,
                                                                     const VC_RECT_T *src_rect, DISPMANX_PROTECTION_T protection,
                                                                     VC_DISPMANX_ALPHA_T *alpha,
                                                                     DISPMANX_CLAMP_T *clamp, DISPMANX_TRANSFORM_T transform );
// Change the source image of a display element
int vc_dispmanx_element_change_source( DISPMANX_UPDATE_HANDLE_T update, DISPMANX_ELEMENT_HANDLE_T element,
                                                        DISPMANX_RESOURCE_HANDLE_T src );
// Change the layer number of a display element
int vc_dispmanx_element_change_layer ( DISPMANX_UPDATE_HANDLE_T update, DISPMANX_ELEMENT_HANDLE_T element,
                                                        int32_t layer );
// Signal that a region of the bitmap has been modified
int vc_dispmanx_element_modified( DISPMANX_UPDATE_HANDLE_T update, DISPMANX_ELEMENT_HANDLE_T element, const VC_RECT_T * rect );
// Remove a display element from its display
int vc_dispmanx_element_remove( DISPMANX_UPDATE_HANDLE_T update, DISPMANX_ELEMENT_HANDLE_T element );
// Ends an update
int vc_dispmanx_update_submit( DISPMANX_UPDATE_HANDLE_T update, DISPMANX_CALLBACK_FUNC_T cb_func, void *cb_arg );
// End an update and wait for it to complete
int vc_dispmanx_update_submit_sync( DISPMANX_UPDATE_HANDLE_T update );
// Query the image formats supported in the VMCS build
int vc_dispmanx_query_image_formats( uint32_t *supported_formats );

//New function added to VCHI to change attributes, set_opacity does not work there.
int vc_dispmanx_element_change_attributes( DISPMANX_UPDATE_HANDLE_T update,
                                                            DISPMANX_ELEMENT_HANDLE_T element,
                                                            uint32_t change_flags,
                                                            int32_t layer,
                                                            uint8_t opacity,
                                                            const VC_RECT_T *dest_rect,
                                                            const VC_RECT_T *src_rect,
                                                            DISPMANX_RESOURCE_HANDLE_T mask,
                                                            VC_IMAGE_TRANSFORM_T transform );

//xxx hack to get the image pointer from a resource handle, will be obsolete real soon
uint32_t vc_dispmanx_resource_get_image_handle( DISPMANX_RESOURCE_HANDLE_T res);

//Call this instead of vc_dispman_init
void vc_vchi_dispmanx_init (VCHI_INSTANCE_T initialise_instance, VCHI_CONNECTION_T **connections, uint32_t num_connections );

// Take a snapshot of a display in its current state.
// This call may block for a time; when it completes, the snapshot is ready.
int vc_dispmanx_snapshot( DISPMANX_DISPLAY_HANDLE_T display,
                                           DISPMANX_RESOURCE_HANDLE_T snapshot_resource,
                                           VC_IMAGE_TRANSFORM_T transform );

These last ones looks like they'd have something to do with updating the screen. But otherwise not really certain about them.
User avatar
Posts: 219
Joined: Wed Jan 25, 2012 9:39 pm
by JonathanGraham » Mon Jul 30, 2012 1:07 am
As ever dealing with an undocumented API is tiresome...after some experimenting I have a good idea how to use dispmanx.

This after noon I was thinking about messing with: vc_dispmanx_resource_write_data_handle()

It was less than helpful when no library actually exposed that interface. :roll:
User avatar
Posts: 39
Joined: Thu Jul 05, 2012 5:55 am
by AndrewS » Mon Jul 30, 2012 3:47 am
JonathanGraham wrote:This after noon I was thinking about messing with: vc_dispmanx_resource_write_data_handle()

It was less than helpful when no library actually exposed that interface. :roll:

Might be worth submitting a bug report? https://github.com/raspberrypi/firmware/issues
(assuming you're already updated to the latest firmware, of course)
User avatar
Posts: 3091
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
by JonathanGraham » Mon Jul 30, 2012 7:42 pm
I just checked the latest firmware from a few days back. No luck. I've registered an issue.
User avatar
Posts: 39
Joined: Thu Jul 05, 2012 5:55 am
by JonathanGraham » Tue Jul 31, 2012 3:55 am
I got a response. Essentially the call doesn't do anything.

This is where some documentation might come in handy.
User avatar
Posts: 39
Joined: Thu Jul 05, 2012 5:55 am
by jdbennet » Tue Jul 31, 2012 7:35 am
As far as I know, you cant much of the SoC docs without an NDA from broadcom?
Posts: 96
Joined: Sun Jul 22, 2012 2:25 pm
by brooc » Thu Jun 27, 2013 1:06 pm
Hi,
I am trying to take snapshots of the display and save them into a file to create images.
I have some code that takes the snapshot using vc_dispmanx_snapshot, and when I send back the resource to be displayed on screen it works fine.

When I come to save the image to a file I have no idea what is the correct way to do so since the image pointer is of type void*.
I tried to find some code on google but with no luck.
I used both RGB888 and RGB565 and nothing works.

Can someone please help me with that?
How are the pixels arranged in this void*?

Thanks
Posts: 58
Joined: Mon Jun 03, 2013 2:30 pm
by brooc » Thu Jun 27, 2013 1:46 pm
BTW if there is a better way to do this, please let me know...
Posts: 58
Joined: Mon Jun 03, 2013 2:30 pm
by dom » Thu Jun 27, 2013 2:05 pm
vc_dispmanx_snapshot fills in a GPU side resource.
vc_dispmanx_resource_read_data will get pixels on arm from a GPU resource.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 3999
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by brooc » Thu Jun 27, 2013 3:32 pm
dom wrote:vc_dispmanx_snapshot fills in a GPU side resource.
vc_dispmanx_resource_read_data will get pixels on arm from a GPU resource.

I know, that is what I did.
After calling vc_dispmanx_snapshot I called vc_dispmanx_resource_read_data to fill the void *data pointer of the resource.
The data is arranged in some fashion that I can't figure out.

I don't understand the pitch parameter and also I found some code that uses a power of 2 of the height and I think also the width instead of the actual dimensions.

I'll post my code once I get home, it'll be very helpful, if you could post an example of how to take the snapshot and save the data to a PPM file or whatever format easiest to save to...
Posts: 58
Joined: Mon Jun 03, 2013 2:30 pm
by jamesh » Thu Jun 27, 2013 6:41 pm
Pitch will need to be a multiple of 16 (so even a width of %16 != 0 will still have a pitch of %16 = 0). Pitch is the number of bytes to go from one row to the same X position on the next.

If YUV planar, then each of YUV will need to finished on a 16 row boundary (so there may be unused pixels in between the planes to pad the gap)

I'm not sure, but it's possible the void * points to a VC_IMAGE_T - see userland/helpers/vc_image/vc_image.h
Soon to be unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11583
Joined: Sat Jul 30, 2011 7:41 pm
by brooc » Thu Jun 27, 2013 10:03 pm
Thank you for the clarification regarding the pitch.
My code still does not work.

This is my code, I take a snapshot of the screen and then I display the image back on screen after moving to another screen so I see both images, one on top of the other.
Then I go to save the image to a PPM file, but that does not work.
What do you think the problem is?

Code: Select all
    int          screenWidth, screenHeight;
    VC_RECT_T      rectangle;
    RECT_VARS_T    *vars;
    uint32_t       screen = 0;
    int            ret;
    VC_IMAGE_TYPE_T type = VC_IMAGE_RGB888;
    int pitch;
    int row;

    vars = &gRectVars;

    bcm_host_init();

    printf("Open display[%i]...\n", screen );
    vars->display = vc_dispmanx_display_open( screen );

    ret = vc_dispmanx_display_get_info( vars->display, &vars->info);
    assert(ret == 0);
    printf( "Display is %d x %d\n", vars->info.width, vars->info.height );

    screenHeight = vars->info.height;
    screenWidth = vars->info.width;

    if(type == VC_IMAGE_RGB888)
       pitch = ALIGN_UP(screenWidth*3, 32);
    else
       pitch = ALIGN_UP(screenWidth*2, 32);

    vars->image = calloc( 1, pitch * screenHeight );
    assert(vars->image);

    vars->resource = vc_dispmanx_resource_create(type,
                                      screenWidth,
                                      screenHeight,
                                      &vars->vc_image_ptr );
    assert( vars->resource );

    vc_dispmanx_snapshot( vars->display,vars->resource, VC_IMAGE_ROT0);

    printf("Creating rectangle\n");
    vc_dispmanx_rect_set( &rectangle, 0, 0, screenWidth, screenHeight);

    vc_dispmanx_resource_read_data(vars->resource, &rectangle, vars->image, pitch);

    ret = vc_dispmanx_resource_write_data(  vars->resource,
                                            type,
                                            pitch,
                                            vars->image,
                                            &rectangle);
    assert( ret == 0 );

    printf("Move to a different screen\n");
    printf("Sleeping for 10 seconds...\n");
    sleep(10);

    vars->update = vc_dispmanx_update_start( 10 );
    assert( vars->update );

    vc_dispmanx_rect_set( &src_rec, 0, 0, screenWidth << 16, screenHeight << 16);

    vars->element = vc_dispmanx_element_add(    vars->update,
                                                vars->display,
                                                2000,               // layer
                                                &rectangle,
                                                vars->resource,
                                                &src_rec,
                                                DISPMANX_PROTECTION_NONE,
                                                &alpha,
                                                NULL,             // clamp
                                                VC_IMAGE_ROT0 );

    ret = vc_dispmanx_update_submit_sync( vars->update );
    assert( ret == 0 );

    printf( "Sleeping for 10 seconds...\n" );
    sleep( 10 );

    vars->update = vc_dispmanx_update_start( 10 );
    assert( vars->update );
    ret = vc_dispmanx_element_remove( vars->update, vars->element );
    assert( ret == 0 );
    ret = vc_dispmanx_update_submit_sync( vars->update );
    assert( ret == 0 );

    if (argc < 2) {
       printf("Usage: %s <filename>\n", argv[0]);
       return -1;
    }

    FILE *fp = fopen(argv[1], "wb+");

    if (!fp) {
       printf("File %s not found.\n", argv[1]);
       return -1;
    }

    fprintf(fp, "P6\n%d %d\n255\n", screenHeight, screenWidth);

    char* image_ptr = (char *)vars->image;

    printf("Size of image %d bytes\n", strlen(image_ptr));

    for(row = 0; row < screenHeight; row++)
    {
       image_ptr = ((char *)vars->image) + pitch * row;

       fwrite(image_ptr, sizeof(char), screenWidth * 3, fp);
    }

    fclose(fp);

    ret = vc_dispmanx_resource_delete( vars->resource );
    assert( ret == 0 );

    ret = vc_dispmanx_display_close( vars->display );
    assert( ret == 0 );

    return 0;
Posts: 58
Joined: Mon Jun 03, 2013 2:30 pm
by brooc » Fri Jun 28, 2013 7:35 am
I just found out what my problem was...
I had the header of the PPM file wrong, I mixed between the height and the width.

Changing this line solved the issue:
Code: Select all
fprintf(fp, "P6\n%d %d\n255\n", screenWidth, screenHeight);
Posts: 58
Joined: Mon Jun 03, 2013 2:30 pm
by brooc » Mon Jul 01, 2013 7:37 am
Hi,
Is it possible to create a splitter after the OMX player has already started?
I think using the snapshot functions is quite expensive (CPU/GPU time-wise)
When I use it repeatedly every half a second or less (i.e every 100 milliseconds) I begin to see flickering on the screen.

I thought maybe, if I could create a splitter, it would be less time consuming and I could get better performance.
If it is possible could someone direct me to an example?

Thanks
Posts: 58
Joined: Mon Jun 03, 2013 2:30 pm
by jamesh » Mon Jul 01, 2013 8:06 am
brooc wrote:Hi,
Is it possible to create a splitter after the OMX player has already started?
I think using the snapshot functions is quite expensive (CPU/GPU time-wise)
When I use it repeatedly every half a second or less (i.e every 100 milliseconds) I begin to see flickering on the screen.

I thought maybe, if I could create a splitter, it would be less time consuming and I could get better performance.
If it is possible could someone direct me to an example?

Thanks


Not after it has started I don't think - but perhaps you could create one at startup but disable the output port when you don't need it?

Snapshot would need lots of copies from GPU to CPU of the frame.
Soon to be unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11583
Joined: Sat Jul 30, 2011 7:41 pm
by brooc » Mon Jul 01, 2013 8:45 am
That would require changing the OMX player code.
I am creating a standalone application the will most likely be run only after the player has already started.

Is there a way to optimize the usage of the snapshot function and avoid flickering on the screen?
Posts: 58
Joined: Mon Jun 03, 2013 2:30 pm
by brooc » Tue Jul 02, 2013 6:08 pm
It turns out that every time I take a snapshot of a full screen no matter at what speed it causes the screen to flicker at that moment.
This seems like a bug in the dispmanx_snapshot function.

Where should I submit this bug?
Posts: 58
Joined: Mon Jun 03, 2013 2:30 pm