6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11216
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Fast method to pass data from framebuffer to MMAL

Fri Apr 30, 2021 1:07 pm

randomher0 wrote:
Fri Apr 30, 2021 12:38 pm
Hi, are there any resources on how to get this to work? I've applied the kernel patch as described above but could not get X11 running with it - only the 16bit console framebuffer worked. I got around 40fps for 1080p which is good. I'd really like to find a way to capture the screen at fast rates without changing the kernel. @6by9 proposed mmal hardware scaler component or transposer/writeback. Are there any sample projects or pointers you can suggest looking into ?
MMAL vc.ril.hvs is incompatible with vc4-kms-v3d. (It can work with vc4-fkms-v3d - the firmware has to be in control of the display pipeline).

The writeback connector is one that I've never understood fully. They were added for vc4 by Bootlin, and there are some igt-gpu-tools tests for them, but I don't know the details. I believe Android's SurfaceFlinger will also use them under some situations.
AIUI you treat it the same as any other DRM connector, assign planes to it, and then feed in an output FB for it to render to. The issue may be that DRM only allows a single master at a time, and X is likely to have claimed that role.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

randomher0
Posts: 24
Joined: Wed Apr 21, 2021 9:54 pm

Re: Fast method to pass data from framebuffer to MMAL

Fri Apr 30, 2021 4:07 pm

Somehow enumeration of resources doesn't work as I expected:
The "drmtoy" lists a single console framebuffer when X11 is killed, "modetest" does not list any framebuffers at all.
There is a fb_count in resources which is just 0 - drmtoy collects the framebuffer ids from the planes and prints them.

I suspect enumeration of connectors might just not work. There are just two connectors (hdmi and composite) enumerated - X11 is not running.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11216
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Fast method to pass data from framebuffer to MMAL

Fri Apr 30, 2021 4:55 pm

From modetest on Pi3B+ I get

Code: Select all

Encoders:
id      crtc    type    possible crtcs  possible clones 
31      83      TMDS    0x00000008      0x00000001
48      0       TVDAC   0x00000008      0x00000002
54      0       Virtual 0x00000001      0x00000004

That Virtual device is the writeback connector.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

randomher0
Posts: 24
Joined: Wed Apr 21, 2021 9:54 pm

Re: Fast method to pass data from framebuffer to MMAL

Fri Apr 30, 2021 5:08 pm

Cool, i just stumbled upon this as well - seems to be some sort of "workaround" to satisfy drm api - I'll dig some more - i don't really have a clue what to do with this Encoder? We need a connector, right?

How to get the corresponding writeback connector for the writeback encoder ("virtual") ?

Client capabilities must be set before discovering to retrieve writeback_connectors:

Code: Select all

drmSetClientCap(fd, DRM_CLIENT_CAP_WRITEBACK_CONNECTORS, 1);
How to set properties on the writeback connector (the output framebuffer) ?

Code: Select all

extern drmModePropertyPtr drmModeGetProperty(int fd, uint32_t propertyId);
extern int drmModeConnectorSetProperty(int fd, uint32_t connector_id, uint32_t property_id,

Trying to bring it together:

Code: Select all

int main() {
    int width = 1920;
    int height = 1080;
    int depth = 24;
    int bpp = 32;

    int fd = -1;
    int ret = -1;
    char* card = "/dev/dri/card0";

    int available = drmAvailable();
    if(!available) {
        fprintf(stderr, "drm is not available\n");
        goto error;
    }

    fd = open(card, O_RDONLY);
    if(fd == -1) {
        fprintf(stderr, "cannot open %s\n", card);
        goto error;
    }

    {
        drmVersionPtr ver = drmGetVersion(fd);
        printf("drm version %d.%d.%d driver=%.*s date=%.*s desc=%.*s\n",
            ver->version_major, ver->version_minor, ver->version_patchlevel,
            ver->name_len, ver->name,
            ver->date_len, ver->date,
            ver->desc_len, ver->desc);
        drmFreeVersion(ver);
    }

    {
        drmVersionPtr ver = drmGetLibVersion(fd);
        printf("drm lib version %d.%d.%d driver=%.*s date=%.*s desc=%.*s\n",
            ver->version_major, ver->version_minor, ver->version_patchlevel,
            ver->name_len, ver->name,
            ver->date_len, ver->date,
            ver->desc_len, ver->desc);
        drmFreeVersion(ver);
    }

    {
        const char *busid = drmGetBusid(fd);
        printf("busid=%s\n", busid);
        drmFreeBusid(busid);
    }

    drmSetClientCap(fd, DRM_CLIENT_CAP_UNIVERSAL_PLANES, 1);
    drmSetClientCap(fd, DRM_CLIENT_CAP_ATOMIC, 1);
    drmSetClientCap(fd, DRM_CLIENT_CAP_WRITEBACK_CONNECTORS, 1);

    drmModeResPtr res = drmModeGetResources(fd);

    drmModeConnectorPtr writebackConnector = NULL;

    for(int i = 0; i < res->count_connectors; i++) {
        drmModeConnectorPtr connector = drmModeGetConnector(fd, res->connectors[i]);
        if(connector->connector_type == DRM_MODE_CONNECTOR_WRITEBACK) {
            writebackConnector = connector;
            break;
        }
        drmModeFreeConnector(connector);
    }

    if(!writebackConnector) {
        fprintf(stderr, "No writeback connector found.\n");
        goto error;
    }

    printf("Writeback Connector found (id=%d):\n", writebackConnector->connector_id);

    struct drm_mode_create_dumb create_request = {
        .width = width,
        .height = height,
        .bpp = bpp
    };

    ret = ioctl(fd, DRM_IOCTL_MODE_CREATE_DUMB, &create_request);
    if (ret) {
        fprintf(stderr, "Dumb Buffer Object Allocation request of %ux%u@%u failed : %s\n",
                create_request.width, create_request.height, create_request.bpp);
        goto error;
    }

    printf("Allocated dumb buffer on GPU:  %ux%u@%u\n",
           create_request.width, create_request.height, create_request.bpp);

    uint32_t output_frame_buffer_id;
    ret = drmModeAddFB(fd, width, height, depth, bpp, create_request.pitch, create_request.handle, &output_frame_buffer_id);
    if (ret) {
        fprintf(stderr,"Could not add a framebuffer using drmModeAddFB : %s\n", strerror(ret));
        goto error;
    }

    printf("drmModeAddFB: added output framebuffer (id=%d).\n", output_frame_buffer_id);
    drmModeFBPtr outputFramebuffer = drmModeGetFB(fd, output_frame_buffer_id);
    printf("Framebuffer properties:\n");
    printf("   - width=%d\n", outputFramebuffer->width);
    printf("   - width=%d\n", outputFramebuffer->height);
    printf("   - depth=%d\n", outputFramebuffer->depth);
    printf("   - bpp=%d\n", outputFramebuffer->bpp);
    printf("   - handle=%d\n", outputFramebuffer->handle);
    printf("   - fb_id=%d\n\n", outputFramebuffer->fb_id);


    drmModePropertyPtr writebackCrtcIdProperty = NULL;
    drmModePropertyPtr writebackOutFenceProperty = NULL;
    drmModePropertyPtr writebackFbIdProperty = NULL;
    drmModePropertyPtr writebackPixelFormatsProperty = NULL;

    for(int i = 0; i < writebackConnector->count_props; i++) {
        drmModePropertyPtr property = drmModeGetProperty(fd, writebackConnector->props[i]);
        if(strcmp(property->name, "WRITEBACK_OUT_FENCE_PTR") == 0) writebackOutFenceProperty = property;
        else if(strcmp(property->name, "WRITEBACK_FB_ID") == 0) writebackFbIdProperty = property;
        else if(strcmp(property->name, "WRITEBACK_PIXEL_FORMATS") == 0) writebackPixelFormatsProperty = property;
        else if(strcmp(property->name, "CRTC_ID") == 0) writebackCrtcIdProperty = property;
        else drmModeFreeProperty(property);
    }

    if(writebackCrtcIdProperty == NULL) {
        fprintf(stderr, "CRTC_ID property not found on writeback connector.\n");
        goto error;
    }

    if(writebackFbIdProperty == NULL) {
        fprintf(stderr, "WRITEBACK_FB_ID property not found on writeback connector.\n");
        goto error;
    }

    //look for an active crtc.
    drmModeCrtcPtr firstActiveCrtc = NULL;
    for(int i = 0; i < res->count_crtcs; i++) {
        drmModeCrtcPtr crtc = drmModeGetCrtc(fd, res->crtcs[i]);
        drmModeObjectPropertiesPtr properties = drmModeObjectGetProperties(fd, crtc->crtc_id, DRM_MODE_OBJECT_CRTC);
        for(int j = 0; j < properties->count_props; j++) {
            drmModePropertyPtr property = drmModeGetProperty(fd, properties->props[j]);
            if(strcmp(property->name, "ACTIVE") == 0 && properties->prop_values[j] == 1) {
                printf("Found active crtc (id=%d)\n", crtc->crtc_id);
                firstActiveCrtc = crtc;
                break;
            }
            drmModeFreeProperty(property);
        }
        drmModeFreeObjectProperties(properties);
        if(!firstActiveCrtc) drmModeFreeCrtc(crtc);
    }

    if(!firstActiveCrtc) {
        fprintf(stderr, "No Active crtc found.\n");
        goto error;
    }

    //attach writeback connector to first active crtc
    ret = drmModeConnectorSetProperty(fd, writebackConnector->connector_id, writebackCrtcIdProperty->prop_id, (uint64_t)firstActiveCrtc->crtc_id);
    if(ret != 0) {
        fprintf(stderr, "Could not connect writeback connector (id=%d) to crtc (id=%d) (error=%d).\n",
                writebackConnector->connector_id,
                firstActiveCrtc->crtc_id,
                ret);
        goto error;
    }

    printf("Attached writeback connector (id=%d) to crtc (id=%d)\n", writebackConnector->connector_id, firstActiveCrtc->crtc_id);

    ret = drmModeConnectorSetProperty(fd, writebackConnector->connector_id, writebackFbIdProperty->prop_id, output_frame_buffer_id);
    if(ret != 0) {
        fprintf(stderr,"Could not connect output framebuffer (id=%d) to writeback connector (id=%d) (error=%d).\n",
                output_frame_buffer_id,
                writebackConnector->connector_id,
                ret);
        goto error;
    }

    printf("Connected output framebuffer to writeback connector.\n");
error:
    close(fd);
    return 0;
}
Output:

Code: Select all

drm version 0.0.0 driver=vc4 date=20140616 desc=Broadcom VC4 graphics
drm lib version 1.3.0 driver= date= desc=
busid=
Writeback Connector found (id=54):
Allocated dumb buffer on GPU:  1920x1080@32
drmModeAddFB: added output framebuffer (id=205).
Framebuffer properties:
   - width=1920
   - width=1080
   - depth=24
   - bpp=32
   - handle=2
   - fb_id=205

Found active crtc (id=82)
Could not connect writeback connector (id=54) to crtc (id=82) (error=-22).

randomher0
Posts: 24
Joined: Wed Apr 21, 2021 9:54 pm

Re: Fast method to pass data from framebuffer to MMAL

Sun May 02, 2021 7:48 pm

6by9 wrote:
Wed Apr 28, 2021 8:54 am
Yes it's texture format. You can import it back into EGL happily, but I suspect that you'll just be taking rendering resource away from X rendering the desktop.
This does not seem to be the case: I can import the screen at good framerate into opengl (https://imgur.com/a/NRpR7UT).
I'll try to write this to another dmabuf in linear format - if that is possible.
I hope I'll be more lucky with this approach than with writeback.. no progress there.

randomher0
Posts: 24
Joined: Wed Apr 21, 2021 9:54 pm

Re: Fast method to pass data from framebuffer to MMAL

Tue May 04, 2021 8:21 pm

I've made some progress with exporting the captured desktop image, however I've hit another roadblock:

The steps are as follows:

1. import to GL via glEGLImageTargetTexture2DOES
2a. render to GL texture and export via eglExportDMABUFImageMESA
2b. render to GL renderbuffer and export via eglExportDMABUFImageMESA
3. import to mmal via vcsm_import_dmabuf and feed to h264 encoder via zero_copy

While 2a) works - the problem is the exported texture also uses T-Format again, so there might be a chance converting it via a fragment shader. However, I think once rendered to a framebuffer the frame is in linear format anyway - so I guess this would be the direct/faster way to do it.

Update: managed to get the 2b running - however the GL renderbuffer is also in T-Format. Is this even possible!?

randomher0
Posts: 24
Joined: Wed Apr 21, 2021 9:54 pm

Re: Fast method to pass data from framebuffer to MMAL

Wed May 05, 2021 5:12 pm

I've managed to get X11 remote desktop working with the kernel changes proposed by @6by9.
It's only working with a 16 bit framebuffer and some custom X11 configuration - but the results are quite good (at least better than than any other vnc solutions I have tried so far) with 45fps and pretty low lag. (see demo here)

Code: Select all

Section "Device"
    Identifier      "VideoCore4-HDMI"
    Driver          "modesetting"
    Option "PageFlip" "false"
    Option "AccelMethod" "none"
    Screen          0
EndSection

Section "Monitor"
    Identifier      "HDMI-1"
    Option          "Primary" "true"
EndSection

Section "Screen"
    Identifier      "HDMI Screen"
    Device          "VideoCore4-HDMI"
    Monitor         "HDMI-1"
    DefaultDepth    16
    #DefaultFbBpp    24
        SubSection "Display"
            Depth    16
            #FbBpp   24
            Modes   "1920x1080"
        EndSubSection
EndSection

Section "ServerLayout"
    Identifier      "SeparateScreens"
    Screen 0        "HDMI Screen"
    #    Screen 0        "DPI Screen"
#    Screen 1        "HDMI Screen" RightOf "DPI Screen"
EndSection

Section "ServerFlags"
    # These options disable screen blanking, which is important for embedded systems
    Option "BlankTime"    "0"
    Option "StandbyTime"  "0"
    Option "SuspendTime"  "0"
    Option "OffTime"      "0"
EndSection
I'd really like to develop this into an app that anyone can use - however right now kernel patch is required and HDMI device must be attached for it to work. I've found this github issue stating that UIF format causes problems when being advertised format by the driver - this sounds like the analogous problem for Pi4.

cleverca22
Posts: 3744
Joined: Sat Aug 18, 2012 2:33 pm

Re: Fast method to pass data from framebuffer to MMAL

Wed May 05, 2021 5:21 pm

randomher0 wrote:
Wed May 05, 2021 5:12 pm
I'd really like to develop this into an app that anyone can use - however right now kernel patch is required and HDMI device must be attached for it to work.
you might be interested in how vnc does it in several different ways

1: x0vncserver, it essentially just abuses the crap out of the screenshot api, to capture the entire screen
2: Xvnc, a fork? of Xorg, which swapped out the gpu drivers, it doesnt connect to any gpu, renders directly to ram, and then shares that out
3: lib/xorg/modules/extensions/libvnc.so (from tigervnc), an xorg plugin (add it to xorg.conf), and it can hook into the rendering routines, to know when the screen is updating

using a method like 2 or 3, you can probably get things working pretty easily, and if the opengl accel is gone, you dont have to deal with t-format anymore

randomher0
Posts: 24
Joined: Wed Apr 21, 2021 9:54 pm

Re: Fast method to pass data from framebuffer to MMAL

Wed May 05, 2021 6:18 pm

Hi, thanks for the pointers.
As soon as there is something going on on the screen all vnc solutions become slideshows and all the snap is gone.This is probably due to the compression they use (at best jpeg - all in software) and due to the the lack of a hardware accelerated framebuffer. Compare this to any recent game streaming service or even RDP for windows. The PI has all the hardware units to support a better experience for remote desktop than is currently available..

randomher0
Posts: 24
Joined: Wed Apr 21, 2021 9:54 pm

Re: Fast method to pass data from framebuffer to MMAL

Thu May 06, 2021 5:47 pm

Update: patched xf86-video modesetting that it uses drmModeAddFB2WithModifiers instead of drmModeAddFb. A modifier of type DRM_FORMAT_MOD_LINEAR can be passed there and all textures will be allocated as linear. One problem solved. :D

Return to “Graphics programming”