MMAL video playback into DRM planes.

[dividuum]

I'm currently working on a DRM/MMAL based video playback using the new Pi4 APIs. Essentially I'm trying to replicate (as a first step) what omxplayer does with dispmanx and OMX. I'm having a few difficulties understanding how that's all supposed to work.

With OMX it was pretty simple: Create a dispmanx layer, create a video_render and point that to this layer. The layer could then be dynamically placed on displays, different layers and the alpha/rotation/source/dest/etc could be updated.

With DRM it seems I now have to create a DRM backed FB object (using drmModeAddFB2), feed that into the ril component. Once that produces a new output video frame I then use drmModeSetPlane to place that. I'm not sure how that then decides which dispmanx layer gets used. I observe that calling drmModeSetPlane the first time creates a new dispmanx layer on layer 1. Is there any way to change which layer is used using DRM? Maybe using DRM properties but haven't been able to find any documentation on how that's supposed to work from userland.

Similarly I'm not sure how rotating a video works now. This code seems to use the MMAL_DISPLAYREGION_T parameter for that (which looks familar to how OMX worked), but I'm not sure if that's the correct approach and that doesn't seem DRM related at all as I then have to feed the decoded MMAL buffers into video_render instead. So how can I rotate videos, set their alpha and other properties?

How is screen tearing prevented if I use drmModeSetPlane? Or is that deferred until a drmModePageFlip is issued and thus magically solved?

It seems there is only a limited number of DRM planes available. So I have to be careful mapping multiple MMAL pipelines to one plane each. Is that correct? Previously I think I could create any number of dispmanx layers until the GPU melted. Now it seems there are 6 planes in total. Or is there a way to raise this limit?

I've tried to run two videos, got two different planes and now the second plane is on the secondary display for some reason despite specifying the same CRTC id in the two drmModeSetPlane calls. See the following output:

Code: Select all

root@raspberrypi:~# vcgencmd dispmanx_list
display:2 format:ARGB8888 transform:0 layer:-127 1920x1080 src:0,0,1920,1080 dst:0,0,1920,1080 cost:1156 lbm:0
display:2 format:YUV420 transform:0 layer:1 640x360 src:0,0,640,360 dst:0,0,960,540 cost:713 lbm:10240 
display:7 format:XRGB8888 transform:0 layer:-127 1920x1080 src:0,0,1920,1080 dst:0,0,1920,1080 cost:1156 lbm:0
display:7 format:YUV420 transform:0 layer:2 640x360 src:0,0,640,360 dst:960,540,960,540 cost:713 lbm:10240 

root@raspberrypi:~# cat /sys/kernel/debug/dri/1/state # slightly cut for brevity
plane[35]: plane-1
        crtc=crtc-0
        fb=117
                allocated by = ib-thpool
                refcount=2
                format=YU12 little-endian (0x32315559)
                modifier=0x0
                size=640x360
                ...
        crtc-pos=960x540+0+0
        src-pos=640.000000x360.000000+0.000000+0.000000
        rotation=1
        normalized-zpos=1
        color-encoding=ITU-R BT.601 YCbCr
        color-range=YCbCr limited range
<snip>  
plane[59]: plane-4
        crtc=crtc-0
        fb=125
                allocated by = ib-thpool
                refcount=2
                format=YU12 little-endian (0x32315559)
                modifier=0x0
                size=640x360
                ...
        crtc-pos=960x540+960+540
        src-pos=640.000000x360.000000+0.000000+0.000000
        rotation=1
        normalized-zpos=2
        color-encoding=ITU-R BT.601 YCbCr
        color-range=YCbCr limited range

plane 35 maps to the second line in the dispmanx_list output while plane 59 maps to the fourth line. Notice that the crtc is crtc-0 in both cases, yet the display is 2 / 7 in dispmanx, which is also what I see on my screens.

[dividuum]

Anything? Anyone?

I was able to figure out how to rotate a DRM plane (although only the rotate-180 bitmask value seem to work) and found the "zpos" property. Looking at the output of modeset (below) and my own output doesn't mark the "zpos" plane property as immutable. I tried setting various values in the range from 0 to 127 and none seem to have any effect when I look at `vcgencmd dispmanx_list`. Is that value ignored? I haven't been able to freely arrange the DRM planes and I get a feeling with its three plane types (Overlay/Primary/Cursor) seems to only provide the minimal features required for something like Kodi. Will it ever be possible to do more using DRM?

Code: Select all

        34 zpos:
                flags: range
                values: 0 127
                value: 0

Similarly, the "rotation" property advertises 4 bitmask values. Again the modeset output (which is confirmed by my own code):

Code: Select all

        31 rotation:
                flags: bitmask
                values: rotate-0=0x1 rotate-180=0x4 reflect-x=0x10 reflect-y=0x20
                value: 1

Only `rotate-180` seems to work. I've not been able to use any of the reflect bits. For example setting 0x10 returns errno=Invalid argument. I also don't see any way of rotating a plane by 90 degree. How can I rotate a video by 90/270 degree, similar to the MMAL_DISPLAY_ROT90 in a video_render component? Or is that not possible through DRM?

[dividuum]

Continuing the journey speaking to myself: The rotation seems to be indeed limited to a few options. It it looks like it's impossible to rotate a video by 90/270 degree right now. Is that something that will be supported in the future? Or is the `video_render` component the only way to achieve this?

[6by9]

It's holiday season in the UK, therefore expect slightly lower levels of support from employees at Pi Towers. I'm back full time as of next Monday, and I'm not sure when James is back.

Correct, FKMS is not exposing transpose as it has to be done via a two pass render using the transposer hardware block. It requires an extra buffer for the transposed element, and the transposer is limited to 1080p when transposing. It also falls over when it comes to updating the elements - DRM is expecting the buffer to be "live", therefore anything written into it should be displayed. In reality you have to kick the transposer to retranspose the element into the hidden "live" buffer.
H and V flips, and rot180 (which is both H&V) can all be done "live" as the display is being composed.
The same restriction is true with the Full KMS driver on Pi 0-3, mainly as you can't then hide buffer allocations for the transpose buffer. Rotation is expected to be done through GL.

The number of planes available is mainly down to the number created for the display
https://github.com/raspberrypi/linux/bl ... _kms.c#L40
https://github.com/raspberrypi/linux/bl ... ms.c#L1684
It looks like I did think about adding multiple overlay planes per display, but missed the for loop around vc4_fkms_plane_init(drm, DRM_PLANE_TYPE_OVERLAY,...)

The firmware is currently configured for 16 elements over all the displays. Again it's only a define so could be increased easily if there were a need. Everything has to start somewhere, and having huge numbers of planes makes the output from modetest nearly unreadable.

The Full KMS driver has an arbitrary 8 overlay layers
https://github.com/raspberrypi/linux/bl ... tc.c#L1201

I didn't think you could change the crtc binding of planes randomly. The code I've written is certainly not expecting it to change. It'd take a bit of digging to work out why/how that is being allowed to happen, as it shouldn't be.

zpos does control the layer used within the HVS config. The DRM core believes it is the absolute master, therefore it normalises the zpos values before composition as some hardware requires contiguous layer numbers (not the Pi as it happens). FKMS maps layer 0 to Dispmanx layer -127, but otherwise passes the numbers through as is. https://github.com/raspberrypi/linux/bl ... kms.c#L541

[dividuum]

Correct, FKMS is not exposing transpose as it has to be done via a two pass render using the transposer hardware block. It requires an extra buffer for the transposed element, and the transposer is limited to 1080p when transposing. It also falls over when it comes to updating the elements - DRM is expecting the buffer to be "live", therefore anything written into it should be displayed. In reality you have to kick the transposer to retranspose the element into the hidden "live" buffer.

Oh. Just to be sure: any 90/270 rotation, even on older Pis went through an extra buffer? I'm wondering because I never noticed any performance difference when rotating through the video_render component compared to what I observed when using display_rotate=1/3 in /boot/config.txt. I think the latter was noticeably slower. My cargo cult senses tell me that for a full screen 90 degree rotated video, the work done should be equivalent.

Rotation is expected to be done through GL.

That's possible in my program already, albeit a lot slower. Maybe I'm doing something wrong, but 1080p30 with OpenGL is at the limit of what I'm able to do. Should I expect more?

zpos does control the layer used within the HVS config. The DRM core believes it is the absolute master, therefore it normalises the zpos values before composition as some hardware requires contiguous layer numbers (not the Pi as it happens). FKMS maps layer 0 to Dispmanx layer -127, but otherwise passes the numbers through as is. https://github.com/raspberrypi/linux/bl ... kms.c#L541

Thanks for all the source code links. I guess I'll have to revisit and look at the vcgencmd dispmanx_list output again while playing with zpos.

[dom]

Oh. Just to be sure: any 90/270 rotation, even on older Pis went through an extra buffer? I'm wondering because I never noticed any performance difference when rotating through the video_render component compared to what I observed when using display_rotate=1/3 in /boot/config.txt. I think the latter was noticeably slower. My cargo cult senses tell me that for a full screen 90 degree rotated video, the work done should be equivalent.

rotating through video_render involves transposing a YUV420 buffer at 1.5 bytes/pixel.
rotating through display_rotate involves transposing an RGBA32 buffer at 4 bytes/pixel. Significantly more work.

If video is scaled up then the different in work needed would increase more.

[dividuum]

rotating through video_render involves transposing a YUV420 buffer at 1.5 bytes/pixel.
rotating through display_rotate involves transposing an RGBA32 buffer at 4 bytes/pixel. Significantly more work.

That explains it. Thanks.

[6by9]

Code: Select all

        31 rotation:
                flags: bitmask
                values: rotate-0=0x1 rotate-180=0x4 reflect-x=0x10 reflect-y=0x20
                value: 1

Only `rotate-180` seems to work. I've not been able to use any of the reflect bits. For example setting 0x10 returns errno=Invalid argument.

I'm looking at this for other reasons and had a bit of a head scratch.

The core DRM code is sanity checking the value - https://elixir.bootlin.com/linux/v4.19. ... mic.c#L903

The generic options are rotate 0/90/180/270, and then adding either reflect x and/or y. It's two independent bitmasks in one property.
A value of 16 (reflect-x) has not specified the rotate at all, therefore is invalid. The core code has thrown that out.

17 would be rotate-0 and reflect-x, or 33 for rotate-0 and reflect-y.

I believe the driver is misinterpreting the final result as isn't breaking it up either when passing it down to the firmware. It depends on exactly how the drm_rotation_simplify function ends up simplifying the transform.