roxlu
Posts: 4
Joined: Wed May 09, 2018 8:17 pm

[solved] video_decode: how to get frame number of decoded or rendered frame

Wed May 09, 2018 8:26 pm

Hi,

When I have a h264 bitstream that I feed into the video_decode component, is there a way to get notified when a frame has been decoded or is either ready for display or being rendered (with the video_render component) already?

One solution that might work is to inspect the bitstream and detect access unit starts/ends and mark the buffer that I feed into the decoder and listen for the mark event. As this makes buffer management/parsing a bit complex (e.g. when filling a full complete buffer contains 2 access units I would need to split it into two separte buffers), I'm trying to find other possibilities.

Another solution could be to create my own component and attach it in a tunnel between the video_decode and video_render component. This is just a wild guess and I'm not sure if this might work.

Any other solutions maybe?

roxlu
Last edited by roxlu on Fri May 11, 2018 9:12 am, edited 1 time in total.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5203
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_decode: how to get frame number of decoded or rendered frame

Thu May 10, 2018 10:05 am

Mark events should work, but as you say you need to have parsed the bitstream first.

You can't create your own OMX components running on the VPU, therefore any tunnelling would pull the data back to the ARM. If you're doing that then it's as easy to do the buffer handling yourself off the "normal" buffer events. Any tunnelling via the ARM will disable the performance optimisations available if the tunnel remains on the VPU.

Personally I'd switch from IL to MMAL. With MMAL you can request buffers out of the decoder to be of type MMAL_ENCODING_OPAQUE which are small buffers passing references to VPU objects. Those can be brought back to the ARM, your app can look at the headers, and then sent back down to the VPU for further processing/rendering. The mmal_connection util functions almost do what you want, but the callback only passes the connection handle and not the buffer details (which have been pushed into a queue). Nothing stopping you copying and amending that functionality though. Most of the time I've used connections with the MMAL_CONNECTION_FLAG_TUNNELLING flag, which is a special one to tell MMAL to do all the callbacks on the VPU rather than the ARM. That minimises the latency, but means you can't get any callbacks on the ARM).
https://github.com/raspberrypi/userland ... _basic_2.c is a reasonable example of video decodeusing MMAL, although it only prints a message saying that it has decoded the frame rather than rendering it (see line 341).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

roxlu
Posts: 4
Joined: Wed May 09, 2018 8:17 pm

Re: video_decode: how to get frame number of decoded or rendered frame

Thu May 10, 2018 11:03 am

Thanks so much for your elaborate answer and suggestions.

Because I've spend qutie some time digging through the 330 paged IL spec I'm not yet ready to look into another spec :D. So I went on and implemented the idea I suggested in my post and that seems to work pretty well. I guess the only downside to this solution is that the decoder is not fully filled with data all the time. This probably means that some parts of the hardware accelerator are doing nothing while they could be doing some work (like parsing the bitstream).

Your MMAL solution seems to be a more appropriate solution... I'll dive into that at some point.

Thanks!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5203
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_decode: how to get frame number of decoded or rendered frame

Thu May 10, 2018 11:15 am

If you are passing in framed data then setting the OMX_BUFFERFLAG_ENDOFFRAME flag in nFlags also kicks the bitstream parser on the assumption that it should be able to guarantee that there is a frame to consume. I can't recall the heuristics if that flag isn't passed in, but I have seen it result in a couple of frames worth of encoded data not being processed for a while.

If you're running with multiple buffers on the input then you should be able to easily keep the hardware busy. As it is video_decode has a moderate sized FIFO at the front end, so I'd expect to see the input buffers being returned to you pretty rapidly for further filling.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

roxlu
Posts: 4
Joined: Wed May 09, 2018 8:17 pm

Re: video_decode: how to get frame number of decoded or rendered frame

Thu May 10, 2018 2:20 pm

Thanks! I'm indeed settings OMX_BUFFERFLAG_ENDOFFRAME. I have to create a proper setup to see if it makes a difference but I'm sure it does. Currently I'm looking into a proper way to shutdown everything, clear buffers, reset tunnels etc.

Return to “OpenMAX”

Who is online

Users browsing this forum: No registered users and 2 guests