fd_
Posts: 59
Joined: Thu Oct 25, 2018 7:35 am

Questions regarding H264 NAL boundaries

Thu Jul 04, 2019 1:37 pm

I'm using OpenMAX for decoding an H264 stream (coming from an iOS AirPlay Mirroring connection) under Raspbian on a Raspberry Pi. I'm receiving data in packets, with (AFAICT) one NAL unit per packet. I'm then forwarding the individual NAL units to the OpenMAX H264 decoder.

In my current working solution, I'm not specifying any OMX_IndexParamBrcmDataUnit explicitly, so I believe the default of OMX_DataUnitArbitraryStreamSection is used. I'm manually adding a four-byte 0x00000001 start code to the beginning of every NAL unit I feed the decoder. In the case that the NAL unit doesn't fit into an OMX input buffer, I'm splitting the NAL into multiple buffers and set the OMX_BUFFER_FLAG_ENDOFNAL flag on the last one (I'm not setting any flag on NAL units that fit into a single buffer). This works, but I'm seeing a bit of latency that I'd like to reduce.

Having read through some more information about H264, I've come to the conclusion that by following the procedure described above, I'm using the NAL Byte Stream Format and let the decoder determine NAL boundaries, when in fact I should have all information needed to tell the decoder where the NAL boundaries are.

Now, I tried to adjust my program to use the NAL Transport Stream Format. I discovered the OMX_IndexParamBrcmDataUnit configuration that has a unit type I set to OMX_DataUnitCodedPicture and an encapsulation type I set to OMX_DataEncapsulationRtpPayload. From some threads on this forum, I found I needed to set the OMX_BUFFER_FLAG_ENDOFFRAME flag on buffers when using the OMX_DataUnitCodedPicture data unit, so I set that to all buffers that contained a whole NAL, and to all buffers that contained the last part of a NAL split into multiple buffers. I didn't know how exactly I should handle SPS and PPS NALs, so I just set OMX_BUFFER_FLAG_ENDOFNAL flag on those. I also didn't add the start codes I previously had added. However, I couldn't get the decoder (and attached renderer) to display any frames.

I figured my NALs probably contain only a part of a frame, so the OMX_BUFFER_FLAG_ENDOFFRAME flag is inappropriate for me to set. But still, there must be a way of getting rid of manually prepending the 0x00000001 start code to every NAL.

So, here are my questions:
1) What is the difference between prepending a NAL unit with the start code and setting the OMX_BUFFER_FLAG_ENDOFNAL flag on a buffer?
2) Is there a way to let the decoder figure out frame boundaries, but explicitly tell it NAL boundaries? I imagine that must improve latency a bit at least if my stream doesn't tell me frame boundaries.
3) What is the difference between the OMX_PARAM_DATAUNITTYPE's unit type and encapsulation type?

Any help is highly appreciated!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7140
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Questions regarding H264 NAL boundaries

Thu Jul 04, 2019 2:31 pm

The most common use case is OMX_NaluFormatStartCodes where you have start codes and emulation prevention, and adding OMX_BUFFER_FLAG_ENDOFFRAME is used as the kick to tell the bitstream parser that it should have a whole frame to process (athough it'll still be happy should there not be a complete frame).

The other option is OMX_NaluFormatOneNaluPerBuffer where you have implicitly delineated NAL units (controlled only by input flags) without emulation prevention. By the looks of it, it still uses OMX_BUFFERFLAG_ENDOFFRAME to flag NAL boundaries. OMX_BUFFERFLAG_ENDOFNAL is only used by video_encode.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

fd_
Posts: 59
Joined: Thu Oct 25, 2018 7:35 am

Re: Questions regarding H264 NAL boundaries

Fri Jul 05, 2019 6:39 pm

6by9 wrote:
Thu Jul 04, 2019 2:31 pm
The most common use case is OMX_NaluFormatStartCodes where you have start codes and emulation prevention, and adding OMX_BUFFER_FLAG_ENDOFFRAME is used as the kick to tell the bitstream parser that it should have a whole frame to process (athough it'll still be happy should there not be a complete frame).

The other option is OMX_NaluFormatOneNaluPerBuffer where you have implicitly delineated NAL units (controlled only by input flags) without emulation prevention. By the looks of it, it still uses OMX_BUFFERFLAG_ENDOFFRAME to flag NAL boundaries. OMX_BUFFERFLAG_ENDOFNAL is only used by video_encode.
Thanks for that hint! I hadn't come across the OMX_IndexParamNalStreamFormatSelect parameter before, but I now found some info about it at http://home.nouwen.name/RaspberryPi/doc ... ecode.html. Will try it and report back how it works!

fd_
Posts: 59
Joined: Thu Oct 25, 2018 7:35 am

Re: Questions regarding H264 NAL boundaries

Mon Jul 08, 2019 2:36 pm

Just a small update: I found myself using an open source library for injecting max_dec_frame_buffering into the SPS NAL, and it's much more convenient to use with the byte stream format, so I won't further experiment on that domain. I guess there isn't too much of performance that can be won here anyway.
BTW, thanks for the hint about OMX_BUFFER_FLAG_ENDOFNAL only being relevant for encoding!

Return to “OpenMAX”