User avatar
gagle
Posts: 82
Joined: Fri Feb 14, 2014 6:54 pm
Contact: Website

[OpenMAX] H264 settings and DRC

Sun Jul 20, 2014 5:40 pm

I'm trying to expose all the h264 settings found in raspivid but I don't find the openmax struct or the field of some of them. These are the h264 settings that I've managed to expose:
  • IDR frame period (intra): OMX_VIDEO_CONFIG_AVCINTRAPERIOD - OMX_IndexConfigVideoAVCIntraPeriod
  • SEI frames (not in raspivid?): OMX_PARAM_BRCMVIDEOAVCSEIENABLETYPE - OMX_IndexParamBrcmVideoAVCSEIEnable
  • EEDE (end to end distortion estimator, not in raspivid?): OMX_VIDEO_EEDE_ENABLE - OMX_IndexParamBrcmEEDEEnable
    I'm not sure if this setting is implemented, I don't know how to test it, it just doesn't throw any error.
And these are the remaining settings:
  • quantisationParameter (variable bitrate)
  • profile
  • inline
-------

And DRC? This seems a common setting between raspistill and raspivid.

------

The main problem I always have is that I don't find the structs/fields that I need...

Any advice?

ethanol100
Posts: 651
Joined: Wed Oct 02, 2013 12:28 pm

Re: [OpenMAX] H264 settings and DRC

Sun Jul 20, 2014 8:00 pm

Just guessing, never looked at openmax before ;)

quantisationParameter: OMX_VIDEO_PARAM_QUANTIZATIONTYPE
which has the quantization properties for I and P frames: nQpI and nQpP

profile: OMX_VIDEO_MPEG4PROFILETYPE

inline: OMX_BUFFERFLAG_CODECCONFIG

And one question to SEI, what additional data do you want to add to the stream. SEI are some kind of "metadata" which are not needed to decode the stream. Mainly user supplied custom fields and information for i.e. 3d streams.

User avatar
gagle
Posts: 82
Joined: Fri Feb 14, 2014 6:54 pm
Contact: Website

Re: [OpenMAX] H264 settings and DRC

Mon Jul 21, 2014 8:15 am

ethanol100 wrote:Just guessing, never looked at openmax before ;)

quantisationParameter: OMX_VIDEO_PARAM_QUANTIZATIONTYPE
which has the quantization properties for I and P frames: nQpI and nQpP

profile: OMX_VIDEO_MPEG4PROFILETYPE

inline: OMX_BUFFERFLAG_CODECCONFIG

And one question to SEI, what additional data do you want to add to the stream. SEI are some kind of "metadata" which are not needed to decode the stream. Mainly user supplied custom fields and information for i.e. 3d streams.
Ok thanks!, I'll take a look in the afternoon. :D

what additional data do you want to add to the stream

Don't know. I've read that SEI messages carry metadata information but I need to investigate this more. For now I simply expose all the possible settings.

User avatar
gagle
Posts: 82
Joined: Fri Feb 14, 2014 6:54 pm
Contact: Website

Re: [OpenMAX] H264 settings and DRC

Sun Jul 27, 2014 5:29 pm

I've been doing some tests with the quantization parameter and I have some questions:

1. The field nQpB doesn't allow any modification, it returns an error. Why? Just curious.
2. What are the maximum values of nQpI and nQpP? From my tests, when nQpP is higher than 50, the video freezes. I'm not sure if this issue it's related with the h264 encoder or the matroska tools that I'm using to wrap the h264 file in a mkv file. nQpI seems to not have any upper limit.
3. What is the meaning of the 0 value? Disabled?
4. raspivid has one QP parameter with a range 10-40, but OpenMAX has two configurable fields. If this can be answered, how does mmal do to simplify 2 parameters in 1? It is just ignoring the nQpI field?

Note: nQpP = 1 (and whatever value for nQpI, including 0) produces a VERY high quality video, 2 seconds at 640x480 and 30fps produce a 7MB file :o, when with regular quality they have 200KB, impressive.

ethanol100
Posts: 651
Joined: Wed Oct 02, 2013 12:28 pm

Re: [OpenMAX] H264 settings and DRC

Sun Jul 27, 2014 8:05 pm

Guessing again:
1. I think the encoder does not support B frames, only I and P are supported?!
2. all Qp should have the range from 0 to 51. They are defined as integer division, and therefore high values create small size videos, neglecting frequencies. 0-10 will create too high bitrates, and should create problems. and 41-51are heavily compressed. (This is explains the range in raspivid.)
3. 0 should correspond to no compression, because we do not filter any frequencies? But could be a special value?
4. I would guess both parameter for I and P are set to the value, additionally the "initial Qp" should be set to a similar value, but I have now idea how this is done in OpenMAX.

nQpP = 1 and whatever value for nQpI: there are 1 I and 59 P frames in a 2s video, the I frame is not so important.

ps. You can use http://sourceforge.net/projects/h264bitstream to analyse the h264 stream and look at what Qp the encoder creates.

User avatar
gagle
Posts: 82
Joined: Fri Feb 14, 2014 6:54 pm
Contact: Website

Re: [OpenMAX] H264 settings and DRC

Mon Jul 28, 2014 9:18 am

Thanks, ethanol100! :D

QP successfully exposed, but I've noticed one issue that is both in raspivid and in openmax.

Code: Select all

 /opt/vc/bin/raspivid -t 2000 -o video.h264 -b 0 -qp 10
This records a 2s video, 1920x1080 @30fps, but when I wrap the file in a mkv file the file duration is less than a second. This doesn't occur with lower resolutions.

If I set a qp = 1 in openmax, the encoder fails, it hangs up and doesn't return more buffers. I need to kill the app. This doesn't occur with a lower resolution like 640x480. Perhaps the rpi doesn't support high rates of data transmission from the camera.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27062
Joined: Sat Jul 30, 2011 7:41 pm

Re: [OpenMAX] H264 settings and DRC

Mon Jul 28, 2014 9:59 am

H264 files do not contain fps information, so when you wrap in mkv, you will need to specify what the fps used to record was.

The camera and encoder support 1080p30 up to about 30Mbits/s IIRC, but tbh, 25 should be easily good enough and you could go as low at 15-18MBits and still get very good image quality.

We do not support B frames.

I would expect QP=1 to simply overload the data transfer from GPU to ARM at 1080p30 - that a hell of a lot of less compressed stuff going across the link.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
gagle
Posts: 82
Joined: Fri Feb 14, 2014 6:54 pm
Contact: Website

Re: [OpenMAX] H264 settings and DRC

Mon Jul 28, 2014 11:40 am

Yes, I'm already setting the fps when using the mkv tools.

AVC profile exposed successfully. The struct was OMX_VIDEO_PARAM_AVCTYPE and OMX_VIDEO_AVCPROFILETYPE. A lot of parameters can be configured, but for now I'll simply expose the profile, like in raspivid.

User avatar
gagle
Posts: 82
Joined: Fri Feb 14, 2014 6:54 pm
Contact: Website

Re: [OpenMAX] H264 settings and DRC

Mon Jul 28, 2014 5:20 pm

Inline SPS/PPS

Certainly the buffer flag OMX_BUFFERFLAG_CODECCONFIG indicates that the buffer is a SPS or PPS frame, but... it's just an informative flag.
typedef struct OMX_PARAM_CODECCONFIGTYPE {
OMX_U32 nSize;
OMX_VERSIONTYPE nVersion;
OMX_U32 nPortIndex;
OMX_U32 bCodecConfigIsComplete;
OMX_U8 nData[1];
} OMX_PARAM_CODECCONFIGTYPE;

/*
This parameter contains opaque data in a format specified by Broadcom
and allows out-of-band information such as cropping rectangles, aspect
ratio information, codec-specific header bytes, and other essential
information to be passed between connected components.

\code{bCodecConfigIsCompete} specifies if the codec config is fully
contained in here and there is no need to wait for OMX_BUFFERFLAG_CODECCONFIG
buffers
*/
I suspect that this parameter has something to do with the SPS/PPS... but Broadcom doesn't specify anything, it's "opaque" data. I only know that I can call GetParameter() and it returns a non-null nData array. Any hints?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27062
Joined: Sat Jul 30, 2011 7:41 pm

Re: [OpenMAX] H264 settings and DRC

Mon Jul 28, 2014 6:46 pm

Check the Raspcam code (cannot remember if SPS supported), then track the MMAL call and see what it turns into in OMX space.

Assuming that's not done inside the GPU of course. Cannot remember where the translation happens...
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

ethanol100
Posts: 651
Joined: Wed Oct 02, 2013 12:28 pm

Re: [OpenMAX] H264 settings and DRC

Mon Jul 28, 2014 7:26 pm

RaspiVid sets the mmal parameter MMAL_PARAMETER_VIDEO_ENCODE_INLINE_HEADER to true,
again guessing:
In OMX_Index.h the OMX_INDEXTYPE contains:
OMX_IndexParamBrcmVideoAVCInlineHeaderEnable, /**< reference: OMX_CONFIG_PORTBOOLEANTYPE*/

so in principle using "OMX_SetConfig" to set this bool to true for the encoder component?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9569
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: [OpenMAX] H264 settings and DRC

Mon Jul 28, 2014 7:50 pm

jamesh wrote:Check the Raspcam code (cannot remember if SPS supported), then track the MMAL call and see what it turns into in OMX space.

Assuming that's not done inside the GPU of course. Cannot remember where the translation happens...
Translation is done on the GPU.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27062
Joined: Sat Jul 30, 2011 7:41 pm

Re: [OpenMAX] H264 settings and DRC

Mon Jul 28, 2014 7:57 pm

6by9 wrote:
jamesh wrote:Check the Raspcam code (cannot remember if SPS supported), then track the MMAL call and see what it turns into in OMX space.

Assuming that's not done inside the GPU of course. Cannot remember where the translation happens...
Translation is done on the GPU.
I had my suspicions...that my memory sucked.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9569
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: [OpenMAX] H264 settings and DRC

Mon Jul 28, 2014 8:08 pm

gagle wrote:Inline SPS/PPS

Certainly the buffer flag OMX_BUFFERFLAG_CODECCONFIG indicates that the buffer is a SPS or PPS frame, but... it's just an informative flag.
If you can feed it back into the decoder then it makes life easier.
gagle wrote:
typedef struct OMX_PARAM_CODECCONFIGTYPE {
OMX_U32 nSize;
OMX_VERSIONTYPE nVersion;
OMX_U32 nPortIndex;
OMX_U32 bCodecConfigIsComplete;
OMX_U8 nData[1];
} OMX_PARAM_CODECCONFIGTYPE;

/*
This parameter contains opaque data in a format specified by Broadcom
and allows out-of-band information such as cropping rectangles, aspect
ratio information, codec-specific header bytes, and other essential
information to be passed between connected components.

\code{bCodecConfigIsCompete} specifies if the codec config is fully
contained in here and there is no need to wait for OMX_BUFFERFLAG_CODECCONFIG
buffers
*/
I suspect that this parameter has something to do with the SPS/PPS... but Broadcom doesn't specify anything, it's "opaque" data. I only know that I can call GetParameter() and it returns a non-null nData array. Any hints?
Ignore it. There are some benefits in being able to feed codec header bytes into the the decoder before providing any buffers, particularly if you're going to be seeking within a stream. Unless you know what you're doing, then don't touch.

Boolean to enable Inline headers via MMAL_PARAMETER_VIDEO_ENCODE_INLINE_HEADER / OMX_IndexParamBrcmVideoAVCInlineHeaderEnable
If I remember correctly, it will then insert both an SPS and PPS before every I-frame, so the GOP dictates how frequently that is, and it makes it easy to split into multiple files at any I-frame if you take the header bytes too.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
gagle
Posts: 82
Joined: Fri Feb 14, 2014 6:54 pm
Contact: Website

Re: [OpenMAX] H264 settings and DRC

Tue Jul 29, 2014 10:06 am

OMX_IndexParamBrcmVideoAVCInlineHeaderEnable
Omg... how embarrassing... I need a second pair of glasses. :oops:

Thanks to all of you, guys ;) . SPS/PPS exposed successfully...
it makes it easy to split into multiple files at any I-frame if you take the header bytes too.
So true. Without SPS/PPS for each IDR frame, a client cannot record a random sequence of frames from an h264 stream (based on my tests, I don't know practically anything about the h264 encoding). http://www.cardinalpeak.com/blog/the-h- ... meter-set/
Both entities contain information that an h.264 decoder needs to decode the video data, for example the resolution and frame rate of the video.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27062
Joined: Sat Jul 30, 2011 7:41 pm

Re: [OpenMAX] H264 settings and DRC

Tue Jul 29, 2014 11:07 am

gagle wrote:
OMX_IndexParamBrcmVideoAVCInlineHeaderEnable
Omg... how embarrassing... I need a second pair of glasses. :oops:

Thanks to all of you, guys ;) . SPS/PPS exposed successfully...
it makes it easy to split into multiple files at any I-frame if you take the header bytes too.
So true. Without SPS/PPS for each IDR frame, a client cannot record a random sequence of frames from an h264 stream (based on my tests, I don't know practically anything about the h264 encoding). http://www.cardinalpeak.com/blog/the-h- ... meter-set/
Both entities contain information that an h.264 decoder needs to decode the video data, for example the resolution and frame rate of the video.
There is an overhead for having this per frame, IIRC 20 bytes or so. Can add up over the length of a decent video.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9569
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: [OpenMAX] H264 settings and DRC

Tue Jul 29, 2014 12:06 pm

jamesh wrote:There is an overhead for having this per frame, IIRC 20 bytes or so. Can add up over the length of a decent video.
Are you sure? Yes, PPS/SPS is about 20bytes, but I could have sworn it was only on the I-frames which can be quite a way apart (default of 60 frames). Overall not such a big overhead.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

ethanol100
Posts: 651
Joined: Wed Oct 02, 2013 12:28 pm

Re: [OpenMAX] H264 settings and DRC

Tue Jul 29, 2014 1:04 pm

Yes this will add 20 bytes each 2 seconds, only direct in front of an I-frame.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27062
Joined: Sat Jul 30, 2011 7:41 pm

Re: [OpenMAX] H264 settings and DRC

Wed Jul 30, 2014 8:05 am

6by9 wrote:
jamesh wrote:There is an overhead for having this per frame, IIRC 20 bytes or so. Can add up over the length of a decent video.
Are you sure? Yes, PPS/SPS is about 20bytes, but I could have sworn it was only on the I-frames which can be quite a way apart (default of 60 frames). Overall not such a big overhead.
Sorry, you are right. They are added every i-frame not frame, so depending on the iframe interval, it can add up over a long recording period.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
gagle
Posts: 82
Joined: Fri Feb 14, 2014 6:54 pm
Contact: Website

Re: [OpenMAX] H264 settings and DRC

Fri Aug 01, 2014 9:33 am

DRC only works in still mode.

OMX_IndexConfigDynamicRangeExpansion - OMX_CONFIG_DYNAMICRANGEEXPANSIONTYPE

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9569
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: [OpenMAX] H264 settings and DRC

Fri Aug 01, 2014 11:01 am

gagle wrote:DRC only works in still mode.

OMX_IndexConfigDynamicRangeExpansion - OMX_CONFIG_DYNAMICRANGEEXPANSIONTYPE
It should work on both stills and preview, but not video encode (IL port 71 tunneled to video_encode, or MMAL output[1] set with MMAL_ENCODING_OPAQUE). Video_encode uses a different image layout to make life easier for the codecs, and the DRC algo doesn't support that.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”