Applicable_Robotics
Posts: 34
Joined: Sat Jun 28, 2014 11:55 pm

OpenCV decode H264 frame-by-frame

Mon Jul 28, 2014 10:00 pm

Hi,
I am attempting to overlay text on a video feed from the Raspberry Pi Camera using OpenCV
Currently I have a feed working (without overlay) using the following commands:

Code: Select all

Send:  raspivid -n -t 0 -w 640 -h 480 -b 1000000 -fps 30 -o - | nc [insert destination IP address here] 1500
Recv:  nc -l -p 1500 | mplayer -fps 45 -cache 1024 -
I would like to replace the mplayer command with a C++ program using OpenCV to decode, modify and display the feed. If OpenCV is unable to decode H264, then any solution that creates an OpenCV Mat is acceptable.
Does anyone know a method for decoding a h264 frame in C/C++?
Thanks.

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: OpenCV decode H264 frame-by-frame

Mon Aug 04, 2014 8:15 pm

You can decode h264 using libav.

Initialize the lib with something like this:

Code: Select all

	av_register_all();
	avcodec_register_all();
	avformat_network_init();
	m_ffmpeg.codec = avcodec_find_decoder(AV_CODEC_ID_H264);
	if (!m_ffmpeg.codec) 
	{
		fprintf(stderr, "Codec not found\n");
		exit(1);
	}
	m_ffmpeg.context = avcodec_alloc_context3(m_ffmpeg.codec);
	if (!m_ffmpeg.context)
	{
		fprintf(stderr, "Could not allocate video codec context\n");
		exit(1);
	}
	avcodec_get_context_defaults3(m_ffmpeg.context, m_ffmpeg.codec);
	m_ffmpeg.context->flags |= CODEC_FLAG_LOW_DELAY;
	m_ffmpeg.context->flags2 |= CODEC_FLAG2_CHUNKS;
	m_ffmpeg.context->thread_count = 4;
 	m_ffmpeg.context->thread_type = FF_THREAD_SLICE;
	m_ffmpeg.context->strict_std_compliance = FF_COMPLIANCE_EXPERIMENTAL;
	if (avcodec_open2(m_ffmpeg.context, m_ffmpeg.codec, nullptr) < 0)
	{
		fprintf(stderr, "Could not open codec\n");
		exit(1);
	}

	m_ffmpeg.frame_yuv = avcodec_alloc_frame();
	if (!m_ffmpeg.frame_yuv) 
	{
		fprintf(stderr, "Could not allocate video frame\n");
		exit(1);
	}
Then, whenever you have data for one frame do this:

Code: Select all

        AVPacket packet;
        av_init_packet(&packet);
        packet.pts = AV_NOPTS_VALUE;
        packet.dts = AV_NOPTS_VALUE;
        packet.data = m_data.data();//your frame data
        packet.size = m_data.size();//your frame data size
        int got_frame = 0;
        int len = avcodec_decode_video2(m_ffmpeg.context, m_ffmpeg.frame_yuv, &got_frame, &packet);
        if (len >= 0 && got_frame)
        {
                //do things with the data, like converting it to RGB, resizing it etc.
        }
You can use sws_getCachedContext together with sws_scale to scale and convert to RGB.

RpiName
Posts: 714
Joined: Sat Jul 06, 2013 3:14 am

Re: OpenCV decode H264 frame-by-frame

Mon Aug 04, 2014 9:38 pm

For full fps, live text overlay over video from the rpi have a look at uv4l at http://linux-projects.org

wrice127
Posts: 2
Joined: Tue Aug 12, 2014 6:46 pm
Location: California
Contact: Website

Re: OpenCV decode H264 frame-by-frame

Wed Aug 13, 2014 6:32 am

jeanleflambeur wrote:You can decode h264 using libav.

Initialize the lib with something like this:

Code: Select all

	av_register_all();
	avcodec_register_all();
	avformat_network_init();
	m_ffmpeg.codec = avcodec_find_decoder(AV_CODEC_ID_H264);
	if (!m_ffmpeg.codec) 
	{
		fprintf(stderr, "Codec not found\n");
		exit(1);
	}
	m_ffmpeg.context = avcodec_alloc_context3(m_ffmpeg.codec);
	if (!m_ffmpeg.context)
	{
		fprintf(stderr, "Could not allocate video codec context\n");
		exit(1);
	}
	avcodec_get_context_defaults3(m_ffmpeg.context, m_ffmpeg.codec);
	m_ffmpeg.context->flags |= CODEC_FLAG_LOW_DELAY;
	m_ffmpeg.context->flags2 |= CODEC_FLAG2_CHUNKS;
	m_ffmpeg.context->thread_count = 4;
 	m_ffmpeg.context->thread_type = FF_THREAD_SLICE;
	m_ffmpeg.context->strict_std_compliance = FF_COMPLIANCE_EXPERIMENTAL;
	if (avcodec_open2(m_ffmpeg.context, m_ffmpeg.codec, nullptr) < 0)
	{
		fprintf(stderr, "Could not open codec\n");
		exit(1);
	}

	m_ffmpeg.frame_yuv = avcodec_alloc_frame();
	if (!m_ffmpeg.frame_yuv) 
	{
		fprintf(stderr, "Could not allocate video frame\n");
		exit(1);
	}
Then, whenever you have data for one frame do this:

Code: Select all

        AVPacket packet;
        av_init_packet(&packet);
        packet.pts = AV_NOPTS_VALUE;
        packet.dts = AV_NOPTS_VALUE;
        packet.data = m_data.data();//your frame data
        packet.size = m_data.size();//your frame data size
        int got_frame = 0;
        int len = avcodec_decode_video2(m_ffmpeg.context, m_ffmpeg.frame_yuv, &got_frame, &packet);
        if (len >= 0 && got_frame)
        {
                //do things with the data, like converting it to RGB, resizing it etc.
        }
You can use sws_getCachedContext together with sws_scale to scale and convert to RGB.
This code snippet looks very interesting. Where can I learn more about libav??
Thanks for ur reply in advance.
http://raspberrypiprogramming.blogspot.com

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: OpenCV decode H264 frame-by-frame

Tue Aug 19, 2014 6:31 am

wrice127 wrote:This code snippet looks very interesting. Where can I learn more about libav??
Thanks for ur reply in advance.
Here are some links:
https://libav.org/documentation.html
http://blog.tomaka17.com/2012/03/libavc ... -tutorial/

The API is pretty well structured and you can find tons of examples if you google specific functions.
I use it to decode H264 video streamed by the raspi to my laptop.

nettercm
Posts: 8
Joined: Wed Jul 16, 2014 2:35 pm

Re: OpenCV decode H264 frame-by-frame

Tue Sep 02, 2014 7:18 pm

Dear jeanleflambeur - thanks for providing the code snippet. I will give this a try. I found code snippets elsewhere, mostly the examples provided with libav and got something like this to work.

However, I get more latency compared to a solution that uses netcat to receive the raw H264-ES stream + mplayer to render it.

I see that you are using flags such as CODEC_FLAG_LOW_DELAY. Are there any other flags or tricks that you can think to ensure that libav does not introduce any delay? Would AVFMT_FLAG_NOBUFFER come into play here at all, for example?

Thanks.

Return to “C/C++”