Andy Armstrong
Posts: 57
Joined: Sat Dec 03, 2011 10:10 am

h264: possible to force SPS/PPS per GOP?

Fri May 17, 2013 3:55 pm

It turns out that raspivid nearly produces h264 in the right format for HLS streaming. If you have the right version of ffmpeg you can try something like this:

https://gist.github.com/AndyA/5599982

The only thing stopping it working is the lack of an SPS/PPS in each segment. Because HLS requires each segment to be individually playable they each need an SPS/PPS pair.

Looking at the various MMAL_PARAMETER_* ids in mmal_parameters_video.h some of them look quite promising - but many of them are only mentioned in the enum that declares them - I can't see any code that refers to them.

Does anyone happen to know if it's possible to persuade the GPU to stuff a SPS/PPS pair in at the start of each group-of-pictures? Failing that I can post-process the stream to add them - but it seems a shame if the encoding pipeline can do it.

Thanks.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23874
Joined: Sat Jul 30, 2011 7:41 pm

Re: h264: possible to force SPS/PPS per GOP?

Fri May 17, 2013 4:06 pm

There is some if'f out code in the RaspiVid app which sets rate control

Code: Select all

  // Set the rate control parameter
   if (0)
   {
      MMAL_PARAMETER_VIDEO_RATECONTROL_T param = {{ MMAL_PARAMETER_RATECONTROL, sizeof(param)}, MMAL_VIDEO_RATECONTROL_DEFAULT};
      status = mmal_port_parameter_set(encoder_output, &param.hdr);
      if (status != MMAL_SUCCESS)
      {
         vcos_log_error("Unable to set ratecontrol");
         goto error;
      }

   }
You could do something similar with the parameters in mmal_parameters_video.h to set up various H264 parameters.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

Andy Armstrong
Posts: 57
Joined: Sat Dec 03, 2011 10:10 am

Re: h264: possible to force SPS/PPS per GOP?

Fri May 17, 2013 4:31 pm

Thanks James.

I can see how to set the parameters. The problem is that the seem to be undocumented :)

Specifically I'm looking for an option that tells the encoder to emit SPS and PPS NALs in the h264 bit stream before each IDR frame - so that each group-of-pictures is self-describing.

I'm not even certain that such an option exists - just wondering if anyone knows.

Andy Armstrong
Posts: 57
Joined: Sat Dec 03, 2011 10:10 am

Re: h264: possible to force SPS/PPS per GOP?

Fri May 17, 2013 10:10 pm

I managed to work round the issue I was having and now have lovely HD HLS streaming served direct from a Wifi connected Model A.

I ended up writing a filter to add the SPS and PPS NALs to the h264 bit stream before every IDR NAL. The code is here:

https://github.com/AndyA/psips

And there's a script here:

https://github.com/AndyA/psips/blob/mas ... les/hls.sh

that you can use to stream HLS via Nginx on a Pi.

Unfortunately you need an up to date build of ffmpeg - I'm using 1.0.5. The version that ships with Raspbian doesn't have the segmented output format and HLS support.

Once it's daylight and I get some more video out of it I'll post a link.

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: h264: possible to force SPS/PPS per GOP?

Fri May 17, 2013 10:21 pm

[just noticed you replied again. but here’s my post anyway]

This is sweet! I was not aware of this type of streaming yet. Much easier to stream across the internet than UDP.

I’ve managed to play the stream using all these

Code: Select all

ffplay http://alarmpi.lan/live/stream.m3u8
vlc http://alarmpi.lan/live/stream.m3u8
gst-launch-1.0 playbin uri=http://alarmpi.lan/live/stream.m3u8
But these work only when started briefly after ffmpeg started chruning out segments.

If started too late ffplay then spams these lines

Code: Select all

[h264 @ 0x7f40e80108c0] non-existing PPS referenced
[h264 @ 0x7f40e80108c0] non-existing PPS 0 referenced
[h264 @ 0x7f40e80108c0] decode_slice_header error
[h264 @ 0x7f40e80108c0] no frame!
What I don’t understand is why I get a consistent delay of about 6 seconds. My segment size is 4 even with segment 8 it’s still 6 seconds delay.

I’ve amended your script slightly. Dunno why you force empty audio. I didn’t need it. Also, you can skip the fifo and pipe directly.

Code: Select all

#!/bin/bash
 
base="/data/live"

cd $base

raspivid -n -w 720 -h 405 -fps 25 -vf -t 86400000 -b 1800000 -o - \
| ffmpeg -y \
    -i - \
    -c:v copy \
    -map 0:0 \
    -f ssegment \
    -segment_time 4 \
    -segment_format mpegts \
    -segment_list "$base/stream.m3u8" \
    -segment_list_size 720 \
    -segment_list_flags live \
    -segment_list_type m3u8 \
    "segments/%08d.ts" 


trap "rm stream.m3u8 segments/*.ts" EXIT

# vim:ts=2:sw=2:sts=2:et:ft=sh

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: h264: possible to force SPS/PPS per GOP?

Fri May 17, 2013 10:39 pm

Just a heads up. The Arch Linux raspberry pi distro has ffmpeg 1.2.0. Almost up-to-date at the time of writing.

You added comments about having problems with piping. With 1.2.0 I’ve noticed no problems with that. But I’d like to find out where the delay comes from. Maybe related to this?

The ffmpeg docs also recommend -f ssegment for streamed ts segments. What’s the difference?

I’ve just compiled psips and it seems to fix the PPS issue. Thanks!

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: h264: possible to force SPS/PPS per GOP?

Sat May 18, 2013 1:41 am

Here’s a shorter variant. I’m not sure what the difference between -f segment and -f hls is. With this one it seems to get stuck now and then. Otherwise it works almost the same.

Code: Select all

#!/bin/bash 
wwwroot="/data/live"

raspivid -n -w 720 -h 405 -fps 25 -awb off -vf -t 86400000 -b 1800000 -o - \
  | /root/psips | ffmpeg -y \
    -analyzeduration 10M \
    -i - \
    -c:v copy \
    -map 0:0 \
    -hls_time 10 \
    -hls_wrap 40 \
    "${wwwroot}/stream.m3u8"

trap "rm ${wwwroot}/{stream.m3u8,stream*.ts}" EXIT

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: h264: possible to force SPS/PPS per GOP?

Sat May 18, 2013 2:21 am

"raspivid --exposure night" doesn’t work with this, because of this ffmpeg bug https://ffmpeg.org/trac/ffmpeg/ticket/2211 .

So ffmpeg interprets the raw h264 always as 25fps. Night mode captures slower at 5fps and setting another fps has no effect. The outcome is that the stream plays as timelapse at first and then stops until the next segment is reached.

Andy Armstrong
Posts: 57
Joined: Sat Dec 03, 2011 10:10 am

Re: h264: possible to force SPS/PPS per GOP?

Sat May 18, 2013 10:38 am

Wow - good work :)

As you say I'm not sure if the fifo is now necessary at all. Under some circumstances ffmpeg seems to have a better stab at analysing an incoming stream if it has a full fifo to read from - so I tend to chuck a fifo and a small delay in if I'm having difficulty getting something working. In this case it's probably just cargo culting.

The latency is inherent in HLS (and all http chunked streaming) I'm afraid. You can read more about how it works in general here: http://www.bbc.co.uk/blogs/bbcinternet/ ... ng_tr.html (that's mainly Adobe HDS but the principle is the same). In this case we're not doing adaptive streaming - because the Pi is only encoding a single bit rate.

The delay can be minimised but not eliminated entirely. The server doesn't advertise a chunk of video until it's completely encoded - so that's one fragment's worth of delay before you start. And the player holds at least one downloaded fragment in hand - in case of adverse network conditions. Typically the lowest latency you can achieve is around 2 * fragment duration - so you can drive the delay down by having, say, 2s fragments - but that impacts on player stability.

The audio is in there because some HLS client implementations seem to require it - I'm interested in using this to stream to anything that's HLS capable - connected TVs / Android / iOS / etc etc (also I intend to inject audio at some point - so I figure I might as well have it in the encoding pipeline from the start). JWPlayer can handle HLS - so that makes it work anywhere Flash is available. If you add up all the iOS devices with everything that supports Flash you get a lot of devices - but it works best if you keep the stream 'normal' - which means including audio.

And it's nice and simple to serve and works over http - which means it pretty much works anywhere. However, while it's acceptable for, say, live event coverage, it's not really suitable for low latency applications such as robotics.

Andy Armstrong
Posts: 57
Joined: Sat Dec 03, 2011 10:10 am

Re: h264: possible to force SPS/PPS per GOP?

Sat May 18, 2013 10:42 am

On the frame rate thing - psips could be used as the starting point for a filter that overrides the frame rate in the h264 stream - although it's a bit more involved than the simple NAL copying that it does at the moment.

And, now I think about it, a generic, stand alone h264 bit stream filter is a useful thing to have... :)

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: h264: possible to force SPS/PPS per GOP?

Sat May 18, 2013 1:01 pm

The cool thing about this segmentation business is that one gets nice little muxed parcels that can be easily concatenated into a long video after the fact.

If you lose power during recording the chunks recorded so far are still there.

I’ve now let it run for 9 hours and have 6.3GB worth of 25fps video in 3259 .ts chunks. 6 542 652 kilobytes / 9 hours = 1 615 Kbps

Now, to mux that into one video, the easiest is to create a ffconcat playlist. This is the format

Code: Select all

# cat stream.ffconcat 
ffconcat version 1.0
file segments/00000000.ts
file segments/00000001.ts
file segments/00000002.ts
So this would be the snippet to mux it all together

Code: Select all

ls segments/*.ts | awk 'BEGIN {print "ffconcat version 1.0"}{print "file " $1}' > segments.ffconcat
ffmpeg -i segments.ffconcat -c copy destination/long.mp4

Andy Armstrong
Posts: 57
Joined: Sat Dec 03, 2011 10:10 am

Re: h264: possible to force SPS/PPS per GOP?

Sat May 18, 2013 1:06 pm

You can actually just concatenate them all together with cat and it works fine. Neat, isn't it? :)

You can also take a live event and turn it into an on-demand HLS video (which plays from the start) by making a couple of changes in the m3u8 that lists the fragments.

You can also have discontinuities in the m3u8 - so that you can do coarse editing by concatenating runs of fragments virtually in the manifest. Discontinuities aren't supported by all players I believe.

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: h264: possible to force SPS/PPS per GOP?

Sat May 18, 2013 1:21 pm

Interesting, if you find cool uses of this please post snippets.

So, the difference between "-segment_list_flags live" and "-segment_list_flags cache" is that it starts from the beginning in the latter case? Would one be able to seek up to the most recent segment and keep playing as new segments arrive? And seek back and forth? I don’t have an iphone and seeking hasn’t been working with my Linux players.

About this FPS thing. Is the raw h264 that the GPU produces amenable to fixes in the firmware or would we be forced to work with whatever we get now?

It would be cool to be able to use very low FPS, maybe even less than 1fps.

Andy Armstrong
Posts: 57
Joined: Sat Dec 03, 2011 10:10 am

Re: h264: possible to force SPS/PPS per GOP?

Sat May 18, 2013 1:34 pm

towolf wrote:Interesting, if you find cool uses of this please post snippets.
The BBC uses it for all the live streaming on iOS devices. I had something to do with that. It's pretty mature and stable by now.
towolf wrote:So, the difference between "-segment_list_flags live" and "-segment_list_flags cache" is that it starts from the beginning in the latter case? Would one be able to seek up to the most recent segment and keep playing as new segments arrive? And seek back and forth? I don’t have an iphone and seeking hasn’t been working with my Linux players.
Yeah, that's the idea. You're supposed to be able to rewind a live stream anywhere within the segments that are advertised in the m3u8. The packager normally retires segments after a certain period of time - often 2 hours.

If you're playing live that means you can do live rewind and then return to the live point. For a pre-recorded video it means that you can seek as normal.

However I've only ever seen live rewind work on iOS devices. Not sure why that is; live HLS plays OK in Safari but attempts to rewind always send you right back to the start and then it won't seek any more.
towolf wrote:About this FPS thing. Is the raw h264 that the GPU produces amenable to fixes in the firmware or would we be forced to work with whatever we get now?

It would be cool to be able to use very low FPS, maybe even less than 1fps.
I don't know much about GPU internals - so I can't comment on that. But unless you have both the FPS fix and the SPS/PPS duplication in GPU you might as well do both in an external filter - you have to have it anyway.

It's quite handy having a stand alone h264 bitstream filter anyway - I can think of unrelated cases in which that might be useful.

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: h264: possible to force SPS/PPS per GOP?

Sun May 19, 2013 6:08 pm

So, raspivid emits an iframe every 60 frames or 2.4 seconds. Is it in any way optimal to set the hls_time to multiples of this? I’ve notived the hlsenc segmenter already chunks according to iframe boundaries.

But the m3u8 file always contains integer second references. What’s up with that?

Andy Armstrong
Posts: 57
Joined: Sat Dec 03, 2011 10:10 am

Re: h264: possible to force SPS/PPS per GOP?

Sun May 19, 2013 7:52 pm

towolf wrote:So, raspivid emits an iframe every 60 frames or 2.4 seconds. Is it in any way optimal to set the hls_time to multiples of this? I’ve notived the hlsenc segmenter already chunks according to iframe boundaries.

But the m3u8 file always contains integer second references. What’s up with that?
I'm getting non-integral segment durations in live.m3u8 that jitter between 9.6 and 7.2 as it attempts to approximate the requested 8s duration. I'm using the 'segment' format.

I imagine that setting the duration to either 9.6 or 7.2 would stabilise things.

At 25fps 12 seconds is 300 frames - which is 5 iframes - so maybe making the segment duration 12s would be a good idea.

I'd experiment but I'm soak testing mine to discover how long it'll run for.

Here's a speeded up video converted from the HLS out of mine this afternoon. I wrote some code to do the 200:1 temporal resampling - rather than just taking one frame every 200 (per stop-motion/time lapse) it effectively speeds the video up 200x.

I also got round to ordering a fibre upgrade for my broadband so I can stream live HD HLS from the Pi and got 4G/LTS on my phone with a view to being able to stream from a field (albeit a field in a major city at the moment...)

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: h264: possible to force SPS/PPS per GOP?

Sun May 19, 2013 8:07 pm

Good idea with the least common multiple. I’ll give 12s a go then. But it works mostly fine at any rate. I have the impression that ffplay continuously playing the m3u8 stream slowly drifts away from 6-8 seconds initially to longer and longer delays. Maybe the advertised fps is not really 25 after all.

Your temporal averaging, or whatever it is, looks very nice and smooth indeed. Would you mind posting the code for it? I’d like to try it.

So far I’ve only sped up video by changing the container to 60fps and playing that, but that bug mentioned above always interferes somehow.

Andy Armstrong
Posts: 57
Joined: Sat Dec 03, 2011 10:10 am

Re: h264: possible to force SPS/PPS per GOP?

Sun May 19, 2013 8:14 pm

Here's the code to do the temporal resampling: https://gist.github.com/AndyA/5608821

I'll tidy it up and release it to github properly when I get a moment - but you should be able to get it working I think :)

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: h264: possible to force SPS/PPS per GOP?

Sun May 19, 2013 8:41 pm

Andy Armstrong wrote:Here's the code to do the temporal resampling: https://gist.github.com/AndyA/5608821

I'll tidy it up and release it to github properly when I get a moment - but you should be able to get it working I think :)
Sweet, thank you.

However, I get ffmpeg errors after the merge:

Code: Select all

[bmp @ 0x26dbaa0] BMP coding -1003094016 not supported
    Last message repeated 5 times
[image2pipe @ 0x26d29c0] decoding for stream 0 failed
[image2pipe @ 0x26d29c0] Could not find codec parameters for stream 0 (Video: bmp): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
This is my command line. I’ve tried on one 12 second chunk only, to rule out external factors:

Code: Select all

ffmpeg -i ssh/live/stream0.ts -c:v bmp -f image2pipe - | ./merge 20 | ffmpeg -y -f image2pipe -c:v bmp -i - -pix_fmt yuv420p -c:v libx264 -b:v 3000k -r:v 25 fast.mkv
Any idea?

Andy Armstrong
Posts: 57
Joined: Sat Dec 03, 2011 10:10 am

Re: h264: possible to force SPS/PPS per GOP?

Sun May 19, 2013 9:03 pm

Yup, I think it's because 12 seconds is only 300 frames of input which would only give you one frame of output (merge outputs one frame for every 200 in) - so the receiving ffmpeg probably doesn't have enough to work with. Give it a try with a longer movie.

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: h264: possible to force SPS/PPS per GOP?

Sun May 19, 2013 9:16 pm

It’s funny. I’m watching your vimeo talk with the Firefox banner in the background and talk to you on here now.

Anyhow, I had used 20 as the argument to ./merge. But now I’ve made the full hour video and it always fails with BMP error of the merge output.

With the default it spits out the error after the first 400 frames have been processed. And then 5 emitted frames later it errors fatally.

I’m looking at the header spec. Is this some kind of 64-bit architecture data type kind of issue? My PC is 64-bit.

Here’s a long paste with debugging on: http://paste.ubuntu.com/5681809/

[edited to correct frame counts]

[edited twice: Also, my image viewer can view the first frame of the BMP that ffmpeg spits out, but not the one merge produces]

Andy Armstrong
Posts: 57
Joined: Sat Dec 03, 2011 10:10 am

Re: h264: possible to force SPS/PPS per GOP?

Sun May 19, 2013 9:44 pm

towolf wrote:It’s funny. I’m watching your vimeo talk with the Firefox banner in the background and talk to you on here now.
That sounds like a recipe for insanity :)
towolf wrote:Anyhow, I had used 20 as the argument to ./merge. But now I’ve made the full hour video and it always fails with BMP error of the merge output.

With the default it spits out the error after the 200 frames have been processed. And then one emitted frame later it errors fatally.

I’m looking at the header spec. Is this some kind of 64-bit architecture data type kind of issue? My PC is 64-bit.

Here’s a long paste with debugging on: http://paste.ubuntu.com/5681809/
Hmm, not sure. Obviously the code hasn't been well tested - I just knocked it up this afternoon and I've only run it on my 1280x720 stream.

The image2pipe format is just a concatenation of images - in this case BMPs, chosen because they're uncompressed and easy to parse.

So you could try capturing the output and using the split command to verify the individual BMPs:

Code: Select all

$ ffmpeg -i cap1.mp4 -c:v bmp -f image2pipe - | ./merge > foo.bmp
$ hexdump -C -n 4 -s 2 foo.bmp 
00000002  36 30 2a 00                                       |60*.|
# so the length of a single BMP in the stream is 0x2a3036
$ split -b $[0x2a3036] foo.bmp 
That will give you a bunch of files named xaa, xab, xac etc. They should contain the individual frames that merge output:

Code: Select all

$ file x*
xaa: PC bitmap, Windows 3.x format, 1280 x 720 x 24
xab: PC bitmap, Windows 3.x format, 1280 x 720 x 24
xac: PC bitmap, Windows 3.x format, 1280 x 720 x 24
xad: PC bitmap, Windows 3.x format, 1280 x 720 x 24
xae: PC bitmap, Windows 3.x format, 1280 x 720 x 24
xaf: PC bitmap, Windows 3.x format, 1280 x 720 x 24
xag: PC bitmap, Windows 3.x format, 1280 x 720 x 24
...

BPK
Posts: 30
Joined: Mon Jun 04, 2012 10:12 am
Location: Bristol, United Kingdom
Contact: Website

Re: h264: possible to force SPS/PPS per GOP?

Sun May 19, 2013 9:47 pm

Hi All,
This is a great thread for HLS!

Andy you mentioned the latency trade off that occurs for the benefits you get with HLS - is 2 seconds the lowest you can go to for each fragment or can you push it down further?

Appreciate you may not want to do this over the Internet but if you had a more reliable network connection you may want to use HLS for the large benefits it gives - player compatibility, firewall friendly etc compared to other approaches (RTSP / RTMP)

Thanks
Barnaby Kent
http://www.pi-cars.com
Control your radio controlled car through your Raspberry Pi

Andy Armstrong
Posts: 57
Joined: Sat Dec 03, 2011 10:10 am

Re: h264: possible to force SPS/PPS per GOP?

Sun May 19, 2013 9:59 pm

BPK wrote:Andy you mentioned the latency trade off that occurs for the benefits you get with HLS - is 2 seconds the lowest you can go to for each fragment or can you push it down further?
I think that's pretty much the practical limit. By design there are always a couple of fragments in the pipeline and as the fragments get very small the request overhead per fragment starts to be significant.
BPK wrote:Appreciate you may not want to do this over the Internet but if you had a more reliable network connection you may want to use HLS for the large benefits it gives - player compatibility, firewall friendly etc compared to other approaches (RTSP / RTMP)

Thanks
If you have very tightly controlled conditions you might be able to get as low as 1s latency - but if you have tightly controlled conditions you can probably use RTP/RTSP/RTMP or just pipe a MPEG-TS stream over a TCP connection without any protocol.

I know it seems like a tantalising prospect but I really don't think HTTP chunked streaming is viable for that kind of real time application (video chat, remote control etc).

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: h264: possible to force SPS/PPS per GOP?

Sun May 19, 2013 10:08 pm

Andy Armstrong wrote:Hmm, not sure. Obviously the code hasn't been well tested - I just knocked it up this afternoon and I've only run it on my 1280x720 stream.
So, to rule that out I’ve switch to 1280x720 too.
The image2pipe format is just a concatenation of images - in this case BMPs, chosen because they're uncompressed and easy to parse.

So you could try capturing the output and using the split command to verify the individual BMPs:

Code: Select all

$ ffmpeg -i cap1.mp4 -c:v bmp -f image2pipe - | ./merge > foo.bmp
$ hexdump -C -n 4 -s 2 foo.bmp 
00000002  36 30 2a 00                                       |60*.|
# so the length of a single BMP in the stream is 0x2a3036
$ split -b $[0x2a3036] foo.bmp 
That will give you a bunch of files named xaa, xab, xac etc. They should contain the individual frames that merge output:

Code: Select all

$ file x*
xaa: PC bitmap, Windows 3.x format, 1280 x 720 x 24
xab: PC bitmap, Windows 3.x format, 1280 x 720 x 24
xac: PC bitmap, Windows 3.x format, 1280 x 720 x 24
xad: PC bitmap, Windows 3.x format, 1280 x 720 x 24
xae: PC bitmap, Windows 3.x format, 1280 x 720 x 24
xaf: PC bitmap, Windows 3.x format, 1280 x 720 x 24
xag: PC bitmap, Windows 3.x format, 1280 x 720 x 24
...
Alright. I think the file utility will not help. That merely looks for the BM magic at the start of the file. ImageMagick has identify and that analyzes all frames in the original from ffmpeg, but it chokes on the merged.bmp and all the splitted out x* bitmaps.

Here’s a full transscript: http://paste.ubuntu.com/5681944/

Are you sure it’s not some kind of overflow? This -1003094016 value looks really suspicious.

So I’ve compiled using "gcc -Wall -O0 -o merge merge.c" on x64 Linux. What are you using?

Return to “Graphics, sound and multimedia”