alnaseh
Posts: 63
Joined: Thu Jun 23, 2016 5:12 am

2 new options for raspivid

Thu Jan 17, 2019 5:38 pm

Hi,

i have just updated the firmware "rpi-update" to the latest one and discovered 2 new options in raspivid

Code: Select all

-stm, --spstimings      : Add in h.264 sps timings
-sl, --slices   : Horizontal slices per frame. Default 1 (off)
i tried to understand their usage from the commits with no luck.
https://github.com/raspberrypi/userland/pull/486
https://github.com/raspberrypi/userland/pull/488

i would appreciate it if you could shed some light on these options and the benefits of them

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7305
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: 2 new options for raspivid

Thu Jan 17, 2019 6:02 pm

--spstiming
In the H264 VUI (Video Usability Information) section of the SPS (Sequence Parameter Set) there is an optional field to insert timing information. It's disabled by default, and this will enable it.

--slices
H264 can encode a frame as one or more independent slices of image. This allows you to alter the number of slices from the default of 1.
Under some circumstances it can improve the robustness of the stream.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
HermannSW
Posts: 1502
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: 2 new options for raspivid

Thu Jan 17, 2019 6:15 pm

"--spstiming" is in official doc, "--slices" not:
https://www.raspberrypi.org/documentati ... /camera.md

Will "--slices" be added to official doc as well?
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://gitlab.freedesktop.org/HermannSW/gst-template
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

ethanol100
Posts: 585
Joined: Wed Oct 02, 2013 12:28 pm

Re: 2 new options for raspivid

Thu Jan 17, 2019 8:10 pm

I just want to add the use cases for the options.

-stm: if you put the created h264 file into a container with programs like MP4Box, ffmpeg or gstreamer, the program can read the frame rate from the h264 file and it can set the right frame rate for the output automatically. This is kind of useful all the time and does not disturb the rest of the file.

-sl: the image is divided in slices and each slice can be decoded separately, allowing parallel decoding, which can increase the decoding efficiency. I don't know if the pi's hardware decoder would use this. On the other side this means the motion estimation an other things can only depend on the information stored in the current slice, which can make the compression less effective. This is really only useful in very special cases.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23686
Joined: Sat Jul 30, 2011 7:41 pm

Re: 2 new options for raspivid

Thu Jan 17, 2019 9:01 pm

ethanol100 wrote:
Thu Jan 17, 2019 8:10 pm
I just want to add the use cases for the options.

-stm: if you put the created h264 file into a container with programs like MP4Box, ffmpeg or gstreamer, the program can read the frame rate from the h264 file and it can set the right frame rate for the output automatically. This is kind of useful all the time and does not disturb the rest of the file.

-sl: the image is divided in slices and each slice can be decoded separately, allowing parallel decoding, which can increase the decoding efficiency. I don't know if the pi's hardware decoder would use this. On the other side this means the motion estimation an other things can only depend on the information stored in the current slice, which can make the compression less effective. This is really only useful in very special cases.
IIRC, slices can also reduce latency, as you can decode a slice as soon as it arrives, rather than waiting for whole frame. That means by the time the last slice arrives, you have already decoded the rest of the image., meaning the complete image is available more quickly (the time to decode one slice).
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
HermannSW
Posts: 1502
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: 2 new options for raspivid

Thu Jan 17, 2019 10:03 pm

ethanol100 wrote:
Thu Jan 17, 2019 8:10 pm
-stm: if you put the created h264 file into a container with programs like MP4Box, ffmpeg or gstreamer, the program can read the frame rate from the h264 file and it can set the right frame rate for the output automatically. This is kind of useful all the time and does not disturb the rest of the file.
Until now if I wanted to achieve the same effect I had to use "--save-pts" and then generate .mkv video from recorded .h264 and saved .pts files using mkvmerge, eg:
viewtopic.php?f=43&t=206047&p=1285069#p1294557

A very convenient option, or did I understand -stm wrong and --save-pts and mkvmerge achieve something different?
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://gitlab.freedesktop.org/HermannSW/gst-template
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

ethanol100
Posts: 585
Joined: Wed Oct 02, 2013 12:28 pm

Re: 2 new options for raspivid

Thu Jan 17, 2019 11:54 pm

jamesh wrote:
Thu Jan 17, 2019 9:01 pm
IIRC, slices can also reduce latency, as you can decode a slice as soon as it arrives, rather than waiting for whole frame. That means by the time the last slice arrives, you have already decoded the rest of the image., meaning the complete image is available more quickly (the time to decode one slice).
That's good to know and makes the option much more useful in my opinion. I will need to try this out. Would you need a special decoder for this, or will programs like ffmpeg automatically use it to decrease latency?
HermannSW wrote: Until now if I wanted to achieve the same effect I had to use "--save-pts" and then generate .mkv video from recorded .h264 and saved .pts files using mkvmerge, eg:
viewtopic.php?f=43&t=206047&p=1285069#p1294557

A very convenient option, or did I understand -stm wrong and --save-pts and mkvmerge achieve something different?
If I understood this correctly it makes quite a difference. The -stm will just include a single constant number for the frame number like 30fps and will not actually measure the time when the frames arrive. It is more like a hint for the decoder. The -pts and mkvmerge option will save the correct time for each single frame(display time stamps), if you would have gaps or change the frame rate mkvmerge will account for it.

alnaseh
Posts: 63
Joined: Thu Jun 23, 2016 5:12 am

Re: 2 new options for raspivid

Fri Jan 18, 2019 2:48 am

thank you all,

so --spstimings is good to have all the time.
for --slices, it helps in reducing the latency but with less compression efficiency. reducing the latency is good for me so i will start playing with it. i guess practical values should be within 1 - 5

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7305
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: 2 new options for raspivid

Fri Jan 18, 2019 7:17 am

As above, spstimings is a hint to the decoder as to the framerate, using the actual timestamps is far more accurate.

Slices can only reduce latency in some circumstances.
It won't give a big gain if decoding via any hardware acceleration (eg a Pi) as you won't have a second hw block to decode the second slice in parallel, in fact it may increase decode time as it requires extra setup.
Reduction in encode latency for the first slice is likely to be a couple of milliseconds out of ~40ms for a 1080p frame - it is not going to be huge.
The one place you may gain is over a low bandwidth link, where you can start sending the data a couple of ms earlier, and get the decoder started a similar few ms sooner.
IIRC it was added originally for a project where the input frame was also produced and fed to the encoder in slices, at which point you could save a bit more time. That option is not available on the Pi.

Tbh neither option is adds a huge amount for 99.9% of use cases, hence why they haven't been worried about before.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23686
Joined: Sat Jul 30, 2011 7:41 pm

Re: 2 new options for raspivid

Fri Jan 18, 2019 8:25 am

6by9 wrote:
Fri Jan 18, 2019 7:17 am
As above, spstimings is a hint to the decoder as to the framerate, using the actual timestamps is far more accurate.

Slices can only reduce latency in some circumstances.
It won't give a big gain if decoding via any hardware acceleration (eg a Pi) as you won't have a second hw block to decode the second slice in parallel, in fact it may increase decode time as it requires extra setup.
Reduction in encode latency for the first slice is likely to be a couple of milliseconds out of ~40ms for a 1080p frame - it is not going to be huge.
The one place you may gain is over a low bandwidth link, where you can start sending the data a couple of ms earlier, and get the decoder started a similar few ms sooner.
IIRC it was added originally for a project where the input frame was also produced and fed to the encoder in slices, at which point you could save a bit more time. That option is not available on the Pi.

Tbh neither option is adds a huge amount for 99.9% of use cases, hence why they haven't been worried about before.
I htink the latency advantage is that even if only decoding one slice at a time, you can be decoding earlier slices whilst later slices are still arriving.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7305
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: 2 new options for raspivid

Fri Jan 18, 2019 9:34 am

jamesh wrote:
Fri Jan 18, 2019 8:25 am
The one place you may gain is over a low bandwidth link, where you can start sending the data a couple of ms earlier, and get the decoder started a similar few ms sooner.
I htink the latency advantage is that even if only decoding one slice at a time, you can be decoding earlier slices whilst later slices are still arriving.
That would be a low bandwidth link then.

Even 802.11g (54Mbit/s) should carry the average 41kB frame of a 10Mbit/s 30fps stream in 0.7ms (assuming my maths is correct). Splitting that into two might save you 0.3ms - I won't be losing sleep over not having that saving.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23686
Joined: Sat Jul 30, 2011 7:41 pm

Re: 2 new options for raspivid

Fri Jan 18, 2019 10:22 am

6by9 wrote:
Fri Jan 18, 2019 9:34 am
jamesh wrote:
Fri Jan 18, 2019 8:25 am
The one place you may gain is over a low bandwidth link, where you can start sending the data a couple of ms earlier, and get the decoder started a similar few ms sooner.
I htink the latency advantage is that even if only decoding one slice at a time, you can be decoding earlier slices whilst later slices are still arriving.
That would be a low bandwidth link then.

Even 802.11g (54Mbit/s) should carry the average 41kB frame of a 10Mbit/s 30fps stream in 0.7ms (assuming my maths is correct). Splitting that into two might save you 0.3ms - I won't be losing sleep over not having that saving.
At a previous job we were using DECT, so pretty low bandwidth, and it made a difference in those circumstances.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

Pilotnbr1
Posts: 10
Joined: Fri Aug 26, 2016 9:08 pm

Re: 2 new options for raspivid

Fri Aug 09, 2019 9:50 pm

Hi guys, I am playing around with the slices option. Do you need an updated or different decoder to best display an h264 stream that has been sliced?

All I did was seperately compile the updated (as in having the slice param) raspivid and then pasted it into my image over my old raspivid... I am streaming my video to another pi over wifi but am seeing more artifacts and latency than my previous non-sliced solution. I am wondering if I need to update the player side so that it can handle slices. I see a dramatic improvement if I use spstiming....

Return to “Camera board”