Ratsima
Posts: 41
Joined: Tue Nov 10, 2015 12:13 am
Location: Korat, Thailand
Contact: Website

GStreamer: omxh264enc Results in out-of-squence first frame

Fri Oct 13, 2017 12:04 am

I'm using GStreamer to produce a time lapse video from a series of JPEG images.

I recently noticed that the first frame of the video is not the first JPEG in the sequence but is instead a JPEG that is about 2/3 of the way through.

Here's the code:

Code: Select all

home="/home/pi/webcam"
mfslocation="$home"/arch/image_%03d.jpg
fslocation="$home"/video/kamvideo.mp4

gst-launch-1.0 \
	-e multifilesrc location="$mfslocation" \
	! image/jpeg, framerate=10/1 \
	! decodebin ! queue ! videoscale \
	! video/x-raw, width=640, height=480 \
	! progressreport name=progress \
	! omxh264enc target-bitrate=1200000 control-rate=variable \
	! video/x-h264, profile=high \
	! h264parse \
	! mp4mux \
	! filesink location="$fslocation"
In all other respects the video is just fine.

Here it is: http://www.mgnewman.com/blogger/omxh264enc_pi.mp4

If you try to view this video under Mac OS High Sierra or iOS 11, it will "stick" on the first frame.

It plays fine on VLC and on OSs prior to Apple's most current. The prior OSs simply play the out of sequence frame. The current OSs display the first frame until the player reaches a frame with a timestamp after the first frame. Let it play to the end and you'll see what I mean.

Any ideas as to how to fix this?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4531
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: GStreamer: omxh264enc Results in out-of-squence first frame

Fri Oct 13, 2017 9:12 am

What is the range of sequence numbers in the filenames? Starting at 000, or 001?

If you use "avprobe -show_packets omxh264enc_pi.mp4" you'll see that the first timestamp (pts = presentation time stamp) comes out as -140462.610 seconds. Negative time? Sounds like something nasty.
At 6 seconds your second keyframe (normal intra-I-period is 60 frames) also has a dubious pts of -140462.610. (There seems to also be a dropped frame as we get 5.9, -140462.61, 6.2)
Likewise the frame that should be at 12.100 seconds has a timestamp of -140462.610.

All the decode time stamps (dts) look rather wacky too being significantly negative, but they are increasing by 0.1s per frame.

I suspect that it is going to be a GStreamer issue as an H264 stream in itself has no timestamp information (that is held in the container or other metadata). You could initially try encoding with something along the lines of

Code: Select all

gst-launch-1.0 \
	-e multifilesrc location="$mfslocation" \
	! image/jpeg, framerate=10/1 \
	! decodebin ! queue ! videoscale \
	! video/x-raw, width=640, height=480 \
	! progressreport name=progress \
	! omxh264enc target-bitrate=1200000 control-rate=variable \
	! video/x-h264, profile=high \
	! filesink location="file.h264"
to save the elementary stream. That will have discarded any timestamp data, but VLC should play it to confirm whether it is just timestamps going wrong that causes the issue, or is your multifilesrc picking up the files in the wrong order.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Ratsima
Posts: 41
Joined: Tue Nov 10, 2015 12:13 am
Location: Korat, Thailand
Contact: Website

Re: GStreamer: omxh264enc Results in out-of-squence first frame

Fri Oct 13, 2017 10:10 am

Thanks for taking the time to reply.

The files are numbered like this:

Code: Select all

-rw-r--r-- 1 pi pi 240770 Oct 13 04:16 image_000.jpg
-rw-r--r-- 1 pi pi 240688 Oct 13 04:20 image_001.jpg
-rw-r--r-- 1 pi pi 240424 Oct 13 04:24 image_002.jpg
...
Until I solve this I have offloaded the video encoding job to a Mac using ffmpeg. It seems to work.

I will also encode tonight's video using GStreamer on the Mac so I can compare the pts_times. I'll post the results tomorrow.

I'll also take your suggestion and save the elementary stream.

I would like to solve this so I can continue encoding on the Pi using the hardware encoder.

Thanks again.

Ratsima
Posts: 41
Joined: Tue Nov 10, 2015 12:13 am
Location: Korat, Thailand
Contact: Website

Re: GStreamer: omxh264enc Results in out-of-squence first frame

Fri Oct 13, 2017 11:52 pm

I tried the same code with GStreamer on a Mac (except with the x264enc encoder) and got the pts results posted below. I'm not sure why the pts_times seem to be out of order, but the video plays normally.

I also tried this on the Pi:

Code: Select all

home="/home/pi/webcam"

mfslocation="$home"/arch/image_%03d.jpg
testlocation="$home"/video/file.h264

gst-launch-1.0 \
	-e multifilesrc location="$mfslocation" \
	! image/jpeg, framerate=10/1 \
	! decodebin ! queue ! videoscale \
	! video/x-raw, width=640, height=480 \
	! progressreport name=progress \
	! omxh264enc target-bitrate=1200000 control-rate=variable \
	! video/x-h264, profile=high \
	! h264parse \
	! filesink location="$testlocation"
But that video won't play. It just shows the first frame and quits. (I tried without h264parse, but the pipeline failed.)

The following is from running GStreamer on a Mac with the same set of JPEGs.

Code: Select all

pi@rasp ~/webcam $ avprobe -show_packets x264enc_mac.mp4 | grep pts_time
avprobe version 11.9-6:11.9-1~deb8u1+rpi1, Copyright (c) 2007-2017 the Libav developers
  built on Apr 26 2017 06:57:28 with gcc 4.9.2 (Raspbian 4.9.2-10)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'x264enc_mac.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41isomiso2
    creation_time   : 2017-10-13 21:02:50
    encoder         : x264
  Duration: 00:00:35.70, start: 0.000000, bitrate: 1604 kb/s
    Stream #0.0(eng): Video: h264 (High), yuvj420p, 640x480 [PAR 1:1 DAR 4:3], 1603 kb/s, 10 fps, 1k tbn, 20 tbc (default)
    Metadata:
      creation_time   : 2017-10-13 21:02:50
pts_time=0.000000
pts_time=0.400000
pts_time=0.200000
pts_time=0.100000
pts_time=0.300000
pts_time=0.800000
pts_time=0.600000
pts_time=0.500000
pts_time=0.700000
pts_time=1.200000
pts_time=1.000000
pts_time=0.900000

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4531
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: GStreamer: omxh264enc Results in out-of-squence first frame

Thu Oct 19, 2017 3:44 pm

Ratsima wrote:
Fri Oct 13, 2017 11:52 pm
I tried the same code with GStreamer on a Mac (except with the x264enc encoder) and got the pts results posted below. I'm not sure why the pts_times seem to be out of order, but the video plays normally.
x264enc supports B-frames which are bidirectionally predicted. That means later frames may have to be delivered before earlier ones in order for the earlier ones to have all the required references.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

tvjon
Posts: 556
Joined: Mon Jan 07, 2013 9:11 am

Re: GStreamer: omxh264enc Results in out-of-squence first frame

Fri Oct 20, 2017 10:27 am

vlc on iOS11 does play it fine & finishes properly.

I can't run High Sierra on my intel "coreduo" mac unfortunately.

On RPi, mpv complains but does play it ok, as does avplay.


$ mpv omxh264enc_pi.mp4

Playing: omxh264enc_pi.mp4

(+) Video --vid=1 (*) (h264)
Using hardware decoding (rpi).
VO: [rpi] 640x480 mmal

[osd/libass] PlayResX undefined, setting to 960
Invalid video timestamp: 0.000000 -> 140462.710000
V: 39:01:08 / 00:00:35 (100%)
Invalid video timestamp: 140468.510000 -> 0.000000
V: 39:01:08 / 00:00:35 (100%)
Invalid video timestamp: 0.000000 -> 140468.810000
V: 39:01:14 / 00:00:35 (100%)
Invalid video timestamp: 140474.610000 -> 0.000000
V: 39:01:14 / 00:00:35 (100%)
Invalid video timestamp: 0.000000 -> 140474.810000
V: 39:01:20 / 00:00:35 (100%)
Invalid video timestamp: 140480.610000 -> 0.000000
V: 39:01:20 / 00:00:35 (100%)
Invalid video timestamp: 0.000000 -> 140480.810000
V: 39:01:26 / 00:00:35 (100%)
Invalid video timestamp: 140486.610000 -> 0.000000
V: 39:01:26 / 00:00:35 (100%)
Invalid video timestamp: 0.000000 -> 140486.810000
V: 39:01:29 / 00:00:35 (100%)

[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: Invalid timestamps stream=0, pts=-140462610, dts=-140456610, size=28110
V: 39:01:32 / 00:00:35 (100%)

Invalid video timestamp: 140492.610000 -> 0.000000
V: 39:01:32 / 00:00:35 (100%)
Invalid video timestamp: 0.000000 -> 140492.810000
V: 39:01:38 / 00:00:35 (100%)


Exiting... (End of file)

If I can find time it will be interesting to try your gstreamer pipeline.

Return to “Graphics, sound and multimedia”

Who is online

Users browsing this forum: No registered users and 15 guests