weshinsley
Posts: 4
Joined: Fri Sep 19, 2014 7:48 am

Best approach for movie with text overlay

Fri Sep 19, 2014 8:02 am

Hi all,

I'd like to write some software for the Pi that displays a looping movie and overlay a page of text (simple html probably) on top of it - song lyrics for my church - and I'm wondering what sort of language (or combination of languages) to go for. I suspect the Pi might be capable of this, since it can play HD movies at full speed, and I'd be connecting to a projector that is probably native 800x600, so theoretically at a lower res, it sounds doable.

My first instincts (for easy programming) were to try a Java transparent window on top of an omxplayer I'd somehow launch, but in the first tests, the transparent window works on Win, but comes out opaque on Pi. I also tried JavaFX, which again worked ok on Windows (and plays the movie itself), but the absence of the Stage and Scene classes in the ARM build was the showstopper on the Pi. In my first test, the text drawing was a bit slow too, so perhaps I'm being too optimistic about Java's performance.

So... it would be a fun project to do, but I wondered if anyone has any ideas about what direction is likely to be possible (and fast enough) ? No need for much detail!

Thanks,
Wes

fruitoftheloom
Posts: 22557
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: Best approach for movie with text overlay

Fri Sep 19, 2014 8:52 am

weshinsley wrote:Hi all,

I'd like to write some software for the Pi that displays a looping movie and overlay a page of text (simple html probably) on top of it - song lyrics for my church - and I'm wondering what sort of language (or combination of languages) to go for. I suspect the Pi might be capable of this, since it can play HD movies at full speed, and I'd be connecting to a projector that is probably native 800x600, so theoretically at a lower res, it sounds doable.

My first instincts (for easy programming) were to try a Java transparent window on top of an omxplayer I'd somehow launch, but in the first tests, the transparent window works on Win, but comes out opaque on Pi. I also tried JavaFX, which again worked ok on Windows (and plays the movie itself), but the absence of the Stage and Scene classes in the ARM build was the showstopper on the Pi. In my first test, the text drawing was a bit slow too, so perhaps I'm being too optimistic about Java's performance.

So... it would be a fun project to do, but I wondered if anyone has any ideas about what direction is likely to be possible (and fast enough) ? No need for much detail!

Thanks,
Wes
GStreamer has a text overlay option ( gstreamer 1.2 is in latest release/dist-upgrade of Wheezy by default ) which may require a plugin installed http://wiki.oz9aec.net/index.php/Gstrea ... xt_Overlay

Openshot & Kdenlive are in the repositories and worth investigating if you want a more comprehensive software suite..
Rather than negativity think outside the box !

Asus ChromeBox 3 Celeron is my other computer.

User avatar
Defiant
Posts: 181
Joined: Tue Oct 30, 2012 6:17 pm
Location: Hamburg, Germany

Re: Best approach for movie with text overlay

Sat Sep 20, 2014 9:41 am

There is also rsvgoverlay for svg.

weshinsley
Posts: 4
Joined: Fri Sep 19, 2014 7:48 am

Re: Best approach for movie with text overlay

Sat Sep 20, 2014 2:30 pm

Many thanks for those - will have a look through those methods.
A further method I wondered about was showing the movie in a browser and seeing if that can handle a transparent layer on top nicely...

Thanks again,
Wes

lanewinfield
Posts: 21
Joined: Tue Apr 07, 2015 2:12 am

Re: Best approach for movie with text overlay

Wed Apr 08, 2015 9:31 pm

Looking to do a very similar project to this—were you ever able to figure out a solution?

Thanks!

Brian

weshinsley
Posts: 4
Joined: Fri Sep 19, 2014 7:48 am

Re: Best approach for movie with text overlay

Wed Apr 08, 2015 9:50 pm

Not yet - but I ran out of steam a little and wrote what I needed in 4 lines of Java on a windows laptop. It would have been cute to do it on a PI, because I think it can do it. But I didn't have the appetite at that time to go into GStreamer, or some of the more intricate ways. The browser method was far too slow, and Java not supporting transparency basically thwarted me. That said, I've not tested a more recent Java 8 and/or my newer quad-core Pi, which might improve things a bit.

For info:

Related topic maybe:
http://www.raspberrypi.org/forums/viewt ... 43&t=44474

And what I really want:-
http://fxexperience.com/2013/08/javafx- ... spberrypi/

(Last paragraph of above is word for word what I'd like. Quote: Its mostly running on the shipping EA of JavaFX 8 Embedded we prototyped a couple changes to the platform .... the ability to draw JavaFX with transparent background on a hardware layer over hardware decoded video.)

uler3161
Posts: 30
Joined: Tue Feb 10, 2015 3:58 pm

Re: Best approach for movie with text overlay

Wed Apr 08, 2015 10:56 pm

I'm also trying to make this happen. A few things I've tried:

Pipe raspivid to GStreamer (version 1.2) and use timeoverlay: Result: Could not get video output. Possibly due to h264 encoding versus raw, but I tried a few things and couldn't get it to work.

GStreamer with v4l2src: This kind of worked, but strange framerate issues. Doing a preview overlay with v4l2-ctl worked fine. But using v4l2src resulted in about 0.5 fps. If I recall correctly, when I removed the timeoverlay, I got smooth video, but it played about twice as fast.

Built latest GStreamer 1.4.5. Using v4l2src I was able to do raw video with timeoverlay and it looked great. However, I couldn't get h264 encoded video to work. First the omx plugins weren't even showing up with gst-inspect. I found out they were blacklisted. Fixed that, but if I recall I wasn't getting any playback. This is where I'm stuck.

So I think the key might be to use a newer version of GStreamer with v4l2src. I'm using the script here: http://www.onepitwopi.com/raspberry-pi/ ... pberry-pi/ and I modified it to point to the 1.4 version. I wish someone out there had a binary for it though. It takes about 8 hours to build on a Pi. Might be faster on a Pi2. Tried that script on a Pi2 and it got stuck. May try it again tonight.

User avatar
Defiant
Posts: 181
Joined: Tue Oct 30, 2012 6:17 pm
Location: Hamburg, Germany

Re: Best approach for movie with text overlay

Thu Apr 09, 2015 5:27 am

uler3161 wrote: Pipe raspivid to GStreamer (version 1.2) and use timeoverlay: Result: Could not get video output. Possibly due to h264 encoding versus raw, but I tried a few things and couldn't get it to work.
You do realize, that you need raw video for the overlay element? What pipeline did you use?
uler3161 wrote:I
GStreamer with v4l2src: This kind of worked, but strange framerate issues. Doing a preview overlay with v4l2-ctl worked fine. But using v4l2src resulted in about 0.5 fps. If I recall correctly, when I removed the timeoverlay, I got smooth video, but it played about twice as fast.
Pipeline please

uler3161
Posts: 30
Joined: Tue Feb 10, 2015 3:58 pm

Re: Best approach for movie with text overlay

Thu Apr 09, 2015 3:17 pm

Unfortunately I didn't save my old pipelines. I realized it was probably because it couldn't overlay without decoding. I had tried a few things that I thought should convert it from h264 to raw, but it didn't work.

I have this pipeline right now:

Code: Select all

gst-launch-1.0 -v v4l2src do-timestamp=true ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! timeoverlay ! omxh264enc ! video/x-h264,width=640,height=480,framerate=30/1 ! decodebin ! autovideosink sync=false
Which just sits there and does nothing. Doesn't even try to put anything on the screen.

Removing the format=I420 doesn't change anything.
Removing format=I420 and the h264 encoding works (and appears to use BGRx format according to caps)
Using format=BGRx and h264 encoding results in: "ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data flow error."
Using format=I420 and removing h264 encoding results in it just sitting there.

I decided to add GST_DEBUG=2 and I see this repeated over and over:

0:00:02.002227621 3848 0x1595490 ERROR default video-frame.c:136:gst_video_frame_map_id: failed to map video frame plane 1

I guess I still don't really know what I'm doing. I've been using GStreamer for a few weeks now and thought I was getting the hang of it. But it seems like this should be pretty easy. Just pull in raw video with v4l2src, do a timeoverlay (or textoverlay, clockoverlay), then use omxh264enc for hardware encoding. Is there something obvious about my pipeline? I'd really like to see a known good pipeline that would do an overlay and h264 hardware encoding. I've done a lot of searching, but i have yet to see one.

uler3161
Posts: 30
Joined: Tue Feb 10, 2015 3:58 pm

Re: Best approach for movie with text overlay

Thu Apr 09, 2015 4:07 pm

I've reverted back to the gstreamer that is installed with raspbian. Here's the pipeline I'm currently playing with:

Code: Select all

 gst-launch-1.0 v4l2src do-timestamp=true ! video/x-raw,width=1024,height=768,framerate=24/1 ! omxh264enc target-bitrate=1000000 control-rate=variable ! video/x-h264,width=1024,height=768,framerate=24/1 ! decodebin ! autovideosink sync=false
I'm not yet doing a timeoverlay. I'm just trying to get it to encode in h264. I've hit two problems:

1. It seems I can't just pick whatever width/height. I was doing a widescreen like recording of width=1024, height=500. Works fine without h264 encoding or if I do it via raspivid. But doing that in the above pipeline results in:
0:00:01.235803546 2759 0x623400 WARN omxvideoenc gstomxvideoenc.c:891:gst_omx_video_enc_loop:<omxh264enc-omxh264enc0> error: Failed to set caps
2. Seems to only be around maybe 5 fps. But h264 works nice and smooth if I'm just doing this:

Code: Select all

gst-launch-1.0 -v v4l2src do-timestamp=true ! video/x-h264,width=1024,height=768,framerate=24/1 ! decodebin ! autovideosink sync=false
Only thing I can think is that omxh264enc is not using hardware acceleration. Maybe in this case it's not able to? In that case I'd guess it's not even possible to do an overlay at this resolution and get decent framerate?

User avatar
Defiant
Posts: 181
Joined: Tue Oct 30, 2012 6:17 pm
Location: Hamburg, Germany

Re: Best approach for movie with text overlay

Thu Apr 09, 2015 5:10 pm

uler3161 wrote: I have this pipeline right now:

Code: Select all

gst-launch-1.0 -v v4l2src do-timestamp=true ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! timeoverlay ! omxh264enc ! video/x-h264,width=640,height=480,framerate=30/1 ! decodebin ! autovideosink sync=false
Which just sits there and does nothing. Doesn't even try to put anything on the screen.

Removing the format=I420 doesn't change anything.
Fun it does. Try:

Code: Select all

gst-launch-1.0 -v v4l2src do-timestamp=true ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! timeoverlay ! autovideosink
vs

Code: Select all

gst-launch-1.0 -v v4l2src do-timestamp=true ! video/x-raw,width=640,height=480,framerate=30/1 ! timeoverlay ! autovideosink
so does the following work for me:

Code: Select all

gst-launch-1.0 -v v4l2src do-timestamp=true ! video/x-raw,format=BGRx,width=640,height=480,framerate=30/1 ! timeoverlay ! videoconvert ! omxh264enc ! video/x-h264,width=640,height=480,framerate=30/1 ! decodebin ! autovideosink sync=false
This also works:

Code: Select all

gst-launch-1.0 -v videotestsrc ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! timeoverlay ! autovideosink
strange. Btw replacing timeoverlay with textoverlay also works:

Code: Select all

gst-launch-1.0 -v v4l2src do-timestamp=true ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! textoverlay ! omxh264enc ! video/x-h264,width=640,height=480,framerate=30/1 ! decodebin ! autovideosink sync=false

uler3161
Posts: 30
Joined: Tue Feb 10, 2015 3:58 pm

Re: Best approach for movie with text overlay

Thu Apr 09, 2015 5:41 pm

Thanks for the suggestions. Here are my results:

Code: Select all

gst-launch-1.0 -v v4l2src do-timestamp=true ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! timeoverlay ! autovideosink
results in no video displayed and with GST_DEBUG=2, this message repeated:
0:00:02.655636749 2599 0x13d5e60 ERROR default video-frame.c:136:gst_video_frame_map_id: failed to map video frame plane 1
This next one works fine:

Code: Select all

gst-launch-1.0 -v v4l2src do-timestamp=true ! video/x-raw,width=640,height=480,framerate=30/1 ! timeoverlay ! autovideosink
Now this:

Code: Select all

gst-launch-1.0 -v v4l2src do-timestamp=true ! video/x-raw,format=BGRx,width=640,height=480,framerate=30/1 ! timeoverlay ! videoconvert ! omxh264enc ! video/x-h264,width=640,height=480,framerate=30/1 ! decodebin ! autovideosink sync=false
This one does work, but framerate seems to be a bit low and there's a bit of a delay (maybe a second). It's almost usable, but would like to do a little better if possible.

Next up:

Code: Select all

gst-launch-1.0 -v videotestsrc ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! timeoverlay ! autovideosink
Bad framerate and I see a number of these messages:
Additional debug info:
gstbasesink.c(2791): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstEglGlesSink:autovideosink0-actual-sink-eglgles:
There may be a timestamping problem, or this computer is too slow.
This last one is kind of interesting:

Code: Select all

gst-launch-1.0 -v v4l2src do-timestamp=true ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! textoverlay ! omxh264enc ! video/x-h264,width=640,height=480,framerate=30/1 ! decodebin ! autovideosink sync=false
It works very smooth and no delay. But, I think it's skipping over the textoverlay because you don't have any text. Once I actually add some text it does the same thing the very first example did with no video and failed to map video frame plane 1 messages. I've tried to figure out what that message means and how to fix it, but I didn't get far. But I'm assuming that's the problem.

uler3161
Posts: 30
Joined: Tue Feb 10, 2015 3:58 pm

Re: Best approach for movie with text overlay

Thu Apr 09, 2015 5:58 pm

After finding this: http://lists.freedesktop.org/archives/g ... 51678.html when researching the failed to map video frame plane 1 error message, I decided to run some more logging to look for errors about stride. I don't know what it means, but I think it's suspicous:

0:00:01.030951269 2807 0x1bd05b0 ERROR v4l2 gstv4l2bufferpool.c:468:gst_v4l2_buffer_pool_set_config:<v4l2bufferpool0> missing GstMetaVideo API in config, default stride: 640, wanted stride 960

User avatar
Defiant
Posts: 181
Joined: Tue Oct 30, 2012 6:17 pm
Location: Hamburg, Germany

Re: Best approach for movie with text overlay

Sat Apr 11, 2015 8:26 am

uler3161 wrote:

Code: Select all

gst-launch-1.0 -v v4l2src do-timestamp=true ! video/x-raw,format=BGRx,width=640,height=480,framerate=30/1 ! timeoverlay ! videoconvert ! omxh264enc ! video/x-h264,width=640,height=480,framerate=30/1 ! decodebin ! autovideosink sync=false
This one does work, but framerate seems to be a bit low and there's a bit of a delay (maybe a second). It's almost usable, but would like to do a little better if possible.
Delay with h264 expected. What do you want to do anyway? I suppose in your final application you do not want to encode/decode h264 in a single pipeline?

uler3161
Posts: 30
Joined: Tue Feb 10, 2015 3:58 pm

Re: Best approach for movie with text overlay

Mon Apr 13, 2015 3:32 pm

Yeah, but without the overlay there is almost zero delay. I do want to encode to h264 in a single pipeline, but this is a video conferencing application with a time overlay (and possibly other text). The delay just isn't great and neither is the low resolution. Doing everything but the overlay works great at high res and very little delay. Adding an overlay seems like such a small thing to accomplish.

I've been trying to get the latest version of gstreamer working on my pi, but having little luck. Apparently eglglessink was removed and replaced with glimagesink which I can't seem to get to show up. Had a lot of issues getting gst-plugins-bad built. Finally did, but still no glimagesink. I'm not sure it's the problem, but I see this in gst-inspect:
(gst-plugin-scanner:15462): GStreamer-WARNING **: Failed to load plugin '/usr/local/lib/gstreamer-1.0/libgstopengl.so': /opt/vc/lib/libEGL.so: undefined symbol: glPointSizePointerOES

uler3161
Posts: 30
Joined: Tue Feb 10, 2015 3:58 pm

Re: Best approach for movie with text overlay

Mon Apr 13, 2015 4:36 pm

I was going to try another alternative. On the receiving end, I currently have this:

Code: Select all

gst-launch-1.0 udpsrc port=$VideoInPort caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! \
        rtph264depay ! \
        decodebin ! \
        autovideosink > logs/videoin.txt &
I thought I could just add a timeoverlay after decodebin since it should be raw at that point. But I'm back to the issue of not having any video show up. Pretty much the same problem I've been trying to fix: "failed to map video frame plane 0" Though I'm also now getting this:
0:00:03.433818610 3792 0x178ea30 ERROR GST_MEMORY gstmemory.c:318:gst_memory_map: mem 0x178b408: subclass map failed
This is with gstreamer 1.2 installed with apt-get.

User avatar
Defiant
Posts: 181
Joined: Tue Oct 30, 2012 6:17 pm
Location: Hamburg, Germany

Re: Best approach for movie with text overlay

Mon Apr 13, 2015 5:07 pm

You need to specify special compile options for eglglessink.

uler3161
Posts: 30
Joined: Tue Feb 10, 2015 3:58 pm

Re: Best approach for movie with text overlay

Wed Apr 15, 2015 3:16 pm

I think I have something figured out.

I have two Pis set up and I am running this for testing:

Code: Select all

gst-launch-1.0 videotestsrc ! omxh264enc ! h264parse ! omxh264dec ! timeoverlay ! autovideosink
Pi #1 (GStreamer installed with apt-get):
Results: Stops at Pipeline is PREROLLING and has Failed to map video frame plane 0 error.
Version:
gst-launch-1.0 version 1.2.0
GStreamer 1.2.0
http://packages.qa.debian.org/gstreamer1.0


Pi #2 (Built with this: http://www.onepitwopi.com/raspberry-pi/ ... pberry-pi/)
Results: Works fine
Version:
gst-launch-1.0 version 1.2.4
GStreamer 1.2.4
Unknown package origin

So I guess the key is to get the latest 1.2 version. I assume the 1.4 version would work too, but I couldn't get everything to build correctly.

Next I'm going to try my actual video source and see whether this is the actual answer.

uler3161
Posts: 30
Joined: Tue Feb 10, 2015 3:58 pm

Re: Best approach for movie with text overlay

Wed Apr 15, 2015 9:31 pm

Working with GStreamer 1.2.4:

Code: Select all

gst-launch-1.0 v4l2src do-timestamp=true ! video/x-h264,width=1024,height=500,framerate=24/1 ! h264parse ! omxh264dec ! timeoverlay ! autovideosink sync=false

weshinsley
Posts: 4
Joined: Fri Sep 19, 2014 7:48 am

Re: Best approach for movie with text overlay

Tue Mar 29, 2016 7:34 am

Hi all,

Just another update on this topic, which I might have another try with, especially with the Raspberry Pi 3 now out. I was thinking about the VLC+Java angle which I can make work on a 10-yr old Windows machine. And recently I came across:

https://ubuntu-mate.community/t/tutoria ... ax-il/3585

That I believe tells you how to compile the VLC player for Pi with hardware acceleration. I wonder if that will then play with the Java bindings (http://capricasoftware.co.uk/#/projects/vlcj). And if that works, then will the PI (especially Pi 3) be fast enough to put a transparent full-screen Java frame on top with the lyrics updating when I want...?

Worth a try!
W.

Return to “General programming discussion”