Problem streaming to Windows with Gstreamer SDK

[CostasVav]

Hi All,

I am attempting to stream my PiCam on my Pi2 to a Windows 7 machine on the same WiFi network.

As low latency is a must, I am not using an VLC RTP stream, but currently testing a Gstreamer RTP-UDP stream.

I just downloaded Gstreamer SDK 2013.6 (Congo) for my Windows machine and did the complete install, but there appear to be a lot of missing plugins that do not allow me to follow the old streaming guides. Plugins like rtpjitterbuffer, avdec_h264 and fpsdisplaysink do not appear to be packaged with Gstreamer SDK anymore.

The Pi command I am using (with the Windows machine being 192.168.1.143):

Code: Select all

raspivid -t 0 -h 720 -w 1280 -fps 15 -hf -vf -b 3000000 -o - | \
gst-launch-1.0 -v fdsrc ! h264parse !  rtph264pay config-interval=5 pt=96 ! udpsink host=192.168.1.143 port=5001

Internet guides usually offer the following code, which fails due to missing pipeline elements:

Code: Select all

gst-launch-1.0 -v udpsrc port=5001 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! \
rtph264depay ! avdec_h264 ! videoconvert ! autovideosink sync=false

I tried adapting the Windows command to not use missing elements, as such:

Code: Select all

gst-launch-1.0 -v udpsrc port=5001 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! \
rtph264depay ! ffdec_h264 ! autovideosink sync=false

But all I get in the console are a few lines and

Code: Select all

Setting pipeline to PLAYING ...
New clock: GetSystemClock

The console just hangs without displaying anything. Any clues on how to get the stream to appear on Windows?

Thanks!

EDIT:
The commands actually work, just forgot to type them correctly...

UPDATE:
Found the following approach which is MUCH better than raspivid/gstreamer/vlc approaches. http://petrkout.com/electronics/low-lat ... er-opencv/

[towolf]

It might be that the receiving side needs to be started first, so that it can catch the start of the stream.

Can you post the commands you are using on both ends?

[CostasVav]

Pi command:
raspivid -t 0 -h 720 -w 1280 -fps 30 -hf -vf -b 4500000 -o - | \
gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=5 pt=96 ! udpsink host=192.168.1.143 port=5001

Windows command:
gst-launch-1.0 -v udpsrc port=5001 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! \
rtph264depay ! ffdec_h264 ! autovideosink sync=false

EDIT:
Nevermind. I found the problem, I was missing the -v on the receiving end.

Video works, but with a 10-20 second lag. The video is also very low FPS and grays out when there is motion.

Is there something I can tweak?

[towolf]

-v just enables verbose logging. That can’t be it.

For me this sender and receiver gives me around 100ms of latency. But my receiver is Linux as well.

Sender

Code: Select all

gst-launch-1.0 -e rpicamsrc bitrate=5000000 do-timestamp=true preview=false  ! 'video/x-h264,width=1024,height=768,framerate=30/1' ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=10.0.0.9 port=9000 sync=false async=true

I compiled rpicamsrc on my Raspberry Pi, which is a project to avoid having to pipe rasivid into GStreamer. But it should not be needed.

Receiver

Code: Select all

gst-launch-1.0 udpsrc port=9000 caps='application/x-rtp, media=video, clock-rate=90000, encoding-name=H264' ! rtph264depay ! h264parse ! avdec_h264 ! autovideosink

Maybe that is ffdec_h264 for you

[CostasVav]

I am encountering lots of lag that is dependent on resolution and framerate.

At low resolutions like 400x300, I get a low latency (guessing about 200-300ms). The preview on the RPi looks normal. There is minimal distortion on the client.

When I bump up the resolution to 720p or 1080p, the preview starts to lag (1 second delay at 720p, 2-5 second delay at 1080p) and the FPS is around 5-10, even if it is defined as 15 or 30. The client can barely stream the 720p feed, with about 10-30 second delay. The 1080p feed is endlessly buffering, with buffer/timestamp errors on both client as server during the process.

It appears that the gstreamer cannot encode the video at the requested resolutions with the above code. The processing overhead appears to be causing issues.

What alternatives are there for encoding?

UPDATE:

Found MJPG-Streamer, which works better. The encoding works without latency on the RPi, and can stream to Windows without too much latency (<500ms), which I blame MS on. Linux has less than 100ms lag, which is perfect. All this at 1080p30 resolution, though JPG quality set to 6 (out of a 100). Still looks darn good though.

For anyone that ended up here with google, go to http://petrkout.com/electronics/low-lat ... er-opencv/ to get information on it.

[thaytan]

The GStreamer 2013.6 SDK from gstreamer.com is the obsolete 0.10 SDK. We wish the people that created gstreamer.com would take it down, it leads to a lot of confusion. These days, we release the Windows build as part of the GStreamer project at http://gstreamer.freedesktop.org/download/

I recommend grabbing the latest 1.4.5 release. Using the old 0.10 version is the reason you have to use 'ffdec_h264' instead of 'avdec_h264'. I assume you are also actually using 'gst-launch-0.10' on Windows, even though you typed 'gst-launch-1.0'

On the RPi side, instead of "raspivid -t 0 -h 720 -w 1280 -fps 15 -hf -vf -b 3000000 -o - | gst-launch-1.0 fdsrc", I recommend using my gst-rpicamsrc component from https://github.com/thaytan/gst-rpicamsrc

That integrates the same functions as raspivid directly into GStreamer, which reduces the overhead, and avoids the huge lag you're seeing - which is usually from not being able to put the data onto the network fast enough, and having it buffer up in the pipe inside the kernel between raspivid and gst-launch. My most recent commits to gst-rpicamsrc bring it up to speed with all the latest raspivid features, fix a few bugs, and add 'proper' timestamping to transfer the capture times from the RPi camera directly onto the outgoing H.264, for a smaller reduction in jitter.

An example streaming pipeline that looks like what you pasted above is something like:

gst-launch-1.0 rpicamsrc bitrate=3000000 hflip=true vflip=true ! video/x-h264,width=1280,height=720,framerate=15/1 ! h264parse ! rtph264pay config-interval=5 pt=96 ! udpsink host=192.168.1.143 port=5001

You might also like the 'intra-refresh-type=both' property for rpicamsrc, which will use H.264 intra-refresh instead of keyframes, for smoother recovery from packet loss. That's one of the new raspivid features I merged recently. Run 'gst-inspect-1.0 rpicamsrc' for a breakdown of all available properties.

Another thing you can do is compile gst-rtsp-server, and use the examples/test-launch utility to make your camera available as an RTSP stream:

./examples/test-launch "( rpicamsrc bitrate=3000000 hflip=true vflip=true ! video/x-h264,width=1280,height=720,framerate=15/1 ! h264parse ! rtph264pay config-interval=5 pt=96 name=pay0 )"

Then, you can connect using any RTSP client on Windows at the URL 'rtsp://192.1.whatever:8554/test'

VLC, or 'gst-launch-1.0 playbin uri=rtsp://ip:8554/test' should work as clients.

[JPelletier]

Wow thanks for those informations thaytan!

I'm new to the PI Community, just received my Pi 2 kit today for my project. Here are my requirements:

1- Live preview + Overlay on the connect TV (Portrait Orientation). I will probably put the preview layer on the background and add an Open GL Layer with clear alpha on top of the preview for the overlay. I don't need/want the overlay in my videos, only over the preview.
2- Stream live to a mobile App (not a webpage) over WiFi. I want to support both Android and iOS at least.
3- Save short video sequences (approx. 10s) on a web server(maybe Wistia) for later watch. Videos will be accessible from a website or the same mobile App.
4- No Audio

Actually I'm reading on the Pi and libraries that I can use. I think that GStreamer is what I need, is the tutorials on GStreamer.com still valid? http://docs.gstreamer.com/display/GstSD ... +tutorials

[thaytan]

Wow thanks for those informations thaytan!

I'm new to the PI Community, just received my Pi 2 kit today for my project. Here are my requirements:

1- Live preview + Overlay on the connect TV (Portrait Orientation). I will probably put the preview layer on the background and add an Open GL Layer with clear alpha on top of the preview for the overlay. I don't need/want the overlay in my videos, only over the preview.
2- Stream live to a mobile App (not a webpage) over WiFi. I want to support both Android and iOS at least.
3- Save short video sequences (approx. 10s) on a web server(maybe Wistia) for later watch. Videos will be accessible from a website or the same mobile App.
4- No Audio

Actually I'm reading on the Pi and libraries that I can use. I think that GStreamer is what I need, is the tutorials on GStreamer.com still valid? http://docs.gstreamer.com/display/GstSD ... +tutorials

All of those things are possible, but non-trivial.

Nothing on gstreamer.com is up to date. There are newer versions of the tutorials for the newer SDK builds from the GStreamer project at http://cgit.freedesktop.org/~slomo/gst-sdk-tutorials/

[JPelletier]

All of those things are possible, but non-trivial.

Nothing on gstreamer.com is up to date. There are newer versions of the tutorials for the newer SDK builds from the GStreamer project at http://cgit.freedesktop.org/~slomo/gst-sdk-tutorials/

I don't expect it to be trivial But I want to make sure that I'm doing it the right way... V4L2/U4VL vs GStreamer vs raspivid based on the fork of RaspberrIPCamera etc...

Thanks for the link, will check the tutorials later this week.

[superrei]

Hi thaytan,

I'm interested in using the rpicamsrc, but I cannot add the package repository under http://vontaene.de/raspbian-updates/. Cause I can't find the public key for it. Would you share how can I fix this? Thanks.

[thaytan]

Run these commands as root to retrieve the GPG key the packages were signed with:

Code: Select all

apt-get install debian-keyring
gpg --keyserver pgp.mit.edu --recv-keys 0C667A3E
gpg --armor --export 0C667A3E | apt-key add -

[JPelletier]

Hi thaytan,

I'm interested in using the rpicamsrc, but I cannot add the package repository under http://vontaene.de/raspbian-updates/. Cause I can't find the public key for it. Would you share how can I fix this? Thanks.

If you have the latest Raspbian image update, you don't need the http://vontaene.de/raspbian-updates/ repo, GStreamer is already included in the latest image! All you have to do is to install Requirements and build everything, as explained in the readme.

To install GStreamer

Code: Select all

sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install gstreamer1.0

[thaytan]

Hi thaytan,

I'm interested in using the rpicamsrc, but I cannot add the package repository under http://vontaene.de/raspbian-updates/. Cause I can't find the public key for it. Would you share how can I fix this? Thanks.

If you have the latest Raspbian image update, you don't need the http://vontaene.de/raspbian-updates/ repo, GStreamer is already included in the latest image! All you have to do is to install Requirements and build everything, as explained in the readme.

Oh! Thanks - I missed when they rolled in GStreamer 1.2 to the Rasbian build.

[superrei]

I'm a bit confused.

I have been checking all these gst packages in Raspbian for a few days. In my knowledge the "official" packages of Gstreamer1.0 under Raspiban are on 1.2.0 version. Which cannot support to build gst-rtsp-server 1.2.3. That's why I need the newer version (1.2.4) from vontaene.de.

Or, did I miss something?

[towolf]

Another thing you can do is compile gst-rtsp-server, and use the examples/test-launch utility to make your camera available as an RTSP stream:

./examples/test-launch "( rpicamsrc bitrate=3000000 hflip=true vflip=true ! video/x-h264,width=1280,height=720,framerate=15/1 ! h264parse ! rtph264pay config-interval=5 pt=96 name=pay0 )"

Then, you can connect using any RTSP client on Windows at the URL 'rtsp://192.1.whatever:8554/test'

VLC, or 'gst-launch-1.0 playbin uri=rtsp://ip:8554/test' should work as clients.

I looked at gst rtsp server and compiled it (n.b. the git master version wanted Gstreamer 1.5.x libs, so I edited configure.ac then it worked with the older version). I also edited test-launch.c for my requirements and put the pipeline into the code itself.

My problem is, with the regular ffmpeg based players, or mpv, or MX Player on Android, they all buffer too much. I can set latency=100 on the receiving end to make it close-to-realtime:

gst-launch-1.0 rtspsrc latency=100 location=rtsp://paitu/video ! rtph264depay ! h264parse ! avdec_h264 ! autovideosink

But playing with playbin or ffplay, or mplayer/mpv or MX Player, they all give ~ 2 seconds delay.

Is there any way, on the serving end to tell clients to not buffer so much?

My end-game is close-to-realtime video stream on Android. With WebRTC it would be possible, but stupid Google is still clinging to banning H.264 from WebRTC in Chrome. And somehow they removed rtsp:// protocol from HTML5 <video> element.

[superrei]

That's bad news for me.

I've been trying to use Android to playback video stream sending from rtsp-server running in RPi. I wish to use the VideoView in Android since I really don't want to use Android NDK at this moment. (Google is promoting Android Studio but it doesn't work with NDK . What are the guys in Google thinking?)

Now the nearest result I can get is sending a videotestsrc to vlc with gst-rtsp-server, but if I change the pipeline to send anything else. VLC refuse to play. I'm using something like

• ./examples/test-launch "( rpicamsrc bitrate=3000000 hflip=true vflip=true ! video/x-h264,width=1280,height=720,framerate=15/1 ! h264parse ! rtph264pay config-interval=5 pt=96 name=pay0 )"

It doesn't work even with smaller resolution and less frame rate.

I'm still looking for solution.

[thaytan]

That's bad news for me.

I've been trying to use Android to playback video stream sending from rtsp-server running in RPi. I wish to use the VideoView in Android since I really don't want to use Android NDK at this moment. (Google is promoting Android Studio but it doesn't work with NDK . What are the guys in Google thinking?)

Now the nearest result I can get is sending a videotestsrc to vlc with gst-rtsp-server, but if I change the pipeline to send anything else. VLC refuse to play. I'm using something like

• ./examples/test-launch "( rpicamsrc bitrate=3000000 hflip=true vflip=true ! video/x-h264,width=1280,height=720,framerate=15/1 ! h264parse ! rtph264pay config-interval=5 pt=96 name=pay0 )"

It doesn't work even with smaller resolution and less frame rate.

Make sure you're using the latest gst-rpicamsrc, and try with baseline profile:

Code: Select all

./examples/test-launch "( rpicamsrc bitrate=3000000 hflip=true vflip=true ! video/x-h264,width=1280,height=720,framerate=15/1,profile=baseline ! h264parse ! rtph264pay config-interval=5 pt=96 name=pay0 )"

[superrei]

Using baseline profile doesn't help. Though I know it is required on Android.

[RpiName]

That's bad news for me.

I've been trying to use Android to playback video stream sending from rtsp-server running in RPi. I wish to use the VideoView in Android since I really don't want to use Android NDK at this moment. (Google is promoting Android Studio but it doesn't work with NDK . What are the guys in Google thinking?)

I'm still looking for solution.

One solution is here:
http://www.linux-projects.org/modules/n ... toryid=174

[JPelletier]

That's bad news for me.

I've been trying to use Android to playback video stream sending from rtsp-server running in RPi. I wish to use the VideoView in Android since I really don't want to use Android NDK at this moment. (Google is promoting Android Studio but it doesn't work with NDK . What are the guys in Google thinking?)

I'm still looking for solution.

One solution is here:
http://www.linux-projects.org/modules/n ... toryid=174

Really interesting! Thanks for the hint!

[thaytan]

Here's some info on how to do NDK with Android Studio:

http://www.shaneenishry.com/blog/2014/0 ... id-studio/

If it were me (a GStreamer dev), I'd use the GStreamer SDK for android, of course. It plays back RTSP fine.

http://cgit.freedesktop.org/~slomo/gst- ... tutorial-5 has a GStreamer SDK example that will play any supported URL into a SurfaceView.

[superrei]

Just did it on Eclipse. Not very straight forward. See how I did it here: http://javaappdesign.blogspot.hk/2015/0 ... -with.html.

I don't know why I need to define NULL myself. Any idea?

[RpiName]

I am attempting to stream my PiCam on my Pi2 to a Windows 7 machine on the same WiFi network.
As low latency is a must

Give a try to UV4L. It works "out of the box" with the standard HTML5 WebRTC, Video & Audio, to Android, Chrome or other WebRTC-capable browsers. The delay is ~200ms. No configuration required. Apart from WebRTC, it also supports streaming in MJPEG, JPEG, H264, YUV over HTTP.

http://www.linux-projects.org/modules/s ... e&artid=14