Baby monitor with sound, video and more

[vadim]

Ladies and gentlemen, meet Nico, a RPi-based baby monitor. Making it was a great learning experience and I hope it will inspire some other expecting parents

http://oi40.tinypic.com/2dtn6fo.jpg
Nico was made to monitor a sleeping baby while parents work in other rooms or cook dinner in the kitchen. It was requested that no radio transmitters go anywhere near baby, Macs used by parents and an old iPod Touch could be used as monitoring devices and the monitor would calm the baby down while parents rush to the nursery.
Everything is fitted inside a standard size RPi case. I hope other RPi developers will find this topic interesting and share some useful hints and ideas

Hardware modification:
1) P5 was populated with sockets for pin connectors
2) An electret mic and an opamp-based preamp was built on a PCB an fitted between USB and composite output. The mic capsule is behind the nose. Free spaces above LEDs and between RCA and audio socket are occupied by capacitors, the opamp resides above the SoC. The board is powered by RPi's 5V, which is very noisy and unstable so I actually had to use quite large electrolytic capacitors.
3) WM8783 ADC (the only SMD component in the project) on an adapter PCB was connected to P5 and the preamp. The master clock is fed from GPCLK0 pin on P1.
4) Camera board was fitted as Nico's left eye.
5) A shutdown button (red, seen in the top view below) was added behind Nico's ear above micro-USB
6) After unsuccessful attempts to unsolder the RCA socket it was decided to be used as the base for tripod mounting. Hanging on a wall will be possible as well (haven't decided yet what mounting to use).
7) UTP cable was modified to carry both Ethernet and 5V power. Thus no twisted pairs are wasted and only one cable is visible.
8) Nico's happy face was drawn and attached to the front thus hiding most of the hardware. Perhaps the SD card will be substituted by a micro-SD adapter that would disappear from sight.


Software part:
1) Nico runs on Raspbian. Move to 3.10 kernel is planned later this year.
2) I built the kernel with a modified I2S driver and a "soundcard" driver that supplies the required master clock from GPCLK0. The maximum usable sample rate is 32000 but that is sufficient.
3) There are two streaming options that I am satisfied with: udp multicast of mpeg TS stream and HLS. Most probably, both will be used - udp multicast to VLC on Macs and HLS will be served to the iPod by nginx when required. ffmpeg captures audio, encodes it to aac, combines it with the video stream from the camera and sends udp packets and/or HLS files. ffmpeg's HLS encoder was slightly modified to work as a live broadcast with bearable latency of 3-4 seconds.
4) A Python script starts at boot and waits for the shutdown button to be pressed, then it, obviously, shuts the system down.
5) Control and monitoring of RPi to be performed using a modified version of RaspCTL. HLS web page is yet to be integrated into this easy to use interface.

Personal touch:
1) Heartbeat and voice of the mother as well as some lullabies performed by the father will be recorded. These sounds can be output to a set of small portable speakers using RaspCTL's functionality embedded in the web interface.

This project wouldn't be possible without this community. Big thanks to the foundation people, Andy Armstrong for psips, Florian Meier for I2S driver, Jan Carreras Prat for RaspCTL.

This topic is not intended to be a step-by-step how-to. However, I will answer specific questions and constructive suggestions

[SeNS]

Hi, nice work, congrats!

Could you please post your scripts and software used for HLS streaming? Also, does your solution used a GPU power for the streaming?

Thanks a lot,

WBR, SeNS

[vadim]

I will direct you to Andy Armstrong's github page with examples and sources.
https://github.com/AndyA/psips
This works 100%.
Video encoding is done in GPU by raspivid.
ffmpeg has it's own implementation of HLS, however it is tailored for video-on-demand. I did some modifications to hlsenc.c and that allowed me to use HLS with nginx for live translation. Unless you do the same my scripts won't be useful.
Please be aware that HLS in general is high latency technology. You can reduce it by reducing segment size to 1 second (my playlist actually is just two segments of 1 second). Another optimisation is to run ffmpeg first and put it in background and only then to start filling in the fifo. Worked for me. Obviously then you will need some way of killing ffmpeg on exit.

PM me if you experience any problems with my suggestion.

[vadim]

http://www.raspberrypi.org/phpBB3/viewt ... 28#p411628
Detailed guide on building compatible ffmpeg from sources. Let it run overnight while you sleep

[vadim]

Nico was officially renamed to George. Oh, those women...

[SeNS]

Thanks, guys, for your feedback. After small research, I decided to keep my current solution - MJPEG streaming from my robot camera to the control software on Windows Phone (actually I don't have RPi cam but small Logitech webcam with microphone - works pretty fine).

HLS is cool but have some significant cons: has a big latency for the live streaming and resource intensive (CPU and battery).

Currently I'm using "M-JPEG streamer" app but will be good to give GPU some work... Could you recommend some RPi specific MJPEG GPU-based streaming solution (I prefer C/C++ code)?

[vadim]

your camera most probably already does jpg compression. what do you need GPU for then?

[SeNS]

your camera most probably already does jpg compression. what do you need GPU for then?

Truly, I don't know. I do have a Logitech C110 webcam and (as I said before) I'm using M-JPEG streamer software. Could you tell how that solution does jpeg compression?

[vadim]

I made a wrong assumption that you use RasPi camera when recommending HLS solution.
I am 99% sure that the cam you use produces jpg frames that are further streamed by mjpg_streamer using input_uvc.so driver. (BTW, it was the first proof of concept for the camera I made). There is no image manipulation in software, nothing to be accelerated by GPU.

[Eugene]

May be you'll find it useful. But here is what we did for similar project (no need in any custom software modification):


Quick Setup Overview

USB Web cam Logitech B500 connected to Reaspberry Pi. But any camera will work. You can check wiki if your cam need extra power supply.

Network is either Ethernet or WiFi. Ethernet is preferable as Web cam will use USB heavily.

ffserver will stream your video over the network.
ffmpeg will prepare your video and sound streams.

Several Linux versions are now switching to avconv instead of ffmpeg. But they are mostly compatible.


What is your Web cam capable of?

Normally your USB web cam will support 2 output streams:
- YUYU (raw video capture)
- MJPEG (compressed images capture)

To see you Web cam supported modes you can use video4linux2 control utility:

Code: Select all

v4l2-ctl --list-formats

Advanced web cams, like Logitech C920 has built-in hardware support for h.264. See later why it is useful.

Web Cam works in one of the predefined modes with fixed resolution and FPS (frames per second). You can find all supported modes as the following:

Code: Select all

v4l2-ctl --list-formats-ext

You will need to know them later to get best performance. If you use "non-standard" resolution of FPS, then RPi CPU is used to adjust stream. And we need all our CPU for compression.



Network streaming quick overview

Streaming pipeline is simply the following:

1.V) Web cam ( /dev/video0 ) sends raw video stream, MJPEG stream (or h.264 stream) to the driver (video4linux2).
1.A) Audio device sends audio stream to audio driver.

2) ffmpeg grabs this video and audio data, combines them, and feeds final raw stream to the ffserver

3) ffserver accepts connections from clients and streams them media content, converting it to the desired format "on the fly" (with another ffmpeg instance)


If you choose to stream MJPEG to the client, all is simple. But it uses network heavily. STILL THIS IS MUST HAVE TO DO, in order to test your whole setup. At least you would know that everything works.

To reduce network traffic, we want to use video stream with compression. Many options are available here. Yet most commonly used are mpeg1, theora (ogg), and h.264. The last one is now unofficial web standard and has hardware support almost everywhere.

Yet there is one problem with video stream. ffmpeg (and all other tools as of today) uses software compression. What is something very slow on Raspberry Pi. With decent resolution 320x240 you won't get more than 2-3 frames per second. And you want to have at least 25 (better 30) for stable view over the network.

Good news, RPi has unlocked GPU support for h.264 encoding. And I'm currently working on wrapping GPU-based encoder/decoder omxtx for ffmpeg. This will give 30 fps even on top resolutions.

Another option is to use Web cam, that has built-in hardware h.264 encoder. Logitech C920 is an example.



MJPEG setup

MJPEG stream is a set of compressed images. Relatively heavy traffic usage. BUT, very simple to setup and supported with almost any client. Drawback is that we do not have audio (but workarounds are possible). Still used in all security surveillance projects due its simplicity.

Step 1. Test your web cam stream:

Code: Select all

ffmpeg -f video4linux2 -r 15 -s 352x288 -vcodec mjmpeg -i /dev/video0 -an /tmp/wcam.avi 

IMPORTANT: All the time you work with Web Cam, you should explicitly limit framerate with the -r parameter. Othervise ffmpeg will read data from your Web cam as fast as it can, and will fail, because data is not ready yet.

FrameRate (-r) and resolution (-s) is better to match one of your Web Cam native modes (discussed earlier).


Step 2. Prepare ffserver configuration

By default ffserver will look for /etc/ffserver.conf configuration file. Here is a simple setup for MJPEG stream:

Code: Select all

Port 8090
BindAddress 0.0.0.0
MaxClients 5
MaxBandwidth 1000
NoDaemon

<Feed wcam.ffm>
	File /tmp/wcam.ffm
	FileMaxSize 5M
	ACL allow 127.0.0.1
</Feed>

<Stream wcam.mjpeg>
	Format mjpeg
	Feed wcam.ffm

	VideoFrameRate 15
	VideoBitRate 500
	VideoSize 352x288
	VideoQMin 1
	VideoQMax 10

	NoAudio
</Stream>

Now we can start ffserver simply with the command bellow. Explicit configuration file path is not necessary, but just in case you want to place it somewhere else.

Code: Select all

ffserver -f /etc/ffserver.conf

It is now running and streaming... black matter.


Step 3. Feed Web Cam stream to ffserver

Code: Select all

ffmpeg -f video4linux2 -r 15 -s 352x288 -vcodec mjmpeg -i /dev/video0 -an http://localhost:8090/wcam.ffm

Step 4. Enjoy your view

Using VLC you can simply open your stream by the URL:

Code: Select all

http://<raspberry pi IP>:8090/wcam.mjpeg

There are also mobile applications available that can view such streams. And another option is preview it in your web browser (Chrome does support it):

Code: Select all

<html><head></head><body>
  <img src="http://<raspberry pi IP>:8090/wcam.mjpeg" />
</body></html>

Now if you want to add audio, you can do some tricks. MJPEG is container-less stream. E.g. just raw video data. In order to add sound you need to wrap it into AVI or some other format. Yes it would become a video stream, but without compression.

One more option would be to stream audio totally separately, and merge it with video on the client. But this approach is not recommended, because audio will be out of sync with picture.


Using video stream

Video stream has 3 parts:
- Video
- Audio
- Container

Container holds meta-information and helps to keep audio and video synced. It also includes key-frames that are necessary to rebuild compressed video and audio streams.

Audio and Video streams are captured from your hardware. If hardware does not compress (encode) them, than ffmpeg has to do it. Yet it only supports software compression, and RPi CPU will be a huge bottleneck here. It is easy to check if you try to grab raw video with 30 fps, and then repeat it, but with h.264 compression with 2 fps and 100% CPU utilization, and even less fps with sound.


Step 1. Grab video and audio streams

Same as with MJPEG, but use different Web Cam mode instead:

Code: Select all

ffmpeg -f video4linux2 -r 2 -s 320x240 -i /dev/video0 -f alsa -ac 1 -i hw:0 /tmp/wcam.avi

IMPORTANT part here is a Frame Rate. You can set it up to 30 (or whatever is the best one your Web Cam can support). BUT. ffmpeg won't be able to process them faster than it encodes. And it would encode 2-3 frames per second at most (unless GPU codec would be available in nearest future).

What would happen if we put "-r 30" for capturing? Well, it would capture all the data and feed it to ffserver buffer in a raw format. But all these frames are actually saved to the disk buffer, until they are streamed to the client. And when we would start streaming, we won't be able to give away more than 2-3 frames per second due to compression speed. And you still getting new frames... and vuala, your buffer is now full, client is still getting images from about 10 minutes ago... and new frames are discarded...

So the lesson here is to try to capture data with the same speed as you give it away. With MJPEG we give it away fast. With compression (unless we have GPU codec) we have to use 2-3 frames per second (depends on resolution).


Step 2. Update ffserver stream configuration

Unfortunately I didn't save my h.246 stream configuration, but it would look something like this:

Code: Select all

Port 8090
BindAddress 0.0.0.0
MaxClients 5
MaxBandwidth 1000
NoDaemon

<Feed wcam.ffm>
	File /tmp/wcam.ffm
	FileMaxSize 5M
	ACL allow 127.0.0.1
</Feed>

<Stream wcam.mp4>
	Format mpeg4
	Feed wcam.ffm

	VideoCodec libx264
	VideoFrameRate 2
	VideoBitRate 512
	VideoSize 320x240
	AVOptionVideo crf 26
	AVOptionVideo preset medium
	AVOptionVideo flags +global_header

	AudioCodec aac
	Strict -2
	AudioBitRate 128
	AudioChannels 2
	AudioSampleRate 44100
	AVOptionAudio flags +global_header
</Stream>

IMPORTANT: there will be errors due to "preset medium". I google it hard and found the workaround... will update the post when find it again.

Try not to stream multiple compressed streams with RPi (ogg and h.264 at the same time). It has no enough CPU even for one stream.

Now start ffserver. You may need to kill old instance, if any.


Step 3. Feed Web Cam stream to ffserver and "enjoy" the view

Code: Select all

ffmpeg -f video4linux2 -r 15 -s 352x288 -vcodec mjmpeg -i /dev/video0 -f alsa -ac 1 -i hw:0 http://localhost:8090/wcam.ffm

You won't realy "enjoy" the video with 2 FPS. This is because your player will buffer the stream... and then show you the video with 25 FPS (everything will move very fast)... then buffer again for the next couple of minutes.


The idea here is to get prepared, use MJPEG stream for a now... and wait a little till RPi GPU codec for ffserver is ready.

Still nobody stops you from using uncompressed MJPEG as video stream, combine it with compressed audion, wrap it with simple AVI container... and stream. I was able to view my home Web Cam with 320x240 MJPEG at 15 FPS (native mode for my Web Cam) over the Internet from my office with no delays at all. Bigger resolution can be used in local home network.


Hope this helps.

[vadim]

Eugene, with all respect, these solutions don't offer anything better than SeNS already has.
I find ffserver useful only when there is need to provide many different streams, which RPi anyway is not capable of producing.
aac encoding on the fly is extremely heavy, I don't get the reason for encoding two channels (there is no stereo source) at 44.1Ksps (when the mic doesn't really capture frequencies above 12-13KHz). These should be reconsidered when an accelerated encoder is available. Until then there's no use.
This contains two mistakes:

Code: Select all

ffmpeg -f video4linux2 -r 15 -s 352x288 -vcodec mjmpeg -i /dev/video0 -an http://localhost:8090/wcam.ffm

Can you spot them?

[jamesth]

I've got a Logitech C920 and I'm really struggling to get this working.

I want to stream video and audio to devices on my network (as the topic suggests, a baby monitor), ideally web browsers. The pi will be headless and connected via ethernet, nothing else required.
The C920 is pretty decent quality and can output h264 directly, so my ignorant brain thinks it should be "trivial" to pass a relatively decent image and equally good audio straight through to a high-bandwidth/powerful client - I just can't figure out the best way of doing this, nor actually make it work!

I'm a novice, but using information in this thread (and 100 others) I'm currently able to get video to the raspberry pi itself.
I've Installed a minimal raspian base / v4l-utils / ffmpeg and can run :

Code: Select all

ffmpeg -f video4linux2 -s 640x480 -i /dev/video0 -f alsa -ac 2 -i hw:0 /tmp/ThisWorks.avi

I haven't managed to get ffserver to work at all with a multitude of .conf attempts.

I'm able to stream to a desktop machine running vlc using...vlc on the pi

Code: Select all

cvlc v4l2:///dev/video0:chroma=h264:width=640:height=480 --sout '#standard{access=http,mux=ts,dst=192.168.1.250:8080,name=stream,mime=video/ts}' -vvv

The downsides to this was that it seems a bit more 'heavy' - The delay both starting and viewing the stream was ~10 seconds (is this what I should expect?) and I also wasn't able to figure out how to add audio. I could also only stream to VLC. MPC-HC sometimes connected, sometimes didn't and all my browsers just wanted to download a file rather than play it.

If anyone could point me in the right direction it would be greatly appreciated.

[vadim]

Jamesth, if ffmpeg works for you stick with it. I assume you compile it from the sources.
You don't need ffserver, ffmpeg can easily stream or output files that a web server can further deliver to the clients. I think you first need to decide what clients to use and then their capabilities. In my case I have two solutions: multicast UDP streaming to vlc running on computers and HLS for cases when an iPod is used (because it's the only damn thing it can play properly). HLS works in web browsers on computers as well but there's a longer latency compared to UDP streaming. HLS requires a web server (nginx), encoding audio to AAC and patching ffmpeg encoder. try streaming to vlc on a computer by specifying udp://ip:port and open the stream in vlc. If that works fine then you can stream to multicast IP address (might require some work on the router). I'll post ffmpeg commands for both types of output someday later, probably this weekend.

[jamesth]

No, I haven't done any compiling/patching - Everything I've tried has been straight from raspbian's standard repository.

Both your usage scenarios match mine - My clients will be desktop machines with access to VLC etc and iOS devices.
I like the idea of doing everything in a web browser, but low latency and UDP also appeals (and the idea of looking into local multicasting interests me!).

I look forward to your your commands very much, thanks so much!

[vadim]

Code: Select all

/usr/bin/ffmpeg -y \
-analyzeduration 100000 \
-v debug -fflags +nobuffer \
-r 25.126 \
-i /home/pi/live.h264 \
-f alsa -ar 32000 -ac 2 -i hw:1,0 \
-c:v copy \
-strict -2 \
-c:a aac -b:a 64k -ac 1 \
-f mpegts udp://224.0.0.1:5000?ttl=4&pkt_size=1400

This is multicast udp stream. Obviously you will need to set your input sources and their parameters.
I would suggest to skip audio in your first try and change ip address to the client's ip. Then in VLC open the stream as udp://@:5000
If that works fine, add sound and encoding to aac (if your ffmpeg supports it). Then try multicast ip address (udp://@224.0.0.1:5000 in vlc)
I must warn you that wifi is very unreliable for udp streaming. Looks like my HLS setup is somehow broken, so I possibly need to reinstall everything. When I succeed I will post some guidelines of how to make it work properly.

This part is for my specific inputs:

Code: Select all

-r 25.126 \
-i /home/pi/live.h264 \
-f alsa -ar 32000 -ac 2 -i hw:1,0 \

Here I define copying video stream as is and encoding audio:

Code: Select all

-c:v copy \
-strict -2 \
-c:a aac -b:a 64k -ac 1 \

This is of most interest to you, jamesth - muxing and streaming. packet size and ttl are specific to my network topology, but should work with yours

Code: Select all

-f mpegts udp://224.0.0.1:5000?ttl=4&pkt_size=1400

[pixster]

Where can I get the Schematic an code to this project?
Don't feel like re-inventing the wheel!

[vadim]

Where can I get the Schematic an code to this project?
Don't feel like re-inventing the wheel!

The purpose of the project was to re-invent baby cameras in order to understand how the damn thing can be built. Since then there has been significant development in Raspbian and Pi peripherals market. I think there is no point in repeating my project as you can easily get mic modules from Adafruit or similar, even a sound card with instructions how to make it work. The code base used is already outdated so I cannot guarantee that patches would work on today's sources. All I can guarantee is that you can get the result faster and easier by doing it differently. Having said that I still can answer specific questions and give advice based on my frustration experience.

[PedroGil91]

Hi I´d like to know if i can use ffmpeg just for encoding audio in AAC
thanks

[vadim]

Hi I´d like to know if i can use ffmpeg just for encoding audio in AAC
thanks

compile ffmpeg with aac support and use "-c:a aac" in the output part of ffmpeg command. see my example above.

[sandyjmacdonald]

Could you say some more about how you got the WM8783 working?

[vadim]

The soundcard driver initialises and starts the appropriate master clock on GPCLK0 that drives the ADC. As the sample rate is fixed the settings for GPCLK0 are fixed as well. The code is just a copy-paste from I2S driver with the only change for the appropriate memory addresses for control registers. I didn't bother about stoping the clock because it's supposed to work until shut down. Sorry, I don't have access to the source code anymore.