Torkel
Posts: 27
Joined: Thu Apr 16, 2015 8:43 pm

Pi-based VR Camera + Telepresence Robot (working project)

Thu Mar 03, 2016 12:05 pm

I made a 360° camera based on the raspberry pi 2 (6 of them actually), and now also a VR telepresence robot. I've gotten help from this forum so I wanted to share the result:
Image
The camera has 6 Raspberry Pi 2 boards and long cables to 6 sensor boards mounted on a small cube. Almost everything on both the camera and robot is 3D-printed. The electronics for the robot is based off an arduino which reads "pwm" from the rc receiver and controls the stepper motors. (I want to move to raspberry there too, so it can be controlled from another city/continent but haven't had the time..) Here is a short film from last night when I showed the robot at the local makerspace here in Linköping: https://www.youtube.com/watch?v=bwLuCmzhrvs
Image
Image
Maybe it's best that I add that I work at the startup doing the live video processing software that runs on the computer. So building this has been both an "8-5" and a "5-12"-time thing for me :)

start78
Posts: 43
Joined: Mon May 23, 2016 8:02 am
Location: Germany

Re: Pi-based VR Camera + Telepresence Robot (working project

Fri Aug 12, 2016 1:45 pm

Would it be possible to use this for
a: record 360°-videos for YouTube and Facebook?
b: lifestream 360° to YouTube and Facebook?

Does the stitching software run on an external PC? What are the system requirements for the software?

I work for a local tv station in germany and i have a hard time convincing my boss to buy a 360° rig. Your system seems to be rather affordable...

Torkel
Posts: 27
Joined: Thu Apr 16, 2015 8:43 pm

Re: Pi-based VR Camera + Telepresence Robot (working project

Sat Aug 13, 2016 5:35 am

Would it be possible to use this for
a: record 360°-videos for YouTube and Facebook?
Yes

b: lifestream 360° to YouTube and Facebook?
Yes

Does the stitching software run on an external PC?
Yes

What are the system requirements for the software?
For streaming/recording it needs an Nvidia GPU (we use the NVENC API for encoding video), or a Decklink card if you want SDI out and do it yourself off that. Just watching needs almost nothing - our software is very efficient.

The startup doing this now, that I'm a co-founder of, sells the software to broadcasters for professional use. It supports any camera rig (almost). Send me an email at torkel@voysys.se if it sounds interesting :)

But that said - if you just want to get started playing around with 360 video then I'd recommend you to get one of the new small consumer cameras that are coming out. That's a very good starting point - and even cheaper than a DIY solution actually.

start78
Posts: 43
Joined: Mon May 23, 2016 8:02 am
Location: Germany

Re: Pi-based VR Camera + Telepresence Robot (working project

Sat Aug 13, 2016 2:11 pm

Most consumer 360° cameras lack in resolution. We planned buying a cube rig for six GoPros, lease some GoPros and use a test licence of the needed stitching software for some first tests before we would have bought all the equipment needed.

Planned combined final costs would have been 4000€.

I already found the homepage of your startup, where you estimated the costs of your solution at about 2000$.

The problem would be the nvidia card, because for on-location-recording /-streaming we would need a laptop, right? Our laptops don't have nvidia cards. We only got one rather old i5 with who-knows-what-kind-of old mobile nvidia chip. I doubt that thing would do the trick!?

Torkel
Posts: 27
Joined: Thu Apr 16, 2015 8:43 pm

Re: Pi-based VR Camera + Telepresence Robot (working project

Sat Aug 13, 2016 3:51 pm

Might work on that laptop though... our SW is very efficient. But also might not work - hard to tell. Decode of h264 streams is CPU intensive.

This solution (Cube we call it) is not very good in image quality either. The raspberry pi cameras have "automatic lens shading" which messes them up.

If you want higher resolution then you will want eg 3 blackmagic micro studio cameras. We make a rig for them. Email me if you want info on that.

We made the Cube as a development kit for industrial customers. They can put it on a machine or something and then remote control that thing from VR.

start78
Posts: 43
Joined: Mon May 23, 2016 8:02 am
Location: Germany

Re: Pi-based VR Camera + Telepresence Robot (working project

Sat Aug 13, 2016 4:25 pm

The Blackmail rig would be more expensive in the end than the Gopro solution, wouldn't it?

Main reason we still don't have a 360°-system are the costs.

Life stitching isn't really necessary for us but it would have been a nice touch, given that your system is even cheaper than the solution we are focused at right now.

And a life 360°- stream is still rare. Maybe the image quality is not such a big issue. And maybe we can sell that to our customers (as a small local tv station we try to make additional money with every kind of production possible).

The pro of the Raspberry cams is that you can build a very small cube which minimizes parallax-errors and makes the stitching process a bit easier and the result cleaner. Am I right?

Torkel
Posts: 27
Joined: Thu Apr 16, 2015 8:43 pm

Re: Pi-based VR Camera + Telepresence Robot (working project

Sat Aug 13, 2016 5:21 pm

Yes, a Blackmagic solution is more expensive. It saves a lot of time because it's live (and live-to-tape) so no lengthy post stitching.

Yes, Cube had minimal parallax. But our software is the best in the market at handling parallax so that's not the biggest plus :)

Maybe a two-camera Kodak SP360 solution could be right for you? That's two 4K images so you get ok resolution and stitch is easier.

Then if you want to live stream that you can contact me and our software could be what you want for that.

sorbonne
Posts: 49
Joined: Thu Jan 14, 2016 10:25 am

Re: Pi-based VR Camera + Telepresence Robot (working project

Sat Aug 13, 2016 10:25 pm

what's the resolution when live streaming over internet and what is the delay? is there audio? can the telepresence be bidirectional (send and receive at the same time)?

Torkel
Posts: 27
Joined: Thu Apr 16, 2015 8:43 pm

Re: Pi-based VR Camera + Telepresence Robot (working project

Sun Aug 14, 2016 12:46 am

Resolution is arbitrary (up to 8K I think if you use h265). But that's for the equrectangular. Really, if you go above 4K it will not inprove anything because the cameras will limit it then. (Each camera is 1440x1080 at 42fps)

Delay is 150ms or so locally from photon hitting camera to photon hitting eye in oculus rift. If your internet is no good then it will make things worse.

We can do audio. Haven't had time to play with that enough yet though so nothing fancy like noise cancelling..

Why do you want bidirectional? Can be done, but just go to fully virtual environment then ;)

sorbonne
Posts: 49
Joined: Thu Jan 14, 2016 10:25 am

Re: Pi-based VR Camera + Telepresence Robot (working project

Sun Aug 14, 2016 12:10 pm

Torkel wrote:Why do you want bidirectional? Can be done, but just go to fully virtual environment then ;)
bidirectional (audio especially) so to possibly have more interaction between you and the remote audience. Anyway what is the "fully virtual environment"? can you give an example?

Can the streams be encrypted?

Torkel
Posts: 27
Joined: Thu Apr 16, 2015 8:43 pm

Re: Pi-based VR Camera + Telepresence Robot (working project

Sun Aug 14, 2016 4:58 pm

Ah, ok, that's what you meant by bidirectional. For the audio yes - that makes sense of course. I think I could integrate it. When I played around with it at the time of posting this though, I just used a mobile phone in speaker mode :)

By fully virtual I mean a synthetic, game-like virtual environment. If two people both have headsets then they are both virtual and thus there is no need for them to meet in the real world.

sorbonne
Posts: 49
Joined: Thu Jan 14, 2016 10:25 am

Re: Pi-based VR Camera + Telepresence Robot (working project

Sun Aug 14, 2016 5:58 pm

What about encryption of the streams? is it possible?

Also, out of my curiosity, why did you call this solution "VR" even if what you stream is a real room? Is there the possibility of also streaming really virtual environments?

Torkel
Posts: 27
Joined: Thu Apr 16, 2015 8:43 pm

Re: Pi-based VR Camera + Telepresence Robot (working project

Sun Aug 14, 2016 7:14 pm

I haven't tried encrypting it, but should not be too difficult to get that too in the pipeline. Tricky part about encryption is how to handle handshakes and keys.

It is VR in that the user experiencing it is in a virtual representation of the location around the camera. My latest developments in this field is looking at if I can make the virtual environment richer so the user can move around the head a bit and getting proper parallax (using light fields).

There are some companies in Silicon Valley looking at light field streaming of virtual scenes. Otoy is one, but I know at least one other startup that is funded but still in stealth.

RpiName
Posts: 717
Joined: Sat Jul 06, 2013 3:14 am

Re: Pi-based VR Camera + Telepresence Robot (working project

Mon Oct 24, 2016 2:43 pm

start78 wrote:I already found the homepage of your startup, where you estimated the costs of your solution at about 2000$.
Isn't it too much? A simpler and more effective solution can be done with $100 at most. I wonder how many people would be interested...

Davidlan
Posts: 1
Joined: Tue Nov 22, 2016 6:05 am

Re: Pi-based VR Camera + Telepresence Robot (working project

Tue Nov 22, 2016 6:10 am

Hi!

I have one question, what software you use for controlling the 360 degree camera and how to use?
Could you give me some guide?

Thanks a lot!!

RpiName
Posts: 717
Joined: Sat Jul 06, 2013 3:14 am

Re: Pi-based VR Camera + Telepresence Robot (working project

Sat Jan 28, 2017 9:08 pm

DEMO for those curious about telepresence robots:

https://youtu.be/-SMKos1NxQI

Return to “Automation, sensing and robotics”