User avatar
Realizator
Posts: 31
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website Twitter

Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Thu Jun 28, 2018 9:47 am

Hello!
I’d like to share with you the details of my last project on Compute Module 3 for playing with stereo video and OpenCV. It could be interesting for those who study computer vision or make drones and robots.
Here is how it looks like with two cameras and Compute Module installed.
Image


It works with a stock Raspbian, you only need to put a dtblob.bin file to a boot partition for enabling second camera. It means you can use raspivid, raspistill and other traditional tools for work with pictures and video.

JFYI stereo mode supported in Raspbian from 2014, you can read implementation story on this forum here.

Before diving into the technical details let me show you some real work examples.

1. Capture image:

Code: Select all

raspistill -3d sbs -w 1280 -h 480 -o 1.jpg
and you get this:
Image
You can download original file here.


2. Capture video:

Code: Select all

raspivid -3d sbs -w 1280 -h 480 -o 1.h264
and you get this:
Image
You can download original captured video fragment (converted to mp4) here.

3. Using Python and OpenCV you can experiment with depth map:
Image

For this example I used slightly modified code from my previous project 3Dberry (https://github.com/realizator/3dberry-turorial)

I used this pair of cameras for taking the pictures in examples above:
Image

For video livestream from drone I use wideangle (160 degrees) cameras like this:
Image


Now – to hardware part.



Front view:
Image



Top view:
Image


Dimensions: 90x40 mm
Camera: 2 x CSI 15 lanes cable
GPIO: 40 classic Raspberry PI GPIO
USB: 2 x USB type A, 1 USB on a pins
Ethernet: RJ45
Storage: Micro SD (for CM3 Lite)
Monitor: HDMI out
Power: 5V DC
Supported Raspberry Pi: Raspberry Pi Compute Module 3, Raspberry Pi CM 3 Lite, Raspberry Pi CM 1
Supported cameras: Raspberry Pi camera OV5647, Raspberry Pi camera Sony IMX 237, HDMI In (single mode)
Firmware update: MicroUSB connector
Power switch: Yes! No more connect-disconnect MicroUSB cable for power reboot!

That’s all that I wanted to cover today. If you have any questions I will be glad to answer.
The site of our project is: http://stereopi.com
Last edited by Realizator on Mon Jul 30, 2018 8:20 am, edited 1 time in total.


JulesThuillier
Posts: 1
Joined: Fri Jul 27, 2018 10:33 am

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Fri Jul 27, 2018 10:38 am

So cool ! Congratulation !
That's the best CM3 based project I've seen so far, you really managed to keep everything very compact.
Are you planning on selling those boards or releasing the schematics ?
I was wondering one thing, why did you not integrate the cameras directly on the board ? That would be cheaper I think.

Cheers

User avatar
Realizator
Posts: 31
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website Twitter

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Mon Jul 30, 2018 8:17 am

JulesThuillier wrote:
Fri Jul 27, 2018 10:38 am
So cool ! Congratulation !
That's the best CM3 based project I've seen so far, you really managed to keep everything very compact.
Are you planning on selling those boards or releasing the schematics ?
I was wondering one thing, why did you not integrate the cameras directly on the board ? That would be cheaper I think.

Cheers
Yes, we plan to make hardware open-source, and also plan to make a batch for sale (now talking with crowdsupply.com)
As for cameras on the board - it will reduce board costs, but also reduce flexibility (like choosing different cameras and optics, and also stereobase size). If current edition of our board will be popular, we plan to develop several modifications including one with onboard sensors.

jholster
Posts: 10
Joined: Fri Nov 27, 2015 5:11 am

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Thu Dec 13, 2018 8:42 am

This is really cool project, I'll immediately order one when available. This is exactly what the foundation should have originally come up with, a compact CMIO board, usable as is for mobile (drones, etc) applications.

I've been building a fixed-wing uav project for a while, utilizign a stereo-video sent over befinitiv's wifibroadcast to virtual glasses. Technically it works, but the major blocker has been the lack of a compact / lightweight CM carrier board. The official CMIO board is just too bulky. Your project finally enables me to proceed!

A little wish: possibility to order the board without bulky full-size USB and RJ45 connectors, and GPIO header unsoldered.

User avatar
Realizator
Posts: 31
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website Twitter

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Thu Dec 13, 2018 12:18 pm

jholster wrote:
Thu Dec 13, 2018 8:42 am
This is really cool project, I'll immediately order one when available. This is exactly what the foundation should have originally come up with, a compact CMIO board, usable as is for mobile (drones, etc) applications.

I've been building a fixed-wing uav project for a while, utilizign a stereo-video sent over befinitiv's wifibroadcast to virtual glasses. Technically it works, but the major blocker has been the lack of a compact / lightweight CM carrier board. The official CMIO board is just too bulky. Your project finally enables me to proceed!

A little wish: possibility to order the board without bulky full-size USB and RJ45 connectors, and GPIO header unsoldered.
@jhosler, we use it with wifibroadcast too. It just works :-) In our case we restream it on the ground part to iPhone installed in plastic Google Board to obtain stereo.
As for your wish: version without soldered USB, Ethernet and gpio will be always available. We call it "slim" version, and plan to manufacture both "slim" and "regular" editions. It really saves a space, and it is very useful in our projects.

Since my last post in this topic we conducted a lot of experiments. For example livestream to Oculus Go, depth map in ROS, livestream 3D video to YouTube and so on. The last one was obtaining 360 photo panorama from two 200-degree cameras. Some experiments we put in our blog, and a brief list is on our website.

Also I'd like to mention that we received 20 first boards from factory chosen for the batch production.

Image

All samples passed all factory tests, now we are doing aggressive high-load tests to be sure everything is Ok. We plan to start crowdfunding soon, so if you're want to have one StereoPi - subscribe to our crowdfunding news here.

neilhendrick
Posts: 1
Joined: Wed Dec 26, 2018 10:01 pm

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Wed Dec 26, 2018 10:11 pm

Will this work on any other versions of the RPi? Is it necessary to use a compute module, or will an Rpi3A+ be fine?

User avatar
mahjongg
Forum Moderator
Forum Moderator
Posts: 12123
Joined: Sun Mar 11, 2012 12:19 am
Location: South Holland, The Netherlands

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Thu Dec 27, 2018 1:38 am

only the compute module has two camera interfaces, and supports stereo recording.

But other RPI’s should be able to do playback.

User avatar
Gavinmc42
Posts: 3622
Joined: Wed Aug 28, 2013 3:31 am

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Thu Dec 27, 2018 7:04 am

That is a really nice piece of hardware.
Khronos want to use the 3B+ as a OpenVX reference.
A stereo version could be done on the CM dev but this one looks much better for robot and drone usage.

For VR/AR research this is great for Universities etc.
Wish you guys every success and hope we see it soon in Adafruit and other distributors stock.

Did make me wonder if RPT are working on new revs of the CM3's with the new metal package chip :D
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
Realizator
Posts: 31
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website Twitter

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Thu Dec 27, 2018 8:33 am

neilhendrick wrote:
Wed Dec 26, 2018 10:11 pm
Will this work on any other versions of the RPi? Is it necessary to use a compute module, or will an Rpi3A+ be fine?
neilhendrick, as mahjongg mentioned calssic RPi supports only 1 camera at a time.
When our project started about 4 years ago, my aim was to test if it is theoretically possible to work with stereoscopic picture on Raspberry Pi. I was afraid of Pi's CPU performance is too slow. For this experiment we used single camera and a mirror system to "split" picture, we call it "3Dberry".

There were two editions of optical splitters.

1st generation:
Image

2nd generation:
Image

And I was able to build such a depth map:

Image

It was a good project for proof-of-concept and it shows us that stereoscopic processing on Raspberry Pi is possible. So we decided to make StereoPi.

Now 3Dberry project is open source, you can find drawings on http://3dberry.org, and lessons codes on a gihub https://github.com/realizator/3dberry-turorial
Please notice we did not updated this repo, so some code may be out-of-date now (OpenCV and Python syntax has changed over these several years).

Let me highlight some technology limitations for 3Dberry:
- It works only with original Raspberry V1 and V2 cameras with narrow angle (~60 degree)
- Each camera use a half of FOV, so for depth map you have near 30 degree FOV
- Plastic mirrors after installing have some bending and distortions not covered by undistortion logic in OpenCV (it is not like lens image distortions). So there are some issues with stereoscopic calibration and rectification.
- stereobase is a very small (~3cm), so it really works on a small distances only (2-3 metres). At far distances stereoscopic effect disappears.

But it was enough for the first experiment, and also for making decision to start development of StereoPi hardware.
Last edited by Realizator on Thu Dec 27, 2018 8:51 am, edited 1 time in total.

User avatar
Realizator
Posts: 31
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website Twitter

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Thu Dec 27, 2018 8:49 am

Gavinmc42 wrote:
Thu Dec 27, 2018 7:04 am
That is a really nice piece of hardware.
Khronos want to use the 3B+ as a OpenVX reference.
A stereo version could be done on the CM dev but this one looks much better for robot and drone usage.

For VR/AR research this is great for Universities etc.
Wish you guys every success and hope we see it soon in Adafruit and other distributors stock.

Did make me wonder if RPT are working on new revs of the CM3's with the new metal package chip :D
Gavinmc42 thank you, we'll do our best to make a batch and to put StereoPi to Adafruit stock and to other resellers :-) Now a lot of things depends on our upcoming crowdfunding campaign.

As for new CM3 I'm curious too. In our case we felt some pain after upgrade from CM1 to CM3 because of not full compatibility. CM3 requires another power voltage. It forced us to make full redesign of PCB, but now we have switchable power option on StereoPi for work with CM1 and CM3. Hope CM[next] will not have such issues.

User avatar
Gavinmc42
Posts: 3622
Joined: Wed Aug 28, 2013 3:31 am

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Thu Dec 27, 2018 9:15 am

Someone should contact doe300 and see if he is interested in doing this.
viewtopic.php?f=67&t=229707

With a Khronos reference OpenVX software and stereo hardware, machine vision on Pi's is getting very close to mainstream.
OpenCV has been done on Pi's but it is a DIY sort of thing.
With stereo camera setup and normal plus NOIR camera with filter for ENDVI, drone crop cameras?
OpenVX plus NNEF = Realtime ENDVI?

Dyson's 360 eye camera on their robot vacuum is an application for a 3A+, wonder where they get that mirror lens?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
Realizator
Posts: 31
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website Twitter

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Thu Dec 27, 2018 10:05 am

Gavinmc42 wrote:
Thu Dec 27, 2018 9:15 am
Someone should contact doe300 and see if he is interested in doing this.
viewtopic.php?f=67&t=229707

With a Khronos reference OpenVX software and stereo hardware, machine vision on Pi's is getting very close to mainstream.
OpenCV has been done on Pi's but it is a DIY sort of thing.
With stereo camera setup and normal plus NOIR camera with filter for ENDVI, drone crop cameras?
OpenVX plus NNEF = Realtime ENDVI?

Dyson's 360 eye camera on their robot vacuum is an application for a 3A+, wonder where they get that mirror lens?
Wow... I found doe300 repos, and https://github.com/doe300/VC4CL looks very interesting. I also have some plans to use H264 Pi's encoder to accelerate depth map. I will add issue in this repo now to start discussion.
UPD> Done: https://github.com/doe300/VC4CL/issues/50

p.s. As for Dyson, IMHO they have not 360 degree. According to drawings and pictures they use one wideangle camera and works with semisphere. Even if they use 360 camera they are unable to see through the robot plastic case. We used similar 200-degree cameras for 360 panorama stitching.

User avatar
Gavinmc42
Posts: 3622
Joined: Wed Aug 28, 2013 3:31 am

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Thu Dec 27, 2018 12:06 pm

As for Dyson, IMHO they have not 360 degree
360 degrees in horizontal plane but maybe 45 in vertical plane?
More a panoramic image than real 360 in every direction, but that's all that is needed for their navigation method?

Depth map is interesting, I have used the older MS Kinect's for AR Sandboxes and Linux Mint box with a Nvidia GPU card.
https://arsandbox.ucdavis.edu/
Always wanted to try a desktop one made with Pi"s.
VC4CL is one way to accelerate, NEON is another, 4 ARM core's yet another.
Some combination?

Cameras are an issue, but hardware is used for jpeg/H264, we don't need jpeg, raw data would be preferred?
Raw data uses a horrible interleaved 8-10bit format.
Some guys having been playing with that and figuring it out.

I expect Khronos would prefer a 64bit reference design for OpenVX.
Can raw image data be got in 64bit mode?
My stuff is not C, I prefer to use Ultibo, just to make things simpler for me? - https://ultibo.org/

H264 is probably better for motion detection, but some combination for optical flow?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
Realizator
Posts: 31
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website Twitter

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Thu Jan 31, 2019 11:18 am

Just want to inform you that yesterday our crowdfunding campaign has been started: https://www.crowdsupply.com/virt2real/stereopi :-)

User avatar
Realizator
Posts: 31
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website Twitter

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Thu Feb 07, 2019 11:27 am

If you have classic Raspberry Pi with the camera, you can repeat all our video livestream experiments. Livestream to YouTube, Android and Oculus Go. Also you can repeat our behind-the-scene experiments with video livestream to WIndows desktop, Mac or any RTMP server.

Today we want to share with you our Raspbian image. We call it SLP (StereoPi Livestream Playground). It supports single-camera mode and also two-camera mode for StereoPi.
You can find image, Android application and brief manual in our Wiki here:
http://wiki.stereopi.com/index.php?titl ... bian_Image

gtoal
Posts: 111
Joined: Sun Nov 18, 2012 12:02 am

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Sat Feb 23, 2019 3:17 am

Two comments from me. 1) I jumped on this as soon as I saw it because I'm a stereo photographer from way back and thought this was going to be hardware-synchronised frame grabbing, especially for video - I was a little disappointed when I read that the cameras are shot sequentially with a 1ms delay between them. We'll just have to see how significant that is, for movies with a lot of motion or for high-speed stills photography. Unfortunately synching 2 cameras isn't enough for the project I really want to do, which is to synch 8 cameras and do real time 3D video using a 3D glasses-free display (the lenticular display from alioscopy.com) - if you can find a way to synch multiple modules, do add that some day!

2) Time to resurrect a project from a couple of years ago... I want to build a system with one regular camera and one IR camera to take superimposed images so they can be processed to false-colour infra-red reflecting surfaces, spot pollution etc. My plan for this is to use a semi-silvered mirror at 45 degrees and have the cameras in a 3d printed jig at 90 degrees apart, so that they are perfectly aligned. (No stereo separation wanted for this project)

User avatar
Realizator
Posts: 31
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website Twitter

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Mon Mar 04, 2019 2:32 pm

gtoal wrote:
Sat Feb 23, 2019 3:17 am
Two comments from me. 1) I jumped on this as soon as I saw it because I'm a stereo photographer from way back and thought this was going to be hardware-synchronised frame grabbing, especially for video - I was a little disappointed when I read that the cameras are shot sequentially with a 1ms delay between them. We'll just have to see how significant that is, for movies with a lot of motion or for high-speed stills photography.
@gtoal, I think for your described usage scenario you'll face other cameras limitations earlier than sync issues. For example, a rolling shutter for high-speed photos. You see, we are practitioners, not theoretics. So if our bakers asks us specific question about synchronization, we prefer to answer with the technical details, but not with advertisers style like "it's Ok". In our experiments we were unable to capture and measure this unsync, and it is very interesting for us to compare our results with other users. I just want to tell you that in practice you need to do a lot of work to force unsync issue and to measure it :-)
gtoal wrote:
Sat Feb 23, 2019 3:17 am
Unfortunately synching 2 cameras isn't enough for the project I really want to do, which is to synch 8 cameras and do real time 3D video using a 3D glasses-free display (the lenticular display from alioscopy.com) - if you can find a way to synch multiple modules, do add that some day!
8 cameras means at least 4 synchronized StereoPis. Yes, it's a challenge!

gtoal wrote:
Sat Feb 23, 2019 3:17 am
2) Time to resurrect a project from a couple of years ago... I want to build a system with one regular camera and one IR camera to take superimposed images so they can be processed to false-colour infra-red reflecting surfaces, spot pollution etc. My plan for this is to use a semi-silvered mirror at 45 degrees and have the cameras in a 3d printed jig at 90 degrees apart, so that they are perfectly aligned. (No stereo separation wanted for this project)
Are you plan to use mirror for cameras to have the same optical axe while looking at your object? I can suggest that this is critical for close objects, but for far objects you can use two cameras side-by-side with minimal stereo base (~25 mm).

User avatar
Realizator
Posts: 31
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website Twitter

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Tue Mar 05, 2019 10:43 am

We've updated our Raspbian image and want to share one of our experiments.
One of our new features is the ability to livestream MPEG-TS. We used this feature to livestream video from StereoPi to OBS (Open Broadcaster Software) with this OBS-VirtualCam plugin installed. OBS creates a virtual webcam accessible to Skype.
Image


Here’s a video of OBS settings: https://vimeo.com/320755542
And here is a screen capture of my iPhone and our first 3D Skype call: https://vimeo.com/320745829

As the iOS screen recorder does not record sounds, I added music to it.

To get all these things to work, we used two tricks. First, we used mic from Logitech webcam connected to the same computer to provide the audio to Skype, since the OBS-VirtualCam cannot emulate a sound device, so sound from StereoPi’s microphone isn’t available to Skype.

Second, we avoid a one-second delay between audio (due to an internal OBS video buffer) by first streaming the video from StereoPi to gstreamer on Windows and then pointing OBS to use the gstreamer window as its video source, which resulted in a delay of only about 100 milliseconds.

This test shows it is possible to use stereoscopic livestream with a lot of common software, like Skype and other video-related programs, all without modification since they already work with a traditional camera.

blimpyway
Posts: 148
Joined: Mon Mar 19, 2018 1:18 pm

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Tue Mar 05, 2019 10:33 pm

Nice.
What are usual frame rates for the depth map mode?
Or otherwise asked, what is the maximum resolution for a depth map updated 10fps?
Thanks

mkeys
Posts: 2
Joined: Tue Mar 12, 2019 6:02 pm

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Tue Mar 12, 2019 6:09 pm

Realizator wrote:
Tue Mar 05, 2019 10:43 am
To get all these things to work, we used two tricks. First, we used mic from Logitech webcam connected to the same computer to provide the audio to Skype, since the OBS-VirtualCam cannot emulate a sound device, so sound from StereoPi’s microphone isn’t available to Skype.

Second, we avoid a one-second delay between audio (due to an internal OBS video buffer) by first streaming the video from StereoPi to gstreamer on Windows and then pointing OBS to use the gstreamer window as its video source, which resulted in a delay of only about 100 milliseconds.

This test shows it is possible to use stereoscopic livestream with a lot of common software, like Skype and other video-related programs, all without modification since they already work with a traditional camera.
As a disclaimer I work for eZuce but I'm not trying to sell you anything.
You can export the video feeds directly from the RPi into a eZuce Vibe client and remove OBS all together, unless you just need obs for something. I used ffmpeg to do this with multiple video sources (including the desktop) on a rpi and can post examples, if desired.

--edit 16:39--
I had an idea to try and split a single video source (usb webcam) into two camera feeds for Vibe and was successful. I used v4l2loopback, then ffmpeg uses loopback devices as inputs. Screenshot: https://owncloud.ezuce.com/index.php/s/89cKtub4lXoYpCE

Regards,
Matt

User avatar
Realizator
Posts: 31
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website Twitter

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Wed Mar 13, 2019 7:39 am

blimpyway wrote:
Tue Mar 05, 2019 10:33 pm
Nice.
What are usual frame rates for the depth map mode?
Or otherwise asked, what is the maximum resolution for a depth map updated 10fps?
Thanks
We didn't provide peak performance tests. When we finish batch production, we will go back to this experiment.

User avatar
Realizator
Posts: 31
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website Twitter

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Wed Mar 13, 2019 7:42 am

mkeys wrote:
Tue Mar 12, 2019 6:09 pm
As a disclaimer I work for eZuce but I'm not trying to sell you anything.
You can export the video feeds directly from the RPi into a eZuce Vibe client and remove OBS all together, unless you just need obs for something. I used ffmpeg to do this with multiple video sources (including the desktop) on a rpi and can post examples, if desired.

--edit 16:39--
I had an idea to try and split a single video source (usb webcam) into two camera feeds for Vibe and was successful. I used v4l2loopback, then ffmpeg uses loopback devices as inputs. Screenshot: https://owncloud.ezuce.com/index.php/s/89cKtub4lXoYpCE

Regards,
Matt
Matt, the idea was to create virtual webcam on the computer, so user can use it with any software. At this moment I can not understand how can I use your solution for this. :roll:

mkeys
Posts: 2
Joined: Tue Mar 12, 2019 6:02 pm

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Wed Mar 13, 2019 12:48 pm

Realizator wrote:
Wed Mar 13, 2019 7:42 am
Matt, the idea was to create virtual webcam on the computer, so user can use it with any software. At this moment I can not understand how can I use your solution for this. :roll:
Ah, I'll try to clarify. The first suggestion was to address the latency mentioned. Vibe can consume the video feed(s) from the rpi without needing obs, yet still achieve a telephony enabled video conference. The second half of my post was to suggest that it seems you don't really need stereo cams to have a stereoscopic video chat. I was able to achieve the same result using a v4l2loopback from one camera source on the rpi (usb webcam), and feeding my vibe client on a remote machine two camera feeds; one from each loopback device on the rpi.

Regards,
Matt

nirvenesh20
Posts: 1
Joined: Sun Mar 24, 2019 1:27 pm

Re: Stereo camera with CM3 inside for OpenCV learners, drones and robotics

Sun Mar 24, 2019 1:33 pm

I've been very much intrested with the stereopi, but was just curious, if the stereo camera requires any GPIO jumpers to be connected. Its a concern as i plan to attach a 3.5 inch touch display (something like this https://www.cytron.io/p-3.5-inches-tft ... 0iEALw_wcB ) to the stereopi, and it wouldn't be possible if there's need for GPIO connectors for the cameras

Return to “Automation, sensing and robotics”