quanghuynguyen
Posts: 6
Joined: Fri May 24, 2019 2:56 pm

Stereo Vision on Raspberry Pi 3B+

Fri May 24, 2019 3:04 pm

Hi every one
My name is Huy, a researcher in Sungkyuankwan University in South Korea.
I am currently working with stereo vision. The available platforms that I am using are Raspberry Pi 3B + and Pi Camera V2 and IVPort Version 2 ( camera multiplexer).
My question is, is there anyone succeeded using these combined platforms to do stereo visions using OpenCV.
It also good for me if anyone kindly shares relevant project. I am glad to hear from you and discuss together.
Thank you so much for your consideration.
Best,

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23380
Joined: Sat Jul 30, 2011 7:41 pm

Re: Stereo Vision on Raspberry Pi 3B+

Fri May 24, 2019 3:32 pm

You could use a Pi Compute module as that has two camera ports and the software available will already combine those two into a stereo pair image.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

quanghuynguyen
Posts: 6
Joined: Fri May 24, 2019 2:56 pm

Re: Stereo Vision on Raspberry Pi 3B+

Fri May 24, 2019 4:51 pm

Hi Jamesh
Thank you so much for your response. As I know at the moment, there are two solutions for my current problem.
I want to take a pair of images from two pi camera at the same time in order to do stereo vision.

1. The first solution is to use two separate Pi Compute board and 2 separate Pi camera for each board and to use GStreamer.
2. The second solution is to use "IVPort Raspberry Pi Camera Module Multiplexer".[https://ivmech.com/magaza/en/developmen ... lexer-p-90].

I do not know which one will be better, and would you mind give me a little more information or relevant links.
That would be a great pleasure for me.

Thank you so much for your consideration.

PiGraham
Posts: 3577
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Stereo Vision on Raspberry Pi 3B+

Fri May 24, 2019 5:49 pm

quanghuynguyen wrote:
Fri May 24, 2019 4:51 pm
As I know at the moment, there are two solutions for my current problem.
I want to take a pair of images from two pi camera at the same time in order to do stereo vision.

1. The first solution is to use two separate Pi Compute board and 2 separate Pi camera for each board and to use GStreamer.
2. The second solution is to use "IVPort Raspberry Pi Camera Module Multiplexer".[https://ivmech.com/magaza/en/developmen ... lexer-p-90].

If you use a Compute Module you only need one. You can connect two Pi Cameras to it.
There is a blog post from 2014 about doing that.
https://www.raspberrypi.org/blog/real-t ... te-module/

A multiplexer does not allow images to be captured from cameras at the same time. It takes some time to switch over. That may not be important to you, but ifthere is motion in the scenes you want to process investigate the switching time. I think it may be a couple of frame times, but I'm not sure.


There is a third option to consider. Use two RPi each with a camera. Network them together to get the two images of the pair together. You can't precisely synchronise the PI Cameras but you should get closer than with the multiplexer (less than one frame time).

A second Pi is quite a bit cheaper than one multiplexer.

User avatar
HermannSW
Posts: 1394
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Stereo Vision on Raspberry Pi 3B+

Fri May 24, 2019 11:27 pm

⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://gitlab.freedesktop.org/HermannSW/gst-template
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

quanghuynguyen
Posts: 6
Joined: Fri May 24, 2019 2:56 pm

Re: Stereo Vision on Raspberry Pi 3B+

Sat May 25, 2019 4:04 am

Hi PiGraham.
Hi HermannSW.
Thank you so much for your information.
Best, Hope we can share experience about this problem more regularly.

quanghuynguyen
Posts: 6
Joined: Fri May 24, 2019 2:56 pm

Re: Stereo Vision on Raspberry Pi 3B+

Sat May 25, 2019 4:48 am

PiGraham wrote:
Fri May 24, 2019 5:49 pm
quanghuynguyen wrote:
Fri May 24, 2019 4:51 pm
As I know at the moment, there are two solutions for my current problem.
I want to take a pair of images from two pi camera at the same time in order to do stereo vision.

1. The first solution is to use two separate Pi Compute board and 2 separate Pi camera for each board and to use GStreamer.
2. The second solution is to use "IVPort Raspberry Pi Camera Module Multiplexer".[https://ivmech.com/magaza/en/developmen ... lexer-p-90].

If you use a Compute Module you only need one. You can connect two Pi Cameras to it.
There is a blog post from 2014 about doing that.
https://www.raspberrypi.org/blog/real-t ... te-module/

A multiplexer does not allow images to be captured from cameras at the same time. It takes some time to switch over. That may not be important to you, but ifthere is motion in the scenes you want to process investigate the switching time. I think it may be a couple of frame times, but I'm not sure.


There is a third option to consider. Use two RPi each with a camera. Network them together to get the two images of the pair together. You can't precisely synchronise the PI Cameras but you should get closer than with the multiplexer (less than one frame time).

A second Pi is quite a bit cheaper than one multiplexer.
Actually, the stereo vision module I am trying to develop will finally be attached to the end-effector of a robot manipulator. That means, it is important to have frames captured precisely at the same time. So as you said, the best option for me at the moment is to use two RPi each with a camera. Then I can precisely synchronize the PI Cameras.

Would you please kindly suggest me any relevant objects or post that network two RPis each with a camera, since I am quite naive in this field. Besides, according to your experience, how many raw frames per second would the RPis will receive if I follow this method.

Thank you so much for your consideration and hope to hear from you soon.

PiGraham
Posts: 3577
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Stereo Vision on Raspberry Pi 3B+

Sat May 25, 2019 12:02 pm

quanghuynguyen wrote:
Sat May 25, 2019 4:48 am
PiGraham wrote:
Fri May 24, 2019 5:49 pm
quanghuynguyen wrote:
Fri May 24, 2019 4:51 pm
As I know at the moment, there are two solutions for my current problem.
I want to take a pair of images from two pi camera at the same time in order to do stereo vision.

1. The first solution is to use two separate Pi Compute board and 2 separate Pi camera for each board and to use GStreamer.
2. The second solution is to use "IVPort Raspberry Pi Camera Module Multiplexer".[https://ivmech.com/magaza/en/developmen ... lexer-p-90].

If you use a Compute Module you only need one. You can connect two Pi Cameras to it.
There is a blog post from 2014 about doing that.
https://www.raspberrypi.org/blog/real-t ... te-module/

A multiplexer does not allow images to be captured from cameras at the same time. It takes some time to switch over. That may not be important to you, but ifthere is motion in the scenes you want to process investigate the switching time. I think it may be a couple of frame times, but I'm not sure.


There is a third option to consider. Use two RPi each with a camera. Network them together to get the two images of the pair together. You can't precisely synchronise the PI Cameras but you should get closer than with the multiplexer (less than one frame time).

A second Pi is quite a bit cheaper than one multiplexer.
Actually, the stereo vision module I am trying to develop will finally be attached to the end-effector of a robot manipulator. That means, it is important to have frames captured precisely at the same time. So as you said, the best option for me at the moment is to use two RPi each with a camera. Then I can precisely synchronize the PI Cameras.

Would you please kindly suggest me any relevant objects or post that network two RPis each with a camera, since I am quite naive in this field. Besides, according to your experience, how many raw frames per second would the RPis will receive if I follow this method.

Thank you so much for your consideration and hope to hear from you soon.
If you need "frames captured precisely at the same time" then you need to look elsewhere, but you should define what precision you actually need. Certainly if you need synchronised pixel clocks and line and frame sync pulses you don't get that with Pi Cameras. But maybe you don't really need that. Maybe you can get good enough with nearly synchronous frames.

There is some discssion in this old thread
viewtopic.php?f=43&t=48238&hilit=sync

And this one describes a board for CM3 for stereo cameras.

viewtopic.php?f=37&t=216940&hilit=stereo

quanghuynguyen
Posts: 6
Joined: Fri May 24, 2019 2:56 pm

Re: Stereo Vision on Raspberry Pi 3B+

Sat May 25, 2019 1:25 pm

Hi PiGraham
Thank you so much for your time. It would be a treasure for me at the moment.
Hope to can discuss more in the very near future.
Best,

quanghuynguyen
Posts: 6
Joined: Fri May 24, 2019 2:56 pm

Re: Stereo Vision on Raspberry Pi 3B+

Sat May 25, 2019 4:24 pm

Hi Pigraham
Would you please suggest me other relevant post which focus more on software matter.
I am currently owning two Raspberry Pi 3 B+ board and two Pi Camera V2. Is it possible to networking them and do stereo vision with them according to your experience.
Thank you so much for any suggestion from you.
Best,

PiGraham
Posts: 3577
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Stereo Vision on Raspberry Pi 3B+

Sun May 26, 2019 1:41 pm

I gave you what I knew. This isn't something I have done.

PiGraham
Posts: 3577
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Stereo Vision on Raspberry Pi 3B+

Mon May 27, 2019 7:52 am

This could be relevant.
Maybe you could use RPi compatible ArduCam camera modules with external reset to synchronise two sensors.

viewtopic.php?f=43&t=241418&sid=b02a0a0 ... 24bb50728e

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7144
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo Vision on Raspberry Pi 3B+

Mon May 27, 2019 8:36 am

PiGraham wrote:
Mon May 27, 2019 7:52 am
This could be relevant.
Maybe you could use RPi compatible ArduCam camera modules with external reset to synchronise two sensors.

viewtopic.php?f=43&t=241418&sid=b02a0a0 ... 24bb50728e
Only if you also have the external strobe lighting required, otherwise the bottom of the image will be overexposed cf the top.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

PiGraham
Posts: 3577
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Stereo Vision on Raspberry Pi 3B+

Mon May 27, 2019 10:40 am

6by9 wrote:
Mon May 27, 2019 8:36 am
PiGraham wrote:
Mon May 27, 2019 7:52 am
This could be relevant.
Maybe you could use RPi compatible ArduCam camera modules with external reset to synchronise two sensors.

viewtopic.php?f=43&t=241418&sid=b02a0a0 ... 24bb50728e
Only if you also have the external strobe lighting required, otherwise the bottom of the image will be overexposed cf the top.
I'm assuming the critical requirement for 3d from stereo is synchronisation between L&R frames, not nice images, and both images should be the same.


Your comment from the other topic is very helpful:
6by9 wrote:
Tue Mar 12, 2019 10:28 am
HermannSW wrote:
Tue Mar 12, 2019 1:15 am
(capturing "global reset AND very short shutter time == global shutter"
No it doesn't.
Global reset and short readout time ~= global shutter.
In some ways global reset and very long exposure time is closer to global shutter, as the variance in exposure due to the readout time is significantly smaller than the configured exposure time.

You trigger a global reset and ALL lines start exposing.
At the end of the exposure time line 1 starts reading out. Lines 2-N keep exposing.
At exposure time + 1 line readout time, line 2 starts reading out. Lines 3-N keep exposing.
At exposure time + N * readout time line N has stopped exposing and completed readout.

If using a mode based on mode 7, then a full line takes 31us to read out. For the full 480 lines of readout you've therefore got 14ms of variation in exposure between first and last lines. Even down at 50 lines you've got 1.5ms of variation.
So if running at, say, 25fps (40ms/frame) the variation is 14/40 = 35%. That might be good enough to still allow the algorithm to find features to match.

Since I don't think we need to worry about latency, just synchronisation between sensors, then can't we just capture the second frame after FREX which will have even exposure on all lines?

User avatar
HermannSW
Posts: 1394
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Stereo Vision on Raspberry Pi 3B+

Wed Jun 26, 2019 5:57 pm

There is a new/interesting/surprising way to get dual camera support for the Pi3B+ (requires carefully de-soldering the main processor and LPDDR2 chip, no Pi clone):
http://www.arducam.com/shrink-your-rasp ... on-module/
Image
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://gitlab.freedesktop.org/HermannSW/gst-template
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

Return to “Camera board”