rspt1908
Posts: 12
Joined: Tue Mar 08, 2016 3:26 pm

Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 4:28 am

Good day,

I am currently working on an obstacle avoiding quadcopter using stereo vision to obtain depth maps. I noticed that the quadcopter would sometimes not steer to the correct direction.

I am using the Raspberry Pi Compute Module IO board which comes with two CSI ports used with two v1 Pi Cameras.

Issue

I soon found out that due to the latency between the cameras, the left and the right images are not in sync thus the errors in the depth map result.

Steps taken

I noticed the image blur when moving the cameras around so I adjusted the shutter speed by setting the UV4l/raspicam driver. With the shutter speed, I also tried to increase the framerate as I've read, it improves the latency issue. In my code which uses the opencv library, I used the grab() and retrieve() commands to replace the read() function so that the frames from both cameras is grabbed at the nearest time possible however it didn't help much :( I also tried using UV4L's timeout parameter on one camera however with no improvement :(

UV4L (Settings for both cameras)
-Framerate set is 30fps
-Shutterspeed is 1500us
-Buffer is set to 5

Does anyone know any possible solutions? Thank you :)

RpiName
Posts: 728
Joined: Sat Jul 06, 2013 3:14 am

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 7:35 am

Run one instance of uv4l with the images placed side by side, see the --side_by_side driver option

rspt1908
Posts: 12
Joined: Tue Mar 08, 2016 3:26 pm

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 8:19 am

I need the images obtained from the two cameras to be separate. As I am also rectifying images from the two cameras based on their respective camera calibration parameters

rspt1908
Posts: 12
Joined: Tue Mar 08, 2016 3:26 pm

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 8:20 am

I need to treat each camera as two separate devices, is there any way to synchronize them? :)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10851
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 8:51 am

rspt1908 wrote:I need to treat each camera as two separate devices, is there any way to synchronize them? :)
https://github.com/waveform80/picamera/pull/279 and viewtopic.php?f=43&t=48238&start=75

You have buffer timestamps taken from a common clock. You have a way of adjusting the framerate dynamically to approx 1/256fps accuracy. Userspace can make a control loop to sync the two together.
The PR implemented this on PiCamera 1.10 and was going to be merged. However I'm not sure whether it finally made 1.11 as waveform80 had to juggling a load of things around it. The ideas should all still work though.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

rspt1908
Posts: 12
Joined: Tue Mar 08, 2016 3:26 pm

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 9:59 am

Then it is only possible to do the adjustments using the MMAL driver and using python? Is this compatible with openCV? I am sorry I quite at a loss.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10851
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 10:15 am

rspt1908 wrote:Then it is only possible to do the adjustments using the MMAL driver and using python? Is this compatible with openCV? I am sorry I quite at a loss.
OpenCV will use one of a number of potential mechanisms to get images from the camera. Either:
- V4L2 (or UV4L2 which also implements the V4L2 API), which then uses MMAL.
- the PiCamera Python library, which then uses MMAL
- directly to MMAL.
- OpenMaxIL if you're insane.
TBH OpenCV is very crude over all the handling of cameras directly.

There is no hardware mechanism for synchronising these sensors.

If you instantiate two camera components then there is no linking at all below your OpenCV code to link the cameras together, so nothing in there is able to sync things together and you need to do it somehow.

If you used the stereoscopic support to produce side-by-side or top/bottom images, then the GPU knows to do the best it can to sync them together. There will be some oscillator drift, but they should be within about 2ms for >30mins (using gsh's numbers from https://www.raspberrypi.org/blog/the-vi ... ment-40713 which should be comparable).
From memory the OpenCV code to take one buffer and treat the one half as one image and the other half as a second is trivial. If you're having to do a rectification operation then you're immediately copying all the wanted pixel data out of the buffer anyway, so the fact they come from a common source buffer is irrelevant.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

rspt1908
Posts: 12
Joined: Tue Mar 08, 2016 3:26 pm

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 10:27 am

Thank you so much :) I will try this :) Thank you for your time :)

rspt1908
Posts: 12
Joined: Tue Mar 08, 2016 3:26 pm

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 11:46 am

I was able to get sychronized output using uv4l. However I encountered a new problem as it seems, the driver does not follow the set width and as a result crops the stiched two images from the two cameras. Is this normal? I do not think there is anything wrong with my opencv code.

I have also heard others having this issue.

Here is the resulting image:
https://drive.google.com/file/d/0B4nF7a ... sp=sharing

Here is the UV4L parameters in the terminal:

Code: Select all

uv4l --driver raspicam --auto-video_nr --stereoscopic-mode=side_by_side --encoding=yuv420 --height=240 --width=640
<notice> [core] Trying driver 'raspicam' from built-in drivers...
<warning> [core] Driver 'raspicam' not found
<notice> [core] Trying driver 'raspicam' from external plug-in's...
<notice> [driver] Dual Raspicam Video4Linux2 Driver v1.9.32 built Sep 16 2015
<notice> [driver] Selected format: 640x240, encoding: yuv420, YUV 4:2:0 Planar (I420)
<notice> [driver] Framerate max. 30 fps
<notice> [driver] ROI: 0, 0, 1, 1
<notice> [core] Device detected!
<notice> [core] Registering device node /dev/video0
Here is the opencv code used to acquire the image:

Code: Select all

int main(int argc, char** argv){

        // ***********************************  Get frames from cameras *********************************** 

	// Declare image storage variables
    	Mat image;

        VideoCapture cap(0); // Get right image from cam 0
        if (!cap.isOpened()) // Check if successful video capture
        {
            cout << "Cannot open right camera (cam 0)" << endl;
            return -1;
        }

        namedWindow("Cap",CV_WINDOW_NORMAL);

        // ************************** Get Camera Parameters from Calibration File **************************

    while (1)
    {

        //Set delay for each frame capture. Use formula delayinms=1000ms/fps
        //max fps=10
        int delayinms=1000/30;
	
	//Capture test images-------------------------------------------------

        //Grab image
        if(!cap.grab())
        {
            cout << "Cant grab image from video0." << endl;
            return -1;
        }


        //Retrieve left image
        if(!cap.retrieve(image))
        {
            cout << "Cannot retrieve image from video0)" << endl;
        }


        //Convert images to grayscale -------------------------------------
        cvtColor(image, image, CV_BGR2GRAY);

        //Gaussian Blur----------------------------------------------------
        GaussianBlur( image, image, Size( 3, 3 ), 0, 0 );
 
	imshow("Cap", image);

	//If key pressed, stop //delayinms
	if(waitKey(1)>=0) 
	{
		cout<<"Exit"<<endl;
		break;
	}
    }
}

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10851
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 12:18 pm

UV4L2 is closed source, and RpiName is about the only one on these forums who supports it.
V4L2 as an API does not have any support for stereoscopic, so exactly how they have set it up is down to them.

There are various other API wrappers written for pulling the Pi camera in to OpenCV (eg http://www.uco.es/investiga/grupos/ava/node/40, or https://robidouille.wordpress.com/2013/ ... th-opencv/) which allow more direct access to the MMAL level.

To educate yourself, please read the main thread on stereoscopic support - viewtopic.php?f=43&t=85012
side-by-side does just that and puts the two images side by side. top/bottom puts them one above each other.
There is then an extra option (decimation) which squeezes the images by a factor of 2 in one direction so that they fit in a buffer of the size specified.

You appear not to have specified a resolution from your application, so I assume you've asked for a 640x480 or other 4:3 aspect ratio OUTPUT image. But you want two images in there side by side, so each one becomes 320x480.
If you want the full FOV from each, your OUTPUT image needs to be 1280x480.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

rspt1908
Posts: 12
Joined: Tue Mar 08, 2016 3:26 pm

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 12:27 pm

I had used two 320 by 240 images and has selected the side_by_side option thus I had set previously the resolution to 640X240 using the --width and --height parameters of the driver in the command line. However it does not seem to respond to changing these parrameters as changing the resolution to other that 320 x 240 does not seem to affect the output

RpiName
Posts: 728
Joined: Sat Jul 06, 2013 3:14 am

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 8:15 pm

rspt1908 wrote:I was able to get sychronized output using uv4l. However I encountered a new problem as it seems, the driver does not follow the set width and as a result crops the stiched two images from the two cameras. Is this normal? I do not think there is anything wrong with my opencv code.

I have also heard others having this issue.

Here is the resulting image:
https://drive.google.com/file/d/0B4nF7a ... sp=sharing

Here is the UV4L parameters in the terminal:

Code: Select all

uv4l --driver raspicam --auto-video_nr --stereoscopic-mode=side_by_side --encoding=yuv420 --height=240 --width=640
<notice> [core] Trying driver 'raspicam' from built-in drivers...
<warning> [core] Driver 'raspicam' not found
<notice> [core] Trying driver 'raspicam' from external plug-in's...
<notice> [driver] Dual Raspicam Video4Linux2 Driver v1.9.32 built Sep 16 2015
<notice> [driver] Selected format: 640x240, encoding: yuv420, YUV 4:2:0 Planar (I420)
<notice> [driver] Framerate max. 30 fps
<notice> [driver] ROI: 0, 0, 1, 1
<notice> [core] Device detected!
<notice> [core] Registering device node /dev/video0
Here is the opencv code used to acquire the image:

Code: Select all

int main(int argc, char** argv){

        // ***********************************  Get frames from cameras *********************************** 

	// Declare image storage variables
    	Mat image;

        VideoCapture cap(0); // Get right image from cam 0
        if (!cap.isOpened()) // Check if successful video capture
        {
            cout << "Cannot open right camera (cam 0)" << endl;
            return -1;
        }

        namedWindow("Cap",CV_WINDOW_NORMAL);

        // ************************** Get Camera Parameters from Calibration File **************************

    while (1)
    {

        //Set delay for each frame capture. Use formula delayinms=1000ms/fps
        //max fps=10
        int delayinms=1000/30;
	
	//Capture test images-------------------------------------------------

        //Grab image
        if(!cap.grab())
        {
            cout << "Cant grab image from video0." << endl;
            return -1;
        }


        //Retrieve left image
        if(!cap.retrieve(image))
        {
            cout << "Cannot retrieve image from video0)" << endl;
        }


        //Convert images to grayscale -------------------------------------
        cvtColor(image, image, CV_BGR2GRAY);

        //Gaussian Blur----------------------------------------------------
        GaussianBlur( image, image, Size( 3, 3 ), 0, 0 );
 
	imshow("Cap", image);

	//If key pressed, stop //delayinms
	if(waitKey(1)>=0) 
	{
		cout<<"Exit"<<endl;
		break;
	}
    }
}
It seems you are using a very old version of uv4l, so I suggest that you upgrade it in the first place, secondly in side by side or other stereo modes the resolutions are limited by the firmware, you can query them with the v4l2 tools, if you try to set an unsupported resolution the driver adjusts it to the nearest supported resolution

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10851
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 9:33 pm

RpiName wrote:It seems you are using a very old version of uv4l, so I suggest that you upgrade it in the first place, secondly in side by side or other stereo modes the resolutions are limited by the firmware, you can query them with the v4l2 tools, if you try to set an unsupported resolution the driver adjusts it to the nearest supported resolution
There are stricter restrictions when doing H264 or MJPEG encoding, but when requesting raw YUV or RGB buffers it should only be that each image needs to be a multiple of 32 pixels wide in SBS, and 16 lines high in TB. Perhaps I communicated that badly in the original stereoscopic post.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

rspt1908
Posts: 12
Joined: Tue Mar 08, 2016 3:26 pm

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Thu Jul 14, 2016 11:55 pm

RpiName wrote:
rspt1908 wrote:I was able to get sychronized output using uv4l. However I encountered a new problem as it seems, the driver does not follow the set width and as a result crops the stiched two images from the two cameras. Is this normal? I do not think there is anything wrong with my opencv code.

I have also heard others having this issue.

Here is the resulting image:
https://drive.google.com/file/d/0B4nF7a ... sp=sharing

Here is the UV4L parameters in the terminal:

Code: Select all

uv4l --driver raspicam --auto-video_nr --stereoscopic-mode=side_by_side --encoding=yuv420 --height=240 --width=640
<notice> [core] Trying driver 'raspicam' from built-in drivers...
<warning> [core] Driver 'raspicam' not found
<notice> [core] Trying driver 'raspicam' from external plug-in's...
<notice> [driver] Dual Raspicam Video4Linux2 Driver v1.9.32 built Sep 16 2015
<notice> [driver] Selected format: 640x240, encoding: yuv420, YUV 4:2:0 Planar (I420)
<notice> [driver] Framerate max. 30 fps
<notice> [driver] ROI: 0, 0, 1, 1
<notice> [core] Device detected!
<notice> [core] Registering device node /dev/video0
Here is the opencv code used to acquire the image:

Code: Select all

int main(int argc, char** argv){

        // ***********************************  Get frames from cameras *********************************** 

	// Declare image storage variables
    	Mat image;

        VideoCapture cap(0); // Get right image from cam 0
        if (!cap.isOpened()) // Check if successful video capture
        {
            cout << "Cannot open right camera (cam 0)" << endl;
            return -1;
        }

        namedWindow("Cap",CV_WINDOW_NORMAL);

        // ************************** Get Camera Parameters from Calibration File **************************

    while (1)
    {

        //Set delay for each frame capture. Use formula delayinms=1000ms/fps
        //max fps=10
        int delayinms=1000/30;
	
	//Capture test images-------------------------------------------------

        //Grab image
        if(!cap.grab())
        {
            cout << "Cant grab image from video0." << endl;
            return -1;
        }


        //Retrieve left image
        if(!cap.retrieve(image))
        {
            cout << "Cannot retrieve image from video0)" << endl;
        }


        //Convert images to grayscale -------------------------------------
        cvtColor(image, image, CV_BGR2GRAY);

        //Gaussian Blur----------------------------------------------------
        GaussianBlur( image, image, Size( 3, 3 ), 0, 0 );
 
	imshow("Cap", image);

	//If key pressed, stop //delayinms
	if(waitKey(1)>=0) 
	{
		cout<<"Exit"<<endl;
		break;
	}
    }
}
It seems you are using a very old version of uv4l, so I suggest that you upgrade it in the first place, secondly in side by side or other stereo modes the resolutions are limited by the firmware, you can query them with the v4l2 tools, if you try to set an unsupported resolution the driver adjusts it to the nearest supported resolution
I tried updating my uv4l-raspicam package but it says my package is up to date

rspt1908
Posts: 12
Joined: Tue Mar 08, 2016 3:26 pm

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Fri Jul 15, 2016 2:24 pm

Thank you for your help :) I just worked with how UV4L's stereoscopic option works with my pi and yey, I can now get accurate depth maps :) God bless :)

shihkai
Posts: 1
Joined: Wed Apr 26, 2017 4:46 am

Re: Stereo Vision Using Compute Module: Syncing Pi Cameras

Wed Apr 26, 2017 5:05 am

hi,
may I ask some questions about max resolution and fps?
if camera module is instead of Sony IMX219, and fps more than 60fps, then what max resolution is?

thanks for your reply first. :)

Return to “Compute Module”