Kozuch
Posts: 59
Joined: Sun Aug 10, 2014 10:35 am

Stereo camera frame synchronization

Tue Aug 12, 2014 10:28 pm

I am interested in stereo camera frame synchronization with the Compute Board and IO Board because of computer vision application. I would like to know what level of synchronization can be reached with the two cameras. Has anyone tried to take still photos or video from both cameras simultaneously and measure the sync level?

A sample visual test could be done against a single stopwatch. I mean like a simple bash script with two "raspstill" or "raspvideo" commands each for one camera. Altering raspstill to take a picture from both in one run would probably be even better.

I would really appreciate if someone has any sync values. I am thinking about getting the Compute Module but it would be a bet for me without knowing that results I could get.

Thanks for advance for your answers!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5364
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo camera frame synchronization

Wed Aug 13, 2014 8:10 am

On the list on www.raspberrypi.org/forums/viewtopic.php?f=43&t=83353 - full stereoscopic support for display, H264 and JPEG encode.
It is possible to do (I have it running on my desk) but there will be a small amount of drift due to their being independent clocks on the two camera boards. Video encode looks to be limited to a max of 1920 wide, so not sure if we'll manage 1080P top/bottom. The half/half modes will work fine though.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5364
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo camera frame synchronization

Wed Aug 13, 2014 8:15 am

BTW You're better off asking camera questions on the camera forum rather than the compute module one.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Kozuch
Posts: 59
Joined: Sun Aug 10, 2014 10:35 am

Re: Stereo camera frame synchronization

Sat Jul 18, 2015 10:15 pm

Thanks for your answer. Ok, so you say there only will be a delay due to the independent clocks on both camera modules. How much can this be? Given the cameras clock at about 25 MHz I guess this may be in a range of microseconds at worst?

Also, I am interested to know how the whole image capture pipeline works in detail (both software and hardware). In the stereoscopic camera capture that you implemented in C (this thread), are the cameras actually triggered sequentially in the code (first line = first camera, second line = second camera) or is the chipset able to trigger them both by a single line of C code? And are the cameras actually read sequentially by the chipset or are there two HW imaging pipelines that will do this simultaneously? I am not technically savvy enough to be able to read this out from some datasheets...

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5364
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo camera frame synchronization

Mon Jul 20, 2015 11:55 am

Kozuch wrote:Thanks for your answer. Ok, so you say there only will be a delay due to the independent clocks on both camera modules. How much can this be? Given the cameras clock at about 25 MHz I guess this may be in a range of microseconds at worst?

Also, I am interested to know how the whole image capture pipeline works in detail (both software and hardware). In the stereoscopic camera capture that you implemented in C (this thread), are the cameras actually triggered sequentially in the code (first line = first camera, second line = second camera) or is the chipset able to trigger them both by a single line of C code? And are the cameras actually read sequentially by the chipset or are there two HW imaging pipelines that will do this simultaneously? I am not technically savvy enough to be able to read this out from some datasheets...
The 25MHz oscillator for the sensor is mounted on the camera module board, not on the Pi. Therefore if you have two camera modules, you have two oscillators. With the best will in the world, they will not stay perfectly synchronised. The sensors solely use that clock (via PLLs and register settings) to determine the rate at which they produce pixel data, so the pixel rate will drift too.
Even if one is at 25.00000MHz, and the other at 25.0000000001MHz, then you still have an issue given long enough. If you want absolute lock, then you have to drive both sensors from the same clock.

The I2C commands are sent to the sensors via the same peripheral (using pin muxing to select which sensor to talk to as they share the same I2C address), so yes they are started one after the other. The delay is probably in the order of 1ms though, so not massive.

The data from the sensor is received and written into memory via two independent peripherals.
Once there, the software instructs the ISP hardware block to process the data and combine it into one output image.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1282
Joined: Sat Sep 10, 2011 11:43 am

Re: Stereo camera frame synchronization

Mon Jul 20, 2015 12:49 pm

It may be possible to add a little patch to the code that I implemented a while back that synchronises the left and right frames by adding / removing a blanking interval line from one of the cameras. Unfortunately I've not got it to hand and would have to pick it back up again but it is possible... This would then make the cameras sync up to the nearest line (approximately)
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5364
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo camera frame synchronization

Mon Jul 20, 2015 1:58 pm

gsh wrote:It may be possible to add a little patch to the code that I implemented a while back that synchronises the left and right frames by adding / removing a blanking interval line from one of the cameras. Unfortunately I've not got it to hand and would have to pick it back up again but it is possible... This would then make the cameras sync up to the nearest line (approximately)
Gordon, if you really want to start playing, then feel free!
I vaguely remember that the very original setup did include dynamic frame timing adjustments, but the later versions relied on using the same PLL on VC to clock both sensors and remain in sync. The frames are timestamped, so it shouldn't be too tricky to fiddle.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1282
Joined: Sat Sep 10, 2011 11:43 am

Re: Stereo camera frame synchronization

Wed Jul 22, 2015 3:49 pm

Yeah,

Might have to hang until I've got some free time (not had a minute in the last two years but you never know!)

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5364
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo camera frame synchronization

Wed Jul 22, 2015 4:00 pm

gsh wrote:Yeah,

Might have to hang until I've got some free time (not had a minute in the last two years but you never know!)
Free time - I've heard of that, but it's generally in fairly short supply for me too :)
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Kozuch
Posts: 59
Joined: Sun Aug 10, 2014 10:35 am

Re: Stereo camera frame synchronization

Sun Oct 04, 2015 8:33 am

I am newly interested in syncing still photography only (no video needed anymore) - I guess this makes the issue a much easier since one does not really have to care about drift in time from independent clocks (as in case of video). So my need is to trigger both cameras for still capture in the same time. I plan to hack raspistill (or raspicam.c library by [email protected]) to init/start the cameras to get pat the initial delay (auto exposure setting etc.) and then to fire them then manually after the are ready for capture.
The I2C commands are sent to the sensors via the same peripheral (using pin muxing to select which sensor to talk to as they share the same I2C address), so yes they are started one after the other. The delay is probably in the order of 1ms though, so not massive.
Can we get a more concrete delay value than that "order of 1ms" (does this actually mean a sub-1ms?)? I mean, if we know the speed of the I2C bus then it may be possible to say what the delay may be more precisely. I am looking for a sub-1ms synchronization, possibly on a microsecond level. I would really appreciate if anyone knows more about this. Thanks!

mmtww
Posts: 1
Joined: Tue Oct 13, 2015 1:26 pm

Re: Stereo camera frame synchronization

Tue Oct 13, 2015 1:29 pm

Can we get a more concrete delay value than that "order of 1ms" (does this actually mean a sub-1ms?)? I mean, if we know the speed of the I2C bus then it may be possible to say what the delay may be more precisely. I am looking for a sub-1ms synchronization, possibly on a microsecond level. I would really appreciate if anyone knows more about this. Thanks!
Did anyone get any figures to this question?

I am currently working on a project trying to sync large numbers of cameras taking still images in a time window of 1-2ms. If the module is able to achieve this across its two connected cameras it would be perfect.

Kozuch
Posts: 59
Joined: Sun Aug 10, 2014 10:35 am

Re: Stereo camera frame synchronization

Tue Oct 13, 2015 2:52 pm

mmtww,

unfortunately I have no new figures. However I think a HW trigger like FREX is needed for precise sync even for compute module with two cameras. The issue is the cameras run "freely" at their FPS and you only read the frames out at the same time but you have no idea when the frames actually were taken.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5364
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo camera frame synchronization

Tue Oct 13, 2015 3:32 pm

Looking at the firmware code, if the sensor has already been programmed with the correct mode (it should have), then the command to start streaming is writing 3 bytes to the sensor over I2C, and it is in a loop for the number of channels. I2C is set at 100kHz (1), so I think that makes it 0.4ms to send the command (4 bytes including I2C address/direction, and 10bits/word)

As stressed before, the sensors are running off independent oscillators so they will drift once started. You're probably OK for a few minutes, but don't try hours.

(1) not sure why it isn't at 400kHz, but I'm not going tweaking it.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

DonSerge84
Posts: 5
Joined: Sun Nov 01, 2015 5:18 pm

Re: Stereo camera frame synchronization

Sun Nov 01, 2015 5:36 pm

good evening!
is there a list of all possible i2c-commands to the sensor available somewhere?
It would be nice to be able to tell the sensor just to make one capture instead of a stream. This way it would be possible to trigger the shots via I2C, and thereby even more baspiboards with cams via gpio. Is there such a "magic command"? Instead of the magic command I tried this with following workaround in a loop:

rpiGpio.writePinHigh(17);
Camera.open();
Camera.grab();
Camera.retrieve ( image );
Camera.release();
rpiGpio.writePinLow(17);

works fine, but only with 2FPS..:(

Kozuch
Posts: 59
Joined: Sun Aug 10, 2014 10:35 am

Re: Stereo camera frame synchronization

Tue Feb 28, 2017 1:08 am

6by9 wrote:As stressed before, the sensors are running off independent oscillators so they will drift once started. You're probably OK for a few minutes, but don't try hours.
Can you guess what will the drift be after 1 hour (if they were started in perfect sync)? I mean a rough guess given the grade of oscillator that the camera board is equipped with... Will it be in the range of us, single digit ms, tenths/hundreds of ms etc? Will the drift depend on FPS and image resolution used?

Kozuch
Posts: 59
Joined: Sun Aug 10, 2014 10:35 am

Re: Stereo camera frame synchronization

Thu Apr 12, 2018 3:32 pm

Bump - could anyone tell what the drift will approximately be after 1 hour (if they were started in perfect sync)?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5364
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo camera frame synchronization

Thu Apr 12, 2018 4:06 pm

Kozuch wrote:
Thu Apr 12, 2018 3:32 pm
Bump - could anyone tell what the drift will approximately be after 1 hour (if they were started in perfect sync)?
No because it will depend on the exact oscillator frequencies and PLL accuracies in the particular pair of modules you are trying to use.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Kozuch
Posts: 59
Joined: Sun Aug 10, 2014 10:35 am

Re: Stereo camera frame synchronization

Thu Apr 12, 2018 4:19 pm

Will I be able to measure the drift by having two RPis with one camera board each to output a GPIO signal when frame is received in the Raspicam C++ API (uco.es) and measure the drift between two GPIO inputs on th 3rd RPi? All implemented in C++.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5364
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo camera frame synchronization

Thu Apr 12, 2018 6:48 pm

If you've got two independent Pis each having one camera then you have no synchronisation at all. You can use the "disable_camera_led=3" trick to get frame start timings on a GPIO though.

When running with one Compute Module having two cameras attached, then the settings are all matched between the two modules, and the synchronisation should initially be <1ms. I'd need to check, but it may well be that "disable_camera_led=3" will work in stereoscopic mode and pulse the LEDs at the appropriate frame start/ends for the two sensors independently, but that depends on exactly how I did it (and I can't remember offhand).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Compute Module”

Who is online

Users browsing this forum: No registered users and 2 guests