User avatar
HermannSW
Posts: 1311
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Stereoscopic camera capture with single v2 camera

Thu Jan 03, 2019 12:10 am

This is work in progress, I am not aware of something like this being done with Raspberry camera or raspivid before, and I am not sure if it will work finally.

I remembered recently my work with I2C command injection in raspivid (I know there is a risk, but normally it works) resulting in every other frame with different shutter speeds:
viewtopic.php?t=212694#p1313054

Yesterday I thought whether it would be possible to do stereoscopic camera capture with single v2 camera using the same I2C command injection trick.

After I added "--lft" option for setting left border for v2 camera to raspivid in my userland fork, I added experimental "--lft3D" option, and it seens to work:
https://github.com/Hermann-SW/userland/ ... 91609eeb03

I did record with mode 7 (640x480) and that has 2x2 binning. That means that "normal" raspivid horizontal view is [1000..2279] from 3280 pixels in total.. "--lft3D X" results in odd frame range [1000-X..2279-X] and even frame range [1000+X..2279+X].

This is sample command execution and frame skip analysis (no skips):

Code: Select all

[email protected]:~ $ raspivid -md 7 -w 640 -h 480 -t 15000 -stf 5 -l3D 500 -o tst.h264 -pts x.pts
[email protected]:~ $ ptsanalyze x.pts 5
creating tstamps.csv
592 frames were captured at 40fps
frame delta time[us] distribution
      6 24867
    375 24868
    208 24869
after skip frame indices (middle column)
0 frame skips (0%)
[email protected]:~ $ 

Running with framerate 50fps results in "mmal: Failed to write register 0x0164" error messages of new code. But this is not high framerate work, and 40fps means that left and right eye get recorded at 20fps each, not that bad. I did upload the unprocessed .h264 video taken by above command to youtube:
https://www.youtube.com/watch?v=bWs0H-J ... e=youtu.be

Colors look OK, and even moving the Arduino Due around seems to result in fine video frames captured.
I have no idea what "--lft3D" values to use, values between 0 and 1000 are possible (video was taken with 500).
You can stop youtube video and single-step fore/backward with "."/"," keys to remove the high flicker.

This is an odd numbered frame (left eye):
Image

And this is the next frame (right eye):
Image


Next steps:
  • post processing tool to separate left/right frames
  • alternatively let raspivid write two .h264 output files in case of "--lft3D" option
  • post processing to combine the left/right videos to 3D video (1280x480)
  • experiment which "--lft3D" arguments result in "good" 3D view
P.S:
Manually copied together, flat CD sleeve in middle of image on monitor and nose at CD sleeve allows first impression:
Image
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://gitlab.freedesktop.org/HermannSW/gst-template
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

User avatar
Gavinmc42
Posts: 3438
Joined: Wed Aug 28, 2013 3:31 am

Re: Stereoscopic camera capture with single v2 camera

Thu Jan 03, 2019 1:17 am

A very interesting experiment.
What about taking one in the middle as well as the reference for 3D.

Full field of 60 degrees or so for normal lens with 2x2 binning for 640x480 that is left 1280, middle 1280, right 1280 pixels.
With 3280 wide image sensor that is a small overlap?
Digital panning? Stereo microscope?

Use two of these displays for stereo viewer, two SPI ports.
https://www.adafruit.com/product/3787
Wonder if it could be used for depth perception in fpv/drone applications?
Most fpv lens are 120+ degrees, clip-on mobile phone wide angle lens?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

ethanol100
Posts: 583
Joined: Wed Oct 02, 2013 12:28 pm

Re: Stereoscopic camera capture with single v2 camera

Thu Jan 03, 2019 10:06 am

This is technically not a stereoscopic setup and you can't get any depth from these images. You use a single lens and a single imaging plane with an offset.

User avatar
HermannSW
Posts: 1311
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Stereoscopic camera capture with single v2 camera

Thu Jan 03, 2019 7:24 pm

ethanol100 wrote:
Thu Jan 03, 2019 10:06 am
This is technically not a stereoscopic setup and you can't get any depth from these images. You use a single lens and a single imaging plane with an offset.
I found many hits for "single lens 3D" on google.
While still using a single lens only, those solutions use in camera mirrors or record two frames with different focus to get 3D.
Since I don't do anything like that you might be right.

Anyway, I searched for the quickest solution to split recorded video into odd and even part, and then combine side-by-side.
Below is most likely not the shortest solution, but it works (completely ffmpeg based).
The bash script contains the links from where I found the different conversion parts.

This is video "stereo1.h264.mp4" uploaded to youtube, generated with "./to3D stereo1.h264".
Defintely "--lft3D 500" was far too big displacement:
https://www.youtube.com/watch?v=AP46mL0 ... e=youtu.be
Image


This is the script:

Code: Select all

$ cat to3D
#!/bin/bash

# https://superuser.com/questions/573747/drop-every-even-or-odd-frames-using-ffmpeg/573753#573753
#
ffmpeg -i $1 -an -vcodec rawvideo -pix_fmt yuv420p $1.yuv

ffmpeg -r 2 -s 640x480 -i $1.yuv -filter:v select="mod(n-1\,2)" -c:v rawvideo -r 1 -format rawvideo -pix_fmt yuv420p -an $1.odd.yuv

ffmpeg -r 2 -s 640x480 -i $1.yuv -filter:v select="not(mod(n-1\,2))" -c:v rawvideo -r 1 -format rawvideo -pix_fmt yuv420p -an $1.even.yuv

# https://stackoverflow.com/questions/15778774/using-ffmpeg-to-losslessly-convert-yuv-to-another-format-for-editing-in-adobe-pr
#
ffmpeg -f rawvideo -vcodec rawvideo -s 640x480 -r 25 -pix_fmt yuv420p -i $1.odd.yuv -c:v libx264 -preset ultrafast -qp 0 $1.odd.mp4

ffmpeg -f rawvideo -vcodec rawvideo -s 640x480 -r 25 -pix_fmt yuv420p -i $1.even.yuv -c:v libx264 -preset ultrafast -qp 0 $1.even.mp4

# https://superuser.com/questions/153160/join-videos-split-screen
#
ffmpeg -i $1.even.mp4 -i $1.odd.mp4 -filter_complex '[0:v]pad=iw*2:ih[int];[int][1:v]overlay=W/2:0[vid]' -map [vid] -c:v libx264 -crf 23 -preset veryfast $1.mp4
$
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://gitlab.freedesktop.org/HermannSW/gst-template
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

Return to “Camera board”