worldmaster
Posts: 10
Joined: Thu Nov 21, 2019 4:26 pm

Stereo camera PI processing

Thu Nov 21, 2019 4:40 pm

Hello. Please tell me how to do the following:
I have two cameras that are connected to the Raspberry PI.
I need to get an original size image from each camera. For example, 1024 * 1024.
Then cut the fragment size 640 * 480.
Mirror these two fragments and merge them into one image of 1280 * 480 size.
Then stream the stream through rtsp to the network.
Compression required h264. FPS 30.

I read that raspivid can set the region of interest and work with 2 cameras simultaneously in stereo.
But can he process the frames individually to solve my problem?
Please help with the solution. Thanks.

User avatar
mahjongg
Forum Moderator
Forum Moderator
Posts: 12583
Joined: Sun Mar 11, 2012 12:19 am
Location: South Holland, The Netherlands

Re: Stereo camera PI processing

Thu Nov 21, 2019 6:41 pm

Compute modules have two camera inputs, and software exists to combine the two camera's for stereo vision, and split screen display of both camera's

Using regular RPI's with two camera's is much harder, and involves a camera multiplexer, and the above software doesn't work.

User avatar
HermannSW
Posts: 1870
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Stereo camera PI processing

Thu Nov 21, 2019 7:26 pm

There is a way to connect two cameras to a single PI, can be any two Arducam cameras, or Raspberry v1/v2 cameras:
https://www.raspberrypi.org/forums/view ... 8#p1551503
Image
worldmaster wrote:
Thu Nov 21, 2019 4:40 pm
Then cut the fragment size 640 * 480.
Mirror these two fragments and merge them into one image of 1280 * 480 size.
The 2MP monochrome global shutter camera has a 640x400 mode.
That would give you 1280x400 size image without any work.
Above image is 2560x800 (two 1280x800 2MP images side-by-side).

P.S:
Scaled down image allows to see 3D image without any viewer (on 15" laptop display).
Just place hand vertically in the middle and make both images overlay:
Image
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

worldmaster
Posts: 10
Joined: Thu Nov 21, 2019 4:26 pm

Re: Stereo camera PI processing

Sun Nov 24, 2019 2:19 pm

I do not need the camera mode 640 * 480. I need to cut a fragment from each camera. A fragment can be anywhere in sight.

gordon77
Posts: 4464
Joined: Sun Aug 05, 2012 3:12 pm

Re: Stereo camera PI processing

Sun Nov 24, 2019 3:10 pm

"I have two cameras that are connected to the Raspberry PI."

What are these 2 cameras ?

worldmaster
Posts: 10
Joined: Thu Nov 21, 2019 4:26 pm

Re: Stereo camera PI processing

Sun Nov 24, 2019 4:03 pm

gordon77 wrote:
Sun Nov 24, 2019 3:10 pm
"I have two cameras that are connected to the Raspberry PI."

What are these 2 cameras ?
https://onpad.ru/catalog/cubie/raspberr ... /1864.html
Производитель: WaveShare
Поддерживает: Raspberry PI A+/B+/PI2
Сенсор: 5 мегапиксельный сенсор OV5647
Размер матрицы: 1/4
Диафрагма (F): 2,8
Фокусное расстояние: 3.37 mm
Диагональ: 72,4 градусов
Разрешение: 1080p
- 2592 × 1944 Разрешение изображения
Размеры: 25 х 24 мм х 9 мм
Поддерживает запись видео: 1080p30, 720p60 and 640x480p60/90

gordon77
Posts: 4464
Joined: Sun Aug 05, 2012 3:12 pm

Re: Stereo camera PI processing

Sun Nov 24, 2019 4:05 pm

How are they connected?

worldmaster
Posts: 10
Joined: Thu Nov 21, 2019 4:26 pm

Re: Stereo camera PI processing

Sun Nov 24, 2019 4:24 pm

gordon77 wrote:
Sun Nov 24, 2019 4:05 pm
How are they connected?
Stereopi
https://www.crowdsupply.com/virt2real/stereopi

User avatar
HermannSW
Posts: 1870
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Stereo camera PI processing

Sun Nov 24, 2019 5:54 pm

worldmaster wrote:
Sun Nov 24, 2019 2:19 pm
I do not need the camera mode 640 * 480. I need to cut a fragment from each camera. A fragment can be anywhere in sight.
Then using stereo hat with v1 camera is what can do that.
The stereo hat hides the fact that two cameras are connected.
You would use raspivid and it will think it takes a 5MP image.
In that image the left half will be from left camera, and right half from right.
Then just use raspivid's "--roi" option to specify region of interest.
Of course that does not make sense if you already use StereoPi.
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

gordon77
Posts: 4464
Joined: Sun Aug 05, 2012 3:12 pm

Re: Stereo camera PI processing

Sun Nov 24, 2019 6:00 pm

The stereopi site mentions using raspivid,

raspivid -3d sbs -w 1280 -h 480 -o 1.h264

can you use - roi with that?

worldmaster
Posts: 10
Joined: Thu Nov 21, 2019 4:26 pm

Re: Stereo camera PI processing

Sun Nov 24, 2019 7:24 pm

I repeat, I need fragments from each of the cameras. These fragments must be combined into a new frame. And then send the data stream to the network.
Image

gordon77
Posts: 4464
Joined: Sun Aug 05, 2012 3:12 pm

Re: Stereo camera PI processing

Sun Nov 24, 2019 7:32 pm

Roi will select a segment BUT I don't know if it will work for 3d, and select a different segment from each camera (which l don't think you mentioned before)

User avatar
HermannSW
Posts: 1870
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Stereo camera PI processing

Mon Nov 25, 2019 9:50 am

worldmaster wrote:
Sun Nov 24, 2019 7:24 pm
I repeat, I need fragments from each of the cameras. These fragments must be combined into a new frame. And then send the data stream to the network.
I have to repeat, the Stereo hat hides the fact that there are two cameras, Raspberry Pi thinks there is only one camera.
gordon77 wrote:
Sun Nov 24, 2019 7:32 pm
Roi will select a segment BUT I don't know if it will work for 3d, and select a different segment from each camera (which l don't think you mentioned before)
Again, Stereo hat provides "normal" frames, Pi does not know about two cameras connected, nor is any 3D option required for raspistill/raspivid.

But I was wrong with "-roi", since the Pi does not know about two cameras, it does not work the way needed.

This is simple stereo capture for this command:

Code: Select all

$ raspistill -md 4 -w 1296 -h 972 -p 22,50,648,486 -o p1.jpg
$
Image


Now with "-roi" I get different regions:

Code: Select all

$ raspistill -md 4 -w 1296 -h 972 -p 22,50,648,486 -o p2.jpg -roi 0.25,0.0,0.5,0.5
$
Image


Besides having received free samples of 0.3/1.0/2.0MP monochrome global shutter, 13MP/16MP color rolling shutter and 0.3MP color/2.0MP monochome stereo cameras from Lee at Arducam, I have no incentive in increasing Arducam sales.

Btw, Arducam stereo hat for Raspberry Pi with two 5MP camera bundle does cost 89.99€
https://www.uctronics.com/index.php/ard ... -2173.html

Comparable StereoPi starter kit does cost 125$ but includes a compute module 3+ lite:
https://www.crowdsupply.com/virt2real/stereopi

So if you already have a Pi, stereo hat is cheaper.

Stereo hat with two 8MP v2 cameras does cost 109.99$.
In addition to StereoPi it allows to get stereo hat with two Arducam cameras as well (all those I listed above).
I do have a sample here at home with two ov9281 monochrome global shutter cameras (my previous posting photo was taken with that, a "2 x 1280x800 = 2560x800" frame):
https://stamm-wilbrandt.de/en/forum/ArduCAM/mode7.jpg
The Arducam cameras do provide full and not half resolution as v1/v2 cameras.
But they only work with Arducam driver, not with raspivid/raspistill like the v1/v2 cameras:
https://github.com/ArduCAM/MIPI_Camera


So "-roi" does what it should above, capture 50% size in both dimensions , starting at coordinate (324,0).
What is needed for the OP request is to be able to specify two regions of interest.
I will come up with such a solution based on raspividyuv and i420toh264 tool (with GPU doing .h264 conversion) soon:
https://github.com/Hermann-SW2/userland ... i420toh264
What I can say for sure now is that that solution will work, and will allow to capture two arbitrarily located regions of interest.
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

User avatar
HermannSW
Posts: 1870
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Stereo camera PI processing

Wed Nov 27, 2019 10:22 am

HermannSW wrote:
Mon Nov 25, 2019 9:50 am
So "-roi" does what it should above, capture 50% size in both dimensions , starting at coordinate (324,0).
What is needed for the OP request is to be able to specify two regions of interest.
I will come up with such a solution based on raspividyuv and i420toh264 tool (with GPU doing .h264 conversion) soon:
https://github.com/Hermann-SW2/userland ... i420toh264
What I can say for sure now is that that solution will work, and will allow to capture two arbitrarily located regions of interest.
Done and commited, with new section added to README.md with details:
https://github.com/Hermann-SW2/userland ... g-pipeline

This command part of bash i420toh264 pipeline (x1,y1,x2,y1,w,d)

Code: Select all

...
./sample_yuv_dbl_roi 640 480 40,160,371,51,240,180 |
...
extracts regions with left top (40,160) and (371,51) from [email protected] raspividyuv video, both of dimension "w x d", and creates new frames of dimension "2w x d" from them for further processing. There are quite some assertions on coordinates, width, height, ... -- the 2nd frame has to be right of left frame (x1+w<=x2) for easy inplace copying to work, besides that their top left coordinates can be selected freely.

This is a frame from generated 480x180 video tst.h264:
Image


For comparison, this is full 640x480 view:
Image
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

worldmaster
Posts: 10
Joined: Thu Nov 21, 2019 4:26 pm

Re: Stereo camera PI processing

Wed Nov 27, 2019 2:41 pm

Thanks. And how now to broadcast the received stream to the network?

User avatar
HermannSW
Posts: 1870
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Stereo camera PI processing

Wed Nov 27, 2019 7:31 pm

That is a completely different question.
Depends how/where you want to receive it.
In browser, vlc, raw .h64 streaam, using rtsp:, ....

There are tons of solutions, many here in forum, or on the web.
You can send and receive with gstreamer pipeline via several protocols.
Or this:
http://frozen.ca/streaming-raw-h-264-fr ... pberry-pi/

In case you want to do it with i420toh264 bash pipeline, you just need to know that that tool accepts "-" as filename in order to send the GPU compressed .h264 stream to stdout like raspivid[yuv], and form there you can use any method for broadcast.
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

worldmaster
Posts: 10
Joined: Thu Nov 21, 2019 4:26 pm

Re: Stereo camera PI processing

Sun Dec 01, 2019 6:09 pm

Thanks for answers. Maybe you can tell a ready-made solution so that the stream received from i420toh264 is broadcast via rtsp with h264 compression?

User avatar
HermannSW
Posts: 1870
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Stereo camera PI processing

Sun Dec 01, 2019 8:53 pm

worldmaster wrote:
Sun Dec 01, 2019 6:09 pm
Thanks for answers. Maybe you can tell a ready-made solution so that the stream received from i420toh264 is broadcast via rtsp with h264 compression?
Replace "raspivid ... -o -" with "raspividyuv ... | ... | ./i420toh264 - width height" for options 1/2/4
https://raspberrypi.stackexchange.com/q ... 1429#31429


From
https://raspberrypi.stackexchange.com/q ... tcat#27083

you can change option 2 above with:

Code: Select all

... | cvlc -vvv stream:///dev/stdin --sout '#standard{access=http,mux=ts,dst=:8090}' :demux=h264
and access on client browser with:

Code: Select all

http://<IP-OF-THE-RPI>:8090
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

worldmaster
Posts: 10
Joined: Thu Nov 21, 2019 4:26 pm

Re: Stereo camera PI processing

Wed Dec 04, 2019 7:30 pm

Thank you so much. You have helped me a lot. I will try.

worldmaster
Posts: 10
Joined: Thu Nov 21, 2019 4:26 pm

Re: Stereo camera PI processing

Sun Jan 05, 2020 4:32 pm

Hello. Please tell me one more question. How did userland files appear on a clean assembly? I don’t have enough files there and I can’t figure out how to update these files ??

Did apt-get upgrade. But it did not help. Tried to make git clone. But there completely different folders downloaded.

I need the hello_pi section https://github.com/Hermann-SW2/userland ... s/hello_pi
Please tell me which team to update it on my stereoPi.

Thanks.

User avatar
HermannSW
Posts: 1870
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Stereo camera PI processing

Sun Jan 05, 2020 5:50 pm

Code: Select all

git clone https://github.com/Hermann-SW2/userland/tree/master/host_applications/linux/apps/hello_pi
cd userland/tree/master/host_applications/linux/apps/hello_pi
make
./i420toh264 ...
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

worldmaster
Posts: 10
Joined: Thu Nov 21, 2019 4:26 pm

Re: Stereo camera PI processing

Mon Jan 20, 2020 5:33 pm

Hello. Everything worked out. But now the problem is that there is a very big delay when streaming video. about 2 to 3 seconds. Is there any way to reduce the delay ?? Read what is hardware coding?

For the streaming test, I use the following code:

Code: Select all

raspivid -o - -3d sbs -w 1280 -h 480 -fps 50 -g 30 -ih -pf baseline -md 7 -k -b 3000000 | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8554/x}' :demux=h264

User avatar
HermannSW
Posts: 1870
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Stereo camera PI processing

Tue Jan 21, 2020 8:58 am

Low latency streaming is a difficult subject, and searching the net for "raspberry cvlc low latency" is difficult because of the many hits.

I use uv4l for low latency camera access, but it is closed source and has licnsing issues for commercial use (both no problem for me).
http://www.linux-projects.org/uv4l/installation/

There is a node solution that allows you to modify the raspivid command used in streaming, so you might add "-3d":
https://www.raspberrypi.org/forums/view ... p?t=204921

I did a quick duckduckgo search and found this solution with 0.23s latency (see attached pic), where you could use "-3d" as well:
https://altax.net/blog/low-latency-rasp ... nsmission/

Direct output of Raspberry v2 camera on HDMI monitor has latency of 66ms(!):
https://www.raspberrypi.org/forums/view ... 6#p1573596
Attachments
lowlatency.png.jpg
lowlatency.jpg
lowlatency.png.jpg (91.16 KiB) Viewed 384 times
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7885
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo camera PI processing

Tue Jan 21, 2020 9:21 am

HermannSW wrote:
Tue Jan 21, 2020 8:58 am
Direct output of Raspberry v2 camera on HDMI monitor has latency of 66ms(!):
https://www.raspberrypi.org/forums/view ... 6#p1573596
Possibly, but you're also now including the refresh rate of your phone display in the equation as well. If that's only running at 60Hz, then your source has a 16ms granularity.

What you really need to measure the latency is one of the calibration devices that has a sequence of running dots that illuminate at fixed intervals (typically 1, 2, or 5ms). I can't remember the product name, but I have one on my desk.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7885
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereo camera PI processing

Tue Jan 21, 2020 9:23 am

worldmaster wrote:
Mon Jan 20, 2020 5:33 pm
Hello. Everything worked out. But now the problem is that there is a very big delay when streaming video. about 2 to 3 seconds. Is there any way to reduce the delay ?? Read what is hardware coding?

For the streaming test, I use the following code:

Code: Select all

raspivid -o - -3d sbs -w 1280 -h 480 -fps 50 -g 30 -ih -pf baseline -md 7 -k -b 3000000 | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8554/x}' :demux=h264
Most of the latency is in the receiving device in an attempt to overcome network jitter. Solving that depends on your receiving software.

There is also some in the use of Linux pipes as they tend to shift 64kB at a time. That can largely be overcome by adding -fl (flush) to the raspivid command line.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”