A c++ camera api


36 posts   Page 1 of 2   1, 2
by wibble82 » Thu Oct 24, 2013 6:43 pm
Hey

Update: New and better version of the api, along with instructions is here: http://robotblogging.blogspot.co.uk/201 ... i-for.html

In case it's useful for anyone, I've started work on a basic api to easily get the camera data in c++ (primarily for my own purposes). Basically a strip down of the raspivid code, combined with some work from Pierre Raus.

Info + link to src code here
http://robotblogging.blogspot.co.uk/201 ... age-2.html

This first version allows you to easily get at the raw data for each frame as an when it comes in. Next I'll be working on fast conversion to RGB, feeding it through open cv, then rendering the results on screen.

-Chris
Last edited by wibble82 on Thu Oct 31, 2013 8:16 am, edited 2 times in total.
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
by jamesh » Thu Oct 24, 2013 7:34 pm
Check out the raspiyuv code - that can output RGB direct from the ISP so no SW conversion required.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 17234
Joined: Sat Jul 30, 2011 7:41 pm
by jbeale » Thu Oct 24, 2013 7:34 pm
Excellent writeup, thanks for doing this! The MMAL stuff seemed very opaque up til now but your page looks very clear on first read through.

Given that the ARMv6 is rather limited in speed compared with the GPU, I'm curious about the possibility of doing some simple processing on low-res video, maybe 320x240 at video rates, and then once certain criterion are satisfied, switching to full-resolution H.264 encoded video which is just passed through into a file. Then after a timeout, changing back to lower resolution frames for processing again. Do you have a feeling for how quickly you can change video parameters on the fly and see a valid stream with the new settings? Can it happen within a single frame time or two, or would it need much longer for setup?
User avatar
Posts: 3251
Joined: Tue Nov 22, 2011 11:51 pm
by wibble82 » Thu Oct 24, 2013 8:23 pm
@jamesh - thought I'd tried that but gave it another go. I think the still port must support colour conversions that the video port doesn't, as setting to BGR24 (like the yuv app does) results in a fail-to-set video port format error.

@jbeale. while I've not tried it, I see no reason why you couldn't just call stop camera, then call start camera with different parameters. I'd imagine it's a pretty fast process although I've not tested it yet. That said, the alternative (which I'm going to play with next) is doing various bits of processing on the GPU. A very simple gpu operation would just be a downsample to a smaller texture.
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
by lagurus » Thu Oct 24, 2013 8:30 pm
jbeale wrote:Excellent writeup, thanks for doing this! The MMAL stuff seemed very opaque up til now but your page looks very clear on first read through.

Given that the ARMv6 is rather limited in speed compared with the GPU, I'm curious about the possibility of doing some simple processing on low-res video, maybe 320x240 at video rates, and then once certain criterion are satisfied, switching to full-resolution H.264 encoded video which is just passed through into a file. Then after a timeout, changing back to lower resolution frames for processing again. Do you have a feeling for how quickly you can change video parameters on the fly and see a valid stream with the new settings? Can it happen within a single frame time or two, or would it need much longer for setup?


I think that is not needed to switch from low to high resoution. mmal offer to use splitter, which can split one video input stream to 2 output streams and then one use for write full H264 video and second (after is resized to 320x240) for image processing.
Posts: 41
Joined: Wed Aug 07, 2013 8:02 am
by wibble82 » Thu Oct 24, 2013 8:47 pm
Do you know if there's a converter component in mmal to get into the RGB space? Documentation on it seems minimal at best!
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
by jamesh » Thu Oct 24, 2013 8:51 pm
wibble82 wrote:Do you know if there's a converter component in mmal to get into the RGB space? Documentation on it seems minimal at best!


You MIGHT be able to do that with the resize component. The HW resizer can, IIRC, do colourspace conversion as well. But don't quote me on it.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 17234
Joined: Sat Jul 30, 2011 7:41 pm
by jbeale » Thu Oct 24, 2013 8:55 pm
lagurus wrote:I think that is not needed to switch from low to high resoution. mmal offer to use splitter, which can split one video input stream to 2 output streams and then one use for write full H264 video and second (after is resized to 320x240) for image processing.

That sounds great, so I just need to figure out how to implement it. I am comfortable with C and somewhat C++ but not so much GPU stuff. Is there a general introduction to mmal programming online somewhere? I found this
http://www.jvcref.com/files/PI/document ... ml#example
but is there anything more?
User avatar
Posts: 3251
Joined: Tue Nov 22, 2011 11:51 pm
by SnowLeopard » Thu Oct 24, 2013 9:25 pm
jbeale wrote:I found this http://www.jvcref.com/files/PI/document ... ml#example but is there anything more?

That's the same as what is in the mmal.h from the userland sources, except nicely formatted and actually legible :)
https://github.com/raspberrypi/userland ... mal/mmal.h
The same person has made OMX stuff available:
http://www.jvcref.com/files/PI/document ... omponents/
which may or may not be of use to you guys.
http://www.jvcref.com/files/PI/document ... amera.html
"Due to limitations in the current underlying layers, it is not currently possible to emit the same frame at multiple resolutions. This means that to obtain both preview and video capture images simultaneously, the two ports must be at the same resolution and of the same image format. If they are different, the capture port takes precedence, and no images will be emitted from the preview port. "
Posts: 106
Joined: Sun Aug 18, 2013 6:10 am
by wibble82 » Thu Oct 24, 2013 9:43 pm
Interesting - I'll have a look at the image converter. The documentation does suggest it'll happily convert formats while resizing. Just getting my outputs actually rendering on screen first though so I can see what's going on :)
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
by wibble82 » Fri Oct 25, 2013 7:08 pm
Latest update: http://robotblogging.blogspot.co.uk/201 ... age-3.html

Got it rendering the results through opengl. No rgb conversion yet, but very promising. This basically means I can now create a pipeline which is camera -> memory -> opencv -> opengl -> screen. First attempt has the 720p version running at almost 30fps which I think I can get back.

Lets all cross our fingers for the mmal image resize component!
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
by wibble82 » Fri Oct 25, 2013 9:51 pm
Update: image resizer works! pretty slow though - seems to be chomping up lots of the gpu time.

http://robotblogging.blogspot.co.uk/201 ... age-4.html

Still, that alone opens up a fair few doors :) I'm going to look at making an easier api (syncronous 'getframe' style calls), and generating multiple resolutions next.
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
by wibble82 » Sat Oct 26, 2013 9:47 am
Now with a syncronous camera api, at 15hz at 720p now. Getting multiple levels of detail in next...

http://robotblogging.blogspot.co.uk/201 ... age-5.html
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
by jbeale » Thu Oct 31, 2013 12:52 am
I have followed the instructions at http://robotblogging.blogspot.com/2013/ ... i-for.html but I was stopped by this error when building the picamdemo. Posting this here in case anyone else does what I did: the problem was my "userland" directory was named /opt/vc/userland but in fact it must be named /opt/vc/userland-master. After renaming it thus, the make process succeeded, and the picamdemo runs.

Code: Select all
pi@raspberrypi:~/picamdemo$ make
[ 25%] Building CXX object CMakeFiles/picamdemo.dir/picam.cpp.o
In file included from /opt/vc/include/interface/vcos/vcos_assert.h:149:0,
                 from /opt/vc/include/interface/vcos/vcos.h:114,
                 from /opt/vc/include/interface/vmcs_host/vc_dispmanx.h:33,
                 from /opt/vc/include/bcm_host.h:46,
                 from /home/pi/picamdemo/mmalincludes.h:9,
                 from /home/pi/picamdemo/camera.h:3,
                 from /home/pi/picamdemo/picam.cpp:3:
/opt/vc/include/interface/vcos/vcos_types.h:38:33: fatal error: vcos_platform_types.h: No such file or directory
compilation terminated.
make[2]: *** [CMakeFiles/picamdemo.dir/picam.cpp.o] Error 1
make[1]: *** [CMakeFiles/picamdemo.dir/all] Error 2
make: *** [all] Error 2
User avatar
Posts: 3251
Joined: Tue Nov 22, 2011 11:51 pm
by wibble82 » Thu Oct 31, 2013 8:12 am
Thanks for finding that one - it occured to me it was bound to cause some issues for people who had already called their userland folder something different.

For people who already have a userland folder and don't want to rename it, the alternative solution is to go into CMakeLists.txt, and replace any references to userland-master with references to their own folder.

Enjoy! Oh, and fyi, first version of the gpu accelerated api should be ready by Monday!
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
by jbeale » Fri Nov 01, 2013 4:55 am
wibble82 wrote:Enjoy! Oh, and fyi, first version of the gpu accelerated api should be ready by Monday!

Looking forward to it (you aren't already using the GPU? Or you mean passing the data to OpenGL without using CPU?)
I have your 16-image demo http://robotblogging.blogspot.com/2013/ ... ng-on.html running, which is really great. I'm eager to try to figure out how to do some advanced processing like the image-cascade to calculate variance and estimate focus... I know what the code would look with a bunch of nested for-loops in C on the CPU, but I feel like I need some remedial reading to figure out the GPU stuff.
User avatar
Posts: 3251
Joined: Tue Nov 22, 2011 11:51 pm
by wibble82 » Fri Nov 01, 2013 8:07 am
jbeale wrote:
wibble82 wrote:Enjoy! Oh, and fyi, first version of the gpu accelerated api should be ready by Monday!

Looking forward to it (you aren't already using the GPU? Or you mean passing the data to OpenGL without using CPU?)
I have your 16-image demo http://robotblogging.blogspot.com/2013/ ... ng-on.html running, which is really great. I'm eager to try to figure out how to do some advanced processing like the image-cascade to calculate variance and estimate focus... I know what the code would look with a bunch of nested for-loops in C on the CPU, but I feel like I need some remedial reading to figure out the GPU stuff.


Hi

Yeah it's already using gpu, but there's a stall in the pipeline where cpu copies it to opengl. I've just about finish v2 of the app which is entirely gpu based, so you can do any graphics processing you want then just read the results back as the final step. Will write it up once I've fixed a few issues - hopefully Monday evening.
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
by arcanon » Fri Nov 01, 2013 9:22 am
Right now I bet the best test is to use 2 Raspberry pis.

Another idea is to mode one camera with 2 lenses. Take to cameras and stick the lenses onto one sensor. Does anyone know if this would give a good enough stereo effect? would the eye separation and so on be enough.
Posts: 36
Joined: Wed Nov 14, 2012 9:18 am
by wibble82 » Fri Nov 01, 2013 9:33 am
2 raspberry pis is the plan!

Image

Once the gpu pipeline is nice and solid, I'll get some feature recognition in there, pull out the feature info from 2 raspberry pis and pipe it to a 3rd. That'll do feature matching, some delunay triangulation and map the results into a point cloud which will progressively build up a 3d view of the surroundings.
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
by peepo » Fri Nov 01, 2013 10:39 am
$ ./picam
picam: /home/pi/picam/graphics.cpp:224: bool GfxShader::LoadVertexShader(const char*): Assertion `f' failed.
Aborted

any ideas?

thanks again
User avatar
Posts: 304
Joined: Sun Oct 21, 2012 9:36 am
by wibble82 » Fri Nov 01, 2013 10:55 am
Hi

That suggests that its failing to find and open a shader file. Are all the .glsl files in the folder that you are running the application from? They should be if you just unzipped the file to a folder in your home directory and followed the instructions. Could you list here the exact actions you took in getting the zip file, where you saved it, what you typed to extract it and where you extracted it to?

Thanks

-Chris
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
by peepo » Fri Nov 01, 2013 11:21 am
many thanks for the prompt response,

I had mistakenly built picam in a folder /build...
now works for me thanks!

watched the video, but all the manual input happens off-screen
are there instructions to accompany the video?
ie how can I mimic it?

is there a really stripped down introduction to this GPU-openCV approach?
ie the codebase is already quite large...

many thanks!!
User avatar
Posts: 304
Joined: Sun Oct 21, 2012 9:36 am
by arcanon » Fri Nov 01, 2013 12:47 pm
and are you going to put your code in a repository? github or the likes
Posts: 36
Joined: Wed Nov 14, 2012 9:18 am
by wibble82 » Fri Nov 01, 2013 1:31 pm
@peepo the controls to replicate the demo are 'a', 's' and 'd' on the keyboard. Sorry - I should have mentioned that. 'q' is also 'quit'.

The picam.cpp code is the main bit that actually shows how to use the api. Hopefully the blog itself should give enough of a guide as to how to do the most basic 'startcamera', then 'readframe' function, which are the only 2 bits that are actually necessary to use the camera system. If it would help I can post up there an example of the simplest possible code to use it?

The rest of the code files are part of the internals of the api and are quite complex as it's none-trivial to get access to the camera data - that's why I made the api!

@arcanon for the moment I'll be uploading zip files, as most of my work is in my own private svn repo - not because I wanted to hide it, just because I already had it running for other projects and it is the most convenient place to back up my work. Unfortunately it's a paid account with limited user access, although I will ask the chaps at codespaces what the deal is with guest access to the data. I may switch to git at some point, but not in the very near future.
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
by peepo » Fri Nov 01, 2013 3:39 pm
Chris,

thanks for the keyboard input details,
I usually connect to RPi over ethernet
and hence save output as .h264 to file or stream using rtsp,
is this possible currently?
my raspivid based attempts failed. eg -o is not accepted
iirc this is simply sending data to stdout,
however as I have no idea about GPU....

Of course, I appreciate the intention of providing an API
but the explanation can be even more useful,
in the sense that one can develop a true understanding.

in my cursory run ~7fps
cpu was not running anything else, beyond system
seems slow, why would this be?

very much looking forward to your further blog reports

thanks again
User avatar
Posts: 304
Joined: Sun Oct 21, 2012 9:36 am