robmurg
Posts: 5
Joined: Thu Jun 13, 2013 8:57 am

Capture raw image to memory buffer (C/C++)

Thu Jun 13, 2013 9:08 am

Could anyone kindly point me to some sample code/information showing how to capture a still image directly to a memory buffer for further processing? I'm hoping to bypass JPEG encoding/decoding or loading from a disk to reduce overhead and free cycles to use processing the image. I'm trying to integrate the camera into a C application I've been tinkering with for a while.

Thanks in advance!

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24139
Joined: Sat Jul 30, 2011 7:41 pm

Re: Capture raw image to memory buffer (C/C++)

Thu Jun 13, 2013 9:33 am

Check out the raspistillYUV application. This saves the uncompressed YUV output from the camera to file - but could easily be adapted to keep it in memory.

JPEG encoding doesn't cost any CPU time as it's all done on the GPU.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

robmurg
Posts: 5
Joined: Thu Jun 13, 2013 8:57 am

Re: Capture raw image to memory buffer (C/C++)

Thu Jun 13, 2013 3:55 pm

Many thanks James, that looks very promising. If I achieve anything interesting or useful I'll post it back here. :)

viilaaja
Posts: 3
Joined: Thu Jun 13, 2013 4:22 pm

Re: Capture raw image to memory buffer (C/C++)

Thu Jun 13, 2013 4:32 pm

Take a look at
http://viilaamo.com/raspicamerapages/raspicamindex.html
there is code that captures the raw data into a buffer and also instructions how to process that using OpenCV (demosaic the raw image).

Hope this helps!

fdimage
Posts: 2
Joined: Tue Jun 18, 2013 8:30 am

Re: Capture raw image to memory buffer (C/C++)

Tue Jun 18, 2013 8:41 am

check the codes of capturing raw image in c#, and you can also use the image app to process images: resize, crop zoom and so on.

robmurg
Posts: 5
Joined: Thu Jun 13, 2013 8:57 am

Re: Capture raw image to memory buffer (C/C++)

Tue Jun 25, 2013 9:32 pm

Thanks all,

I managed to capture an image to a buffer by looking at raspistillYUV (easier than I expected), and convert it from YUV to RGB (more difficult than I expected).

I wonder: Can I hope to establish a continuous stream of images from the camera via the callback without having to re-initialise the camera? I've tried putting
(mmal_port_parameter_set_boolean(camera_still_port, MMAL_PARAMETER_CAPTURE, 1)
vcos_semaphore_wait(&callback_data.complete_semaphore)
in a loop but that didn't appear to work. Am I missing something or barking up the wrong tree?

Thanks for help so far!

Rob

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24139
Joined: Sat Jul 30, 2011 7:41 pm

Re: Capture raw image to memory buffer (C/C++)

Wed Jun 26, 2013 8:31 am

robmurg wrote:Thanks all,

I managed to capture an image to a buffer by looking at raspistillYUV (easier than I expected), and convert it from YUV to RGB (more difficult than I expected).

I wonder: Can I hope to establish a continuous stream of images from the camera via the callback without having to re-initialise the camera? I've tried putting
(mmal_port_parameter_set_boolean(camera_still_port, MMAL_PARAMETER_CAPTURE, 1)
vcos_semaphore_wait(&callback_data.complete_semaphore)
in a loop but that didn't appear to work. Am I missing something or barking up the wrong tree?

Thanks for help so far!

Rob
Look in the raspistill app at the timelapse mode, which takes continuous pictures without turning off the camera.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

User avatar
dozencrows
Posts: 172
Joined: Sat Aug 04, 2012 6:02 pm

Re: Capture raw image to memory buffer (C/C++)

Wed Jun 26, 2013 1:07 pm

I've been having a similar issue to robmurg when trying to continuously capture still YUV images. I suspect it's my own problem, but my code is closely based on raspistill timelapse... and it will only take one image and then stop.

Here's my original post about this: http://www.raspberrypi.org/phpBB3/viewt ... 05#p370505

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24139
Joined: Sat Jul 30, 2011 7:41 pm

Re: Capture raw image to memory buffer (C/C++)

Wed Jun 26, 2013 1:50 pm

I'm making some changes to the apps at the moment, I'll take the opportunity to add the timelapse support to RaspiYUV, and see if it works OK.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

robmurg
Posts: 5
Joined: Thu Jun 13, 2013 8:57 am

Re: Capture raw image to memory buffer (C/C++)

Wed Jun 26, 2013 6:35 pm

I've made some progress too - I included the loop to send buffers to the camera_still_port in each capture loop
for (q=0;q<num;q++) {
MMAL_BUFFER_HEADER_T *buffer = mmal_queue_get(state.camera_pool->queue);
etc.
and changed the
.one_shot_stills = 0 to .one_shot_stills = 1
when creating the camera component. Not really sure what I'm doing but I'm managing to capture frames in sequence now... I'm seeing about 1.5 fps at 1280x1024 without doing any additional processing. I'll keep at it.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24139
Joined: Sat Jul 30, 2011 7:41 pm

Re: Capture raw image to memory buffer (C/C++)

Thu Jun 27, 2013 8:45 am

I've also found a problem in the callback code where is is using the EOS flag is tested to end capture, which needs to be removed. It was causing a seg fault in my tests.

I've made the changes and done some testingand will push them today to Dom for inclusion in the userland tree.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

User avatar
dozencrows
Posts: 172
Joined: Sat Aug 04, 2012 6:02 pm

Re: Capture raw image to memory buffer (C/C++)

Sat Jun 29, 2013 2:52 pm

I've resolved my issue -

Code: Select all

one_shot_stills
needed to be 1 in order to repeatedly capture frames.

That seems to be the wrong way around, though!

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24139
Joined: Sat Jul 30, 2011 7:41 pm

Re: Capture raw image to memory buffer (C/C++)

Sun Jun 30, 2013 10:54 am

I fixed and tested a timelapse mode for raspistillYUV which should be pushed to userland in the near future,
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

Dweller
Posts: 10
Joined: Sun Jun 30, 2013 7:14 pm

Re: Capture raw image to memory buffer (C/C++)

Sun Jun 30, 2013 7:29 pm

I've been playing with code to repeatedly grab jpgs.. like the guys over in http://www.raspberrypi.org/phpBB3/viewt ... 43&t=45178
I've also coded a custom mjpg streamer plugin, and found if the interval goes below ~100ms, that the images just go black.. (actual threshold seems to be around 63ms for a 640x480 capture.. )

So I have a few questions around the usage of the image capture api.

I noticed in the Raspistill code there's this little bit that configures the camera..

Code: Select all

         MMAL_PARAMETER_CAMERA_CONFIG, sizeof(cam_config) },
         .max_stills_w = state->width,
         .max_stills_h = state->height,
         .stills_yuv422 = 0,
         .one_shot_stills = 1,
         .max_preview_video_w = state->preview_parameters.previewWindow.width,
         .max_preview_video_h = state->preview_parameters.previewWindow.height,
         .num_preview_video_frames = 3,
         .stills_capture_circular_buffer_height = 0,
         .fast_preview_resume = 0,
         .use_stc_timestamp = MMAL_PARAM_TIMESTAMP_MODE_RESET_STC
And I'm wondering if that stills_capture_circular_buffer_height might let me retrieve images faster than 100ms apart ? Are there docs for the one_shot_stills and stills_capture_circular_buffer_height values?

I notice I'm being sent the jpeg spread across 3 or so buffer callbacks.. would it be faster if I could tell it to use larger buffers so it could send back an entire image in one callback ?

Does it help to raise the frame rate on the stills format? I've tried with it at 30, but can't really tell if it helps (based from another post where it was said that stills grabs the next frame, after any frame in progress, so hopefully more frames = less interval lost waiting).

Aiming for the lowest latency image streaming I can get to (which is why I'm on mjpg, not h264, as although the h264 stream had a lowish overhead, converting it to anything I could use in a browser added seconds of latency)..

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24139
Joined: Sat Jul 30, 2011 7:41 pm

Re: Capture raw image to memory buffer (C/C++)

Sun Jun 30, 2013 9:30 pm

There isn't really anything you can do with the settings you reference that will make it go any faster.

The max frame rate of the sensor during stills capture at full resolution is 15fps. We cannot actually get that speed out of the ISP, not sure exactly what speed you can get, but I doubt more than 5fps.

Video capture is upto 30fps at 1080p

100ms between captures seems pretty good to me, but I don't know what resolution you are using to get that.

Note in stills capture, the ISP has to change camera mode from preview (a video) mode to stills mode, it then needs to capture a frame, then go back to preview mode - that takes a fair while - a few frames at 30ms each. That setup each side of the actual capture is going to add up to around 100ms between shots I reckon.

For fast frame rates you should be using video mode. Not sure why H264 is proving problematic in browsers - its very much the standard.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

fastmapper
Posts: 27
Joined: Wed Mar 27, 2013 3:54 am

Re: Capture raw image to memory buffer (C/C++)

Sun Jun 30, 2013 10:57 pm

jamesh wrote:Note in stills capture, the ISP has to change camera mode from preview (a video) mode to stills mode, it then needs to capture a frame, then go back to preview mode - that takes a fair while - a few frames at 30ms each. That setup each side of the actual capture is going to add up to around 100ms between shots I reckon.
What about cases where preview is not needed? Real time image processing often does not require any preview. In such cases, can we expect to get stills at 15 frames per second?

Dweller
Posts: 10
Joined: Sun Jun 30, 2013 7:14 pm

Re: Capture raw image to memory buffer (C/C++)

Mon Jul 01, 2013 7:04 am

For fast frame rates you should be using video mode. Not sure why H264 is proving problematic in browsers - its very much the standard.
Oh sure, the browsers can decode h264, but not when it's an elementary video stream, the options come down to..
  • MJPG - Built in support for all browsers, each frame a full jpeg, easy to add to a page with javascript/native support, around 3-500ms latency.
  • MP4 - need to mux the h264 video into an mp4 container, current tooling to do that tends to buffer, and add latency (5secs), plus once that's achieved, some browsers like to try to download the entire stream before playing, others buffer adding yet more latency (5-10secs), (and playback via a plugin tends to have similar issues)
  • TS - more local buffering/latency to packetise the h264 stream into the mpeg2-ts, adds around 6 seconds or so via ffmpeg.
  • HLS - Http live streaming, again local buffering/latency to get it into the right container, and then HLS sends in chunks, which get downloaded by the player before playback starts, adding at least 2-4 seconds of latency.
  • NetCat - For this to work in the browser, you really have to cheat & use a plugin that can handle h264 elementary streams, like vlc, and even then you'll need to add in the SPS/PPS (another thread on this will lead you to here via here) once all this is running, it can give lowish latency (.5-1sec), but multiple connections are problematic
Having worked through all those, mjpg was by far the simplest to get running with low latency. Although the javascript reports a framerate of around 4 to 5fps, even when I get the delay down to 63ms (@640x480, which calculates out to around 15.87 fps theoretical, I wonder why it's only putting out 4 to 5)

You can observe a similar effect with small raspistill time intervals when writing to a ram disk, where it will run for longer than the time limit, because it couldn't capture the number of frames requested in the time expected. Internally raspistill loops for a calculated number of frames, based on the interval, and time limit requested, with a sleep within the loop to wait for each interval. When the interval is large, the difference is less obvious, but as you push it lower it adds up. It might just be the loop overhead, but I'd hope it could process that fast enough that a few ms wouldn't make a difference.. but inside the loop it does do a semaphore wait on the mutex controlled by the callback so the time taken for the callback to aggregate a full frame will be added to the loop time..

If the stills capture really can't go any faster, (I'm sure it can with a little more encouragement.. at least to see an fps in the browser similar the to the 15 or so the stills is running at).. then I guess if we reduce the iframe interval on the h264 stream to something nice and low, and then pull the iframes from the h264 stream, and recode them as jpegs, we might go faster, but that's not a small amount of work to test ;p

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24139
Joined: Sat Jul 30, 2011 7:41 pm

Re: Capture raw image to memory buffer (C/C++)

Mon Jul 01, 2013 8:06 am

fastmapper wrote:
jamesh wrote:Note in stills capture, the ISP has to change camera mode from preview (a video) mode to stills mode, it then needs to capture a frame, then go back to preview mode - that takes a fair while - a few frames at 30ms each. That setup each side of the actual capture is going to add up to around 100ms between shots I reckon.
What about cases where preview is not needed? Real time image processing often does not require any preview. In such cases, can we expect to get stills at 15 frames per second?
The underlying code isn't really designed to stay in capture mode - it needs to switch - but I will double check that.

You still won't get 15fps at full resolution- the ISP isn't fast enough for cope with the total bandwidth (5MP at 15fps is a lot of pixels). Isnt 1080p30 enough?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24139
Joined: Sat Jul 30, 2011 7:41 pm

Re: Capture raw image to memory buffer (C/C++)

Mon Jul 01, 2013 8:30 am

jamesh wrote:
fastmapper wrote:
jamesh wrote:Note in stills capture, the ISP has to change camera mode from preview (a video) mode to stills mode, it then needs to capture a frame, then go back to preview mode - that takes a fair while - a few frames at 30ms each. That setup each side of the actual capture is going to add up to around 100ms between shots I reckon.
What about cases where preview is not needed? Real time image processing often does not require any preview. In such cases, can we expect to get stills at 15 frames per second?
The underlying code isn't really designed to stay in capture mode - it needs to switch - but I will double check that.

You still won't get 15fps at full resolution- the ISP isn't fast enough for cope with the total bandwidth (5MP at 15fps is a lot of pixels). Isnt 1080p30 enough?
Turns out I am talking complete nonsense. 15fps at 5MP is about 75Mpixels/s which is well inside the performance envelope. Don't know what I was thinking!

And it should be possible to force the system to use full resolution mode as a preview, which means no mode switch would be required to capture which should make it faster. So, I'm going to try setting a preview resolution to the full res if I can find the right parameters.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24139
Joined: Sat Jul 30, 2011 7:41 pm

Re: Capture raw image to memory buffer (C/C++)

Mon Jul 01, 2013 8:57 am

OK, in raspistill, there is some code to set up the camera preview port.

You need to modify that to set the resolution and frame rate.

So something like this...(do search on code to find the right places)

Code: Select all

.max_preview_video_w = state->width;
.max_preview_video_h = state->height;
.
.
.
format->es->video.width = state->width;
format->es->video.height = state->height;
format->es->video.crop.width = state->width;
format->es->video.crop.height = state->height;
format->es->video.frame_rate.num = 15;
format->es->video.frame_rate.den = 1;

status = mmal_port_format_commit(preview_port);

Also do the last two for the camera stills ports as well.

Code: Select all

format->es->video.frame_rate.num = 15;
format->es->video.frame_rate.den = 1;

status = mmal_port_format_commit(still_port);
That should make the preview run in full res mode at 15fps. So the FOV will also match the capture. I've seen some colour issues between preview and capture which I will need to figure out, but give that a go to see if it improves the speed.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

Dweller
Posts: 10
Joined: Sun Jun 30, 2013 7:14 pm

Re: Capture raw image to memory buffer (C/C++)

Mon Jul 01, 2013 2:04 pm

Tried the suggested changes inside my mjpg plugin.. no real difference, avg of around 4fps as reported by the javascript.. I'll try adding some timestamped logs to the plugin next, see where the time goes.. at least that will answer if the callbacks aggregating the image take longer than the intra frame delay sleep does. It'll also answer if it's the capture loop running slow, or the mjpg streamer/display client.. those changes will have to wait till later tho..

Also, I'm still only running at 640x480, higher resolutions definitely slow the fps in the client.. by 1024x768, it's down to 3fps or so.

The circular buffer sounded interesting in the code, as I was hopeful it could quick grab the stills into the buffer, and I'd pull the stills out ;p maybe then it could be grabbing the next while I was processing the last buffer etc..

Same with the repeated callbacks to send a single 640x480 jpegs worth of data.. reducing the no of callbacks via a larger buffer size might reduce the overhead of the callbacks, plus the memcpys to aggregate the data, and each callback test for complete..

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24139
Joined: Sat Jul 30, 2011 7:41 pm

Re: Capture raw image to memory buffer (C/C++)

Mon Jul 01, 2013 3:12 pm

I'm not seeing the expected performance increase either - need to talk to someone about it because it should be faster. I've modded the raspistill app to do the change on the command line which may prove useful to people. I'll push to Dom for inclusion in userland.

As for buffers, internally, the movement of data from camera to encoder is done using handles - no memory copies, so is very fast. It's also triple buffered I think. The buffer handling from encoder to the ARM is over multiple buffers as well I think, so that is already working efficiently - making one bug buffer would probably slow it down. You might have some mileage in handling the encoder output whilst capturing the next frame, but I'm not sure how well the GPU code would cope with that - it's not a standard use case.

I'm not sure where the time delay is - we should be able to get up to 10fps with some effort I would hope. I'm also seeing a network drop out when I use this in timelapse mode with a small delay between images, which is unexpected (but might be my Pi)
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

Dweller
Posts: 10
Joined: Sun Jun 30, 2013 7:14 pm

Re: Capture raw image to memory buffer (C/C++)

Mon Jul 01, 2013 5:57 pm

No network dropouts here thankfully, else the whole attempt at low latency would be for nowt.. and mines an original pi with the polyfuses & 256mb..

Did see some issues with an app using i2c from java which seemed to introduce rather bursty lag to the mjpg stream, both work fine independently, so I'm guessing there might be some sort of resource issue caused by spamming the camera grabs with only 65ms delays .. I'm going to see what kind of fps I can get from a usb webcam (got a c920 and vx-2000 to test with).. I need to test yet, but I know some webcams do mjpg as a v4l2 protocol.. so there might be less overhead overall by just not using the pi-cam.

brooklyn22
Posts: 2
Joined: Tue Oct 15, 2013 10:03 am

Re: Capture raw image to memory buffer (C/C++)

Tue Oct 15, 2013 11:40 am

Could anyone help me with information/code for capturing a frame from camera and use it for further processing with OpenGL ES. I do not wish to use OpenCV.
I'm hoping for C++ camera library creation and linking it with OpenGL ES shaders.
Thanks in advance.

RpiName
Posts: 712
Joined: Sat Jul 06, 2013 3:14 am

Re: Capture raw image to memory buffer (C/C++)

Tue Oct 15, 2013 12:16 pm

brooklyn22 wrote:Could anyone help me with information/code for capturing a frame from camera and use it for further processing with OpenGL ES. I do not wish to use OpenCV.
I'm hoping for C++ camera library creation and linking it with OpenGL ES shaders.
Use Video4Linux2 to capture and the GPU for further processing. It's not the fastest way, but it's simple.

For the Video4Linux2 part:

API: http://linuxtv.org/downloads/v4l-dvb-apis/
Camera Driver: http://www.linux-projects.org/modules/s ... e&artid=14
Capture Example in C: http://linuxtv.org/downloads/v4l-dvb-ap ... ample.html

Return to “Camera board”