User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

feed MMAL video into jpeg encoder or get faster stills

Wed Jun 12, 2013 9:04 am

I'm working on an Mjpeg streamer plugin for the raspicam.

Is it possible to pass video into the jpeg encoder component? Or will that only take images from the camera still port?

If not, is it possible to tell the camera to only do the light levels once when opened? as oppose to every shot? or some way to get faster sequential shots? any delay under 100ms between releasing the buffer mutex lock with:

Code: Select all

vcos_semaphore_post(&(pData->complete_semaphore));
then sending the buffers to the encoder output and starting the capture with:

Code: Select all

mmal_port_parameter_set_boolean(camera_still_port, MMAL_PARAMETER_CAPTURE, 1)
just gives me a black image.

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: feed MMAL video into jpeg encoder or get faster stills

Wed Jun 12, 2013 1:57 pm

jacksonliam wrote:I'm working on an Mjpeg streamer plugin for the raspicam.

Is it possible to pass video into the jpeg encoder component? Or will that only take images from the camera still port?

If not, is it possible to tell the camera to only do the light levels once when opened? as oppose to every shot? or some way to get faster sequential shots? any delay under 100ms between releasing the buffer mutex lock with:

Code: Select all

vcos_semaphore_post(&(pData->complete_semaphore));
then sending the buffers to the encoder output and starting the capture with:

Code: Select all

mmal_port_parameter_set_boolean(camera_still_port, MMAL_PARAMETER_CAPTURE, 1)
just gives me a black image.
Don't you think mjpeg will be covered by the upcoming V4L kernel driver?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26685
Joined: Sat Jul 30, 2011 7:41 pm

Re: feed MMAL video into jpeg encoder or get faster stills

Wed Jun 12, 2013 3:02 pm

I'm not sure MJPEG will be in the V4L2 driver. It still needs to be supported by the underlying encoder to get decent speed out of it. Although I am not sure why it's not supported. Probably because its never been required.

To the OP - the reason you get a black screen if you leave less than 100ms delay between end of one capture and start of next is that the process the system goes through when captuing is as follows :

Run camera in preview mode
Capture requested
Wait for end of frame, switch camera mode to stills capture (takes 1 frame minimum at 30fps = 30-60ms)
Grab frame (assume a max fame rate of 15fps from the sensor itself, means 60ms approx to capture frame)
Switch camera mode to preview (takes 1 frame minimum at 30fps = 30-60ms)

So, that comes to more than 100ms approx.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

Re: feed MMAL video into jpeg encoder or get faster stills

Wed Jun 12, 2013 4:47 pm

jamesh wrote:I'm not sure MJPEG will be in the V4L2 driver. It still needs to be supported by the underlying encoder to get decent speed out of it. Although I am not sure why it's not supported. Probably because its never been required.

To the OP - the reason you get a black screen if you leave less than 100ms delay between end of one capture and start of next is that the process the system goes through when captuing is as follows :

Run camera in preview mode
Capture requested
Wait for end of frame, switch camera mode to stills capture (takes 1 frame minimum at 30fps = 30-60ms)
Grab frame (assume a max fame rate of 15fps from the sensor itself, means 60ms approx to capture frame)
Switch camera mode to preview (takes 1 frame minimum at 30fps = 30-60ms)

So, that comes to more than 100ms approx.
Thanks,
Okay, Since I'm not using preview, can I keep the camera in stills mode, 15fps would probably be acceptable.
If not do you know If I can feed video frames into the jpeg encoder, or will it only read from the stills port on the camera (Don't wanna waste time trying if its not possible).

Otherwise, I think I might have to pull some kinda frame from the camera and send it off to the OMX jpeg encoder.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26685
Joined: Sat Jul 30, 2011 7:41 pm

Re: feed MMAL video into jpeg encoder or get faster stills

Wed Jun 12, 2013 5:14 pm

jacksonliam wrote:
jamesh wrote:I'm not sure MJPEG will be in the V4L2 driver. It still needs to be supported by the underlying encoder to get decent speed out of it. Although I am not sure why it's not supported. Probably because its never been required.

To the OP - the reason you get a black screen if you leave less than 100ms delay between end of one capture and start of next is that the process the system goes through when captuing is as follows :

Run camera in preview mode
Capture requested
Wait for end of frame, switch camera mode to stills capture (takes 1 frame minimum at 30fps = 30-60ms)
Grab frame (assume a max fame rate of 15fps from the sensor itself, means 60ms approx to capture frame)
Switch camera mode to preview (takes 1 frame minimum at 30fps = 30-60ms)

So, that comes to more than 100ms approx.
Thanks,
Okay, Since I'm not using preview, can I keep the camera in stills mode, 15fps would probably be acceptable.
If not do you know If I can feed video frames into the jpeg encoder, or will it only read from the stills port on the camera (Don't wanna waste time trying if its not possible).

Otherwise, I think I might have to pull some kinda frame from the camera and send it off to the OMX jpeg encoder.
You could feed continuous frames in to the JPEG encoder as timelapse mode does, but not as fast as any sort of video - the sensor is capable of 15fps, but the internals of the GPU cannot go that fast at full resolution.

To keep the camera in stills mode would require a firmware change, unless the feature is there already, which I would need to check, but I think something like continuous shot mode would be limited to a small number of images (just like a normal PAS camera)
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

Re: feed MMAL video into jpeg encoder or get faster stills

Wed Jun 12, 2013 5:33 pm

Thanks, think I'm pretty stuck at the ~7FPS I've got then.
I do it exactly how raspistill time-lapse does, but without creating a preview or doing anything with the preview port at all and I memcpy to the mjpeg-streamer buffer rather than write to a file.

I did see something about 'one shot stills' (its set to 1, as is raspistill) and 'circular buffer height' (set to 0) but I had a play and it doesn't seem to do anything.

As a Side note, the jpeg encoder doesn't seem to spit out all the flags, only MMAL_BUFFER_HEADER_FLAG_FRAME_END with the buffer that ends the frame. Had me stumped because I was looking for MMAL_BUFFER_HEADER_FLAG_FRAME_START, had to assume start would be the next buffer after FRAME_END.

User avatar
jbeale
Posts: 3682
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: feed MMAL video into jpeg encoder or get faster stills

Wed Jun 12, 2013 8:03 pm

jacksonliam wrote:Thanks, think I'm pretty stuck at the ~7FPS I've got then.
I do it exactly how raspistill time-lapse does, but without creating a preview or doing anything with the preview port at all and I memcpy to the mjpeg-streamer buffer rather than write to a file.
Are you able to write full-frame JPEG images to a file at a sustained rate of 7 fps? That is certainly better than the ~ 1 fps that we have now (as far as I know?) Even if it was downscaled in pixel resolution (but still covering the full sensor field of view) that would be quite useful. Actually, if you look at the images pixels at 100% you seldom really approach 5Mpixel resolution, but if you downscale to 50% the image becomes crisp and really quite good at the pixel level.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26685
Joined: Sat Jul 30, 2011 7:41 pm

Re: feed MMAL video into jpeg encoder or get faster stills

Wed Jun 12, 2013 8:06 pm

jacksonliam wrote:Thanks, think I'm pretty stuck at the ~7FPS I've got then.
I do it exactly how raspistill time-lapse does, but without creating a preview or doing anything with the preview port at all and I memcpy to the mjpeg-streamer buffer rather than write to a file.

I did see something about 'one shot stills' (its set to 1, as is raspistill) and 'circular buffer height' (set to 0) but I had a play and it doesn't seem to do anything.

As a Side note, the jpeg encoder doesn't seem to spit out all the flags, only MMAL_BUFFER_HEADER_FLAG_FRAME_END with the buffer that ends the frame. Had me stumped because I was looking for MMAL_BUFFER_HEADER_FLAG_FRAME_START, had to assume start would be the next buffer after FRAME_END.
The JPEG encoder is designed for one frame at a time, not a continuous stream. So you will only get the frame end, not a frame start as it assume the start of capture is the start, and that the software using it knows that. Although this is an interesting point - might ask the mmal team about that.

However, I doubt much will change as dev time is aimed in a slightly difference direction at the moment.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

Re: feed MMAL video into jpeg encoder or get faster stills

Wed Jun 12, 2013 8:51 pm

jbeale wrote:
jacksonliam wrote:Thanks, think I'm pretty stuck at the ~7FPS I've got then.
I do it exactly how raspistill time-lapse does, but without creating a preview or doing anything with the preview port at all and I memcpy to the mjpeg-streamer buffer rather than write to a file.
Are you able to write full-frame JPEG images to a file at a sustained rate of 7 fps? That is certainly better than the ~ 1 fps that we have now (as far as I know?) Even if it was downscaled in pixel resolution (but still covering the full sensor field of view) that would be quite useful. Actually, if you look at the images pixels at 100% you seldom really approach 5Mpixel resolution, but if you downscale to 50% the image becomes crisp and really quite good at the pixel level.
I don't think anything is written to a file with mjpeg-streamer, unless you use the file output plugin.
I think raspistill can achieve the same framerate, just run it in timelapse mode with a delay of 100.
jamesh wrote:
jacksonliam wrote:Thanks, think I'm pretty stuck at the ~7FPS I've got then.
I do it exactly how raspistill time-lapse does, but without creating a preview or doing anything with the preview port at all and I memcpy to the mjpeg-streamer buffer rather than write to a file.

I did see something about 'one shot stills' (its set to 1, as is raspistill) and 'circular buffer height' (set to 0) but I had a play and it doesn't seem to do anything.

As a Side note, the jpeg encoder doesn't seem to spit out all the flags, only MMAL_BUFFER_HEADER_FLAG_FRAME_END with the buffer that ends the frame. Had me stumped because I was looking for MMAL_BUFFER_HEADER_FLAG_FRAME_START, had to assume start would be the next buffer after FRAME_END.
The JPEG encoder is designed for one frame at a time, not a continuous stream. So you will only get the frame end, not a frame start as it assume the start of capture is the start, and that the software using it knows that. Although this is an interesting point - might ask the mmal team about that.

However, I doubt much will change as dev time is aimed in a slightly difference direction at the moment.
That makes sense. Cheers for the replies, pitty I can't make this any better but never mind, it works fairly well as it is!

ppumkin
Posts: 82
Joined: Tue May 29, 2012 10:22 pm

Re: feed MMAL video into jpeg encoder or get faster stills

Wed Nov 13, 2013 8:42 pm

Thanks for the great work. The main advantage is that CPU usage is low. Without any over clocking about ~10%

I tried gstreamer pipe to jpenc but that goes to 85% for full frame size for a less FPS. Also tried ffmpeg combination with tmpfs and got about 3~5fps @ 640x480 but CPU was also 60% any could not get it more than 5 FPS

Really great works! Thank you very much again!

Return to “Camera board”