[SOLVED] mmal_port_send_buffer - ENOMEM?

[ShiftPlusOne]

In raspicam, I see snippets of code like this:

Code: Select all

               num = mmal_queue_length(state.camera_pool->queue);

               for (q=0;q<num;q++)
               {
                  MMAL_BUFFER_HEADER_T *buffer = mmal_queue_get(state.camera_pool->queue);

                  if (!buffer)
                     vcos_log_error("Unable to get a required buffer %d from pool queue", q);

                  if (mmal_port_send_buffer(camera_still_port, buffer)!= MMAL_SUCCESS)
                     vcos_log_error("Unable to send a buffer to camera output port (%d)", q);
               }

I gather we're taking a queue of free buffers from a pool we created earlier and are passing each buffer to the port. No problems there.

Code: Select all

	mmal_buffer_header_release(buffer);

	// and send one back to the port (if still open)
	if (!port->is_enabled)
		return;
	new_buffer = mmal_queue_get(pool->queue);

	if (new_buffer)
		status = mmal_port_send_buffer(port, new_buffer);

	if (!new_buffer || status != MMAL_SUCCESS)
		printf("Unable to return a buffer to the encoder port. \n");

Here, we release a used buffer onto the queue, ask for another one and send it to the port. Again, makes sense.

I have identical code, but instead of using opaque buffers and connecting the camera's video port to the encoder, I'm using yuv encoding and taking the output straight from the video port. I would think that it would all work in a similar way, but that first snippet of code doesn't quite work.

Code: Select all

	for (i=0;i<queue_length;i++){
		printf ("Sending buffer %i/%i...\n", i+1, queue_length);
		buffer = mmal_queue_get(pool->queue);
		if (!buffer)
			printf("Unable to get a required buffer %d from pool queue.\n", i);

		status = mmal_port_send_buffer(camera->output[MMAL_CAMERA_VIDEO_PORT], buffer);
		if (status != MMAL_SUCCESS)
			printf("Unable to send a buffer to encoder output port (%d): error %d\n", i, status);
	}

And I get the following output:

Sending buffer 1/3...
Sending buffer 2/3...
mmal: mmal_port_send_buffer: vc.ril.camera:out:1(I420): send failed: ENOMEM
Unable to send a buffer to video output port (1): error 1
Sending buffer 3/3...
mmal: mmal_port_send_buffer: vc.ril.camera:out:1(I420): send failed: ENOMEM
Unable to send a buffer to video output port (2): error 1

What's the issue? In both cases, I'm sending three buffers to a port, yet I get ENOMEM here. I have lots of free ARM and GPU memory, so I don't believe that's the issue.

[jamesh]

Have you crossed referenced against the raspiyuv code - that takes YUV buffers directly from the camera port and sends them to file, so is making a round trip to the ARM.

[ShiftPlusOne]

Looking at raspistillyuv, it seems like you're using the still port where I'm using the video port. What's the actual difference between them? For that matter, what's the difference between a the preview port as well? Is it all just different resolutions, or is there more to it?

[6by9]

Port usage comes originally from the IL spec (find it on khronos.org) where they defined a preview and video (capture) port. That was extended for use cases such as video snapshot where you want a still whilst video encoding. IL is all based around making a graph of tunneled components, and only events get relayed to the client.

preview and video ports need to run at the same resolution, and will get the same frames during normal operation. If they are both enabled with different parameters, then the video port takes preference and you'll get no preview.
stills port enables lots of extra image processing, and is designed to work with a different resolution.


As to your error, you're likely to have a mismatch between the number of buffers you created in your pool vs the format->buffer_num.
Unless you enable zero copy (needs VCSM which I don't believe is enabled on the Pi due to issues in the cutdown firmware), then when you enable a port there is an allocation done on the GPU side for format->buffer_num buffers of format->buffer_size. When you then send a buffer from userspace to the GPU, it tries to associate the provided buffer with a GPU side buffer - if there isn't one, then the send fails.(It will also copy buffer->length bytes from source to destination in VCHI. ARM side buffers don't have to be contiguous in memory, GPU side does).

[ethanol100]

I have uploaded a version of raspivid which uses YUV to github. I have just mixed RaspiVid.c and RaspiStillYUV.c. Getting YUV data form the video port works in the same way as for the still port.
https://github.com/ethanol100/RaspiVidYUV

[ShiftPlusOne]

Port usage comes originally from the IL spec (find it on khronos.org) where they defined a preview and video (capture) port. That was extended for use cases such as video snapshot where you want a still whilst video encoding. IL is all based around making a graph of tunneled components, and only events get relayed to the client.

preview and video ports need to run at the same resolution, and will get the same frames during normal operation. If they are both enabled with different parameters, then the video port takes preference and you'll get no preview.
stills port enables lots of extra image processing, and is designed to work with a different resolution.

Ah, that's all good to know. I've been trying to get by with raspicam source code and the doxygen documentation, but there are a lot of holes there. I guess I'll need to dig into the openmax specs as well.

As to your error, you're likely to have a mismatch between the number of buffers you created in your pool vs the format->buffer_num.
Unless you enable zero copy (needs VCSM which I don't believe is enabled on the Pi due to issues in the cutdown firmware), then when you enable a port there is an allocation done on the GPU side for format->buffer_num buffers of format->buffer_size. When you then send a buffer from userspace to the GPU, it tries to associate the provided buffer with a GPU side buffer - if there isn't one, then the send fails.(It will also copy buffer->length bytes from source to destination in VCHI. ARM side buffers don't have to be contiguous in memory, GPU side does).

Code: Select all

124             pool = mmal_port_pool_create(camera->output[MMAL_CAMERA_VIDEO_PORT], camera->output[MMAL_CAMERA_VIDEO_PORT]->buffer_num, camera->output[MMAL_CAMERA_VIDEO_PORT]->buffer_size);
(gdb) p *camera->output[1]
$1 = {priv = 0x612c0, name = 0x61548 "vc.ril.camera:out:1(I420)", type = MMAL_PORT_TYPE_OUTPUT, index = 1, index_all = 2, is_enabled = 1, format = 0x418c0, buffer_num_min = 1, buffer_size_min = 460800, buffer_alignment_min = 0, 
  buffer_num_recommended = 1, buffer_size_recommended = 460800, buffer_num = 3, buffer_size = 460800, component = 0x41650, userdata = 0x0, capabilities = 2}
(gdb) p *camera->output[1]->format
$2 = {type = MMAL_ES_TYPE_VIDEO, encoding = 808596553, encoding_variant = 808596553, es = 0x418e0, bitrate = 0, flags = 0, extradata_size = 0, extradata = 0x0}
(gdb) p *camera->output[1]->format->es
$3 = {audio = {channels = 640, sample_rate = 480, bits_per_sample = 0, block_align = 0}, video = {width = 640, height = 480, crop = {x = 0, y = 0, width = 640, height = 480}, frame_rate = {num = 1966080, den = 65536}, par = {num = 0, 
      den = 0}, color_space = 0}, subpicture = {x_offset = 640, y_offset = 480}}
(gdb) n
125             if (!pool)
(gdb) p *pool
$4 = {queue = 0x63870, headers_num = 3, header = 0x63920}

I have uploaded a version of raspivid which uses YUV to github. I have just mixed RaspiVid.c and RaspiStillYUV.c. Getting YUV data form the video port works in the same way as for the still port.
https://github.com/ethanol100/RaspiVidYUV

Thanks. I started with raspivid, cut out all the jamesh-isms and configuration options to get the minimal version that does just the things I need and then started hacking away. Now that I know what a little more about how it all fits together, maybe binning what I have so far and starting with your raspividyuv might help avoid whatever silly mistake I made.

[6by9]

Code: Select all

124             pool = mmal_port_pool_create(camera->output[MMAL_CAMERA_VIDEO_PORT], camera->output[MMAL_CAMERA_VIDEO_PORT]->buffer_num, camera->output[MMAL_CAMERA_VIDEO_PORT]->buffer_size);
(gdb) p *camera->output[1]
$1 = {priv = 0x612c0, name = 0x61548 "vc.ril.camera:out:1(I420)", type = MMAL_PORT_TYPE_OUTPUT, index = 1, index_all = 2, is_enabled = 1, format = 0x418c0, buffer_num_min = 1, buffer_size_min = 460800, buffer_alignment_min = 0, 
  buffer_num_recommended = 1, buffer_size_recommended = 460800, buffer_num = 3, buffer_size = 460800, component = 0x41650, userdata = 0x0, capabilities = 2}
(gdb) p *camera->output[1]->format
$2 = {type = MMAL_ES_TYPE_VIDEO, encoding = 808596553, encoding_variant = 808596553, es = 0x418e0, bitrate = 0, flags = 0, extradata_size = 0, extradata = 0x0}
(gdb) p *camera->output[1]->format->es
$3 = {audio = {channels = 640, sample_rate = 480, bits_per_sample = 0, block_align = 0}, video = {width = 640, height = 480, crop = {x = 0, y = 0, width = 640, height = 480}, frame_rate = {num = 1966080, den = 65536}, par = {num = 0, 
      den = 0}, color_space = 0}, subpicture = {x_offset = 640, y_offset = 480}}
(gdb) n
125             if (!pool)
(gdb) p *pool
$4 = {queue = 0x63870, headers_num = 3, header = 0x63920}

Were those the values as set when the port was enabled, or have you changed them since? The GPU allocates on port enable, so any changes afterwards are going to be bad news.

You have GPU source, so you should be able to enable MMAL logging (interface/mmal/core/mmal_logging.h IIRC) to see what is going on there.

[ShiftPlusOne]

Were those the values as set when the port was enabled, or have you changed them since? The GPU allocates on port enable, so any changes afterwards are going to be bad news.

Bang on! I was setting buffer_num right after enabling the port. All works as expected now. Thank you so much, I've been banging my head on this for quite a while today.

[ShiftPlusOne]

While you and james are here, I want to make sure that what I want to do is sensible.

1) I want to get a low res, grayscale opencv image for each frame without using any CPU.
2) a) Display full-res, 30fps colour preview with a dispmanx overlay
OR
b) Get a full-res 30fps colour opengl es frame, apply a shader for fun stuff along with an overlay and display that scene instead.

1 and 2a seem simple and I already have most of it. 1 and 2b seems a bit tricky because I'm not sure what's involved in getting an egl texture out.

I'll dig through raspitex when I get the easy version done, but was wondering if it's sensible and I won't run into an issue where I can't get a texture and a low res frame at the same time without eating CPU.

[6by9]

Hmm, I know how the texture format stuff was handled on other platforms, but not exactly how Tim managed it on Pi. Raspitex is probably your only source of info there.

Sounds like you want:
camera output[0] -> video_render or GL.
camera output[1] -> resize -> app for CV. Just take the luma plane from I420.

Resize will cost a moderate amount on VC, as will the conversion to a texture for GL, but both are reasonable to expect to be fine unless you're trying 1080P.

[ShiftPlusOne]

How would you do the resize?

Currently, I'm just changing the format resolution on the video port format. You said that preview and video ports will have the same resolution, so if I take output[0] for gl, it will be low res.

Would it not make sense to use video and still ports instead? Then I get something I can use for cv and a full-res frame for gl for free, no?

If I need to do a resize as a separate step, that's not really an issue. Although I'm trying to minimize the work done by the ARM, eventually opencv will need to process the grayscale image and I will be dropping the grayscale frames which come during that time without doing anything to them. So it's not like I need to do a resize for every single frame, only whatever opencv can keep up with.

[jamesh]

I pressume Dave is referring to the resize component. IIRC this is done by the VPU so pretty quick, but not in dedicated HW, so there is a impact on VC performance, as Dave says. It does leave the ARM well alone though.

[ShiftPlusOne]

Ah, didn't know there was a resize component. Thanks James, I'll give that a go if I find the approach I'm taking isn't going to work.

It's really nice to see how flexible the whole setup is. When looking at how other people are doing opencv, it's really shame to see how much CPU they use. Pretty much everyone does a memcpy from the buffer, rather than using cvSetData. I've seen people do YUB2RGB, RGB2BGR conversion, resizing and rendering all using the ARM.

[jamesh]

I think its down to not knowing what is available, and the actual complexity of implemented either mmal or OpenMAX stuff, which is non-trivial.

I've currently got fingers crossed that the resize component is supplied - I'm pretty sure it is...

[6by9]

resize should certainly be included as I know PiCamera uses it.

Conversions shouldn't be necessary now on the ARM where they are very expensive - the camera component supports YUV, YVU, RGB, and BGR, so that should deal with most situations. Several of the other components are significantly more restricted, so don't expect all of those from the output of resize. As James says, it's down to knowing what is available, and some things have been added over time rather than always having been there.

[ShiftPlusOne]

Fair enough, thanks again for the help.

[ShiftPlusOne]

Agh, after switching from the video port to the still port, the video buffer callback occurs only once. I guess it makes sense for the still port and might have something to do with one of the default parameters. I'll pick this up tomorrow, since it's 2AM and I have a home to go to.

In case someone stumbles upon this thread with similar questions, this site is very helpful
http://www.jvcref.com/files/PI/document ... amera.html
http://www.jvcref.com/files/PI/document ... index.html

[6by9]

Stills port clears the capturing flag after each capture, so you need to reset MMAL_PARAMETER_CAPTURING after each image is complete.

There is a doc by TJG for all the IL components which you may have access to (ask Dom), otherwise the scripts for generating it are in the VC source tree.

[ShiftPlusOne]

Stills port clears the capturing flag after each capture, so you need to reset MMAL_PARAMETER_CAPTURING after each image is complete.

Is this something that can't be changed or is there a parameter that can be twiddled to prevent that behaviour?

There is a doc by TJG for all the IL components which you may have access to (ask Dom), otherwise the scripts for generating it are in the VC source tree.

Got it, thank you. Seems to be the same thing as the pages I linked in the last post though.

Last question (I swear): Can I connect the video output to the resizer AND enable a buffer callback or do I need to use a splitter?

I think I've got plenty of information to go on now, so I won't be bugging you guys too much now.

[6by9]

Stills port clears the capturing flag after each capture, so you need to reset MMAL_PARAMETER_CAPTURING after each image is complete.

Is this something that can't be changed or is there a parameter that can be twiddled to prevent that behaviour?

Nominally the one_shot parameter in MMAL_PARAMETER_CAMERA_CONFIG_T would allow continuous capture, but it hadn't been used in literally years so may not still work, and also you will then not get any images from the video port at all as it will spend all its time doing stills. You should still get images from the preview port at the same rate as from stills, as it creates a preview sized image based on the capture frame.

There is a doc by TJG for all the IL components which you may have access to (ask Dom), otherwise the scripts for generating it are in the VC source tree.

Got it, thank you. Seems to be the same thing as the pages I linked in the last post though.

Hmm, surprised that is in the public domain. That is the doc I was referring to though.

Last question (I swear): Can I connect the video output to the resizer AND enable a buffer callback or do I need to use a splitter?

Nope. One source goes to one sink. Splitters required for any one to many connections.

[ShiftPlusOne]

Makes sense. Currently I'm trying to get preview connected to the video renderer, video port to resizer, resizer callback to my opencv stuff. Then when I get to the egl stuff, either use a splitter on the video port, or try to use the preview port. So yeah, plenty to play around with for now.

[jwatte]

Currently I'm trying to get preview connected to the video renderer, video port to resizer, resizer callback to my opencv stuff

Did you get that working? Any notes, samples, or suggestions?

[ShiftPlusOne]

Yeah, I've got it all on my computer somewhere. The plan was to clean it up and push it as a hello_mmal example on github. Unfortunately, I got carried away with other stuff and didn't get around to it. It dropped down the priority list when I saw that there's not much of a performance improvement over what people are already doing.

As far as the code itself goes, IIRC I had to use yuv instead of opaque when passing things between certain components, but everything works as advertised.