User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

mmal encoding errors [fixed]

Wed Jun 18, 2014 9:03 am

Hi,

I'm working on a quadcopter controlled by the raspberry pi and I want to use its camera for FPV and recording.
I managed to use the mmal resizer after 3 days of struggle and right now I have everything working except 2 random errors:

Code: Select all

mmal: mmal_port_buffer_header_callback: vc.ril.resize:in:0(I420): buffer headers in transit < 0 (-2)
The counter (-2) keeps decreasing. The message comes from mmal_port.c:893 and seems to mean the reference count for a buffer is decremented too much.
Although all the increments/decrements are protected by a mutex, the reading of the counter (IN_TRANSIT_COUNT macro) is not, and since the counter is not atomic one might get spurious errors like the one above.
However, the message I'm getting is continuous - a buffer counter is definitely -1. This can only mean that a buffer is sent and its reference is incremented but the callback (mmal_port_buffer_header_callback) is never called to decrement it.

The second error:

Code: Select all

mmal: mmal_port_event_send: event lost on port 1,0 (buffer header callback not defined)
This happens randomly and once it does the encoding stops. Once I setup everything I don't call any mmal api except for sending buffers in callbacks. I also never deactivate ports - the only place where the callback is put to null that I could find.

sudo vcdbg log msg -f shows the following:

Code: Select all

2529119.476: mmalsrv: on_bulk_event: bulk tx aborted 63
2529146.994: mmalsrv: send_buffer_to_host: tx failed:size 292 st -1
2529147.012: mmalsrv: send_buffer_to_host: tx failed:size 292 st -1
2529147.029: mmalsrv: send_buffer_to_host: tx failed:size 292 st -1
My setup is:
- camera output with 3 buffer pool goes into
- 720p H264 encoder
- 360p resizer, with 3 buffer pool that goes into a 360p encoder

Since the raspberry will also execute the navigation and some of the stability code (as the rate PID is running on a crius board), I need this to be extremely solid. My only solution right now is to destroy all the mmal components when this happens - but seems like a very ugly thing to do at the beginning of a project.

Any help is much appreciated.
Last edited by jeanleflambeur on Wed Jun 18, 2014 7:22 pm, edited 1 time in total.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7136
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal encoding errors

Wed Jun 18, 2014 11:23 am

The first error sounds like a buffer callback has been made twice somehow. You haven't managed to send the same buffer twice before the buffer was returned?

The second error is that you haven't enabled the control port and provided a callback function.
The component has produced an event that it wants to deliver, but without a callback it can't, and hence drops the event and logs a message. Doing that would probably give you some more information on what is getting upset and why.

Having mmalsrv errors logged is a little worrying. bulk tx aborted implies either the client crashed and we had to tidy up, or a buffer sizing issue. Are all the buffers that you create the same size as set in the port format buffer_size field? A mismatch is the normal reason for an abort when it gets told to copy more data than the receiving buffer size.

You haven't said what encodings you are using on each port (MMAL_ENCODING_OPAQUE recommended between camera and encode, but you have to stick with I420 when using resize), nor whether you are using mmal_connections between the components (I'd recommend it if possible, particularly with the MMAL_CONNECTION_FLAG_TUNNELLING flag as then it is all done on the GPU). Are you using an up to date release too (sudo raspi-update)?
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: mmal encoding errors

Wed Jun 18, 2014 1:34 pm

I actually had a control callback registered, but it wasn't releasing the buffer:

Code: Select all

static void camera_control_callback(MMAL_PORT_T* port, MMAL_BUFFER_HEADER_T* buffer)
{
    QASSERT(port && buffer);
    //GCamera->OnCameraControlCallback(port, buffer);
    BRAIN_INFO("Camera control callback called: {}, {}", port, buffer);
    mmal_buffer_header_release(buffer); <<< --- this was missing
}
Could this explain both errors?

I'm using I420 as the encoding - I found out yesterday that I need this when using the resizer.
All the connections are made using callbacks now in order to debug everything. I'll switch to using tunnelling after everything works with the callbacks.
The buffer sizes are correct as far as I can tell since they are setup at startup and never changed. The encoding works perfectly for 1-5 minutes before the errors.

I'll try now with the release buffer in the control callback and come back with details.

Regarding the resizer - is there any way to get it to perform faster if resizing to half? At 720p it can barely keep up at 30fps on an overclocked rpi (1000Mhz). I tried to do the resize manually without any bilinear but it's (2x) slower than what the resizer can do. Right now settled for 1024x768 and I can get 30 fps with an occasional frame drop but at 50% cpu usage (could it be the lack of tunnelling?)

I am using the latest release as of yesterday.

Thanks a lot for the hints.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7136
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal encoding errors

Wed Jun 18, 2014 2:53 pm

Not releasing the buffers won't have helped as it will slowly leak them and eventually not be able to allocate one when it needs to. There aren't many situations where it needs these buffers though so you may be OK, although just thinking through the way that events are handled, it may well result in a bulk transfer abort if the ARM side buffers aren't released.

If you use connections but without the MMAL_CONNECTION_FLAG_TUNNELLING flag, then all the callbacks still come back to the ARM to deal with, so you could have added debugging in there if needed. When it was all working you could then add the flag to improve the performance. Hindsight is always great ;)

On performance, exactly what ports are you connecting to where? I would recommend that you use:
- camera output[0] as I420 to resize input[0]. resize output[0] to video_encode#1 input[0]. video_encode#1 output[0] to the application.
- camera output[1] as MMAL_ENCODING_OPAQUE to video_encode#2 input[0]. video_encode#2 output[0] to the application.
The encoder hardware actually needs two buffers in a particular format to do the encoding. camera output[1] when set to OPAQUE can provide both directly. If you use I420 or output[0] then it has to do a convert and munge which obviously takes GPU processor time.

There doesn't appear to be a faster version of resize that can be used within the resize component. I'm surprised you're stressing the ARM much at all with this setup - almost all the heavy lifting is done on the GPU. Switching the higher res feed to OPAQUE (as mentioned above) will reduce that load quite significantly, and tunneled connections will improve the buffer handling latency which may be tripping you up.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: mmal encoding errors

Wed Jun 18, 2014 3:59 pm

I added the missing release in the control callback and this fixed the "buffer header callback not defined" error - or so it seems. Didn't get it in one hour
The other error - buffer headers in transit < 0 (-2) - was caused by my camera output callback. There was a case where the buffer could have been released to 0 references _before_ sending it to the encoder and this probably caused the pool callback to trigger adding it in the pool again.

Also applied all the changes you recommended:
- changed all manual buffer handling for tunneled connections
- high-res encoding from the video port (OPAQUE) and resizing from the preview port (I420)

Everything goes much better now - I can encode at 1280x960 + resize and encode at 640x480 30 and 40 fps without any frame drops. The code is shorter and cleaner.

2 other important discoveries from the past 2 days are:
- resize works only with I420
- the encoder for I420 doesn't emit any output buffers if the MMAL_PARAMETER_VIDEO_ENCODE_H264_LOW_LATENCY parameter is set. It works with opaque. No errors are generated.

Thanks a lot 6by9, your info was priceless.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7136
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal encoding errors

Wed Jun 18, 2014 4:24 pm

jeanleflambeur wrote:2 other important discoveries from the past 2 days are:
- resize works only with I420
Hopefully not true. It should work with MMAL_ENCODINGs I420, RGBA, BGRA, and RGB16. If that's not the case, then we ought to raise an issue on https://github.com/raspberrypi/firmware/issues.
All MMAL components support mmal_port_parameter_get(MMAL_PARAMETER_SUPPORTED_ENCODINGS) to retrieve the list of encodings that it supports.
jeanleflambeur wrote:- the encoder for I420 doesn't emit any output buffers if the MMAL_PARAMETER_VIDEO_ENCODE_H264_LOW_LATENCY parameter is set. It works with opaque. No errors are generated.
LOW_LATENCY mode is not a mode intended for general use. There was a specific use case for it where the source could feed the image in a stripe at a time, and the encoder would take the data as it was available. There were a large number of limitations to using it, but it fulfilled the purpose. This is the downside of having released the full MMAL headers without sanitising first - people see interesting looking parameters and tweak. At that point it is user beware!
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: mmal encoding errors

Wed Jun 18, 2014 4:47 pm

6by9 wrote: Hopefully not true. It should work with MMAL_ENCODINGs I420, RGBA, BGRA, and RGB16. If that's not the case, then we ought to raise an issue on https://github.com/raspberrypi/firmware/issues.
All MMAL components support mmal_port_parameter_get(MMAL_PARAMETER_SUPPORTED_ENCODINGS) to retrieve the list of encodings that it supports.
My bad - what I meant is that the resizer doesn't work with OPAQUE. I should have tested the supported formats before rewriting all my camera code 3 times - hindsight is great indeed :)
6by9 wrote:LOW_LATENCY mode is not a mode intended for general use. There was a specific use case for it where the source could feed the image in a stripe at a time, and the encoder would take the data as it was available. There were a large number of limitations to using it, but it fulfilled the purpose. This is the downside of having released the full MMAL headers without sanitising first - people see interesting looking parameters and tweak. At that point it is user beware!
I see it as more of a compromise than a downside. Take the resizer component for example. While not officially there and not properly tested - I still get to use it in my quadcopter and get nice features like high-res encoding to file, medium res streaming and low-res emergency streaming.
I would much rather have an imperfect implementation that allows me to do what I need than a perfect but limited one.

Regarding the LOW_LATENCY param - an error message when setting it with an unsupported encoding would fix all issues.

Anyway, thanks for all the great help.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7136
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal encoding errors

Wed Jun 18, 2014 5:01 pm

Hmm, not sure if OPAQUE will get listed from mmal_port_parameter_get(MMAL_PARAMETER_SUPPORTED_ENCODINGS).
Almost all components have been sorted to allow OPAQUE to some extent, but not resize at the moment. Resize only has potential support for the input port being OPAQUE - supporting it on the output would be quite a lot of work.

LOW_LATENCY - it's the old favourite of being able to change the encoding and the parameter in either order. The only point at which you could reliably flag the error is on enabling the port, and that gets awkward and confusing.

Glad you're all sorted now anyway.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”