jldeon
Posts: 18
Joined: Thu Apr 18, 2013 2:45 pm

MMAL resizer?

Wed Jun 05, 2013 2:48 pm

I'm trying to do some realtime video scaling in my RPi app. I'm using the MMAL library to interface with the RPi camera as part of the project as well.

Right now I'm taking the camera output and piping it into a video splitter component, which I then plan to attach multiple encoder elements to in order to get 264 data out at several simultaneous resolutions. To do this, I'm going to need to insert some kind of scaler between each output of the splitter and the input of each encoder.

The MMAL library appears to expose/wrap the OMX components that I've used in the past. However, I notice that the "resizer" is missing from mmal_default_components.h.

I can successfully create one by passing "vc.ril.resize" to mmal_component_create(), but trying to set any of the port settings returns ENOSYS, which appears to mean "unimplemented" in the MMAL code.

Can I create a "resize" element via the MMAL API? Is there a better/more efficient way to independently scale the outputs of the splitter?

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: MMAL resizer?

Mon Jun 16, 2014 6:09 am

Hi,
I'm getting exactly the same error when changing the port format for the resizer. Did you manage find out what it is?

jldeon
Posts: 18
Joined: Thu Apr 18, 2013 2:45 pm

Re: MMAL resizer?

Mon Jun 16, 2014 1:05 pm

Nope, I eventually gave up on it. I think the code in the library just doesn't exist to make it work.

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: MMAL resizer?

Mon Jun 16, 2014 4:21 pm

I just compiled picam from wibble82 and the resize component actually works in his code but not in mine.
You can find it here: http://www.raspberrypi.org/forums/viewt ... 43&t=59093
I actually traced all the calls to mmal up to the format_commit that fails and they are the same. Next I'll check the arguments to the function calls themselves but at this point I believe the problem might be somewhere else...
Here are the traces:

Code: Select all

picam:
MMAL: mmal_component_create("vc.ril.camera", &camera)
MMAL: mmal_port_enable(camera->control, CameraControlCallback)
MMAL: mmal_port_parameter_set(camera->control, &cam_config.hdr)
MMAL: mmal_port_format_commit(preview_port)
MMAL: mmal_port_format_commit(video_port)
MMAL: mmal_port_format_commit(still_port)
MMAL: mmal_component_enable(camera)
MMAL: mmal_component_create("vc.ril.video_splitter", &splitter)
MMAL: mmal_format_copy(input_port->format,video_output_port->format)
MMAL: mmal_port_format_commit(input_port)
MMAL: mmal_format_copy(output_port->format,input_port->format)
MMAL: mmal_port_format_commit(output_port)
MMAL: mmal_format_copy(output_port->format,input_port->format)
MMAL: mmal_port_format_commit(output_port)
MMAL: mmal_format_copy(output_port->format,input_port->format)
MMAL: mmal_port_format_commit(output_port)
MMAL: mmal_format_copy(output_port->format,input_port->format)
MMAL: mmal_port_format_commit(output_port)
MMAL: mmal_connection_create(&vid_to_split_connection, video_port, splitter->input[0], 0x1 | 0x2)
MMAL: mmal_connection_enable(vid_to_split_connection)
MMAL: mmal_component_create("vc.ril.resize", &resizer)
MMAL: mmal_format_copy(input_port->format,video_output_port->format)
MMAL: mmal_port_format_commit(input_port)

Code: Select all

my code:
MMAL: mmal_component_create(name, &component)
MMAL: mmal_port_enable(port, cb)
MMAL: mmal_port_parameter_set(m_impl->camera->control, &cam_config.hdr)
MMAL: mmal_port_format_commit(m_impl->camera->output[0])
MMAL: mmal_port_format_commit(m_impl->camera_video_output)
MMAL: mmal_port_format_commit(m_impl->camera->output[2])
MMAL: mmal_component_enable(component.get())
MMAL: mmal_component_create(name, &component)
MMAL: mmal_format_copy(splitter->input[0]->format, src->format)
MMAL: mmal_port_format_commit(splitter->input[0])
MMAL: mmal_format_copy(output->format, src->format)
MMAL: mmal_port_format_commit(output)
MMAL: mmal_format_copy(output->format, src->format)
MMAL: mmal_port_format_commit(output)
MMAL: mmal_format_copy(output->format, src->format)
MMAL: mmal_port_format_commit(output)
MMAL: mmal_format_copy(output->format, src->format)
MMAL: mmal_port_format_commit(output)
MMAL: mmal_connection_create(&connection, output, input, 0x1 | 0x2)
MMAL: mmal_connection_enable(connection.get())
MMAL: mmal_component_create(name, &component)
MMAL: mmal_format_copy(dst->format, src->format)
MMAL: mmal_port_format_commit(dst) [color=#FF0000]<<<------ mmal: mmal_vc_port_info_set: failed to set port info (2:0): ENOSYS[/color]
I actually need 3 encoders at different resolutions for my quadcopter camera. High res to save in a file, medium res to stream to the ground station and very low res in case of huge packet loss.

I spent 2 days on this and now it's personal - I have to figure it out.

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: MMAL resizer?

Mon Jun 16, 2014 5:48 pm

Got it!
The resizer returns ENOSYS when using MMAL_ENCODING_OPAQUE for the encoding and MMAL_ENCODING_I420 for the encoding variant.
Using MMAL_ENCODING_I420 for both (as picam does) works.

I'm not sure if there will be a performance impact due to this though.
Now I'm trying to check why the I get an EINTVAL when using crop with the resizer...

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: MMAL resizer?

Tue Jun 17, 2014 4:06 pm

As it turns out, changing the encoding from MMAL_ENCODING_OPAQUE to MMAL_ENCODING_I420 fixed the resizer error but causes the decoder to stop emitting output buffers and to cause a freeze in bcm_host_init when restarting - so I had to reboot the rpi after each test.
After one day of reading my code and other people's code line by line, I manage to make it work by commenting this:

Code: Select all

if (mmal_port_parameter_set_boolean(output, MMAL_PARAMETER_VIDEO_ENCODE_H264_LOW_LATENCY, 1) != MMAL_SUCCESS)
    {
        WARNING("failed to set MMAL_PARAMETER_VIDEO_ENCODE_H264_LOW_LATENCY");
        return nullptr;
    }
So it looks like low latency encoding only works with opaque buffers. It's not a big problem for me as the low latency toggle didn't have any visible effects - at least in my 720p streaming test.

tjlusco
Posts: 7
Joined: Sun Feb 12, 2017 3:32 am

Re: MMAL resizer?

Tue May 22, 2018 8:09 am

Sorry to resurrect an old issue, but its hard to beat a classic. The solution of using MMAL_ENCODING_I420 works, but it is too slow when using large frames from the camera with multiple resized encoder outputs. Using MMAL_ENCODING_OPAQUE_OPAQUE should work, and it does!

A quick recap of how to reproduce this issue:
  • Setup the camera output for opaque pointers.
  • Setup the output of the ISP (resizer) for I420. This is required for the component.
  • Construct the pipeline: Camera -> resizer "vc.ril.isp" -> Renderer. No issues.
  • Construct the pipeline: Camera -> splitter "vc.ril.video_splitter" -> renderer. No issues.
  • This proves the hardware is at least capable of doing what we need.
  • Now try: camera -> splitter -> resizer -> renderer.
  • Depending on how you setup the ports/connections, this will not work.
Everything seems to be pointing to an issue with the handling of port formats inside the VC components. I've found the following issues:
  • When connecting the camera->output to splitter->input, the splitter->output->format's are updated but the splitter->output->buffer_size/buffer_num are not. This appears to be the root of the problem.
  • Setting a port format can change the buffer requirements. On the ISP the default buffer_num == 1, but you need buffer_num == 3 when using MMAL_ENCODING_OPAQUE.
  • Using mmal_format_copy from a VC component uses an internal format buffer_num/size, which in the case of the splitter is incorrect, even when explicitly set.
  • Also a possibility, on the VC side it could be trying to update the buffer->num/size before updating the buffer_size_min/buffer_num_min, causing the update to fail.
The work around is fairly simple
  • Setup the ISP input port manually, or at the very least don’t copy from the splitter!
  • Don't use any of the mmal_connection_create convenience functions which copy the port format, for the above reason.
Here is a code example which shows where the failures happen and the fix. This was a really painful bug to track down I hope this saves someone else a few days.

It would be great if someone could have a quick look at the code on the VC side for the splitter component. :D
Looking at https://github.com/raspberrypi/userland ... splitter.c (which I know isn’t the same component) it seems to be up to the component to setup the output port buffer_size/num, which isn’t happening. This line is as far as I could track the issue, it is where the port format on the ISP fails when trying to set it from the splitter.https://github.com/raspberrypi/userland ... pi.c#L1015. See log below.

Code: Select all

mmal: mmal_connection_create: out 0x11e07c0, in 0x11e4cc0, flags 1, vc.ril.video_splitter:out:0/vc.ril.isp:in:0
mmal: mmal_port_format_commit: vc.ril.isp(2:0) port 0x11e4cc0 format 3:OPQV
mmal: mmal_vc_port_info_set: set port info (2:0)
mmal: mmal_vc_port_info_set: failed to set port info (2:0): EINVAL
mmal: mmal_vc_port_set_format: mmal_vc_port_info_set failed 0x11e4cc0 (EINVAL)
mmal: mmal_connection_create: format not set on input port

Code: Select all

#include <assert.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <fcntl.h>
#include <errno.h>
#include <math.h>

#include "bcm_host.h"
#include "interface/vcsm/user-vcsm.h"
#include "interface/vcos/vcos.h"
#include "interface/vmcs_host/vc_tvservice.h"

#include "interface/mmal/mmal.h"
#include "interface/mmal/mmal_format.h"
#include "interface/mmal/util/mmal_default_components.h"
#include "interface/mmal/util/mmal_component_wrapper.h"
#include "interface/mmal/util/mmal_connection.h"
#include "interface/mmal/util/mmal_graph.h"
#include "interface/mmal/util/mmal_util_params.h"

#include "GLES/gl.h"
#include "GLES/glext.h"
#include "GLES2/gl2.h"
#include "GLES2/gl2ext.h"
#include "EGL/egl.h"
#include "EGL/eglext.h"
#include "EGL/eglext_brcm.h"

typedef struct pipeline_t {
    MMAL_WRAPPER_T *camera;             /**< Camera component source */
    MMAL_WRAPPER_T *splitter;           /**< Port Splitter */
    MMAL_WRAPPER_T *renderer;           /**< Debugging Preview */
    MMAL_WRAPPER_T *isp;                /**< ISP resizer and pixel format conversion */
    MMAL_WRAPPER_T *encoder;            /**< H264/MJPEG Encoder */
    MMAL_WRAPPER_T *copy;
} pipeline_t;

void camera_pipeline_init(pipeline_t *pipeline, int width, int height, int fps)
{
    MMAL_STATUS_T status;
    MMAL_PORT_T *port;
    MMAL_ES_FORMAT_T *format;
    MMAL_CONNECTION_T *connection;

    /* Create camera component */
    status = mmal_wrapper_create(&pipeline->camera, MMAL_COMPONENT_DEFAULT_CAMERA);
    assert(status == MMAL_SUCCESS);

    /* Setup Camera */
    port = pipeline->camera->output[0];
    format = port->format;
    format->encoding = MMAL_ENCODING_OPAQUE;
    format->encoding_variant = MMAL_ENCODING_I420;
    format->es->video.width = VCOS_ALIGN_UP(width, 32);
    format->es->video.height = VCOS_ALIGN_UP(height, 16);
    format->es->video.crop.x = 0;
    format->es->video.crop.y = 0;
    format->es->video.crop.width = width;
    format->es->video.crop.height = height;
    format->es->video.frame_rate.num = fps * 25;
    format->es->video.frame_rate.den = 1 * 27;
    status = mmal_port_format_commit(port);
    assert(status == MMAL_SUCCESS);

    /* Create Splitter */
    status = mmal_wrapper_create(&pipeline->splitter, "vc.ril.video_splitter");
    // status = mmal_wrapper_create(&pipeline->splitter, "splitter");       // Works ok, tends to crash VC when it doesnt..
    // status = mmal_wrapper_create(&pipeline->splitter, "vc.splitter");
    assert(status == MMAL_SUCCESS);

    /* Setup splitter */
    mmal_format_copy(pipeline->splitter->input[0]->format, pipeline->camera->output[0]->format);
    status = mmal_port_format_commit(pipeline->splitter->input[0]);
    assert(status == MMAL_SUCCESS);
    mmal_format_copy(pipeline->splitter->output[0]->format, pipeline->splitter->input[0]->format);
    status = mmal_port_format_commit(pipeline->splitter->output[0]);
    assert(status == MMAL_SUCCESS);

    /* Connect splitter */
    status = mmal_connection_create(&connection, pipeline->camera->output[0], pipeline->splitter->input[0], MMAL_CONNECTION_FLAG_TUNNELLING);
    assert(status == MMAL_SUCCESS);
    status = mmal_connection_enable(connection);
    assert(status == MMAL_SUCCESS);

    /* Create ISP */
    status = mmal_wrapper_create(&pipeline->isp, "vc.ril.isp");
    assert(status == MMAL_SUCCESS);

    /* Check port formatting, notice splitter->output buffer num/sizes haven't been updated */
    port = pipeline->splitter->input[0];
    mmal_log_dump_format(port);
    printf("buf_num:     %3d  bufsize:      %d\n", port->buffer_num, port->buffer_size);
    printf("buf_num_min: %3d  buf_size_min: %d\n", port->buffer_num_min, port->buffer_size_min);
    port = pipeline->splitter->output[0];
    mmal_log_dump_format(port);
    printf("buf_num:     %3d  bufsize:      %d\n", port->buffer_num, port->buffer_size);
    printf("buf_num_min: %3d  buf_size_min: %d\n", port->buffer_num_min, port->buffer_size_min);
    port = pipeline->isp->input[0];
    mmal_log_dump_format(port);
    printf("buf_num:     %3d  bufsize:      %d\n", port->buffer_num, port->buffer_size);
    printf("buf_num_min: %3d  buf_size_min: %d\n", port->buffer_num_min, port->buffer_size_min);

    /************
     * LOOK HERE
     *************/

    /* Setup ISP */
    /* This does not work! Error somewhere on the VC side */
    mmal_format_copy(pipeline->isp->input[0]->format, pipeline->splitter->output[0]->format);
    status = mmal_port_format_commit(pipeline->isp->input[0]);
    // assert(status == MMAL_SUCCESS);

    /* Manually setting up the port works */
    port = pipeline->isp->input[0];
    format = port->format;
    format->encoding = MMAL_ENCODING_OPAQUE;
    format->encoding_variant = MMAL_ENCODING_I420;
    port->buffer_num = 3;
    port->buffer_size = 128;
    status = mmal_port_format_commit(port);
    assert(status == MMAL_SUCCESS);

    /* This does work?
     * The only difference I can find between camera->output and splitter->output is the buffer sizes
     */
    mmal_format_copy(pipeline->isp->input[0]->format, pipeline->camera->output[0]->format);
    status = mmal_port_format_commit(pipeline->isp->input[0]);
    assert(status == MMAL_SUCCESS);

    /* Setup Resize */
    mmal_format_copy(pipeline->isp->output[0]->format, pipeline->isp->input[0]->format);
    pipeline->isp->output[0]->format->encoding = MMAL_ENCODING_I420;
    pipeline->isp->output[0]->format->es->video.width = VCOS_ALIGN_UP(160, 32);
    pipeline->isp->output[0]->format->es->video.height = VCOS_ALIGN_UP(120, 16);
    pipeline->isp->output[0]->format->es->video.crop.x = 0;
    pipeline->isp->output[0]->format->es->video.crop.y = 0;
    pipeline->isp->output[0]->format->es->video.crop.width = 160;
    pipeline->isp->output[0]->format->es->video.crop.height = 120;
    pipeline->isp->output[0]->format->es->video.frame_rate.num = 0;
    pipeline->isp->output[0]->format->es->video.frame_rate.den = 1;
    status = mmal_port_format_commit(pipeline->isp->output[0]);
    assert(status == MMAL_SUCCESS);

    /************
     * LOOK HERE
     *************/

    /* Connect ISP */
    /* This doesn't work, mmal_connection_create uses the above mmal_format_copy call internally and will fail, even if the port is already setup correctly */
    status = mmal_connection_create(&connection, pipeline->splitter->output[0], pipeline->isp->input[0], MMAL_CONNECTION_FLAG_TUNNELLING);
    // assert(status == MMAL_SUCCESS);
    if (status == MMAL_SUCCESS) {
        status = mmal_connection_enable(connection);
        assert(status == MMAL_SUCCESS);
    }

    /* Setup a direct tunnelled connection */
    /* This works */
    status = mmal_port_connect(pipeline->splitter->output[0], pipeline->isp->input[0]);
    assert(status == MMAL_SUCCESS);
    status = mmal_port_enable(pipeline->splitter->output[0], NULL);
    assert(status == MMAL_SUCCESS);

    /* Create Renderer */
    status = mmal_wrapper_create(&pipeline->renderer, MMAL_COMPONENT_DEFAULT_VIDEO_RENDERER);
    assert(status == MMAL_SUCCESS);

    /* Setup Renderer */
    mmal_format_copy(pipeline->renderer->input[0]->format, pipeline->isp->output[0]->format);
    mmal_port_format_commit(pipeline->renderer->input[0]);
    assert(status == MMAL_SUCCESS);

    /* Connect Renderer */
    status = mmal_connection_create(&connection, pipeline->isp->output[0], pipeline->renderer->input[0], MMAL_CONNECTION_FLAG_TUNNELLING | MMAL_CONNECTION_FLAG_ALLOCATION_ON_INPUT);
    assert(status == MMAL_SUCCESS);
    status = mmal_connection_enable(connection);
    assert(status == MMAL_SUCCESS);
}

int main(int argc, char **argv)
{
    pipeline_t *camera_pipeline = malloc(sizeof(pipeline_t));

    bcm_host_init();

    camera_pipeline_init(camera_pipeline, 640, 480, 25);

    while (1) {
        usleep(1000);
    }
}

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5663
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: MMAL resizer?

Tue May 22, 2018 2:36 pm

You've resurrected a 4 year old post about vc.ril.resize with stuff on vc.ril.isp. It would have been more sensible to start a new thread.
vc.ril.isp didn't exist in 2014, and is still considered to be beta code and under active development (as time permits).

mmal_dump_format takes a MMAL_ES_FORMAT_T *, not a MMAL_PORT_T *. All calls should therefore be "mmal_log_dump_format(port->format);", or call mmal_log_dump_port instead. You ought to be including "interface/mmal/util/mmal_util.h" too so you get the prototypes.
I'm assuming you've also found out that you want to do export VC_LOGLEVEL="mmal:info" to get the relevant lofgging messages out.

Your test app is hard to follow as you appear to have 3 different approaches all trying to do the same thing. It would have been much easier to follow if you'd put the options as independent code paths that could each be exercised via #if / #else / #endif blocks
A build line would have been nice too to save time recreating it.

vc.ril.video_splitter isn't preserving format->encoding_variant.
Camera cares because it is a source that can produce opaque buffers as either YUV420 or YUV422. Video splitter doesn't as it just copies (and potentially format converts) the input to output. It'll wait for the opaque buffer to find out what the contents is on the assumption it can handle it.
vc.ril.isp is currently validating the input format on mmal_port_format_commit, and with MMAL_ENCODING_OPAQUE and variant 0 it fails on that. Yes, with mmal_connection_create it will do a full copy and hit this issue.
I will fix the firmware to not sanity check the input format if using opaque buffers and hope we can handle the buffers that finally arrive.

With that fixed I can chop the back end of your camera_pipeline_init down to just

Code: Select all

    /* Create ISP */
    status = mmal_component_create("vc.ril.isp", &pipeline->isp);
    assert(status == MMAL_SUCCESS);

    mmal_log_dump_port(pipeline->splitter->input[0]);
    mmal_log_dump_port(pipeline->splitter->output[0]);
    mmal_log_dump_port(pipeline->isp->input[0]);

    /* Connect ISP */
    status = mmal_connection_create(&connection, pipeline->splitter->output[0], pipeline->isp->input[0], MMAL_CONNECTION_FLAG_TUNNELLING);
    assert(status == MMAL_SUCCESS);
    if (status == MMAL_SUCCESS) {
        status = mmal_connection_enable(connection);
        assert(status == MMAL_SUCCESS);
    }

    /* Setup Resize Output */
    mmal_format_copy(pipeline->isp->output[0]->format, pipeline->isp->input[0]->format);
    pipeline->isp->output[0]->format->encoding = MMAL_ENCODING_I420;
    pipeline->isp->output[0]->format->es->video.width = VCOS_ALIGN_UP(160, 32);
    pipeline->isp->output[0]->format->es->video.height = VCOS_ALIGN_UP(120, 16);
    pipeline->isp->output[0]->format->es->video.crop.x = 0;
    pipeline->isp->output[0]->format->es->video.crop.y = 0;
    pipeline->isp->output[0]->format->es->video.crop.width = 160;
    pipeline->isp->output[0]->format->es->video.crop.height = 120;
    pipeline->isp->output[0]->format->es->video.frame_rate.num = 0;
    pipeline->isp->output[0]->format->es->video.frame_rate.den = 1;
    status = mmal_port_format_commit(pipeline->isp->output[0]);
    assert(status == MMAL_SUCCESS);

    /* Create Renderer */
    status = mmal_component_create(MMAL_COMPONENT_DEFAULT_VIDEO_RENDERER, &pipeline->renderer);
    assert(status == MMAL_SUCCESS);

    /* Connect Renderer */
    status = mmal_connection_create(&connection, pipeline->isp->output[0], pipeline->renderer->input[0], MMAL_CONNECTION_FLAG_TUNNELLING | MMAL_CONNECTION_FLAG_ALLOCATION_ON_INPUT);
    assert(status == MMAL_SUCCESS);
    status = mmal_connection_enable(connection);
    assert(status == MMAL_SUCCESS);
which is the expected behaviour. I'll sort out getting that patch to vc.ril.isp released.

BTW mmal_wrapper is doing nothing for you in your current use case. You'd be better off with just MMAL_COMPONENT_T * and mmal_component_create.
On a mmal_connection, MMAL_CONNECTION_FLAG_ALLOCATION_ON_INPUT is going to do nothing if MMAL_CONNECTION_FLAG_TUNNELLING is set.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

tjlusco
Posts: 7
Joined: Sun Feb 12, 2017 3:32 am

Re: MMAL resizer?

Tue May 22, 2018 11:07 pm

Sorry this might have been more suitable for a new thread, but from memory the same issue pops up when using the "vc.ril.resize" component. If you follow the log trace from the above posts they hit the same error at the same point. That's how I found the thread!
MMAL: mmal_connection_create(&connection, output, input, 0x1 | 0x2)
MMAL: mmal_connection_enable(connection.get())
MMAL: mmal_component_create(name, &component)
MMAL: mmal_format_copy(dst->format, src->format)
MMAL: mmal_port_format_commit(dst) <<<------ mmal: mmal_vc_port_info_set: failed to set port info (2:0): ENOSYS
I also could have made the test app a little clear, I was working through figuring out the exact reason I couldn't connect the ports, and left in all of the failed attempts. When comparing the port formats of camera->output[0] with splitter->output[0], I could find no differences (including the encoding variant), so it was a bit of a mystery as to why it wasn't working. I thought it best to just dump the code so you could follow the process I used to stumble on the problem, or at the very least utilize Cunningham Law.

Your explanation of what was happening makes sense. Thanks for tracking it down and replying so quickly! I thought the use would be more on the splitter side? The ISP input port seems to validate ok with the the format from the camera. its that the splitter didn't seem to be presenting the same format information.

Is this related to the the video splitter being able to do format conversion? Such that it won't overwrite the output formats because you want to be able to change them?

I know the wrapper is doing nothing in this example, but its port queue/pool handling capability is quite nice when you actually do use it! It save a lot of boilerplate. The wrapper does no harm until I try to mix MMAL_COMPONENT_T with MMAL_WRAPPER_T, so to keep things consistent I use the wrapper everywhere.
On a mmal_connection, MMAL_CONNECTION_FLAG_ALLOCATION_ON_INPUT is going to do nothing if MMAL_CONNECTION_FLAG_TUNNELLING is set.
See https://github.com/raspberrypi/userland ... id.c#L2290 :D The raspicam applications are essentially the canonical examples, and every piece of mmal code I've seen copied it. Even above:
MMAL: mmal_connection_create(&vid_to_split_connection, video_port, splitter->input[0], 0x1 | 0x2)
Even picamera used it!
https://github.com/waveform80/picamera/ ... j.py#L2144
flags = mmal.MMAL_CONNECTION_FLAG_ALLOCATION_ON_INPUT
if callback is None:
flags |= mmal.MMAL_CONNECTION_FLAG_TUNNELLING
I didn't want to break tradition! Its harmless.

tjlusco
Posts: 7
Joined: Sun Feb 12, 2017 3:32 am

Re: MMAL resizer?

Wed May 23, 2018 1:18 am

Just to reiterate, here is a "minimal" example that shows the same issue with the resize component, as in the original post of the thread.

I am quite confident the real problem is with the video_splitter, I'm just not sure whether its intended behaviour or not. Setting the input format has no effect on the output format. Also setting the output format doesn't seem to have the intended effect? It is as though the port format is unset on the VC side of the component even though it appears fine from userland. The port format copy uses the internal VC port format, not what you can see from userspace.

I remember reading somewhere that the splitter was capable of format conversion (viewtopic.php?t=189004, and https://github.com/waveform80/picamera/issues/420), in which case maybe this behaviour would be intended? When you connect a port to the input of the splitter, the format would propagate to the output, overwriting the original output port settings, probably not what you want.

Steps:
  • Make splitter and resizer
  • Setup splittter input format
  • Notice the splitter output format is unset
  • Force splitter output port format to match input format
  • Try to copy splitter output port format to resizer, this fails. My hunch is that the splitter output format is unset internally on the VC side.
  • Set the port format on the resizer to exactly the same as for the splitter input. This works.
Log output:

Code: Select all

mmal: mmal_vc_component_create: vc.ril.video_splitter: handle 0x40 status 0 reply status 0
mmal: mmal_component_create_core: created 'vc.ril.video_splitter' 0 0x1b9f3e0
mmal: mmal_vc_component_create: vc.ril.resize: handle 0x38 status 0 reply status 0
mmal: mmal_component_create_core: created 'vc.ril.resize' 1 0x1ba2de0

mmal: mmal_log_dump_port: vc.ril.video_splitter:in:0(OPQV)(0x1b9fd20)
mmal: mmal_log_dump_format: type: video, fourcc: OPQV
mmal: mmal_log_dump_format:  bitrate: 0, framed: 0
mmal: mmal_log_dump_format:  extra data: 0, (nil)
mmal: mmal_log_dump_format:  width: 320, height: 240, (0,0,320,240)
mmal: mmal_log_dump_format:  pixel aspect ratio: 0/0, frame rate: 0/65536
mmal: mmal_log_dump_port:  buffers num: 3(opt 10, min 3), size: 15360(opt 128, min: 128), align: 0
5651504f,0

mmal: mmal_log_dump_port: vc.ril.video_splitter:out:0(0x1ba0040)
mmal: mmal_log_dump_format: type: video, fourcc:
mmal: mmal_log_dump_format:  bitrate: 0, framed: 0
mmal: mmal_log_dump_format:  extra data: 0, (nil)
mmal: mmal_log_dump_format:  width: 320, height: 240, (0,0,320,240)
mmal: mmal_log_dump_format:  pixel aspect ratio: 0/0, frame rate: 0/65536
mmal: mmal_log_dump_port:  buffers num: 1(opt 1, min 1), size: 15360(opt 0, min: 0), align: 0
0,0


mmal: mmal_log_dump_port: vc.ril.video_splitter:in:0(OPQV)(0x1b9fd20)
mmal: mmal_log_dump_format: type: video, fourcc: OPQV
mmal: mmal_log_dump_format:  bitrate: 0, framed: 0
mmal: mmal_log_dump_format:  extra data: 0, (nil)
mmal: mmal_log_dump_format:  width: 320, height: 240, (0,0,320,240)
mmal: mmal_log_dump_format:  pixel aspect ratio: 0/0, frame rate: 0/65536
mmal: mmal_log_dump_port:  buffers num: 3(opt 10, min 3), size: 15360(opt 128, min: 128), align: 0
5651504f,0

mmal: mmal_log_dump_port: vc.ril.video_splitter:out:0(OPQV)(0x1ba0040)
mmal: mmal_log_dump_format: type: video, fourcc: OPQV
mmal: mmal_log_dump_format:  bitrate: 0, framed: 0
mmal: mmal_log_dump_format:  extra data: 0, (nil)
mmal: mmal_log_dump_format:  width: 320, height: 240, (0,0,320,240)
mmal: mmal_log_dump_format:  pixel aspect ratio: 0/0, frame rate: 0/65536
mmal: mmal_log_dump_port:  buffers num: 3(opt 10, min 3), size: 15360(opt 128, min: 128), align: 0
5651504f,0

mmal: mmal_vc_port_info_set: failed to set port info (2:0): EINVAL
mmal: mmal_vc_port_set_format: mmal_vc_port_info_set failed 0x1ba3670 (EINVAL)
Port format setting success
Code:

Code: Select all

int main()
{
    MMAL_STATUS_T status;
    MMAL_COMPONENT_T *splitter;
    MMAL_COMPONENT_T *resizer;
    MMAL_ES_FORMAT_T *format;
    MMAL_PORT_T *port;
    uint32_t diff;

    /* Create components */
    status = mmal_component_create("vc.ril.video_splitter", &splitter);
    status |= mmal_component_create("vc.ril.resize", &resizer);
    assert(status == MMAL_SUCCESS);

    /* Setup Splitter Input Port */
    port = splitter->input[0];
    format = port->format;
    format->encoding = MMAL_ENCODING_OPAQUE;
    format->encoding_variant = MMAL_ENCODING_I420;
    format->es->video.width = VCOS_ALIGN_UP(320, 32);
    format->es->video.height = VCOS_ALIGN_UP(240, 16);
    format->es->video.crop.x = 0;
    format->es->video.crop.y = 0;
    format->es->video.crop.width = 320;
    format->es->video.crop.height = 240;
    format->es->video.frame_rate.num = 0;
    format->es->video.frame_rate.den = 1;
    status = mmal_port_format_commit(port);
    assert(status == MMAL_SUCCESS);

    /* Check port formats */
    /* export VC_LOGLEVEL="mmal:trace" */
    /* Notice the splitter output port formats have not been set */
    printf("\n");
    mmal_log_dump_port(splitter->input[0]);
    printf("%x,%x\n\n", splitter->input[0]->format->encoding, splitter->input[0]->format->encoding_variant);
    mmal_log_dump_port(splitter->output[0]);
    printf("%x,%x\n\n", splitter->output[0]->format->encoding, splitter->output[0]->format->encoding_variant);
    diff = mmal_format_compare(splitter->input[0]->format, splitter->output[0]->format);
    // assert(diff == 0); // 0 == no difference between formats, skip because this will fail

    /* Force Splitter output port */
    port = splitter->output[0];
    format = port->format;
    format->encoding = MMAL_ENCODING_OPAQUE;
    format->encoding_variant = MMAL_ENCODING_I420;
    format->es->video.width = VCOS_ALIGN_UP(320, 32);
    format->es->video.height = VCOS_ALIGN_UP(240, 16);
    format->es->video.crop.x = 0;
    format->es->video.crop.y = 0;
    format->es->video.crop.width = 320;
    format->es->video.crop.height = 240;
    format->es->video.frame_rate.num = 0;
    format->es->video.frame_rate.den = 1;
    status = mmal_port_format_commit(port);
    assert(status == MMAL_SUCCESS);

    /* Check port formats match */
    /* Notice the port formats "definitely" match now from userspace */
    printf("\n");
    mmal_log_dump_port(splitter->input[0]);
    printf("%x,%x\n\n", splitter->input[0]->format->encoding, splitter->input[0]->format->encoding_variant);
    mmal_log_dump_port(splitter->output[0]);
    printf("%x,%x\n\n", splitter->output[0]->format->encoding, splitter->output[0]->format->encoding_variant);
    diff = mmal_format_compare(splitter->input[0]->format, splitter->output[0]->format);
    assert(diff == 0); // 0 == no difference between formats

    /* Copy format to resizer */
    /* This call fails */
    mmal_format_copy(resizer->input[0]->format, splitter->output[0]->format);
    status = mmal_port_format_commit(resizer->input[0]);
    // assert(status == MMAL_SUCCESS); // Skip the assert to try something

    /* Try manually setting the resizer port format to the same as above */
    port = resizer->input[0];
    format = port->format;
    format->encoding = MMAL_ENCODING_OPAQUE;
    format->encoding_variant = MMAL_ENCODING_I420;
    format->es->video.width = VCOS_ALIGN_UP(320, 32);
    format->es->video.height = VCOS_ALIGN_UP(240, 16);
    format->es->video.crop.x = 0;
    format->es->video.crop.y = 0;
    format->es->video.crop.width = 320;
    format->es->video.crop.height = 240;
    format->es->video.frame_rate.num = 0;
    format->es->video.frame_rate.den = 1;
    status = mmal_port_format_commit(port);
    assert(status == MMAL_SUCCESS);
    printf("Port format setting success\n");
}

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5663
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: MMAL resizer?

Wed May 23, 2018 1:26 pm

Bug found in the MMAL framework which results encoding_variant field being incorrectly handled for video domain components (of which video_splitter is one).
It all seems to be working fine now with either isp or resize.

I can only work on the information given. There was no mention in your first post about resize, hence looking at only the ISP component.
The time available to triage issues is limited (we get bored easily ;) ), so I was looking in the obvious place for a solution. Knowing it affected multiple components meant I looked elsewhere.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

tjlusco
Posts: 7
Joined: Sun Feb 12, 2017 3:32 am

Re: MMAL resizer?

Thu May 24, 2018 1:35 am

Awesome! That's great news. Thank you for all of your hard work :D

Return to “Graphics programming”

Who is online

Users browsing this forum: No registered users and 23 guests