beath
Posts: 2
Joined: Wed Jul 25, 2018 8:05 am

MMAL Splitter output port cannot be changed between opaque and non-opaque

Wed Jul 25, 2018 8:17 am

I need multiple H264 video streams running in parallel at varying resolutions.
The configuration of choice is a splitter connected to the camera video port and a suitable number of video-encoders connected to the splitter outputs, either directly or using a resizer for smaller resolutions. For high resolution streams it is essential to use opaque buffers for optimal performance. For lower resolutions opaque cannot be used because the resizer does not support opaque buffers. The splitter output ports therefore need to be configured for opaque or non-opaque buffers depending on the desired resolution. With the ports of a newly created splitter this works fine.
However, once connected with opaque buffers the ports cannot be reconfigured to use non-opaque buffers, even if the ports and the connected components are correctly disabled and disconnected. Same for a change from non-opaque to opaque.

In the non-opaque to opaque case the mmal_port_format_commit returns an EINVAL error when the port is reconfigured.

In the opaque to non-opaque case no error appears but no data ever arrives at the port.

Now my questions:
Why is a disabled and disconnected splitter port different from a newly created one?
Is this an implementation bug?
Is there a trick to reset a splitter port to the initial state?

I am using a lengthy C program but the problem can easily be reproduced using a simple Picamera program:
Tested on a Raspeberry Pi3 model B+ with March 13 firmware (6e08617e7767b09ef97b3d6cee8b75eba6d7ee0b) and camera V2.1.

Program 1 (non-opaque to opaque case):

--------------------------------------------------------------------
import picamera

with picamera.PiCamera(resolution = (1280, 720)) as camera:

camera.start_recording('low.h264', resize=(640, 360))
camera.wait_recording(2)
camera.stop_recording()

camera.start_recording('hi.h264')
camera.wait_recording(2)
camera.stop_recording()
--------------------------------------------------------------------
The first recording (low) works fine but the second (hi) writes an EINVAL error to the log when the splitter port is newly configured. The hi recording still seems to work but this is because Picamera silently retries with non-opaque if an opaque connection fails.


Program 2 (opaque to non-opaque case):

--------------------------------------------------------------------
import picamera

with picamera.PiCamera(resolution = (1280, 720)) as camera:

camera.start_recording('hi.h264')
camera.wait_recording(2)
camera.stop_recording()

camera.start_recording('low.h264', resize=(640, 360))
camera.wait_recording(2)
camera.stop_recording()
--------------------------------------------------------------------
The first recording (hi) works fine. The second (low) does not throw an error but the resulting file is empty.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5954
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: MMAL Splitter output port cannot be changed between opaque and non-opaque

Wed Jul 25, 2018 1:59 pm

beath wrote:
Wed Jul 25, 2018 8:17 am
For lower resolutions opaque cannot be used because the resizer does not support opaque buffers.
Where did you get that information from? vc.ril.resize supports opaque buffers on the input side quite happily.
beath wrote:I am using a lengthy C program but the problem can easily be reproduced using a simple Picamera program:
Tested on a Raspeberry Pi3 model B+ with March 13 firmware (6e08617e7767b09ef97b3d6cee8b75eba6d7ee0b) and camera V2.1.

Program 1 (non-opaque to opaque case):

Code: Select all

import picamera

with picamera.PiCamera(resolution = (1280, 720)) as camera:
   
    camera.start_recording('low.h264', resize=(640, 360))
    camera.wait_recording(2)
    camera.stop_recording()
 
    camera.start_recording('hi.h264')
    camera.wait_recording(2)
    camera.stop_recording()
The first recording (low) works fine but the second (hi) writes an EINVAL error to the log when the splitter port is newly configured. The hi recording still seems to work but this is because Picamera silently retries with non-opaque if an opaque connection fails.


Program 2 (opaque to non-opaque case):

Code: Select all

import picamera

with picamera.PiCamera(resolution = (1280, 720)) as camera:
 
    camera.start_recording('hi.h264')
    camera.wait_recording(2)
    camera.stop_recording()
   
    camera.start_recording('low.h264', resize=(640, 360))
    camera.wait_recording(2)
    camera.stop_recording()
The first recording (hi) works fine. The second (low) does not throw an error but the resulting file is empty.
I'm sorry, I haven't got time to dig through PiCamera to work out the sequence of calls it is making.
If you can create a simple test case in C that I can run then I'll look at it, but not in any other language.

(And please use the code tags for code. (a) it looks nicer, and (b) it retains tabs etc which are vital in some languages).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

beath
Posts: 2
Joined: Wed Jul 25, 2018 8:05 am

Re: MMAL Splitter output port cannot be changed between opaque and non-opaque

Fri Jul 27, 2018 1:07 pm

6by9 wrote:
Wed Jul 25, 2018 1:59 pm
beath wrote:
Wed Jul 25, 2018 8:17 am
For lower resolutions opaque cannot be used because the resizer does not support opaque buffers.
Where did you get that information from? vc.ril.resize supports opaque buffers on the input side quite happily.
Connecting a resizer directly to the camera works fine with opaque buffers. Connecting to a splitter output does not:
mmal: mmal_vc_port_info_set: failed to set port info (2:0): EINVAL
mmal: mmal_vc_port_set_format: mmal_vc_port_info_set failed 0x892a40 (EINVAL)
mmal: mmal_connection_create: format not set on input port
mmal: mmal_connection_destroy_internal: connection vc.ril.video_splitter:out:0/vc.ril.resize:in:0 could not be cleared
Same behaviour for "vc.ril.isp" instead of "vc.ril.resize".
You can easily reproduce this by changing the encoding types in the sample below.
6by9 wrote:
Wed Jul 25, 2018 1:59 pm
I'm sorry, I haven't got time to dig through PiCamera to work out the sequence of calls it is making.
If you can create a simple test case in C that I can run then I'll look at it, but not in any other language.

(And please use the code tags for code. (a) it looks nicer, and (b) it retains tabs etc which are vital in some languages).
Sure, here it is:

Code: Select all

#include <stdio.h>
#include <stdlib.h> // exit()
#include <sys/time.h>
#include <time.h>

#include "bcm_host.h"
#include "interface/vcos/vcos.h"
#include "interface/mmal/mmal.h"
#include "interface/mmal/mmal_logging.h"
#include "interface/mmal/mmal_buffer.h"
#include "interface/mmal/util/mmal_util.h"
#include "interface/mmal/util/mmal_util_params.h"
#include "interface/mmal/util/mmal_default_components.h"
#include "interface/mmal/util/mmal_connection.h"


#define CAMERA_PREVIEW_PORT (0)
#define CAMERA_VIDEO_PORT   (1)
#define CAMERA_STILL_PORT   (2)

#define RESIZER "vc.ril.resize"
//#define RESIZER "vc.ril.isp"


#define AssertSuccess(r) if (r != MMAL_SUCCESS) { \
    printf(" OPERATION FAILED: %s! [%s:%d]\n", mmal_status_to_string(r), __FILE__, __LINE__); \
    exit(EXIT_FAILURE); } 
    
    
static MMAL_COMPONENT_T *camera, *splitter, *resizer, *encoder;
static MMAL_CONNECTION_T *splitterConn, *resizerConn, *encoderConn;
static MMAL_POOL_T *encoderPool;


static void ControlCallback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
    printf("Control callback command: %08X\n", buffer->cmd);
    mmal_buffer_header_release(buffer);
}

static void EncoderCallback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
    if (buffer->length) {
        printf(".");
    }
    
    // Cleanup and recycle buffer.
    mmal_buffer_header_release(buffer);
    if (encoderPool && port->is_enabled) {
        MMAL_BUFFER_HEADER_T *buf = mmal_queue_get(encoderPool->queue);
        if (buf) {
            mmal_port_send_buffer(port, buf);
        }
    }
}

static MMAL_STATUS_T SetVideoFormat(MMAL_PORT_T *port, int w, int h, int framerate)
{
    MMAL_ES_FORMAT_T *format = port->format;
    MMAL_ES_SPECIFIC_FORMAT_T *es = format->es;
    
    int alignedW = (w + 31) & ~31;
    int alignedH = (h + 15) & ~15;
    
    es->video.width = alignedW;
    es->video.height = alignedH;
    es->video.crop.width = w;
    es->video.crop.height = h;
    if (framerate) {
        es->video.frame_rate.num = framerate;
        es->video.frame_rate.den = 1;
    }
    
    return mmal_port_format_commit(port);
}

static void SetupCamera(int w, int h)
{
    MMAL_STATUS_T res;
    
    res = mmal_component_create("vc.ril.camera", &camera);
    AssertSuccess(res);

    res = mmal_port_enable(camera->control, ControlCallback);
    AssertSuccess(res);
    
    // configure camera
    MMAL_PARAMETER_CAMERA_CONFIG_T cam_config = {
        { MMAL_PARAMETER_CAMERA_CONFIG, sizeof(cam_config) },
        .max_stills_w = w,
        .max_stills_h = h,
        .max_preview_video_w = w,
        .max_preview_video_h = h,
        .num_preview_video_frames = 3,
        .use_stc_timestamp = MMAL_PARAM_TIMESTAMP_MODE_RAW_STC
    };
    res = mmal_port_parameter_set(camera->control, &cam_config.hdr);
    AssertSuccess(res);
    
    // setup camera output port
    res = SetVideoFormat(camera->output[CAMERA_VIDEO_PORT], w, h, 30);
    AssertSuccess(res);
    
    res = mmal_component_enable(camera);
    AssertSuccess(res);
}    

static void ReleaseCamera()
{
    if (camera) {
        mmal_component_destroy(camera);
        camera = NULL;
    }
}

static void ConnectSplitter(MMAL_PORT_T *sourcePort)
{
    MMAL_STATUS_T res;
    
    res = mmal_component_create("vc.ril.video_splitter", &splitter);
    AssertSuccess(res);
    
    sourcePort->format->encoding = MMAL_ENCODING_OPAQUE;
    sourcePort->format->encoding_variant = MMAL_ENCODING_I420;
    res = mmal_port_format_commit(sourcePort);
    AssertSuccess(res);
        
    res = mmal_connection_create(&splitterConn, sourcePort, splitter->input[0],
        MMAL_CONNECTION_FLAG_TUNNELLING | MMAL_CONNECTION_FLAG_ALLOCATION_ON_INPUT);
    AssertSuccess(res);
    
    res = mmal_component_enable(splitter);
    AssertSuccess(res);
    
    res = mmal_connection_enable(splitterConn);
    AssertSuccess(res);
}

static void DisconnectSplitter()          
{
    if (splitterConn) {
        mmal_connection_destroy(splitterConn);
        splitterConn = NULL;
    }
    if (splitter) {
        mmal_component_destroy(splitter);
        splitter = NULL;
    }
}

static void ConnectResizer(MMAL_PORT_T *sourcePort, uint32_t encoding, int w, int h)
{
    MMAL_STATUS_T res;
    
    res = mmal_component_create(RESIZER, &resizer);
    AssertSuccess(res);

    sourcePort->format->encoding = encoding;
    sourcePort->format->encoding_variant = MMAL_ENCODING_I420;
    res = mmal_port_format_commit(sourcePort);
    AssertSuccess(res);

    MMAL_PORT_T *inPort = resizer->input[0];
    res = mmal_connection_create(&resizerConn, sourcePort, inPort,
        MMAL_CONNECTION_FLAG_TUNNELLING | MMAL_CONNECTION_FLAG_ALLOCATION_ON_INPUT);
    AssertSuccess(res);

    MMAL_PORT_T *outPort = resizer->output[0];
    mmal_format_copy(outPort->format, inPort->format);
    res = SetVideoFormat(outPort, w, h, 0);
    AssertSuccess(res);
        
    res = mmal_component_enable(resizer);
    AssertSuccess(res);
    
    res = mmal_connection_enable(resizerConn);
    AssertSuccess(res);
}

static void DisconnectResizer()          
{
    if (resizerConn) {
        mmal_connection_destroy(resizerConn);
        resizerConn = NULL;
    }
    if (resizer) {
        mmal_component_destroy(resizer);
        resizer = NULL;
    }
}

static void ConnectEncoder(MMAL_PORT_T *sourcePort, uint32_t encoding)
{
    MMAL_STATUS_T res;
    
    res = mmal_component_create("vc.ril.video_encode", &encoder);
    AssertSuccess(res);

    sourcePort->format->encoding = encoding;
    sourcePort->format->encoding_variant = MMAL_ENCODING_I420;
    res = mmal_port_format_commit(sourcePort);
    AssertSuccess(res);

    MMAL_PORT_T *inPort = encoder->input[0];
    res = mmal_connection_create(&encoderConn, sourcePort, inPort,
        MMAL_CONNECTION_FLAG_TUNNELLING | MMAL_CONNECTION_FLAG_ALLOCATION_ON_INPUT);
    AssertSuccess(res);
    
    MMAL_PORT_T *outPort = encoder->output[0];
    mmal_format_copy(outPort->format, inPort->format);
    outPort->format->encoding = MMAL_ENCODING_H264;
    outPort->format->bitrate = 1000000;
    res = mmal_port_format_commit(outPort);
    AssertSuccess(res);
        
    res = mmal_component_enable(encoder);
    AssertSuccess(res);
    
    res = mmal_connection_enable(encoderConn);
    AssertSuccess(res);
    
    encoderPool = mmal_port_pool_create(outPort, outPort->buffer_num, outPort->buffer_size);
    
    res = mmal_port_enable(outPort, EncoderCallback);
    AssertSuccess(res);
    
    MMAL_BUFFER_HEADER_T *buffer;
    while ((buffer = mmal_queue_get(encoderPool->queue))) {
        res = mmal_port_send_buffer(outPort, buffer);
        AssertSuccess(res);
    }
}

static void DisconnectEncoder()          
{
    if (encoder && encoder->output[0]->is_enabled) {
        mmal_port_disable(encoder->output[0]);
    }
    if (encoderConn) {
        mmal_connection_destroy(encoderConn);
        encoderConn = NULL;
    }
    if (encoder) {
        mmal_component_destroy(encoder);
        encoder = NULL;
    }
}

int main()
{
    bcm_host_init();
    
    SetupCamera(1280, 720);
    ConnectSplitter(camera->output[CAMERA_VIDEO_PORT]);
    mmal_port_parameter_set_boolean(camera->output[CAMERA_VIDEO_PORT], MMAL_PARAMETER_CAPTURE, 1);
    
    // Test low resolution
    printf("test with non-opaque connection:\n");
    ConnectResizer(splitter->output[0], MMAL_ENCODING_I420, 640, 360); // does not work with MMAL_ENCODING_OPAQUE
    ConnectEncoder(resizer->output[0], MMAL_ENCODING_I420);            // does not work with MMAL_ENCODING_OPAQUE
    vcos_sleep(1000);
    printf("\n");
    DisconnectEncoder();
    DisconnectResizer();
    
    // Test high resolution
    printf("test with opaque connection:\n");
    ConnectEncoder(splitter->output[0], MMAL_ENCODING_OPAQUE);
    vcos_sleep(1000);
    printf("\n");
    DisconnectEncoder();

    mmal_port_parameter_set_boolean(camera->output[CAMERA_VIDEO_PORT], MMAL_PARAMETER_CAPTURE, 0);
    DisconnectSplitter();
    ReleaseCamera();
        
    return 0;
}
Swap the two test blocks in main() for the second case.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5954
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: MMAL Splitter output port cannot be changed between opaque and non-opaque

Sat Jul 28, 2018 3:01 pm

Thanks. I'll give it a try when I get a chance.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Graphics programming”