marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Need advice with tuneled JPEG decoder + resizer

Wed Oct 31, 2018 4:43 pm

I'm running a test program that loads 10 files from filesystem. Those 10 files are: 5 jpeg files with different resolutions, 4 jpeg files with header but incomplete data, 1 non jpeg file.

The program runs forever and after each iteration an action is taken depending on RNG: try decode one of the above files or reset de decoder. After a lot of iterations -hours, even days- the program stucks at reseting decoder right in this line:

Code: Select all

if ((omxerr = ilclient_wait_for_command_complete(deco->comp_deco, OMX_CommandPortEnable, deco->out_deco)) < 0)
Is something wrong in the code? At first I though the program stuck when reseting the decoder (destroy+init) after trying to load a jpeg file with incomplete data but it also happens when reseting after decoding a good jpeg file. Thanks.

Code: Select all

struct OMX_JPEG_DECODER
{
    ILCLIENT_T *ilclient;
    COMPONENT_T *comp_deco;
    COMPONENT_T *comp_resz;
    OMX_HANDLETYPE h_deco;
    OMX_HANDLETYPE h_resz;

    int in_deco;
    int out_deco;

    int in_resz;
    int out_resz;
};

void errorcb(void *userdata, COMPONENT_T *comp, OMX_U32 data)
{
    fprintf(stderr, "ERROR in component %p: %X\n", comp, data);
}

void configchangecb(void *userdata, COMPONENT_T *comp, OMX_U32 data)
{
    fprintf(stderr, "CONF CHANGE in component %p: %d\n", comp, data);
}

void emptybuffercb(void *userdata, COMPONENT_T *comp)
{
    fprintf(stderr, "BUFFER READ in component %p\n", comp);
}

void fillbuffercb(void *userdata, COMPONENT_T *comp)
{
    fprintf(stderr, "BUFFER FILLED in component %p\n", comp);
}

void eoscb(void *userdata, COMPONENT_T *comp, OMX_U32 data)
{
    fprintf(stderr, "EOS in component %p: %d\n", comp, data);
}

void psettingscb(void *userdata, COMPONENT_T *comp, OMX_U32 data)
{
    fprintf(stderr, "PORT SETTINGS CHANGED in component %p: %d\n", comp, data);
}

void init_jpeg_decoder(OMX_JPEG_DECODER *deco)
{
    int omxerr;
    deco->ilclient = ilclient_init();

    ilclient_set_error_callback(deco->ilclient, errorcb, deco);
    ilclient_set_configchanged_callback(deco->ilclient, configchangecb, deco);
    ilclient_set_empty_buffer_done_callback(deco->ilclient, emptybuffercb, deco);
    ilclient_set_eos_callback(deco->ilclient, eoscb, deco);
    ilclient_set_fill_buffer_done_callback(deco->ilclient, fillbuffercb, deco);
    ilclient_set_port_settings_callback(deco->ilclient, psettingscb, deco);

    if ((omxerr = ilclient_create_component(deco->ilclient, &deco->comp_deco, (char*)"image_decode", (ILCLIENT_CREATE_FLAGS_T)(ILCLIENT_DISABLE_ALL_PORTS|ILCLIENT_ENABLE_INPUT_BUFFERS))) != 0)
    {
        fprintf(stderr, "error creating component %X\n", omxerr);
    }

    deco->h_deco = ILC_GET_HANDLE(deco->comp_deco);

    OMX_PORT_PARAM_TYPE port;

    port.nSize = sizeof(OMX_PORT_PARAM_TYPE);
    port.nVersion.nVersion = OMX_VERSION;

    if ((omxerr = OMX_GetParameter(deco->h_deco, OMX_IndexParamImageInit, &port)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_GetParameter(OMX_IndexParamImageInit): %X\n", omxerr);
    }

    deco->in_deco = port.nStartPortNumber;
    deco->out_deco = port.nStartPortNumber + 1;

    if ((omxerr = ilclient_create_component(deco->ilclient, &deco->comp_resz, (char*)"resize", (ILCLIENT_CREATE_FLAGS_T)(ILCLIENT_DISABLE_ALL_PORTS|ILCLIENT_ENABLE_OUTPUT_BUFFERS))) != 0)
    {
        fprintf(stderr, "error creating component %X\n", omxerr);
    }

    deco->h_resz = ILC_GET_HANDLE(deco->comp_resz);

    port.nSize = sizeof(OMX_PORT_PARAM_TYPE);
    port.nVersion.nVersion = OMX_VERSION;

    if ((omxerr = OMX_GetParameter(deco->h_resz, OMX_IndexParamImageInit, &port)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_GetParameter(OMX_IndexParamImageInit): %X\n", omxerr);
    }

    deco->in_resz = port.nStartPortNumber;
    deco->out_resz = port.nStartPortNumber + 1;


    if ((omxerr = ilclient_change_component_state(deco->comp_deco, OMX_StateIdle)) < 0)
    {
        fprintf(stderr, "ilclient_change_component_state(Idle): %d\n", omxerr);
    }

    OMX_IMAGE_PARAM_PORTFORMATTYPE portfmt;


    memset(&portfmt, 0, sizeof(OMX_IMAGE_PARAM_PORTFORMATTYPE));
    portfmt.nSize = sizeof(OMX_IMAGE_PARAM_PORTFORMATTYPE);
    portfmt.nVersion.nVersion = OMX_VERSION;
    portfmt.nPortIndex = deco->in_deco;
    portfmt.eCompressionFormat = OMX_IMAGE_CodingJPEG;

    if ((omxerr = OMX_SetParameter(deco->h_deco, OMX_IndexParamImagePortFormat, &portfmt)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SetParameter(OMX_IndexparamImagePortFormat): %X\n", omxerr);
    }

    if ((omxerr = ilclient_enable_port_buffers(deco->comp_deco, deco->in_deco, NULL, NULL, NULL)) < 0)
    {
        fprintf(stderr, "ilclient_enable_port_buffers(deco, in): %d\n", omxerr);
    }

    if ((omxerr = OMX_SetupTunnel(deco->h_deco, deco->out_deco, deco->h_resz, deco->in_resz)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SetupTunnel(deco, out, resize, in): %d\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_deco, OMX_CommandPortEnable, deco->out_deco, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(deco, OMX_CommandPortEnable, out): %d\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_resz, OMX_CommandPortEnable, deco->in_resz, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(resz, OMX_CommandPortEnable, in): %d\n", omxerr);
    }

    if ((omxerr = ilclient_change_component_state(deco->comp_resz, OMX_StateIdle)) < 0)
    {
        fprintf(stderr, "ilclient_change_component_state(resize, Idle): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_deco, OMX_CommandPortEnable, deco->out_deco)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(deco, OMX_CommandPortEnable, out): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_resz, OMX_CommandPortEnable, deco->in_resz)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(resz, OMX_CommandPortEnable, in): %d\n", omxerr);
    }
}

void destroy_jpeg_decoder(OMX_JPEG_DECODER *deco)
{
    COMPONENT_T *components[] = { deco->comp_deco, deco->comp_resz, NULL };
    int omxerr;
    int timeout = 500;

    if ((omxerr = OMX_SendCommand(deco->h_deco, OMX_CommandFlush, deco->in_deco, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(deco, OMX_CommandFlush, in): %X\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_deco, OMX_CommandFlush, deco->out_deco, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(deco, OMX_CommandFlush, out): %X\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_resz, OMX_CommandFlush, deco->in_resz, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(resize, OMX_CommandFlush, in): %X\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_resz, OMX_CommandFlush, deco->out_resz, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(resize, OMX_CommandFlush, out): %X\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_event(deco->comp_deco, OMX_EventCmdComplete, OMX_CommandFlush, 0, deco->in_deco, 0, ILCLIENT_EVENT_ERROR, timeout)) != 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(deco, OMX_CommandFlush, in): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_event(deco->comp_deco, OMX_EventCmdComplete, OMX_CommandFlush, 0, deco->out_deco, 0, ILCLIENT_EVENT_ERROR, timeout)) != 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(deco, OMX_CommandFlush, out): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_event(deco->comp_resz, OMX_EventCmdComplete, OMX_CommandFlush, 0, deco->in_resz, 0, ILCLIENT_EVENT_ERROR, timeout)) != 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(resize, OMX_CommandFlush, in): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_event(deco->comp_resz, OMX_EventCmdComplete, OMX_CommandFlush, 0, deco->out_resz, 0, ILCLIENT_EVENT_ERROR, timeout)) != 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(resize, OMX_CommandFlush, out): %d\n", omxerr);
    }

    ilclient_disable_port_buffers(deco->comp_deco, deco->in_deco, NULL, NULL, NULL);
    ilclient_disable_port_buffers(deco->comp_resz, deco->out_resz, NULL, NULL, NULL);

    if ((omxerr = OMX_SendCommand(deco->h_deco, OMX_CommandPortDisable, deco->out_deco, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(deco, OMX_CommandPortDisable, out): %X\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_resz, OMX_CommandPortDisable, deco->in_resz, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(resize, OMX_CommandPortDisable, in): %X\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_deco, OMX_CommandPortDisable, deco->out_deco)) != 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(deco, OMX_CommandPortDisable, out): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_resz, OMX_CommandPortDisable, deco->in_resz)) != 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(resize, OMX_CommandPortDisable, in): %d\n", omxerr);
    }

    if ((omxerr = OMX_SetupTunnel(deco->h_deco, deco->out_deco, NULL, 0)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SetupTunnel(deco, out, NULL, 0): %X\n", omxerr);
    }

    if ((omxerr = OMX_SetupTunnel(deco->h_resz, deco->in_resz, NULL, 0)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SetupTunnel(resize, in, NULL, 0): %X\n", omxerr);
    }

    ilclient_state_transition(components, OMX_StateLoaded);

    ilclient_cleanup_components(components);
    ilclient_destroy(deco->ilclient);
}

void apply_port_settings(OMX_JPEG_DECODER *deco)
{
    int omxerr;
    int width;
    int height;
    OMX_PARAM_PORTDEFINITIONTYPE portdef;
    OMX_STATETYPE last_state;

    portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
    portdef.nVersion.nVersion = OMX_VERSION;
    portdef.nPortIndex = deco->out_deco;


    if ((omxerr = OMX_SendCommand(deco->h_deco, OMX_CommandPortDisable, deco->out_deco, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(deco, OMX_CommandPortDisable, out): %d\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_resz, OMX_CommandPortDisable, deco->in_resz, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(resz, OMX_CommandPortDisable, in): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_deco, OMX_CommandPortDisable, deco->out_deco)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(deco, OMX_CommandPortDisable, out): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_resz, OMX_CommandPortDisable, deco->in_resz)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(resz, OMX_CommandPortDisable, in): %d\n", omxerr);
    }

    if ((omxerr = OMX_GetParameter(deco->h_deco, OMX_IndexParamPortDefinition, &portdef)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_GetParameter(deco, OMX_IndexParamPortDefinition, out): %X\n", omxerr);
    }
    else
    {
        width = portdef.format.image.nFrameWidth;
        height = portdef.format.image.nFrameHeight;
    }

    portdef.nPortIndex = deco->in_resz;

    if ((omxerr = OMX_SetParameter(deco->h_resz, OMX_IndexParamPortDefinition, &portdef)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SetParameter(resz, OMX_IndexParamPortDefinition, in): %X\n", omxerr);
    }

    if ((omxerr = OMX_GetState(deco->h_resz, &last_state)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_GetState(resz): %X\n", omxerr);
    }

    if ((omxerr = ilclient_change_component_state(deco->comp_resz, OMX_StateIdle)) < 0)
    {
        fprintf(stderr, "ilclient_change_component_state(resize, Idle): %d\n", omxerr);
    }

    ilclient_disable_port_buffers(deco->comp_resz, deco->out_resz, NULL, NULL, NULL);

    portdef.nPortIndex = deco->out_resz;

    if ((omxerr = OMX_GetParameter(deco->h_resz, OMX_IndexParamPortDefinition, &portdef)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_GetParameter(deco, OMX_IndexParamPortDefinition, out): %X\n", omxerr);
    }

    portdef.format.image.eCompressionFormat = OMX_IMAGE_CodingUnused;
    portdef.format.image.eColorFormat = OMX_COLOR_Format32bitARGB8888;
    portdef.format.image.nFrameWidth = width;
    portdef.format.image.nFrameHeight = height;
    portdef.format.image.nStride = 0;
    portdef.format.image.nSliceHeight = 0;
    portdef.format.image.bFlagErrorConcealment = OMX_FALSE;

    if ((omxerr = OMX_SetParameter(deco->h_resz, OMX_IndexParamPortDefinition, &portdef)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SetParameter(resz, OMX_IndexParamPortDefinition, in): %X\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_deco, OMX_CommandPortEnable, deco->out_deco, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(deco, OMX_CommandPortEnable, out): %d\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_resz, OMX_CommandPortEnable, deco->in_resz, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(resz, OMX_CommandPortEnable, in): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_deco, OMX_CommandPortEnable, deco->out_deco)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(deco, OMX_CommandPortEnable, out): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_resz, OMX_CommandPortEnable, deco->in_resz)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(resz, OMX_CommandPortEnable, in): %d\n", omxerr);
    }

    ilclient_enable_port_buffers(deco->comp_resz, deco->out_resz, NULL, NULL, NULL);

    if ((omxerr = ilclient_wait_for_event(deco->comp_resz, OMX_EventPortSettingsChanged, deco->out_resz, 0, 0, 1, ILCLIENT_EVENT_ERROR, 0)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_event(deco, OMX_eventPortSettingsChanged, out): %d\n", omxerr);
    }

    if (last_state != OMX_StateIdle && (omxerr = ilclient_change_component_state(deco->comp_resz, OMX_StateExecuting)) < 0)
    {
        fprintf(stderr, "ilclient_change_component_state(resize, Exec): %d\n", omxerr);
    }
}

int jpeg_decode(OMX_JPEG_DECODER *deco, const std::vector<uint8_t> &src, LwImage &dst)
{
    COMPONENT_T *components[] = { deco->comp_deco, deco->comp_resz, NULL };
    int omxerr;
    int result = -1;
    const uint8_t *srcOffset = src.data();
    size_t pendingbytes = src.size();
    bool portSettingsChanged = false;

    ilclient_state_transition(components, OMX_StateExecuting);

    while (pendingbytes > 0)
    {
        OMX_BUFFERHEADERTYPE *buffer = ilclient_get_input_buffer(deco->comp_deco, deco->in_deco, 1);

        if (pendingbytes > buffer->nAllocLen)
            buffer->nFilledLen = buffer->nAllocLen;
        else
            buffer->nFilledLen = pendingbytes;

        memcpy(buffer->pBuffer, srcOffset, buffer->nFilledLen);
        srcOffset += buffer->nFilledLen;
        pendingbytes -= buffer->nFilledLen;
        buffer->nOffset = 0;
        buffer->nFlags = (pendingbytes > 0) ? 0 : OMX_BUFFERFLAG_EOS;

        if ((omxerr = OMX_EmptyThisBuffer(deco->h_deco, buffer)) != OMX_ErrorNone)
        {
            fprintf(stderr, "OMX_EmptyThisBuffer(deco, in): %X\n", omxerr);
        }
    }

    if ((omxerr = ilclient_wait_for_event(deco->comp_deco, OMX_EventPortSettingsChanged, deco->out_deco, 0, 0, 1, ILCLIENT_EVENT_ERROR | ILCLIENT_CONFIG_CHANGED | ILCLIENT_PARAMETER_CHANGED | ILCLIENT_BUFFER_FLAG_EOS, 50)) != -2)
    {
        if (omxerr == 0)
            apply_port_settings(deco);

        if ((omxerr = ilclient_wait_for_event(deco->comp_deco, OMX_EventBufferFlag, deco->out_deco, 0, OMX_BUFFERFLAG_EOS, 0, ILCLIENT_EVENT_ERROR | ILCLIENT_CONFIG_CHANGED | ILCLIENT_BUFFER_FLAG_EOS, 50)) == 0)
        {
            OMX_BUFFERHEADERTYPE *buffer = ilclient_get_output_buffer(deco->comp_resz, deco->out_resz, 1);

            if ((omxerr = OMX_FillThisBuffer(deco->h_resz, buffer)) != OMX_ErrorNone)
            {
                fprintf(stderr, "OMX_FillThisBuffer(resize, out): %X\n", omxerr);
            }

            OMX_PARAM_PORTDEFINITIONTYPE portdef;
            portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
            portdef.nVersion.nVersion = OMX_VERSION;
            portdef.nPortIndex = deco->out_resz;

            if (ilclient_remove_event(deco->comp_deco, OMX_EventError, 0, 1, 0, 1) < 0 && (omxerr = ilclient_wait_for_event(deco->comp_resz, OMX_EventBufferFlag, deco->out_resz, 0, OMX_BUFFERFLAG_EOS, 0, ILCLIENT_EVENT_ERROR | ILCLIENT_CONFIG_CHANGED | ILCLIENT_BUFFER_FLAG_EOS, 500)) < 0)
            {
                fprintf(stderr, "ilclient_wait_for_event(resize, OMX_EventBufferFlag, out, EOS): %d\n", omxerr);

                if ((omxerr = ilclient_remove_event(deco->comp_deco, OMX_EventError, 0, 1, 0, 1)) < 0)
                {
                    fprintf(stderr, "ilclient_remove_event(deco, OMX_EventError, *): %d\n", omxerr);
                }
                else
                {
                    fprintf(stderr, "!!! ERROR REMOVED: there wasn't an error? %d\n", omxerr);
                }
            }
            else
            {
                if ((omxerr = OMX_GetParameter(deco->h_resz, OMX_IndexParamPortDefinition, &portdef)) == OMX_ErrorNone)
                {
                    dst.width = portdef.format.image.nFrameWidth;
                    dst.height = portdef.format.image.nFrameHeight;
                    dst.stride = portdef.format.image.nStride;
                    dst.format = MAKE_PIXEL_FORMAT(4, 8, 0, 0);
                    dst.pixels = std::vector<uint8_t>(buffer->pBuffer, buffer->pBuffer + dst.stride * dst.height);
                    result = 0;
                }
                else
                {
                    fprintf(stderr, "OMX_GetParameter(resize, OMX_IndexParamPortDefinition, out): %X\n", omxerr);
                }
            }
        }
        else if (omxerr == -2)
        {
            //fprintf(stderr, "ilclient_wait_for_event(deco, OMX_EventBufferFlag, out, EOS): %d\n", omxerr);
            if ((omxerr = ilclient_remove_event(deco->comp_deco, OMX_EventError, 0, 1, 0, 1)) < 0)
            {
                fprintf(stderr, "ilclient_remove_event(deco, OMX_EventError, *): %d\n", omxerr);
            }
        }

    }
    else
    {
        if ((omxerr = ilclient_remove_event(deco->comp_deco, OMX_EventError, 0, 1, 0, 1)) < 0)
        {
            fprintf(stderr, "ilclient_remove_event(deco, OMX_EventError, *): %d\n", omxerr);
        }
    }

    ilclient_state_transition(components, OMX_StateIdle);

    return result;
}

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5808
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Need advice with tuneled JPEG decoder + resizer

Wed Oct 31, 2018 9:55 pm

Give me everything I need to run your test and I can take a look. Seeing as there is no main() in that code I can't run it without making assumptions about how you are setting things up.
It sounds like a memory leak, but the components have had a fairly good hammering over the years to fix those. Hopefully any issues would be noticeable after a few iterations, and you're just seeing the final stages where resources are totally drained.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Fri Nov 02, 2018 9:20 am

Thanks for your help. Remember that the bug shows after a lot of hours even days on my pi.

Code: Select all

#ifdef __cplusplus
extern "C"
{
#endif
#include <ilclient.h>
#ifdef __cplusplus
}
#endif
#include <bcm_host.h>
#include <stdio.h>
#include <vector>
#include <iostream>
#include <fstream>
#include <unistd.h>
#include <dirent.h>

#define MAKE_PIXEL_FORMAT(channels, depth, datatype, order) ((channels) | ((depth) << 4) | ((datatype) << 14) | ((order) << 24))

class LwImage
{
public:
    LwImage() : width(0), height(0), stride(0), format(0) {}
    ~LwImage() {}
    std::vector<uint8_t> pixels;
    int width;
    int height;
    int stride;
    int format;
};

struct OMX_JPEG_DECODER
{
    ILCLIENT_T *ilclient;
    COMPONENT_T *comp_deco;
    COMPONENT_T *comp_resz;
    OMX_HANDLETYPE h_deco;
    OMX_HANDLETYPE h_resz;

    int in_deco;
    int out_deco;

    int in_resz;
    int out_resz;
};

void errorcb(void *userdata, COMPONENT_T *comp, OMX_U32 data)
{
    fprintf(stderr, "ERROR in component %p: %X\n", comp, data);
}

void configchangecb(void *userdata, COMPONENT_T *comp, OMX_U32 data)
{
    fprintf(stderr, "CONF CHANGE in component %p: %d\n", comp, data);
}

void emptybuffercb(void *userdata, COMPONENT_T *comp)
{
    fprintf(stderr, "BUFFER READ in component %p\n", comp);
}

void fillbuffercb(void *userdata, COMPONENT_T *comp)
{
    fprintf(stderr, "BUFFER FILLED in component %p\n", comp);
}

void eoscb(void *userdata, COMPONENT_T *comp, OMX_U32 data)
{
    fprintf(stderr, "EOS in component %p: %d\n", comp, data);
}

void psettingscb(void *userdata, COMPONENT_T *comp, OMX_U32 data)
{
    fprintf(stderr, "PORT SETTINGS CHANGED in component %p: %d\n", comp, data);
}

void init_jpeg_decoder(OMX_JPEG_DECODER *deco)
{
    int omxerr;
    deco->ilclient = ilclient_init();

    ilclient_set_error_callback(deco->ilclient, errorcb, deco);
    ilclient_set_configchanged_callback(deco->ilclient, configchangecb, deco);
    ilclient_set_empty_buffer_done_callback(deco->ilclient, emptybuffercb, deco);
    ilclient_set_eos_callback(deco->ilclient, eoscb, deco);
    ilclient_set_fill_buffer_done_callback(deco->ilclient, fillbuffercb, deco);
    ilclient_set_port_settings_callback(deco->ilclient, psettingscb, deco);

    if ((omxerr = ilclient_create_component(deco->ilclient, &deco->comp_deco, (char*)"image_decode", (ILCLIENT_CREATE_FLAGS_T)(ILCLIENT_DISABLE_ALL_PORTS|ILCLIENT_ENABLE_INPUT_BUFFERS))) != 0)
    {
        fprintf(stderr, "error creating component %X\n", omxerr);
    }

    deco->h_deco = ILC_GET_HANDLE(deco->comp_deco);

    OMX_PORT_PARAM_TYPE port;

    port.nSize = sizeof(OMX_PORT_PARAM_TYPE);
    port.nVersion.nVersion = OMX_VERSION;

    if ((omxerr = OMX_GetParameter(deco->h_deco, OMX_IndexParamImageInit, &port)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_GetParameter(OMX_IndexParamImageInit): %X\n", omxerr);
    }

    deco->in_deco = port.nStartPortNumber;
    deco->out_deco = port.nStartPortNumber + 1;

    if ((omxerr = ilclient_create_component(deco->ilclient, &deco->comp_resz, (char*)"resize", (ILCLIENT_CREATE_FLAGS_T)(ILCLIENT_DISABLE_ALL_PORTS|ILCLIENT_ENABLE_OUTPUT_BUFFERS))) != 0)
    {
        fprintf(stderr, "error creating component %X\n", omxerr);
    }

    deco->h_resz = ILC_GET_HANDLE(deco->comp_resz);

    port.nSize = sizeof(OMX_PORT_PARAM_TYPE);
    port.nVersion.nVersion = OMX_VERSION;

    if ((omxerr = OMX_GetParameter(deco->h_resz, OMX_IndexParamImageInit, &port)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_GetParameter(OMX_IndexParamImageInit): %X\n", omxerr);
    }

    deco->in_resz = port.nStartPortNumber;
    deco->out_resz = port.nStartPortNumber + 1;


    if ((omxerr = ilclient_change_component_state(deco->comp_deco, OMX_StateIdle)) < 0)
    {
        fprintf(stderr, "ilclient_change_component_state(Idle): %d\n", omxerr);
    }

    OMX_IMAGE_PARAM_PORTFORMATTYPE portfmt;


    memset(&portfmt, 0, sizeof(OMX_IMAGE_PARAM_PORTFORMATTYPE));
    portfmt.nSize = sizeof(OMX_IMAGE_PARAM_PORTFORMATTYPE);
    portfmt.nVersion.nVersion = OMX_VERSION;
    portfmt.nPortIndex = deco->in_deco;
    portfmt.eCompressionFormat = OMX_IMAGE_CodingJPEG;

    if ((omxerr = OMX_SetParameter(deco->h_deco, OMX_IndexParamImagePortFormat, &portfmt)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SetParameter(OMX_IndexparamImagePortFormat): %X\n", omxerr);
    }

    if ((omxerr = ilclient_enable_port_buffers(deco->comp_deco, deco->in_deco, NULL, NULL, NULL)) < 0)
    {
        fprintf(stderr, "ilclient_enable_port_buffers(deco, in): %d\n", omxerr);
    }

    if ((omxerr = OMX_SetupTunnel(deco->h_deco, deco->out_deco, deco->h_resz, deco->in_resz)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SetupTunnel(deco, out, resize, in): %d\n", omxerr);
    }
    if ((omxerr = OMX_SendCommand(deco->h_deco, OMX_CommandPortEnable, deco->out_deco, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(deco, OMX_CommandPortEnable, out): %d\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_resz, OMX_CommandPortEnable, deco->in_resz, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(resz, OMX_CommandPortEnable, in): %d\n", omxerr);
    }

    if ((omxerr = ilclient_change_component_state(deco->comp_resz, OMX_StateIdle)) < 0)
    {
        fprintf(stderr, "ilclient_change_component_state(resize, Idle): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_deco, OMX_CommandPortEnable, deco->out_deco)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(deco, OMX_CommandPortEnable, out): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_resz, OMX_CommandPortEnable, deco->in_resz)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(resz, OMX_CommandPortEnable, in): %d\n", omxerr);
    }
}

void destroy_jpeg_decoder(OMX_JPEG_DECODER *deco)
{
    COMPONENT_T *components[] = { deco->comp_deco, deco->comp_resz, NULL };
    int omxerr;
    int timeout = 500;

    if ((omxerr = OMX_SendCommand(deco->h_deco, OMX_CommandFlush, deco->in_deco, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(deco, OMX_CommandFlush, in): %X\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_deco, OMX_CommandFlush, deco->out_deco, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(deco, OMX_CommandFlush, out): %X\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_resz, OMX_CommandFlush, deco->in_resz, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(resize, OMX_CommandFlush, in): %X\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_resz, OMX_CommandFlush, deco->out_resz, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(resize, OMX_CommandFlush, out): %X\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_event(deco->comp_deco, OMX_EventCmdComplete, OMX_CommandFlush, 0, deco->in_deco, 0, ILCLIENT_EVENT_ERROR, timeout)) != 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(deco, OMX_CommandFlush, in): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_event(deco->comp_deco, OMX_EventCmdComplete, OMX_CommandFlush, 0, deco->out_deco, 0, ILCLIENT_EVENT_ERROR, timeout)) != 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(deco, OMX_CommandFlush, out): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_event(deco->comp_resz, OMX_EventCmdComplete, OMX_CommandFlush, 0, deco->in_resz, 0, ILCLIENT_EVENT_ERROR, timeout)) != 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(resize, OMX_CommandFlush, in): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_event(deco->comp_resz, OMX_EventCmdComplete, OMX_CommandFlush, 0, deco->out_resz, 0, ILCLIENT_EVENT_ERROR, timeout)) != 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(resize, OMX_CommandFlush, out): %d\n", omxerr);
    }

    ilclient_disable_port_buffers(deco->comp_deco, deco->in_deco, NULL, NULL, NULL);
    ilclient_disable_port_buffers(deco->comp_resz, deco->out_resz, NULL, NULL, NULL);

    if ((omxerr = OMX_SendCommand(deco->h_deco, OMX_CommandPortDisable, deco->out_deco, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(deco, OMX_CommandPortDisable, out): %X\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_resz, OMX_CommandPortDisable, deco->in_resz, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(resize, OMX_CommandPortDisable, in): %X\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_deco, OMX_CommandPortDisable, deco->out_deco)) != 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(deco, OMX_CommandPortDisable, out): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_resz, OMX_CommandPortDisable, deco->in_resz)) != 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(resize, OMX_CommandPortDisable, in): %d\n", omxerr);
    }

    if ((omxerr = OMX_SetupTunnel(deco->h_deco, deco->out_deco, NULL, 0)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SetupTunnel(deco, out, NULL, 0): %X\n", omxerr);
    }

    if ((omxerr = OMX_SetupTunnel(deco->h_resz, deco->in_resz, NULL, 0)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SetupTunnel(resize, in, NULL, 0): %X\n", omxerr);
    }

    ilclient_state_transition(components, OMX_StateLoaded);

    ilclient_cleanup_components(components);
    ilclient_destroy(deco->ilclient);
}

void apply_port_settings(OMX_JPEG_DECODER *deco)
{
    int omxerr;
    int width;
    int height;
    OMX_PARAM_PORTDEFINITIONTYPE portdef;
    OMX_STATETYPE last_state;

    portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
    portdef.nVersion.nVersion = OMX_VERSION;
    portdef.nPortIndex = deco->out_deco;


    if ((omxerr = OMX_SendCommand(deco->h_deco, OMX_CommandPortDisable, deco->out_deco, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(deco, OMX_CommandPortDisable, out): %d\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_resz, OMX_CommandPortDisable, deco->in_resz, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(resz, OMX_CommandPortDisable, in): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_deco, OMX_CommandPortDisable, deco->out_deco)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(deco, OMX_CommandPortDisable, out): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_resz, OMX_CommandPortDisable, deco->in_resz)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(resz, OMX_CommandPortDisable, in): %d\n", omxerr);
    }

    if ((omxerr = OMX_GetParameter(deco->h_deco, OMX_IndexParamPortDefinition, &portdef)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_GetParameter(deco, OMX_IndexParamPortDefinition, out): %X\n", omxerr);
    }
    else
    {
        width = portdef.format.image.nFrameWidth;
        height = portdef.format.image.nFrameHeight;
    }

    portdef.nPortIndex = deco->in_resz;

    if ((omxerr = OMX_SetParameter(deco->h_resz, OMX_IndexParamPortDefinition, &portdef)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SetParameter(resz, OMX_IndexParamPortDefinition, in): %X\n", omxerr);
    }

    if ((omxerr = OMX_GetState(deco->h_resz, &last_state)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_GetState(resz): %X\n", omxerr);
    }

    if ((omxerr = ilclient_change_component_state(deco->comp_resz, OMX_StateIdle)) < 0)
    {
        fprintf(stderr, "ilclient_change_component_state(resize, Idle): %d\n", omxerr);
    }

    ilclient_disable_port_buffers(deco->comp_resz, deco->out_resz, NULL, NULL, NULL);

    portdef.nPortIndex = deco->out_resz;

    if ((omxerr = OMX_GetParameter(deco->h_resz, OMX_IndexParamPortDefinition, &portdef)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_GetParameter(deco, OMX_IndexParamPortDefinition, out): %X\n", omxerr);
    }

    portdef.format.image.eCompressionFormat = OMX_IMAGE_CodingUnused;
    portdef.format.image.eColorFormat = OMX_COLOR_Format32bitARGB8888;
    portdef.format.image.nFrameWidth = width;
    portdef.format.image.nFrameHeight = height;
    portdef.format.image.nStride = 0;
    portdef.format.image.nSliceHeight = 0;
    portdef.format.image.bFlagErrorConcealment = OMX_FALSE;

    if ((omxerr = OMX_SetParameter(deco->h_resz, OMX_IndexParamPortDefinition, &portdef)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SetParameter(resz, OMX_IndexParamPortDefinition, in): %X\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_deco, OMX_CommandPortEnable, deco->out_deco, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(deco, OMX_CommandPortEnable, out): %d\n", omxerr);
    }

    if ((omxerr = OMX_SendCommand(deco->h_resz, OMX_CommandPortEnable, deco->in_resz, NULL)) != OMX_ErrorNone)
    {
        fprintf(stderr, "OMX_SendCommand(resz, OMX_CommandPortEnable, in): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_deco, OMX_CommandPortEnable, deco->out_deco)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(deco, OMX_CommandPortEnable, out): %d\n", omxerr);
    }

    if ((omxerr = ilclient_wait_for_command_complete(deco->comp_resz, OMX_CommandPortEnable, deco->in_resz)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_command_complete(resz, OMX_CommandPortEnable, in): %d\n", omxerr);
    }

    ilclient_enable_port_buffers(deco->comp_resz, deco->out_resz, NULL, NULL, NULL);

    if ((omxerr = ilclient_wait_for_event(deco->comp_resz, OMX_EventPortSettingsChanged, deco->out_resz, 0, 0, 1, ILCLIENT_EVENT_ERROR, 0)) < 0)
    {
        fprintf(stderr, "ilclient_wait_for_event(deco, OMX_eventPortSettingsChanged, out): %d\n", omxerr);
    }

    if (last_state != OMX_StateIdle && (omxerr = ilclient_change_component_state(deco->comp_resz, OMX_StateExecuting)) < 0)
    {
        fprintf(stderr, "ilclient_change_component_state(resize, Exec): %d\n", omxerr);
    }
}

int jpeg_decode(OMX_JPEG_DECODER *deco, const std::vector<uint8_t> &src, LwImage &dst)
{
    COMPONENT_T *components[] = { deco->comp_deco, deco->comp_resz, NULL };
    int omxerr;
    int result = -1;
    const uint8_t *srcOffset = src.data();
    size_t pendingbytes = src.size();
    bool portSettingsChanged = false;

    ilclient_state_transition(components, OMX_StateExecuting);

    while (pendingbytes > 0)
    {
        OMX_BUFFERHEADERTYPE *buffer = ilclient_get_input_buffer(deco->comp_deco, deco->in_deco, 1);

        if (pendingbytes > buffer->nAllocLen)
            buffer->nFilledLen = buffer->nAllocLen;
        else
            buffer->nFilledLen = pendingbytes;

        memcpy(buffer->pBuffer, srcOffset, buffer->nFilledLen);
        srcOffset += buffer->nFilledLen;
        pendingbytes -= buffer->nFilledLen;
        buffer->nOffset = 0;
        buffer->nFlags = (pendingbytes > 0) ? 0 : OMX_BUFFERFLAG_EOS;

        if ((omxerr = OMX_EmptyThisBuffer(deco->h_deco, buffer)) != OMX_ErrorNone)
        {
            fprintf(stderr, "OMX_EmptyThisBuffer(deco, in): %X\n", omxerr);
        }
    }

    if ((omxerr = ilclient_wait_for_event(deco->comp_deco, OMX_EventPortSettingsChanged, deco->out_deco, 0, 0, 1, ILCLIENT_EVENT_ERROR | ILCLIENT_CONFIG_CHANGED | ILCLIENT_PARAMETER_CHANGED | ILCLIENT_BUFFER_FLAG_EOS, 50)) != -2)
    {
        //bool port_settings_triggered = (omxerr == 0);
        if (omxerr == 0)
            apply_port_settings(deco);
        //else
        //    fprintf(stderr, "ilclient_wait_for_event(deco, OMX_eventPortSettingsChanged, out): %d\n", omxerr);

        if ((omxerr = ilclient_wait_for_event(deco->comp_deco, OMX_EventBufferFlag, deco->out_deco, 0, OMX_BUFFERFLAG_EOS, 0, ILCLIENT_EVENT_ERROR | ILCLIENT_CONFIG_CHANGED | ILCLIENT_BUFFER_FLAG_EOS, 50)) == 0)
        {
            OMX_BUFFERHEADERTYPE *buffer = ilclient_get_output_buffer(deco->comp_resz, deco->out_resz, 1);

            if ((omxerr = OMX_FillThisBuffer(deco->h_resz, buffer)) != OMX_ErrorNone)
            {
                fprintf(stderr, "OMX_FillThisBuffer(resize, out): %X\n", omxerr);
            }

            OMX_PARAM_PORTDEFINITIONTYPE portdef;
            portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
            portdef.nVersion.nVersion = OMX_VERSION;
            portdef.nPortIndex = deco->out_resz;

            if (ilclient_remove_event(deco->comp_deco, OMX_EventError, 0, 1, 0, 1) < 0 && (omxerr = ilclient_wait_for_event(deco->comp_resz, OMX_EventBufferFlag, deco->out_resz, 0, OMX_BUFFERFLAG_EOS, 0, ILCLIENT_EVENT_ERROR | ILCLIENT_CONFIG_CHANGED | ILCLIENT_BUFFER_FLAG_EOS, 500)) < 0)
            {
                fprintf(stderr, "ilclient_wait_for_event(resize, OMX_EventBufferFlag, out, EOS): %d\n", omxerr);

                if ((omxerr = ilclient_remove_event(deco->comp_deco, OMX_EventError, 0, 1, 0, 1)) < 0)
                {
                    fprintf(stderr, "ilclient_remove_event(deco, OMX_EventError, *): %d\n", omxerr);
                }
                else
                {
                    fprintf(stderr, "!!! ERROR REMOVED: there wasn't an error? %d\n", omxerr);
                }
            }
            else
            {
                if ((omxerr = OMX_GetParameter(deco->h_resz, OMX_IndexParamPortDefinition, &portdef)) == OMX_ErrorNone)
                {
                    dst.width = portdef.format.image.nFrameWidth;
                    dst.height = portdef.format.image.nFrameHeight;
                    dst.stride = portdef.format.image.nStride;
                    dst.format = MAKE_PIXEL_FORMAT(4, 8, 0, 0);
                    dst.pixels = std::vector<uint8_t>(buffer->pBuffer, buffer->pBuffer + dst.stride * dst.height);
                    result = 0;
                }
                else
                {
                    fprintf(stderr, "OMX_GetParameter(resize, OMX_IndexParamPortDefinition, out): %X\n", omxerr);
                }
            }
        }
        else if (omxerr == -2)
        {
            if ((omxerr = ilclient_remove_event(deco->comp_deco, OMX_EventError, 0, 1, 0, 1)) < 0)
            {
                fprintf(stderr, "ilclient_remove_event(deco, OMX_EventError, *): %d\n", omxerr);
            }
        }

    }
    else
    {
        if ((omxerr = ilclient_remove_event(deco->comp_deco, OMX_EventError, 0, 1, 0, 1)) < 0)
        {
            fprintf(stderr, "ilclient_remove_event(deco, OMX_EventError, *): %d\n", omxerr);
        }
    }

    ilclient_state_transition(components, OMX_StateIdle);

    return result;
}

std::vector<uint8_t> read_file(const std::string &filename)
{
    std::ifstream stream(filename.c_str(), std::ios::in | std::ios::binary);
    return std::vector<uint8_t>(std::istreambuf_iterator<char>(stream), std::istreambuf_iterator<char>());
}

int main()
{
    OMX_JPEG_DECODER deco;
    int omxerr;

    std::vector<std::pair<std::string,std::vector<uint8_t>>> files;

    bcm_host_init();

    if ((omxerr = OMX_Init()) != OMX_ErrorNone)
        fprintf(stderr, "could'nt initilize OMX: %X", omxerr);

    init_jpeg_decoder(&deco);

    DIR *dp;
    struct dirent *dirp;
    std::string data_path = "./data";

    if ((dp = opendir(data_path.c_str())) != NULL)
    {
        while ((dirp = readdir(dp)) != NULL)
        {
            std::string cf = std::string(dirp->d_name);
            if (cf != "." && cf != "..")
            {
                std::vector<uint8_t> filedata = read_file(data_path + "/" + cf);

                if (filedata.size() > 0)
                {
                    files.push_back(std::pair<std::string, std::vector<uint8_t>>(cf, filedata));
                    std::cout << cf << ": " << filedata.size() << " bytes." << std::endl;
                }
            }

        }
    }

    srand(time(NULL));

    while (true)
    {
        int action = rand() % (files.size() + 1);

        if (action == files.size())
        {
            destroy_jpeg_decoder(&deco);
            init_jpeg_decoder(&deco);
        }
        else
        {
            std::vector<uint8_t> &v = files[action].second;
            LwImage img;

            if (jpeg_decode(&deco, v, img) == 0)
                std::cout << "image decoded: " << files[action].first << "(" << img.width << "x" << img.height << ")" << std::endl;
            else
                std::cout << "!!!error decoding: " << files[action].first << std::endl;
        }
    }

    return 0;
}
libraries: rt,vcos,vchiq_arm,openmaxil,bcm_host,m,ilclient,pthread
defines: OMX_SKIP64BIT

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Fri Nov 02, 2018 9:29 am

Attached files used in tests. As I said some are jpeg, some are incomplet (truncated) jpegs and some are not related to jpeg
Attachments
dbgitms.7z
files used in test
(67.27 KiB) Downloaded 6 times

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Tue Nov 06, 2018 10:05 am

6by9 wrote:
Wed Oct 31, 2018 9:55 pm
Give me everything I need to run your test and I can take a look. Seeing as there is no main() in that code I can't run it without making assumptions about how you are setting things up.
It sounds like a memory leak, but the components have had a fairly good hammering over the years to fix those. Hopefully any issues would be noticeable after a few iterations, and you're just seeing the final stages where resources are totally drained.
I don't want to hurry up but did you find anything? I'm running the test app in 3 raspberies (2x rpi 3B+, 1x rpi 2B) and the test got stuck at different times in all of them.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5808
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Need advice with tuneled JPEG decoder + resizer

Tue Nov 06, 2018 11:44 am

marranxo wrote:
Tue Nov 06, 2018 10:05 am
6by9 wrote:
Wed Oct 31, 2018 9:55 pm
Give me everything I need to run your test and I can take a look. Seeing as there is no main() in that code I can't run it without making assumptions about how you are setting things up.
It sounds like a memory leak, but the components have had a fairly good hammering over the years to fix those. Hopefully any issues would be noticeable after a few iterations, and you're just seeing the final stages where resources are totally drained.
I don't want to hurry up but did you find anything? I'm running the test app in 3 raspberies (2x rpi 3B+, 1x rpi 2B) and the test got stuck at different times in all of them.
I've wasted time trying to get your application to compile.
You haven't provided a makefile or full gcc line, therefore I've wasted 15mins time trying to sort includes etc.

You didn't even provide instructions for where the data had to go, therefore I've been wasting time on that and had to reverse engineer your code to find out that they need to go in a folder called data. Without that I just get

Code: Select all

ilclient_wait_for_command_complete(resize, OMX_CommandFlush, out): -1
PORT SETTINGS CHANGED in component 0xc014c0: 61
repeated.
...
Sorry, you've had almost an hour of my time with most of it wasted.
I have seen that you've made the standard blunder - OMX_FillThisBuffer does not block, so calling

Code: Select all

            OMX_BUFFERHEADERTYPE *buffer = ilclient_get_output_buffer(deco->comp_resz, deco->out_resz, 1);

            if ((omxerr = OMX_FillThisBuffer(deco->h_resz, buffer)) != OMX_ErrorNone)
            {
                fprintf(stderr, "OMX_FillThisBuffer(resize, out): %X\n", omxerr);
            }
<snip>
                {
                    dst.width = portdef.format.image.nFrameWidth;
                    dst.height = portdef.format.image.nFrameHeight;
                    dst.stride = portdef.format.image.nStride;
                    dst.format = MAKE_PIXEL_FORMAT(4, 8, 0, 0);
                    dst.pixels = std::vector<uint8_t>(buffer->pBuffer, buffer->pBuffer + dst.stride * dst.height); <<<<<<<<<<<
                    result = 0;
                }
                else
                {
                    fprintf(stderr, "OMX_GetParameter(resize, OMX_IndexParamPortDefinition, out): %X\n", omxerr);
                }
            }
That means you are referencing a buffer that you no longer "own". nFilledLen and nFlags will probably have immediately been zeroed, and the contents of the buffer may have changed.
If you call FillThisBuffer you MUST wait for the FillBufferDone callback, or the equivalent via ilclient (ie being able to retrieve the buffer via ilclient_get_output_buffer). Once you have processed the output buffer, then call OMX_FillThisBuffer with it.

I don't see any obvious memory leaks. I'll leave it running overnight with the debugger to see if that throws anything up.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Tue Nov 06, 2018 12:41 pm

6by9 wrote:
Tue Nov 06, 2018 11:44 am
marranxo wrote:
Tue Nov 06, 2018 10:05 am
6by9 wrote:
Wed Oct 31, 2018 9:55 pm
Give me everything I need to run your test and I can take a look. Seeing as there is no main() in that code I can't run it without making assumptions about how you are setting things up.
It sounds like a memory leak, but the components have had a fairly good hammering over the years to fix those. Hopefully any issues would be noticeable after a few iterations, and you're just seeing the final stages where resources are totally drained.
I don't want to hurry up but did you find anything? I'm running the test app in 3 raspberies (2x rpi 3B+, 1x rpi 2B) and the test got stuck at different times in all of them.
I've wasted time trying to get your application to compile.
You haven't provided a makefile or full gcc line, therefore I've wasted 15mins time trying to sort includes etc.

You didn't even provide instructions for where the data had to go, therefore I've been wasting time on that and had to reverse engineer your code to find out that they need to go in a folder called data. Without that I just get

Code: Select all

ilclient_wait_for_command_complete(resize, OMX_CommandFlush, out): -1
PORT SETTINGS CHANGED in component 0xc014c0: 61
repeated.
...
Sorry, you've had almost an hour of my time with most of it wasted.
I have seen that you've made the standard blunder - OMX_FillThisBuffer does not block, so calling

Code: Select all

            OMX_BUFFERHEADERTYPE *buffer = ilclient_get_output_buffer(deco->comp_resz, deco->out_resz, 1);

            if ((omxerr = OMX_FillThisBuffer(deco->h_resz, buffer)) != OMX_ErrorNone)
            {
                fprintf(stderr, "OMX_FillThisBuffer(resize, out): %X\n", omxerr);
            }
<snip>
                {
                    dst.width = portdef.format.image.nFrameWidth;
                    dst.height = portdef.format.image.nFrameHeight;
                    dst.stride = portdef.format.image.nStride;
                    dst.format = MAKE_PIXEL_FORMAT(4, 8, 0, 0);
                    dst.pixels = std::vector<uint8_t>(buffer->pBuffer, buffer->pBuffer + dst.stride * dst.height); <<<<<<<<<<<
                    result = 0;
                }
                else
                {
                    fprintf(stderr, "OMX_GetParameter(resize, OMX_IndexParamPortDefinition, out): %X\n", omxerr);
                }
            }
That means you are referencing a buffer that you no longer "own". nFilledLen and nFlags will probably have immediately been zeroed, and the contents of the buffer may have changed.
If you call FillThisBuffer you MUST wait for the FillBufferDone callback, or the equivalent via ilclient (ie being able to retrieve the buffer via ilclient_get_output_buffer). Once you have processed the output buffer, then call OMX_FillThisBuffer with it.

I don't see any obvious memory leaks. I'll leave it running overnight with the debugger to see if that throws anything up.
Sorry for wasting your time. I should have included the full gcc line but that didn't came to my mind, I'm really sorry.

I know that OMX_FillThisBuffer doesn't block that's why I check for errors and wait until EOS before reading the supplied buffer. So I assumed the workflow was:
1. Request an empty buffer to the component (ilclient_get_output_buffer)
2. Tell the component to write on that buffer (OMX_FillThisBuffer)
3. Wait for the component to process data and write to the provided buffer if no errors where found (ilclient_wait_for_event(deco->comp_resz, OMX_EventBufferFlag, deco->out_resz, 0, OMX_BUFFERFLAG_EOS, 0, ILCLIENT_EVENT_ERROR | ILCLIENT_CONFIG_CHANGED | ILCLIENT_BUFFER_FLAG_EOS, 500))
4. Now we got the reliable data in the buffer, copy it to our memory and work with it. (dst.pixels = std::vector<uint8_t>(buffer->pBuffer, buffer->pBuffer + dst.stride * dst.height))

But from your response I understand that OMX_FillThisBuffer must be called after processing the buffer to recicle it. Is this right and was I using wrong logic?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5808
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Need advice with tuneled JPEG decoder + resizer

Tue Nov 06, 2018 1:46 pm

marranxo wrote:
Tue Nov 06, 2018 12:41 pm
6by9 wrote:
Tue Nov 06, 2018 11:44 am
That means you are referencing a buffer that you no longer "own". nFilledLen and nFlags will probably have immediately been zeroed, and the contents of the buffer may have changed.
If you call FillThisBuffer you MUST wait for the FillBufferDone callback, or the equivalent via ilclient (ie being able to retrieve the buffer via ilclient_get_output_buffer). Once you have processed the output buffer, then call OMX_FillThisBuffer with it.

I don't see any obvious memory leaks. I'll leave it running overnight with the debugger to see if that throws anything up.
Sorry for wasting your time. I should have included the full gcc line but that didn't came to my mind, I'm really sorry.

I know that OMX_FillThisBuffer doesn't block that's why I check for errors and wait until EOS before reading the supplied buffer. So I assumed the workflow was:
1. Request an empty buffer to the component (ilclient_get_output_buffer)
2. Tell the component to write on that buffer (OMX_FillThisBuffer)
3. Wait for the component to process data and write to the provided buffer if no errors where found (ilclient_wait_for_event(deco->comp_resz, OMX_EventBufferFlag, deco->out_resz, 0, OMX_BUFFERFLAG_EOS, 0, ILCLIENT_EVENT_ERROR | ILCLIENT_CONFIG_CHANGED | ILCLIENT_BUFFER_FLAG_EOS, 500))
4. Now we got the reliable data in the buffer, copy it to our memory and work with it. (dst.pixels = std::vector<uint8_t>(buffer->pBuffer, buffer->pBuffer + dst.stride * dst.height))

But from your response I understand that OMX_FillThisBuffer must be called after processing the buffer to recicle it. Is this right and was I using wrong logic?
Certainly at the lowest level EOS is transferred via a buffer with nFlags including OMX_BUFFERFLAG_EOS. The core picks up on that flag and signals an event on it, but should also deliver that buffer. Some components send the EOS on the last filled buffer, others as an empty buffer with the flag set.

Should you have more than one buffer on the port then you are going to get very odd behaviour.

Steps 1 & 2 are correct. As you get the buffers delivered to your app there is little point in point in waiting on the EOS event, just look at the nFlags field of the buffers that are returned. The EOS callback is more for audio_render and video_render where they consume data and need to notify the client at some point that it has finished the stream.
Steps 3 & 4 are risky.

I've had a quick hack around with your code to handle it as I'd expect it - see https://github.com/6by9/omx_imgdecode_resize
I hate IL so it's not the cleanest, but it demonstrates the point. As noted, you really want a semaphore or completion that is waited on (instead of a 10ms sleep) which is set in at least errorcb, fillbuffercb and eoscb.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Tue Nov 06, 2018 3:20 pm

6by9 wrote:
Tue Nov 06, 2018 1:46 pm
Certainly at the lowest level EOS is transferred via a buffer with nFlags including OMX_BUFFERFLAG_EOS. The core picks up on that flag and signals an event on it, but should also deliver that buffer. Some components send the EOS on the last filled buffer, others as an empty buffer with the flag set.

Should you have more than one buffer on the port then you are going to get very odd behaviour.

Steps 1 & 2 are correct. As you get the buffers delivered to your app there is little point in point in waiting on the EOS event, just look at the nFlags field of the buffers that are returned. The EOS callback is more for audio_render and video_render where they consume data and need to notify the client at some point that it has finished the stream.
Steps 3 & 4 are risky.

I've had a quick hack around with your code to handle it as I'd expect it - see https://github.com/6by9/omx_imgdecode_resize
I hate IL so it's not the cleanest, but it demonstrates the point. As noted, you really want a semaphore or completion that is waited on (instead of a 10ms sleep) which is set in at least errorcb, fillbuffercb and eoscb.
Thank you for your patience. I'm running the test with your modifications to see how it works in the long run.

When you talk about a sempahore is needed I asume that its initial value will be the number of output buffers available in the resizer, but as long as there's state information involved like errors, isn't a conditional variable and a status flag a more accurate approach?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5808
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Need advice with tuneled JPEG decoder + resizer

Tue Nov 06, 2018 4:26 pm

marranxo wrote:
Tue Nov 06, 2018 3:20 pm
When you talk about a sempahore is needed I asume that its initial value will be the number of output buffers available in the resizer, but as long as there's state information involved like errors, isn't a conditional variable and a status flag a more accurate approach?
TBH I'm not fussed over exactly which construct is used.
You want to sleep in the while loop until something happens that needs acting on, and the callbacks want to trigger the thread to wake up whenever any of them occurs.

Two things:
1) It wants to be

Code: Select all

while(1) {
  while((buffer = ilclient_get_output_buffer(deco->comp_resz, deco->out_resz, 0)) != NULL)
  {
     //process buffers
  }
so that all buffers get handled and passed to the component in the first pass.
I've added a commit that does that. (I've also checked that it works should you have more than one buffer on the resizer output port).

2) You don't want a counting semaphore as it'll be triggered by more than just the the buffer callback so the counting will go wrong. Fixing 1 avoids the issue I think you were thinking of.
We want to wake up on event, process everything that is available, and then sleep again.
VCOS has vcos_event_flags_[create | set | get | delete ] which offer the correct construct of waiting on something to change, and atomically clear the mask of what has happened as well as waking up the thread. The Linux kernel completion will do the same sort of thing without the bitmask of events (which aren't needed in this case). I don't think a condition variable and status flag will, but they aren't constructs I tend to use.
I've pushed a change with that too.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Tue Nov 06, 2018 4:48 pm

6by9 wrote:
Tue Nov 06, 2018 4:26 pm
TBH I'm not fussed over exactly which construct is used.
You want to sleep in the while loop until something happens that needs acting on, and the callbacks want to trigger the thread to wake up whenever any of them occurs.

Two things:
1) It wants to be

Code: Select all

while(1) {
  while((buffer = ilclient_get_output_buffer(deco->comp_resz, deco->out_resz, 0)) != NULL)
  {
     //process buffers
  }
so that all buffers get handled and passed to the component in the first pass.
I've added a commit that does that. (I've also checked that it works should you have more than one buffer on the resizer output port).

2) You don't want a counting semaphore as it'll be triggered by more than just the the buffer callback so the counting will go wrong. Fixing 1 avoids the issue I think you were thinking of.
We want to wake up on event, process everything that is available, and then sleep again.
VCOS has vcos_event_flags_[create | set | get | delete ] which offer the correct construct of waiting on something to change, and atomically clear the mask of what has happened as well as waking up the thread. The Linux kernel completion will do the same sort of thing without the bitmask of events (which aren't needed in this case). I don't think a condition variable and status flag will, but they aren't constructs I tend to use.
I've pushed a change with that too.
That was exactly my question. I'm debuging the changes right now. Thank you a lot for your support, 10/10.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5808
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Need advice with tuneled JPEG decoder + resizer

Wed Nov 07, 2018 12:56 pm

It did stall for me overnight. The GPU was waiting for buffers to be allocated before it could complete the port enable call.

Having looked at the code you appear to set up the tunnel in odd places. Why create and enable the tunnel before you've had the OMX_EventPortSettingsChanged from the decoder output?
There's also no reason to wait for a OMX_EventPortSettingsChanged on the resizer output port as you explicitly set that resolution.
IL is always a touch fussy about when/why/how things occur, so I've hacked away further at your code to use ilclient_setup_tunnel instead. That was the "approved" way of driving IL, so always safest.

I'm leaving it running over lunch to see what happens.

Sorry, really daft question. What is resize doing for you? I can't see a resize operation occuring, and image_decode natively supports OMX_COLOR_Format32bitARGB8888, so you may as well drop the resize totally.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Wed Nov 07, 2018 2:59 pm

6by9 wrote:
Wed Nov 07, 2018 12:56 pm
It did stall for me overnight. The GPU was waiting for buffers to be allocated before it could complete the port enable call.

Having looked at the code you appear to set up the tunnel in odd places. Why create and enable the tunnel before you've had the OMX_EventPortSettingsChanged from the decoder output?
There's also no reason to wait for a OMX_EventPortSettingsChanged on the resizer output port as you explicitly set that resolution.
IL is always a touch fussy about when/why/how things occur, so I've hacked away further at your code to use ilclient_setup_tunnel instead. That was the "approved" way of driving IL, so always safest.

I'm leaving it running over lunch to see what happens.

Sorry, really daft question. What is resize doing for you? I can't see a resize operation occuring, and image_decode natively supports OMX_COLOR_Format32bitARGB8888, so you may as well drop the resize totally.
Yes that's the problem that I'm also facing.

In my first approach I used to create and enable the tunnel after receiving OMX_EventPortSettingsChanged in the decoder output but as long I had those strange hangouts I thought that building the tunnel before would grant more stability to the components despite of the port definitions could be unreliable at that moment.
You're right, but when I started this project I used to call apply_port_settings not only as an event triggered from OMX_EventPortSettingsChanged and that zero seconds wait event would remove the event from the ilclient event manager anyway.
Well at first I tryed to use the ilcient_setup_tunnel function but I had problems with the query to OMX_IndexParamNumAvailableStreams due to setting up the tunnel before any EventPortSettingsChanged received. So I examined the ilclient source code and addressed the issue with my limited resources. Also I understood that that query was related to media with multiple audio/video streams witch was not the case.

Well... the resizer was there to change colorspace. I started the project in rpi 2B and image_decode did not support OMX_COLOR_Format32bitARGB8888 in the past. I'm going to check this again but I did some -failed- tests some time ago with any RGB format available.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5808
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Need advice with tuneled JPEG decoder + resizer

Wed Nov 07, 2018 3:12 pm

marranxo wrote:
Wed Nov 07, 2018 2:59 pm
6by9 wrote:
Wed Nov 07, 2018 12:56 pm
It did stall for me overnight. The GPU was waiting for buffers to be allocated before it could complete the port enable call.

Having looked at the code you appear to set up the tunnel in odd places. Why create and enable the tunnel before you've had the OMX_EventPortSettingsChanged from the decoder output?
There's also no reason to wait for a OMX_EventPortSettingsChanged on the resizer output port as you explicitly set that resolution.
IL is always a touch fussy about when/why/how things occur, so I've hacked away further at your code to use ilclient_setup_tunnel instead. That was the "approved" way of driving IL, so always safest.

I'm leaving it running over lunch to see what happens.

Sorry, really daft question. What is resize doing for you? I can't see a resize operation occuring, and image_decode natively supports OMX_COLOR_Format32bitARGB8888, so you may as well drop the resize totally.
Yes that's the problem that I'm also facing.
The updated version ran for 2 hours from just before lunch. I'll kick it off again tonight.
I've pushed my hacked version - I commented out large sections instead of removing, so that could be cleaned up.
marranxo wrote:In my first approach I used to create and enable the tunnel after receiving OMX_EventPortSettingsChanged in the decoder output but as long I had those strange hangouts I thought that building the tunnel before would grant more stability to the components despite of the port definitions could be unreliable at that moment.
You're right, but when I started this project I used to call apply_port_settings not only as an event triggered from OMX_EventPortSettingsChanged and that zero seconds wait event would remove the event from the ilclient event manager anyway.
Well at first I tryed to use the ilcient_setup_tunnel function but I had problems with the query to OMX_IndexParamNumAvailableStreams due to setting up the tunnel before any EventPortSettingsChanged received. So I examined the ilclient source code and addressed the issue with my limited resources. Also I understood that that query was related to media with multiple audio/video streams witch was not the case.
JPEG can support two streams - the main image and the thumbnail.
marranxo wrote:Well... the resizer was there to change colorspace. I started the project in rpi 2B and image_decode did not support OMX_COLOR_Format32bitARGB8888 in the past. I'm going to check this again but I did some -failed- tests some time ago with any RGB format available.
Hang on, my bad. If the codec spits out RGB (eg GIF, BMP, etc) then OMX_COLOR_Format32bitARGB8888 is valid. No, the component doesn't do format conversion, and the codec always decodes directly into the provided buffer.
video_splitter also supports format conversion and won't go through a resize path at the same time.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Wed Nov 07, 2018 4:58 pm

Thank you a lot for all your time and effort. I see that you've simplified most of my code.

About your changes:
1. Is it healthy to change port settings even if PortSettingsChangedEvent was not received?
2. I used to call ilclient_change_component_state to make state transitions of tunneled components but that can hang the tunnel because that call is OMX_SendCommand + ilclient_wait_for_event/ilclient_wait_for_command_complete (witch blocks before sending the OMX_SendCommand to the next tunneled component). A state transition must first OMX_SendCommand to all components and then wait for all the state change events.
3. You're creating a tunnel each iteration (apply_port_settings) but never tearing down them until destroy call. That seems strange to me.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5808
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Need advice with tuneled JPEG decoder + resizer

Wed Nov 07, 2018 5:12 pm

marranxo wrote:
Wed Nov 07, 2018 4:58 pm
Thank you a lot for all your time and effort. I see that you've simplified most of my code.

About your changes:
1. Is it healthy to change port settings even if PortSettingsChangedEvent was not received?
Potentially unnecesary, but no harm. I did that as if you decode the same image twice then the output format on the decoder doesn't change, therefore you get no PortSettingsChangedEvent but I want the tunnel set up.
marranxo wrote:2. I used to call ilclient_change_component_state to make state transitions of tunneled components but that can hang the tunnel because that call is OMX_SendCommand + ilclient_wait_for_event/ilclient_wait_for_command_complete (witch blocks before sending the OMX_SendCommand to the next tunneled component). A state transition must first OMX_SendCommand to all components and then wait for all the state change events.
It's been a fair while since I had to write IL code (it truly is horrid - use MMAL!). Generally for decode paths you'd
  1. set up the first component input port (disable any outputs)
  2. transition it to Executing
  3. feed in the first few packets
  4. wait for the PortSettingsChangedEvent
  5. Set up the tunnel to the next component
  6. enable the tunnel
  7. transition the component to Executing.
  8. Go back to 4 for the next component
Use ilclient_state_transition to change the state of lots of components at the same time.
Use ilclient_disable_tunnel or ilclient_teardown_tunnels to clear tunnels safely.

Mixing raw OMX calls with ilclient generally means doing something dubious. The exception was setting up clock tunnels which always got done manually for some reason (never worked out why).
marranxo wrote:3. You're creating a tunnel each iteration (apply_port_settings) but never tearing down them until destroy call. That seems strange to me.
Not intentional, but it seems to be working. The tunnel is always between the same two components and ports, so I probably get away with it.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5808
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Need advice with tuneled JPEG decoder + resizer

Wed Nov 07, 2018 5:15 pm

Oops, think there's a seg fault should it try to close/reopen the codec as the first step! Error checking needed.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Thu Nov 08, 2018 8:22 am

The problem with MMAL is that there're not many examples available and I haven't found any documentation behind.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5808
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Need advice with tuneled JPEG decoder + resizer

Thu Nov 08, 2018 10:12 am

marranxo wrote:
Thu Nov 08, 2018 8:22 am
The problem with MMAL is that there're not many examples available and I haven't found any documentation behind.
There's Doxygen markup in all the headers. Someone has compiled it and hosts it at http://www.jvcref.com/files/PI/document ... index.html
(I have asked if we can get it and the IL component docs hosted on raspberrypi.org but apparently it's difficult to host anything except Github style markup).
I did start trying to create MMAL equivalents of the hello_pi apps. https://github.com/6by9/userland/commits/hello_mmal has a video_decode and image_decode example, although there may need to be some further cleanup required (the memory is going these days). There are other more pressing tasks for today, but I will see if I can do a simple extension to hello_mmal_jpeg to include video_splitter as a format converter.

My mods have still stalled in a backtrace

Code: Select all

#0  0x76f0f014 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x47364) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait ([email protected]=0x47364, abstime=0x0) at sem_waitcommon.c:115
#2  0x76f0f158 in __new_sem_wait_slow (sem=0x47364, abstime=0x0) at sem_waitcommon.c:282
#3  0x76eebc54 in ilcs_execute_function_ex () from /opt/vc/lib/libopenmaxil.so
#4  0x76eec6f8 in ilcs_execute_function () from /opt/vc/lib/libopenmaxil.so
#5  0x76eeccb8 in vcil_out_GetState () from /opt/vc/lib/libopenmaxil.so
#6  0x00019980 in ilclient_enable_tunnel (tunnel=0x7efff090) at ilclient.c:589
#7  0x0001acb4 in ilclient_setup_tunnel (tunnel=0x7efff090, portStream=0, timeout=0) at ilclient.c:1076
#8  0x000144ec in apply_port_settings (deco=0x7efff058) at main.cpp:378
#9  0x00014808 in jpeg_decode (deco=0x7efff058, src=std::vector of length 13988, capacity 13988 = {...}, dst=...) at main.cpp:441
#10 0x00015048 in main () at main.cpp:575
Sorry, I can't invest any more effort in looking at this. GetState is a simple call, so it's either a really subtle timing issue or a message getting dropped.
If you can narrow down the failure to make an easily reproducible use case then I may be able to find more time to take it further, but there is a desire to deprecate IL so significant investigation is wasted effort.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Thu Nov 08, 2018 11:14 am

6by9 wrote: There's Doxygen markup in all the headers. Someone has compiled it and hosts it at http://www.jvcref.com/files/PI/document ... index.html
(I have asked if we can get it and the IL component docs hosted on raspberrypi.org but apparently it's difficult to host anything except Github style markup).
That's some good documentation to start with. Thanks.
6by9 wrote: I did start trying to create MMAL equivalents of the hello_pi apps. https://github.com/6by9/userland/commits/hello_mmal has a video_decode and image_decode example, although there may need to be some further cleanup required (the memory is going these days). There are other more pressing tasks for today, but I will see if I can do a simple extension to hello_mmal_jpeg to include video_splitter as a format converter.
That would be great because if you say that IL is going to get deprecated it would be good for me to migrate to MMAL. I need three abstracted components: jpeg decode->rgb (image_decode+resize/splitter), h264 encoder and jpeg encoder. H264/jpeg encoders look very easy to implement with the component_wrapper but I don't have any idea about connecting the jpeg decoder to the video splitter yet (still didn't read the documentation you provided me).
6by9 wrote: Sorry, I can't invest any more effort in looking at this. GetState is a simple call, so it's either a really subtle timing issue or a message getting dropped.
If you can narrow down the failure to make an easily reproducible use case then I may be able to find more time to take it further, but there is a desire to deprecate IL so significant investigation is wasted effort.
It's fine. Following your advice I'm going to move to MMAL.

One more time thank you for your time and efford.

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Thu Nov 08, 2018 4:24 pm

I've been playing with your sample https://github.com/6by9/userland/blob/4 ... l_encode.c and it's working fine with common resolutions (320x240, 640x480, 1024x768, 1920x1080...) but it's not working with uncommon resolutions ie 80x60 (mmal_port_format_commit fails with EINVAL).

With the ilclient API I used to adjust the stride and the slice height to the next 16 multiple like this:

Code: Select all

OMX_PARAM_PORTDEFINITIONTYPE def;
memset(&def, 0, sizeof(OMX_PARAM_PORTDEFINITIONTYPE));
def.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
def.nVersion.nVersion = OMX_VERSION;
def.nPortIndex = INPUT_PORT;
def.format.image.nFrameWidth = img.width;
def.format.image.nFrameHeight = img.height;
def.format.image.eCompressionFormat = OMX_IMAGE_CodingUnused;
def.format.image.nStride = img.width * 4; //ARGB
def.format.image.nSliceHeight = (img.height % 16 == 0) ? img.height : img.height + 16 - img.height % 16;
def.format.image.eColorFormat = OMX_COLOR_Format32bitARGB8888;
but I can't see any parameter like that on MMAL_VIDEO_FORMAT_T

Code: Select all

typedef struct
{
    uint32_t  width;
    uint32_t  height;
    MMAL_RECT_T     crop;
    MMAL_RATIONAL_T frame_rate;
    MMAL_RATIONAL_T par;
    MMAL_FOURCC_T   color_space;
} MMAL_VIDEO_FORMAT_T;
Also vcdbg log msg shows

Code: Select all

1999873.854: mmal: mmal_ril_set_port_settings: ril.image_encode:in:0: failed to set port definition (4) buffers 1/1/20480
1999873.913: mmalsrv: mmal_server_do_port_info_set: ril.image_encode:in:0(RGBA): failed (handle 1, status 3)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5808
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Need advice with tuneled JPEG decoder + resizer

Thu Nov 08, 2018 4:35 pm

Code: Select all

typedef struct
{
    uint32_t  width;
    uint32_t  height;
    MMAL_RECT_T     crop;
    MMAL_RATIONAL_T frame_rate;
    MMAL_RATIONAL_T par;
    MMAL_FOURCC_T   color_space;
} MMAL_VIDEO_FORMAT_T;
width/height define the overall buffer geometry (ie nStride/bytes_per_pixel and nSliceHeight)
crop.width and crop.height define the active area of the buffer (ie nFrameWidth and nFrameHeight).

That's the reason for https://github.com/6by9/userland/blob/h ... ode.c#L117

Code: Select all

   portIn->format->es->video.width = VCOS_ALIGN_UP(WIDTH, 32);
   portIn->format->es->video.height = VCOS_ALIGN_UP(HEIGHT, 16);
   portIn->format->es->video.crop.x = 0;
   portIn->format->es->video.crop.y = 0;
   portIn->format->es->video.crop.width = WIDTH;
portIn->format->es->video.crop.height = HEIGHT;
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Thu Nov 08, 2018 4:57 pm

Well, the sample https://github.com/6by9/userland/blob/h ... ode.c#L117 just throws error if you set WIDTH to 80 and HEIGHT to 60. Something should be wrong there.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5808
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Need advice with tuneled JPEG decoder + resizer

Thu Nov 08, 2018 5:34 pm

marranxo wrote:
Thu Nov 08, 2018 4:57 pm
Well, the sample https://github.com/6by9/userland/blob/h ... ode.c#L117 just throws error if you set WIDTH to 80 and HEIGHT to 60. Something should be wrong there.
I'm testing with latest firmware plus a couple of bits and the port_format_commit succeeds at 80x60. However I'm also on the 4.19 kernel and appear to be seeing VCHIQ issues so some of the output files are corrupt. Sorry, investigating those takes priority (and also explains some of the other issues I've been having).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

marranxo
Posts: 28
Joined: Thu Dec 03, 2015 2:16 pm

Re: Need advice with tuneled JPEG decoder + resizer

Fri Nov 09, 2018 8:19 am

6by9 wrote:
Thu Nov 08, 2018 5:34 pm
I'm testing with latest firmware plus a couple of bits and the port_format_commit succeeds at 80x60. However I'm also on the 4.19 kernel and appear to be seeing VCHIQ issues so some of the output files are corrupt. Sorry, investigating those takes priority (and also explains some of the other issues I've been having).
I'm on the 4.14 kernel version. If you find some information of interest about this issue please let me know.

Return to “OpenMAX”