Question about brcmjpeg implementation in a program.

[cmisip]

I tried a test program with brcmjpeg and it is awesome. I want to implement it in zoneminder and I need a little advice on how to do this.

Zoneminder has a Monitor class object that is working on a lot of objects of class Image. There could be hundreds of Image objects being manipulated by a Monitor object. The Image object contains a uint8_t *buffer which holds RGB data. Operations on the RGB buffer include rotation, scaling, text annotation, highlighting, etc. At some point the buffer is converted to jpeg and written to disk. It made sense that since Image owns the buffer, the operations on the buffer are through Image class member functions. So if frame ( the image objects are actually RGB representations of video frame data ) is an object of Image class, Monitor would simply call frame.Annotate("Frame") and then frame.WriteJPEG().

Where could I put brcmjpeg_create? I am thinking I should not put it in Image class initialization because I would have hundreds of instances of it. However I read in brcmjpeg source that if there is an instance of the encoder already created, it will be reused. Does this mean that I can put it as part of Image class initialization and be assured that there is only ever just one instance of it despite hundreds of Image class objects in existence? Or should I try to put it in Monitor class initialization routine and make the JPEG writing function a member of the Monitor class. That way I am assured there is only one instance of brcmjpeg encoder per Monitor class.

Thanks,
Chris

[6by9]

You do really love finding all the old obscure code!
brcmjpeg was there as a simple library for use by a Broadcom customer who wanted hardware JPEG acceleration.

I think the first simple question is are you aiming to run on a 0/0W/A/B, because if not then libjpeg is probably going to be almost as fast. That would be particularly true if you are not using vcsm to allocate the buffers and pass it in via [input|output]_handle.
(Oh look, more use of mmal_component_wrapper. Hmm)

brcmjpeg is using internal static variables to keep track of a single encoder and decoder. Each process will create a maximum one of each component.
The codec block itself only supports a single client at a time (it can't save out all the internal state), and therefore everything is serialised by both the firmware and brcmjpeg (brcmjpeg_process calling LOCK_PROCESS). I'm not sure how that works when in slice mode as you could end up with a deadlock situation there.

As to where to put the initialisation, sorry can't advise there. Somewhere with a sensible scope would make sense as if the refcount on brcmjpeg hits zero then it will destroy the components.

[cmisip]

You do really love finding all the old obscure code!

I thought it was something you guys had around the corner yet to publish. So that's why I had to manually copy over the library and the include file.

I think the first simple question is are you aiming to run on a 0/0W/A/B

I will be running it on a Pi3B or Pi3B+. Will it be faster than libjpeg-turbo? If brcmjpeg is just as fast as libjepg-turbo, will creating the jpeg encoder the usual way be faster than libjpeg-turbo?

That would be particularly true if you are not using vcsm to allocate the buffers and pass it in via [input|output]_handle.

I think you are talking about zero copy. If I understand correctly, the GPU buffers are essentially also the ARM side buffers so no copying between GPU and ARM when passing data across the ports. I think example_basic_2.c demonstrates this. I tried it once but got a weird vc error after running the program for a while. However, I was also using mmal_buffer_header_mem_lock/unlock which I probably should't be doing when using vcsm to allocate buffers. mmal_buffer_header_mem_lock/unlock worked fine in the older (non zero copy) version of the code.

The codec block itself only supports a single client at a time (it can't save out all the internal state), and therefore everything is serialised by both the firmware and brcmjpeg (brcmjpeg_process calling LOCK_PROCESS)

So If I have 4 processes running concurrently each with its own instance of brcmjpeg, they will have to take turns as only one client can be serviced at a time. Is this true also for the other components initialized in the usual fashion (no wrapper) ? Or is this just a limitation of brcmjpeg's implementation ? I remember being able to run 4 concurrent hardware h264 encode sessions ( when I had 4 cameras to play with last year ) and I thought they were running parallel pipelines.

Thanks,
Chris

[6by9]

You do really love finding all the old obscure code!

I thought it was something you guys had around the corner yet to publish. So that's why I had to manually copy over the library and the include file.

I think the first simple question is are you aiming to run on a 0/0W/A/B

I will be running it on a Pi3B or Pi3B+. Will it be faster than libjpeg-turbo? If brcmjpeg is just as fast as libjepg-turbo, will creating the jpeg encoder the usual way be faster than libjpeg-turbo?

I can't say for definite, but I'd suspect that it will be close. On similar ARM cores we had a threshold for where we used the hardware acceleration via the VPU, and where it was faster to just use libjpeg-turbo, but I can't remember the exact level.

That would be particularly true if you are not using vcsm to allocate the buffers and pass it in via [input|output]_handle.

I think you are talking about zero copy. If I understand correctly, the GPU buffers are essentially also the ARM side buffers so no copying between GPU and ARM when passing data across the ports. I think example_basic_2.c demonstrates this. I tried it once but got a weird vc error after running the program for a while. However, I was also using mmal_buffer_header_mem_lock/unlock which I probably should't be doing when using vcsm to allocate buffers. mmal_buffer_header_mem_lock/unlock worked fine in the older (non zero copy) version of the code.

You shouldn't need to manually mem_lock and unlock the buffers - the MMAL core should do that for you if set up correctly. If you check the implementation then it should be a nop.
It is critical to lock and unlock buffers on the VPU, hence why the calls are in the API.

The codec block itself only supports a single client at a time (it can't save out all the internal state), and therefore everything is serialised by both the firmware and brcmjpeg (brcmjpeg_process calling LOCK_PROCESS)

So If I have 4 processes running concurrently each with its own instance of brcmjpeg, they will have to take turns as only one client can be serviced at a time. Is this true also for the other components initialized in the usual fashion (no wrapper) ? Or is this just a limitation of brcmjpeg's implementation ? I remember being able to run 4 concurrent hardware h264 encode sessions ( when I had 4 cameras to play with last year ) and I thought they were running parallel pipelines.

It's a limitation of specifically the JPEG hardware block on the chip.
If you have 4 processes and therefore 4 image_encode components, submitting a job to one will block the others from starting a job until the first completes. The buffers can be being transferred to/from the VPU, just the actual JPEG encode/decode can not run in parallel. If you try decoding a JPEG and feed in bits of the source file at a time, starting another image_encode or decode will show you what happens (it won't process any input data until the first one has completed).

H264 is split into multiple smaller stages, and can save state between the stages, therefore 4 H264 video encodes will run substantially in parallel. Try the same thing with MJPEG and again it will largely serialise the encodes.

[cmisip]

Thanks for this info. Zoneminder uses multiple Monitor instances and if the access to the hardware JPEG encoder is serialized, then it will be slower than doing parallel software jpeg encodes with libjpeg. So this is not an avenue for optimization.

Are there other components that are limited similarly like the JPEG encoder where we might not expect a performance boost by running processes in parallel? I am using vc.ril.isp to replace swscale when converting I420 to RGB. Each monitor instance has a H264 decoder (via h264_mmal in libavcodec), H264 encoder component and a vc.ril.isp component. Seems to have worked fine when I had 4 cameras. vc.ril.isp and h264 decoder probably worked substantially in parallel as there was no issue running 4 of those at the same time.

You shouldn't need to manually mem_lock and unlock the buffers - the MMAL core should do that for you if set up correctly. If you check the implementation then it should be a nop. It is critical to lock and unlock buffers on the VPU, hence why the calls are in the API.

I don't understand what you mean by this. I assumed it was standard to mmal_buffer_header_mem_lock/unlock whenever I memcpy to and from a buffer with any component I am using.

BTW, I see the green band artifact on the lower portion of the jpeg image with brcmjpeg as well when the height of the video frame is not a multiple of 16. I think there is an issue with vcos_align_up when the input data is in I420 for the JPEG encoder. This problem goes away when there is an intermediate conversion to RGB.

Thanks,
Chris

[6by9]

Are there other components that are limited similarly like the JPEG encoder where we might not expect a performance boost by running processes in parallel? I am using vc.ril.isp to replace swscale when converting I420 to RGB. Each monitor instance has a H264 decoder (via h264_mmal in libavcodec), H264 encoder component and a vc.ril.isp component. Seems to have worked fine when I had 4 cameras. vc.ril.isp and h264 decoder probably worked substantially in parallel as there was no issue running 4 of those at the same time.

Most components will have some level of contention for hardware or VPU time - finite resources and all that.

The ISP hardware can context switch between batches of tiles, but with vc.ril.isp is given the entire frame in one hit. Seeing as it does chomp through it at >200MPix/s that generally appears to be fully parallel.
H264 claims the various hardware pipeline sections (eg CABAC, motion estimation, reconstruction, etc) as the operation requires it.
vc.ril.resize is purely on the VPU. IIRC The VRF (part of the SIMD pipe) is acquired whilst processing a stripe of 16 lines, but released between each stripe so scheduling could come into play.
The JPEG block (whether used for MJPEG or JPEG) has to complete the operation on the full frame before context switching. I can't remember the headline number for processing, but I think it was around 120MPix/s.

You shouldn't need to manually mem_lock and unlock the buffers - the MMAL core should do that for you if set up correctly. If you check the implementation then it should be a nop. It is critical to lock and unlock buffers on the VPU, hence why the calls are in the API.

I don't understand what you mean by this. I assumed it was standard to mmal_buffer_header_mem_lock/unlock whenever I memcpy to and from a buffer with any component I am using.

Have a look at the implementation. https://github.com/raspberrypi/userland ... fer.c#L157

Code: Select all

** Lock the data buffer contained in the buffer header */
MMAL_STATUS_T mmal_buffer_header_mem_lock(MMAL_BUFFER_HEADER_T *header)
{
#ifdef __VIDEOCORE__
   uint8_t *data = mem_lock((MEM_HANDLE_T)header->data);
   if (!data)
      return MMAL_EINVAL;
   header->priv->payload_handle = (void *)header->data;
   header->data = data;
#else
   MMAL_PARAM_UNUSED(header);
#endif

   return MMAL_SUCCESS;
}

__VIDEOCORE__ will be undefined when building the ARM side library, therefore it is a nop.

If doing zero copy, then the magic happens with calls to mmal_vc_shm at https://github.com/raspberrypi/userland ... api.c#L511 and https://github.com/raspberrypi/userland ... api.c#L637 to convert buffer->data between the required MEM_HANDLE_T and ARM pointer.

BTW, I see the green band artifact on the lower portion of the jpeg image with brcmjpeg as well when the height of the video frame is not a multiple of 16. I think there is an issue with vcos_align_up when the input data is in I420 for the JPEG encoder. This problem goes away when there is an intermediate conversion to RGB.

What are you viewing the resulting JPEG with? What size does it report? It sounds like it is the renderer which is incorrect, although unused lines/columns in a macroblock should be filled with a repeat of the last active line/column normally. (Green = 0 in YUV space, so it sounds like someone has done a memset).
I need an example to work from as all the use cases I have work fine.

[cmisip]

I am actually using imagemagick display to display the jpeg file. The test program I put together saves the jpeg output to a file. Connecting the decoder output to jpeg encoder directly results in a green band on the bottom. If I connect vc.ril.isp between the decoder and jpeg encoder to do an intermediate conversion to RGBA, the green band is replaced by a black band. This is not as distracting as a green band. The source video is an rtsp stream at 640x360 from an SV3C camera. Here is the test code I use, just in case you may spot errors. I don't know why I get a run of bad jpegs (gray pixels only) at the start. Its ok after about 120 frames.

Thanks,
Chris

Code: Select all

#include "interface/mmal/client/brcmjpeg/brcmjpeg.h"


#include "bcm_host.h"
#include "interface/mmal/mmal.h"
#include "interface/mmal/util/mmal_default_components.h"
#include "interface/mmal/util/mmal_util_params.h"
#include "interface/vcos/vcos.h"
#include "interface/mmal/vc/mmal_vc_api.h"
#include "interface/mmal/util/mmal_connection.h"

#include <stdio.h>
#include <fcntl.h>
#include <mutex>

extern "C" {
#include <libavcodec/avcodec.h>
#include <libavutil/motion_vector.h>
#include <libavutil/imgutils.h>
#include <libavformat/avformat.h>
#include "libswscale/swscale.h"
#include <libavutil/frame.h>
#include "libavformat/avformat.h"
#include <libswscale/swscale.h>
}




//FUNCTIONS
#define CHECK_STATUS(status, msg) if (status != MMAL_SUCCESS) { fprintf(stderr, msg"\n"); }

struct CONTEXT_T {
   MMAL_QUEUE_T *queue=NULL;
   MMAL_STATUS_T status;
} context;

MMAL_STATUS_T status = MMAL_EINVAL;
MMAL_COMPONENT_T *decoder, *rgbcoder = NULL;
MMAL_POOL_T *pool_in = NULL, *pool_out = NULL;
MMAL_PORT_T *input_port=NULL;
MMAL_PORT_T *output_port=NULL;
MMAL_CONNECTION_T *connection;

BRCMJPEG_T *jcoder;
BRCMJPEG_STATUS_T jstatus;
BRCMJPEG_REQUEST_T jpeg_request;


//hold the jpeg buffer and size
struct JPEG_T {
   //uint32_t data_size=0;
   long unsigned int data_size=0;
   uint32_t buffer_length=0;
   uint8_t *data=NULL;
   
   JPEG_T(uint32_t isize){
	  data=(uint8_t*)av_mallocz(isize); 
	  buffer_length=isize; 
	}   
   ~JPEG_T(){
	  if (data)
	    av_free(data);  
	}   
   
};	



AVDictionary *opts = NULL;
int ret;
AVStream *st;
const char *src_filename = NULL;
AVFormatContext *fmt_ctx = NULL;
AVCodecContext *video_dec_ctx = NULL;
AVStream *video_stream = NULL;
AVCodec *dec = NULL;
AVCodecContext *dec_ctx = NULL;
int video_stream_idx = -1;
AVFrame *frame = NULL;
AVPacket pkt;



MMAL_STATUS_T connect_ports(MMAL_PORT_T *output_port, MMAL_PORT_T *input_port, MMAL_CONNECTION_T **connection)
{
   MMAL_STATUS_T status;

   status =  mmal_connection_create(connection, output_port, input_port, MMAL_CONNECTION_FLAG_TUNNELLING | MMAL_CONNECTION_FLAG_ALLOCATION_ON_OUTPUT);

   if (status == MMAL_SUCCESS)
   {
      status =  mmal_connection_enable(*connection);
      if (status != MMAL_SUCCESS)
         mmal_connection_destroy(*connection);
   }

   return status;
}


/** Callback from the control port.
 * Component is sending us an event. */
void control_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
   struct CONTEXT_T *ctx = (struct CONTEXT_T *)port->userdata;

   switch (buffer->cmd)
   {
   case MMAL_EVENT_EOS:
      /* Only sink component generate EOS events */
      fprintf(stderr,"EOS\n");
      break;
   case MMAL_EVENT_ERROR:
      /* Something went wrong. Signal this to the application */
      ctx->status = *(MMAL_STATUS_T *)buffer->data;
      fprintf(stderr,"ERROR: %d\n", ctx->status);
      break;
   default:
      break;
   }

   /* Done with the event, recycle it */
   mmal_buffer_header_release(buffer);

}



void input_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
   //struct CONTEXT_T *ctx = (struct CONTEXT_T *)port->userdata;
   /* The encoder is done with the data, just recycle the buffer header into its pool */
   mmal_buffer_header_release(buffer);

}

/** Callback from the output port.
 * Buffer has been produced by the port and is available for processing. */
void output_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer )
{
   struct CONTEXT_T *ctx = (struct CONTEXT_T *)port->userdata;

   /* Queue the decoded video frame */
   mmal_queue_put(ctx->queue, buffer);

}

uint8_t ffmpeg_camera_initialize(){
	
	avformat_network_init();

    av_register_all();

    ret = avformat_open_input(&fmt_ctx, src_filename, NULL, NULL) ;
    if (ret <0 ) {
        fprintf(stderr, "Could not open source %s\n", src_filename);
        return ret;
    }


    ret = avformat_find_stream_info(fmt_ctx, NULL);
    if (ret < 0) {
        fprintf(stderr, "Could not find stream information\n");
        return ret;
    }

	
	ret = av_find_best_stream(fmt_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, &dec, 0);  //1 this version creates dec
    
    if (ret < 0) {
        fprintf(stderr, "Could not find %s stream in input file '%s'\n",
                av_get_media_type_string(AVMEDIA_TYPE_VIDEO), src_filename);
        return ret;
    } else {
        int stream_idx = ret;
        st = fmt_ctx->streams[stream_idx];
        
        dec_ctx = avcodec_alloc_context3(NULL);
        if (!dec_ctx) {
            fprintf(stderr, "Failed to allocate codec\n");
            return AVERROR(EINVAL);
        }

        ret = avcodec_parameters_to_context(dec_ctx, st->codecpar);
        if (ret < 0) {
            fprintf(stderr, "Failed to copy codec parameters to codec context\n");
            return ret;
        }
        
        video_stream_idx = stream_idx;
        video_stream = fmt_ctx->streams[video_stream_idx];
        video_dec_ctx = dec_ctx;
        
	    av_dump_format(fmt_ctx, 0, src_filename, 0);

        if (!video_stream) {
           fprintf(stderr, "Could not find video stream in the input, aborting\n");
           ret = 1;
        }
        
} 
        return ret;
};



uint8_t run(AVPacket *packet, JPEG_T *outjpeg)
{   
	MMAL_BUFFER_HEADER_T *buffer;
	if (input_port) { 
	if ((buffer = mmal_queue_get(pool_in->queue)) != NULL)
      {  
         
         memcpy(buffer->data,packet->data,packet->size);
         buffer->length=packet->size;
         
         uint8_t nal_type=buffer->data[4] & 0x1f;
         fprintf(stderr,"NAL TYPE --------------------> %d\n",nal_type);
         
         //PRINT THE BUFFER CONTENTS
         for (uint32_t i = 0; ((i < buffer->length) && (i < 20)); ++i) {
                    fprintf(stderr, "\\%02x", (unsigned char)buffer->data[i]);
         }           
         fprintf(stderr,"\n");
         
         buffer->pts = buffer->dts = MMAL_TIME_UNKNOWN;
         buffer->flags=packet->flags;
        
            
         
         fprintf(stderr, "%s sending packet >>>>> %i bytes\n", decoder->input[0]->name, (int)buffer->length);
         status = mmal_port_send_buffer(input_port, buffer);
         CHECK_STATUS(status, "failed to send buffer to input port");
      }

      }
      
      if (output_port) {
      while ((buffer = mmal_queue_get(context.queue)) != NULL)
      {
         
         
         fprintf(stderr, "%s receiving %d bytes <<<<< frame\n", output_port->name, buffer->length);
         
         if (!buffer->cmd) {
         
          //SEND TO HARDWARE JPEG ENCODER
          if (outjpeg) {
            jpeg_request.input_size = buffer->length;
            jpeg_request.input = buffer->data;
            jpeg_request.output=outjpeg->data;
            jpeg_request.output_alloc_size = outjpeg->buffer_length;
            fprintf(stderr,"OUT SIZE %d", outjpeg->buffer_length);
            outjpeg->data_size=0;
            if (brcmjpeg_process(jcoder, &jpeg_request) > 0 ) {
              fprintf(stderr,"  Error creating JPEG\n");
  	        } else {
	  	      fprintf(stderr,"  JPEG created %d\n",jpeg_request.output_size);
	          outjpeg->data_size=jpeg_request.output_size;
            }
	      }
	     }
	     
         
         mmal_buffer_header_release(buffer);
      }

     
      while ((buffer = mmal_queue_get(pool_out->queue)) != NULL)
      {
         status = mmal_port_send_buffer(output_port, buffer);
         CHECK_STATUS(status, "failed to send buffer to output port");
      }
      
  }
      
      
     return status;    
}	



int main(int argc, char **argv) {
   if (argc != 2) {
        fprintf(stderr, "Usage: %s rtsp://<user>:<pass>@url\n", argv[0]);
        return 1;
   }
    
   src_filename=argv[1];
   
   bcm_host_init();
     
   //Initialize ffmpeg
   ffmpeg_camera_initialize();
    
   uint16_t iwidth=video_dec_ctx->width;
   uint16_t iheight=video_dec_ctx->height; 
   
   //DECODER SETUP
   
   status = mmal_component_create(MMAL_COMPONENT_DEFAULT_VIDEO_DECODER, &decoder);
   CHECK_STATUS(status, "failed to create decoder");
   
   // CONTROL PORT SETTINGS
   decoder->control->userdata = (MMAL_PORT_USERDATA_T *)&context;
   status = mmal_port_enable(decoder->control, control_callback);
   CHECK_STATUS(status, "failed to enable decoder control port");
   
   
   //INPUT PORT SETTINGS
   MMAL_ES_FORMAT_T *format_in = decoder->input[0]->format;
   format_in->type = MMAL_ES_TYPE_VIDEO;
   format_in->encoding = MMAL_ENCODING_H264;
   
   format_in->es->video.width = VCOS_ALIGN_UP(iwidth, 32);
   format_in->es->video.height = VCOS_ALIGN_UP(iheight,16);
   format_in->es->video.frame_rate.num = 30;
   format_in->es->video.frame_rate.den = 1;
   format_in->es->video.par.num = 1;
   format_in->es->video.par.den = 1;
   format_in->es->video.crop.width = iwidth;
   format_in->es->video.crop.height = iheight;
   
   //ALLOCATE Extradata, copying from avcodec context
   status = mmal_format_extradata_alloc(format_in, video_dec_ctx->extradata_size);
   fprintf(stderr,"FORMAT pack size %d\n", video_dec_ctx->extradata_size);
   format_in->extradata_size = video_dec_ctx->extradata_size;
   if (format_in->extradata_size)
      memcpy(format_in->extradata, video_dec_ctx->extradata, video_dec_ctx->extradata_size);
   
   //PRINT the contents of extradata
   fprintf(stderr,"Extradata\n");
   for (size_t i = 0; i != format_in->extradata_size+1; ++i) {
                    fprintf(stderr, "\\%02x", (unsigned char)format_in->extradata[i]);
   }
   fprintf(stderr,"\n");
   
  
   /* Display the input port format */
   fprintf(stderr,"---------------------------------------------------\n");
   fprintf(stderr, "INPUT %s\n", decoder->input[0]->name);
   fprintf(stderr, " type: %i, fourcc: %4.4s\n", format_in->type, (char *)&format_in->encoding);
   fprintf(stderr, " bitrate: %i, framed: %i\n", format_in->bitrate,
           !!(format_in->flags & MMAL_ES_FORMAT_FLAG_FRAMED));
   fprintf(stderr, " extra data: %i, %p\n", format_in->extradata_size, format_in->extradata);
   fprintf(stderr, " width: %i, height: %i, (%i,%i,%i,%i)\n",
           format_in->es->video.width, format_in->es->video.height,
           format_in->es->video.crop.x, format_in->es->video.crop.y,
           format_in->es->video.crop.width, format_in->es->video.crop.height);
       
       
   MMAL_PARAMETER_BOOLEAN_T error_concealment;
   error_concealment.hdr.id = MMAL_PARAMETER_VIDEO_DECODE_ERROR_CONCEALMENT;
            error_concealment.hdr.size = sizeof(MMAL_PARAMETER_BOOLEAN_T);
            error_concealment.enable = MMAL_FALSE;
            status = mmal_port_parameter_set(decoder->input[0], &error_concealment.hdr);
            CHECK_STATUS(status, "failed to set error concealment");    
   
   
   decoder->input[0]->buffer_num = decoder->input[0]->buffer_num_recommended;
   decoder->input[0]->buffer_size = decoder->input[0]->buffer_size_recommended;
   fprintf(stderr,"DECODER INPUT BUFFER SIZE %d NUM %d\n",decoder->input[0]->buffer_size,decoder->input[0]->buffer_num_recommended);
   
   status = mmal_port_format_commit(decoder->input[0]);
   CHECK_STATUS(status, "failed to commit decoder input format");  
   
   
   //OUTPUT PORT SETTINGS
   
   MMAL_ES_FORMAT_T *format_out = decoder->output[0]->format;
   
   format_out->type = MMAL_ES_TYPE_VIDEO;
   format_out->encoding = MMAL_ENCODING_I420;
   
   format_out->es->video.width = VCOS_ALIGN_UP(iwidth, 32);
   format_out->es->video.height = VCOS_ALIGN_UP(iheight,16);
   format_out->es->video.frame_rate.num = 30;
   format_out->es->video.frame_rate.den = 1;
   format_out->es->video.par.num = 0; 
   format_out->es->video.par.den = 1;
   format_out->es->video.crop.width = iwidth;
   format_out->es->video.crop.height = iheight;
   
    /* Display the output port format */
   fprintf(stderr,"---------------------------------------------------\n");
   fprintf(stderr, "OUTPUT %s\n", decoder->output[0]->name);
   fprintf(stderr, " type: %i, fourcc: %4.4s\n", format_out->type, (char *)&format_out->encoding);
   fprintf(stderr, " bitrate: %i, framed: %i\n", format_out->bitrate,
           !!(format_out->flags & MMAL_ES_FORMAT_FLAG_FRAMED));
   fprintf(stderr, " extra data: %i, %p\n", format_out->extradata_size, format_out->extradata);
   fprintf(stderr, " width: %i, height: %i, (%i,%i,%i,%i)\n",
           format_out->es->video.width, format_out->es->video.height,
           format_out->es->video.crop.x, format_out->es->video.crop.y,
           format_out->es->video.crop.width, format_out->es->video.crop.height);
          
  
	
   decoder->output[0]->buffer_num = decoder->output[0]->buffer_num_recommended;
   decoder->output[0]->buffer_size = decoder->output[0]->buffer_size_recommended; 
   fprintf(stderr,"DECODER OUTPUT BUFFER SIZE %d NUM %d\n",decoder->output[0]->buffer_size_recommended,decoder->output[0]->buffer_num_recommended);
   
   
   status = mmal_port_format_commit(decoder->output[0]);
   CHECK_STATUS(status, "failed to commit decoder output format");   
   
   status = mmal_component_enable(decoder);
   CHECK_STATUS(status, "failed to enable decoder output format"); 
   
   //RGB coder 
   
   status = mmal_component_create("vc.ril.isp", &rgbcoder);
   CHECK_STATUS(status, "failed to create isp");
   
   // CONTROL PORT SETTINGS
   rgbcoder->control->userdata = (MMAL_PORT_USERDATA_T *)&context;
   status = mmal_port_enable(rgbcoder->control, control_callback);
   CHECK_STATUS(status, "failed to enable control port");
   
   
   //INPUT PORT SETTINGS
   format_in = rgbcoder->input[0]->format;
   format_in->type = MMAL_ES_TYPE_VIDEO;
   format_in->encoding = MMAL_ENCODING_I420;
   
   format_in->es->video.width = VCOS_ALIGN_UP(iwidth, 32);
   format_in->es->video.height = VCOS_ALIGN_UP(iheight,16);
   format_in->es->video.frame_rate.num = 30;
   format_in->es->video.frame_rate.den = 1;
   format_in->es->video.par.num = 1;
   format_in->es->video.par.den = 1;
   format_in->es->video.crop.width = iwidth;
   format_in->es->video.crop.height = iheight;
   
   
   
  
   /* Display the input port format */
   fprintf(stderr,"---------------------------------------------------\n");
   fprintf(stderr, "INPUT %s\n", rgbcoder->input[0]->name);
   fprintf(stderr, " type: %i, fourcc: %4.4s\n", format_in->type, (char *)&format_in->encoding);
   fprintf(stderr, " bitrate: %i, framed: %i\n", format_in->bitrate,
           !!(format_in->flags & MMAL_ES_FORMAT_FLAG_FRAMED));
   fprintf(stderr, " extra data: %i, %p\n", format_in->extradata_size, format_in->extradata);
   fprintf(stderr, " width: %i, height: %i, (%i,%i,%i,%i)\n",
           format_in->es->video.width, format_in->es->video.height,
           format_in->es->video.crop.x, format_in->es->video.crop.y,
           format_in->es->video.crop.width, format_in->es->video.crop.height);
       
   
   rgbcoder->input[0]->buffer_num = rgbcoder->input[0]->buffer_num_recommended;
   rgbcoder->input[0]->buffer_size = rgbcoder->input[0]->buffer_size_recommended;
   fprintf(stderr,"RGBCODER INPUT BUFFER SIZE %d NUM %d\n",rgbcoder->input[0]->buffer_size,rgbcoder->input[0]->buffer_num_recommended);
   
   status = mmal_port_format_commit(rgbcoder->input[0]);
   CHECK_STATUS(status, "failed to commit rgbcoder input format");  
   
   //OUTPUT PORT SETTINGS
   
   format_out = rgbcoder->output[0]->format;
   
   format_out->type = MMAL_ES_TYPE_VIDEO;
   format_out->encoding = MMAL_ENCODING_RGBA;
   
   format_out->es->video.width = VCOS_ALIGN_UP(iwidth, 32);
   format_out->es->video.height = VCOS_ALIGN_UP(iheight,16);
   format_out->es->video.frame_rate.num = 30;
   format_out->es->video.frame_rate.den = 1;
   format_out->es->video.par.num = 0; 
   format_out->es->video.par.den = 1;
   format_out->es->video.crop.width = iwidth;
   format_out->es->video.crop.height = iheight;
   
    /* Display the output port format */
   fprintf(stderr,"---------------------------------------------------\n");
   fprintf(stderr, "OUTPUT %s\n", rgbcoder->output[0]->name);
   fprintf(stderr, " type: %i, fourcc: %4.4s\n", format_out->type, (char *)&format_out->encoding);
   fprintf(stderr, " bitrate: %i, framed: %i\n", format_out->bitrate,
           !!(format_out->flags & MMAL_ES_FORMAT_FLAG_FRAMED));
   fprintf(stderr, " extra data: %i, %p\n", format_out->extradata_size, format_out->extradata);
   fprintf(stderr, " width: %i, height: %i, (%i,%i,%i,%i)\n",
           format_out->es->video.width, format_out->es->video.height,
           format_out->es->video.crop.x, format_out->es->video.crop.y,
           format_out->es->video.crop.width, format_out->es->video.crop.height);
          
  
   rgbcoder->output[0]->buffer_num = rgbcoder->output[0]->buffer_num_recommended;
   rgbcoder->output[0]->buffer_size = rgbcoder->output[0]->buffer_size_recommended; 
   fprintf(stderr,"RGBCODER OUTPUT BUFFER SIZE %d NUM %d\n",rgbcoder->output[0]->buffer_size_recommended,rgbcoder->output[0]->buffer_num_recommended);
   
   
   status = mmal_port_format_commit(rgbcoder->output[0]);
   CHECK_STATUS(status, "failed to commit rgbcoder output format");  
   
   status = mmal_component_enable(rgbcoder);
   CHECK_STATUS(status, "failed to enable rgbcoder output format");  
   
   //CONNECT DECODER TO RGBCODER
   connect_ports(decoder->output[0],rgbcoder->input[0],&connection);
   pool_in = mmal_pool_create(decoder->input[0]->buffer_num,
                                decoder->input[0]->buffer_size);
                                
   pool_out = mmal_pool_create(rgbcoder->output[0]->buffer_num,
                               rgbcoder->output[0]->buffer_size);
                               
   decoder->input[0]->userdata = (struct MMAL_PORT_USERDATA_T *)&context;
   if ( mmal_port_enable(decoder->input[0], input_callback) != MMAL_SUCCESS ) {
        fprintf(stderr,"failed to enable decoder input port");
   } 
   
   rgbcoder->output[0]->userdata = (struct MMAL_PORT_USERDATA_T *)&context;
   if ( mmal_port_enable(rgbcoder->output[0], output_callback) != MMAL_SUCCESS ) {
      fprintf(stderr,"failed to enable rgbcoder output port");
   }
   context.queue = mmal_queue_create();
   input_port=decoder->input[0];
   output_port=rgbcoder->output[0];


   //JPEG ENCODER
   if (brcmjpeg_create(BRCMJPEG_TYPE_ENCODER, &jcoder) > 0)
      fprintf(stderr,"Failed to create jpeg encoder");
   
      
   jpeg_request.input_size = 0;
   jpeg_request.width = format_out->es->video.width;
   jpeg_request.height = format_out->es->video.height; 
   jpeg_request.quality = 10;
   jpeg_request.pixel_format = PIXEL_FORMAT_RGBA;
   
   JPEG_T jpeg_result(format_out->es->video.width*format_out->es->video.height*24);
   fprintf(stderr,"JPEG BUFFER length %d, data_size %lu \n",jpeg_result.buffer_length,jpeg_result.data_size);
   
   
   FILE *JPEGFile;
   char JPEGFName[256];
   std::mutex m_jcoder;
   
   //PAUSE 
   
   fprintf(stderr,"Press Enter to Continue\n");
   fprintf(stderr,"Press Enter again to stop\n");
   getchar();
    
   /* Start decoding */
   fprintf(stderr, "start decoding\n");
    
   int framecount=0;
   char c;
   int n, tem;
    
    
   tem = fcntl(0, F_GETFL, 0);
   fcntl (0, F_SETFL, (tem | O_NDELAY));
    
   av_init_packet(&pkt);
    
     
   //Send the sps and pps as first packet
   pkt.data=(uint8_t*)av_mallocz(video_dec_ctx->extradata_size);
   memcpy(pkt.data,video_dec_ctx->extradata,video_dec_ctx->extradata_size);
   pkt.size=video_dec_ctx->extradata_size; 
   
   run(&pkt,NULL);
   av_packet_unref(&pkt);
   
   //Send the rest of the packets in a loop
   while ((av_read_frame(fmt_ctx, &pkt) >= 0) && (framecount <500 )) {
        if (pkt.stream_index == video_stream_idx) {
           
            run(&pkt,&jpeg_result);
            
            m_jcoder.lock();  //lock outbuffer then read
      
            if (jpeg_result.data_size) {
            sprintf(JPEGFName, "frame%d.jpg", framecount);
            JPEGFile = fopen(JPEGFName, "wb");
            fwrite(jpeg_result.data, 1, jpeg_result.data_size, JPEGFile);
            fclose(JPEGFile);
		    }
            m_jcoder.unlock();
            
            
            av_packet_unref(&pkt);
        }
        framecount++;
        fprintf(stderr, "Frame number %d\n",framecount);
        
        
        n = read(0, &c, 1);
        if (n > 0) break;
      
   }
    
    
   fcntl(0, F_SETFL, tem);
   /* End decoding */
   fprintf(stderr, "end decoding\n");
    
   avformat_network_deinit();
   
   
   if (video_dec_ctx)
      avcodec_free_context(&video_dec_ctx);
   if (fmt_ctx)
      avformat_close_input(&fmt_ctx);
   
      
   mmal_port_disable(decoder->input[0]);
   mmal_port_disable(decoder->control); 
      
   mmal_port_flush(decoder->input[0]);
   mmal_port_flush(decoder->output[0]);
   mmal_port_flush(decoder->control); 
   
   mmal_port_disable(rgbcoder->output[0]);
   mmal_port_disable(rgbcoder->control); 
      
   mmal_port_flush(rgbcoder->input[0]);
   mmal_port_flush(rgbcoder->output[0]);
   mmal_port_flush(rgbcoder->control); 
   
   
   mmal_connection_disable(connection);
   
   if (connection)
       mmal_connection_destroy(connection);
   
   
   if (decoder)
      mmal_component_destroy(decoder);
   if (rgbcoder)
      mmal_component_destroy(rgbcoder);
   if (pool_in)
      mmal_pool_destroy(pool_in);
   if (pool_out)
      mmal_pool_destroy(pool_out);
   if (context.queue)
      mmal_queue_destroy(context.queue);
    
     
}

[6by9]

0x000000 in YUV space is green. 0x000000 in RGB space is black. Something is looking at memory that has been memset to 0.

I'm not going to try and run that program as it has dependencies on avcodec and others.
It's only from reading the code that I see you are connecting video_decode to image_encode - I'd assumed you were using image_decode. The devil is always in the detail.

The issue looks simple:

Code: Select all

   jpeg_request.width = format_out->es->video.width;
   jpeg_request.height = format_out->es->video.height; 

Wrong. Those are the padded width/heights, which will always be a multiple of the macroblock size, and actually a multiple of 32 horizontally and 16 vertically due to MMAL. You're telling the encoder that the image is 640x368, therefore it encodes a file of that size.

Code: Select all

   jpeg_request.width = format_out->es->video.crop.width;
   jpeg_request.height = format_out->es->video.crop.height; 

should give you the correct results.

Having said that, I'm not sure lines 335-338 in brcmjpeg.c will give the correct results, and would be tempted to make it manually do the align on port_in->format->es->video.[width|height] before commiting the format. mmal_port_format_commit should align width/height up automatically, but components are now allowing differing alignments, therefore it's generally better to manually specify it to avoid issues.

[cmisip]

Thanks again 6by9. That fixed the issue. The lesson here then is to just copy from the output buffer what you need. Don't copy the entire output buffer as it padded in the case when the resolutions were adjusted upward to multiples of 32 or 16. I'm doing this with the rgb output of vc.ril.isp ( just reading width*height*colordepth, instead of reading the entire buffer->length ) which is eventually encoded to jpeg by libjpeg-turbo and I don't see the green artifact anymore.

Do you know why the output of brcmjpeg at least for the first 120 frames is just a grey jpeg. After 120 frames, I get normal jpeg output. Might it be necessary to "warm up" the image encoder by running 120 frames through it before assuming valid useable jpeg data can be obtained? I have to yet test if this will happen with the Jpeg encoder that is setup up using the normal procedure. The other components don't have this latency before issuing valid output. I am still thinking of using the jpeg encoder in a limited fashion, hoping the performance degradation with multiple simultanous clients will not be too big of an issue if the jpeg encoder is only ever used when motion is detected which hopefully is not at the same time with multiple cameras.

Thanks,
Chris

[6by9]

There is absolutely no reason for any startup latency like that - something sounds wrong in either brcmjpeg or your use of it.

Grey is also a slightly odd colour to end up with (YUV 128,128,128).
Is it a fully formed JPEG that you get, or a truncated bitstream? Use something like JPEGSnoop to actually analyse the bitstream, as many viewers cheat and display another colour (possibly grey) for pixels that aren't present in the bitstream.