cmisip
Posts: 94
Joined: Tue Aug 25, 2015 12:38 am

JPEG encoder output tearing

Sun Sep 16, 2018 4:43 am

It's as if the jpeg encoder did not finish writing the frame but returned the output buffer anyway. The top half of the jpeg gets misaligned with the bottom half. It doesn't happen all the time. I'll look into it further.

Thanks,
Chris

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5536
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: JPEG encoder output tearing

Sun Sep 16, 2018 8:22 am

The buffer handling in the image_encode component holds on to them until the encode is completed (or aborted). If you're still filling them when you pass the buffer in then it will encode what it sees in memory.

As you haven't even stated what your source is, there's nearly zero that we can extract from your post. Please give sufficient information to work with in any new post.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

cmisip
Posts: 94
Joined: Tue Aug 25, 2015 12:38 am

Re: JPEG encoder output tearing

Mon Sep 17, 2018 2:30 am

Here is the jpeg encoder initialization:

Code: Select all

int FfmpegCamera::OpenMmalJPEG(AVCodecContext *mVideoCodecContext){  
   

   // Create the jcoder component.
   if ( mmal_component_create(MMAL_COMPONENT_DEFAULT_IMAGE_ENCODER, &jcoder)  != MMAL_SUCCESS) {
      Fatal("failed to create mmal jpeg encoder");
   }   
   
   // CONTROL PORT SETTINGS
   jcoder->control->userdata = (MMAL_PORT_USERDATA_T *)&context;
   if ( mmal_port_enable(jcoder->control, control_callback) != MMAL_SUCCESS ) {
     Fatal("failed to enable mmal jpeg encoder control port");
   }  
   
   /* Get statistics on the input port */
   MMAL_PARAMETER_CORE_STATISTICS_T stats = {{0}};
   stats.hdr.id = MMAL_PARAMETER_CORE_STATISTICS;
   stats.hdr.size = sizeof(MMAL_PARAMETER_CORE_STATISTICS_T);
   if (mmal_port_parameter_get(jcoder->input[0], &stats.hdr) != MMAL_SUCCESS) {
     Info("failed to get jpeg encoder port statistics");
   }
   else {
     Info("JPEG encoder stats: %i, %i", stats.stats.buffer_count, stats.stats.max_delay);
   }
   
   // Set the zero-copy parameter on the input port 
   MMAL_PARAMETER_BOOLEAN_T zc = {{MMAL_PARAMETER_ZERO_COPY, sizeof(zc)}, MMAL_TRUE};
   if (mmal_port_parameter_set(jcoder->input[0], &zc.hdr) != MMAL_SUCCESS)
     Info("Failed to set zero copy on jpeg encoder input");
   // Set the zero-copy parameter on the output port 
   if (mmal_port_parameter_set_boolean(jcoder->output[0], MMAL_PARAMETER_ZERO_COPY, MMAL_TRUE) != MMAL_SUCCESS)
     Info("Failed to set zero copy on jpeg encoder output");
    
    
   /* Set format of jpeg encoder input port */
   MMAL_ES_FORMAT_T *format_in = jcoder->input[0]->format;
   format_in->type = MMAL_ES_TYPE_VIDEO;
   
   if ( colours == ZM_COLOUR_RGB32 ) {
       format_in->encoding = MMAL_ENCODING_RGBA;
   } else if ( colours == ZM_COLOUR_RGB24 ) {
       format_in->encoding = MMAL_ENCODING_RGB24;
   } else if(colours == ZM_COLOUR_GRAY8) { 
       format_in->encoding = MMAL_ENCODING_I420;
   }
   
   
   format_in->es->video.width = VCOS_ALIGN_UP(width, 32);
   format_in->es->video.height = VCOS_ALIGN_UP(height,16);
   format_in->es->video.crop.width = width;
   format_in->es->video.crop.height = height;
   
   format_in->es->video.frame_rate.num = 24000;
   format_in->es->video.frame_rate.den = 1001;
   format_in->es->video.par.num = mVideoCodecContext->sample_aspect_ratio.num;
   format_in->es->video.par.den = mVideoCodecContext->sample_aspect_ratio.den;
   format_in->flags = MMAL_ES_FORMAT_FLAG_FRAMED;
 

   
   if ( mmal_port_format_commit(jcoder->input[0]) != MMAL_SUCCESS ) {
      Fatal("failed to commit mmal jpeg encoder input format");
   }   

   MMAL_ES_FORMAT_T *format_out = jcoder->output[0]->format;
   format_out->type = MMAL_ES_TYPE_VIDEO;
   format_out->encoding = MMAL_ENCODING_JPEG;
   
   format_out->es->video.width = VCOS_ALIGN_UP(width, 32);
   format_out->es->video.height = VCOS_ALIGN_UP(height,16);
   format_out->es->video.crop.width = width;
   format_out->es->video.crop.height = height;
   
   
   if ( mmal_port_format_commit(jcoder->output[0]) != MMAL_SUCCESS ) {
     Fatal("failed to commit jpeg encoder output format");
   }
   
   //FIXME, should get from config
   if (mmal_port_parameter_set_uint32(jcoder->output[0], MMAL_PARAMETER_JPEG_Q_FACTOR, config.jpeg_file_quality) != MMAL_SUCCESS) {
   Fatal("failed to set jpeg quality for mmal jpeg encoder to %d",config.jpeg_file_quality);
   }   

   /* Display the input port format */
   display_format(&jcoder->input[0],&format_in);
   
   display_format(&jcoder->output[0],&format_out);
   

   /* The format of both ports is now set so we can get their buffer requirements and create
    * our buffer headers. We use the buffer pool API to create these. */
   jcoder->input[0]->buffer_num = jcoder->input[0]->buffer_num_min;
   jcoder->input[0]->buffer_size = jcoder->input[0]->buffer_size_min;
   jcoder->output[0]->buffer_num = jcoder->output[0]->buffer_num_min;
   jcoder->output[0]->buffer_size = jcoder->output[0]->buffer_size_min;
   
   pool_inj = mmal_port_pool_create(jcoder->input[0],jcoder->input[0]->buffer_num,
                              jcoder->input[0]->buffer_size);
   pool_outj = mmal_port_pool_create(jcoder->output[0],jcoder->output[0]->buffer_num,
                               jcoder->output[0]->buffer_size);
   /*                           
   pool_inj = mmal_pool_create(jcoder->input[0]->buffer_num,
                              jcoder->input[0]->buffer_size);
   pool_outj = mmal_pool_create(jcoder->output[0]->buffer_num,
                               jcoder->output[0]->buffer_size);                            
   */
   /* Create a queue to store our decoded video frames. The callback we will get when
    * a frame has been decoded will put the frame into this queue. */
   context.jqueue = mmal_queue_create();

   /* Store a reference to our context in each port (will be used during callbacks) */
   jcoder->input[0]->userdata = (MMAL_PORT_USERDATA_T *)&context;
   jcoder->output[0]->userdata = (MMAL_PORT_USERDATA_T *)&context;
   
   // Enable all the input port and the output port.
   if ( mmal_port_enable(jcoder->input[0], input_callback) != MMAL_SUCCESS ) {
     Fatal("failed to enable mmal jpeg encoder input port");
   }  
   
   if ( mmal_port_enable(jcoder->output[0], output_callbackj) != MMAL_SUCCESS ) {
     Fatal("failed to enable mmal jpeg encoder output port");
   }
   
   /* Component won't start processing data until it is enabled. */
   if ( mmal_component_enable(jcoder) != MMAL_SUCCESS ) {
     Fatal("failed to enable mmal jpeg encoder component");
   }  

   return 0;

}

Here is the code that passes the input and output buffers.

Code: Select all

int  FfmpegCamera::mmal_jpeg(uint8_t** jbuffer) {   //uses mFrame data
	MMAL_BUFFER_HEADER_T *buffer;
	if ((buffer = mmal_queue_get(pool_inj->queue)) != NULL) { 
		 
         av_image_copy_to_buffer(buffer->data, bufsize_r, (const uint8_t **)mFrame->data, mFrame->linesize,
                                 encoderPixFormat, mFrame->width, mFrame->height, 1);
         buffer->length=bufsize_r;
         
         buffer->pts = buffer->dts = MMAL_TIME_UNKNOWN;
         //buffer->flags=packet->flags;
         buffer->flags|=MMAL_BUFFER_HEADER_FLAG_FRAME_END;
         
         buffer->alloc_size = jcoder->input[0]->buffer_size;
            
         if (mmal_port_send_buffer(jcoder->input[0], buffer) != MMAL_SUCCESS) {
                 Warning("failed to send RGB buffer to jpeg encoder for frame %d\n", frameCount);
                  
         }
         
      }
      
      while ((buffer = mmal_queue_get(context.jqueue)) != NULL){
         if (buffer->length < jpeg_limit) {
           memcpy((*jbuffer),&buffer->length,4);
           memcpy((*jbuffer)+4,buffer->data,buffer->length);
         } else {
		   Info("JPEG buffer is too small at %d, while actual jpeg size is %d for quality %d", jpeg_limit, buffer->length, config.jpeg_file_quality);
	       int zero_size=0;
           memcpy((*jbuffer),&zero_size,4);
	     }
         mmal_buffer_header_release(buffer);
      }

     
      //if ((buffer = mmal_queue_get(pool_outr->queue)) != NULL) {
      while ((buffer = mmal_queue_get(pool_outj->queue)) != NULL) {
                   if (mmal_port_send_buffer(jcoder->output[0], buffer) != MMAL_SUCCESS) {
                      Warning("failed to send buffer to jpeg encoder output for frame %d\n", frameCount);
                   }
		  
      }
     return (0);    
}	
Here is the link to the source code for context.
https://raw.githubusercontent.com/cmisi ... camera.cpp

As far as I know, the code proceeds in step wise fashion. Packets are read from the input stream, decoded by mmal decoder, saved into an AVFrame mRawFrame, mRawFrame buffer copied into input buffer of vc.ril.isp which resizes it, output buffer of vc.ril.isp stored in an AVFrame mFrame, mFrame buffer copied into input buffer of jpeg encoder. The jpeg encoder should be getting full buffers.

Thanks,
Chris

Return to “Advanced users”

Who is online

Users browsing this forum: No registered users and 13 guests