JPEG encoder output tearing

[cmisip]

It's as if the jpeg encoder did not finish writing the frame but returned the output buffer anyway. The top half of the jpeg gets misaligned with the bottom half. It doesn't happen all the time. I'll look into it further.

Thanks,
Chris

[6by9]

The buffer handling in the image_encode component holds on to them until the encode is completed (or aborted). If you're still filling them when you pass the buffer in then it will encode what it sees in memory.

As you haven't even stated what your source is, there's nearly zero that we can extract from your post. Please give sufficient information to work with in any new post.

[cmisip]

Here is the jpeg encoder initialization:

Code: Select all

int FfmpegCamera::OpenMmalJPEG(AVCodecContext *mVideoCodecContext){  
   

   // Create the jcoder component.
   if ( mmal_component_create(MMAL_COMPONENT_DEFAULT_IMAGE_ENCODER, &jcoder)  != MMAL_SUCCESS) {
      Fatal("failed to create mmal jpeg encoder");
   }   
   
   // CONTROL PORT SETTINGS
   jcoder->control->userdata = (MMAL_PORT_USERDATA_T *)&context;
   if ( mmal_port_enable(jcoder->control, control_callback) != MMAL_SUCCESS ) {
     Fatal("failed to enable mmal jpeg encoder control port");
   }  
   
   /* Get statistics on the input port */
   MMAL_PARAMETER_CORE_STATISTICS_T stats = {{0}};
   stats.hdr.id = MMAL_PARAMETER_CORE_STATISTICS;
   stats.hdr.size = sizeof(MMAL_PARAMETER_CORE_STATISTICS_T);
   if (mmal_port_parameter_get(jcoder->input[0], &stats.hdr) != MMAL_SUCCESS) {
     Info("failed to get jpeg encoder port statistics");
   }
   else {
     Info("JPEG encoder stats: %i, %i", stats.stats.buffer_count, stats.stats.max_delay);
   }
   
   // Set the zero-copy parameter on the input port 
   MMAL_PARAMETER_BOOLEAN_T zc = {{MMAL_PARAMETER_ZERO_COPY, sizeof(zc)}, MMAL_TRUE};
   if (mmal_port_parameter_set(jcoder->input[0], &zc.hdr) != MMAL_SUCCESS)
     Info("Failed to set zero copy on jpeg encoder input");
   // Set the zero-copy parameter on the output port 
   if (mmal_port_parameter_set_boolean(jcoder->output[0], MMAL_PARAMETER_ZERO_COPY, MMAL_TRUE) != MMAL_SUCCESS)
     Info("Failed to set zero copy on jpeg encoder output");
    
    
   /* Set format of jpeg encoder input port */
   MMAL_ES_FORMAT_T *format_in = jcoder->input[0]->format;
   format_in->type = MMAL_ES_TYPE_VIDEO;
   
   if ( colours == ZM_COLOUR_RGB32 ) {
       format_in->encoding = MMAL_ENCODING_RGBA;
   } else if ( colours == ZM_COLOUR_RGB24 ) {
       format_in->encoding = MMAL_ENCODING_RGB24;
   } else if(colours == ZM_COLOUR_GRAY8) { 
       format_in->encoding = MMAL_ENCODING_I420;
   }
   
   
   format_in->es->video.width = VCOS_ALIGN_UP(width, 32);
   format_in->es->video.height = VCOS_ALIGN_UP(height,16);
   format_in->es->video.crop.width = width;
   format_in->es->video.crop.height = height;
   
   format_in->es->video.frame_rate.num = 24000;
   format_in->es->video.frame_rate.den = 1001;
   format_in->es->video.par.num = mVideoCodecContext->sample_aspect_ratio.num;
   format_in->es->video.par.den = mVideoCodecContext->sample_aspect_ratio.den;
   format_in->flags = MMAL_ES_FORMAT_FLAG_FRAMED;
 

   
   if ( mmal_port_format_commit(jcoder->input[0]) != MMAL_SUCCESS ) {
      Fatal("failed to commit mmal jpeg encoder input format");
   }   

   MMAL_ES_FORMAT_T *format_out = jcoder->output[0]->format;
   format_out->type = MMAL_ES_TYPE_VIDEO;
   format_out->encoding = MMAL_ENCODING_JPEG;
   
   format_out->es->video.width = VCOS_ALIGN_UP(width, 32);
   format_out->es->video.height = VCOS_ALIGN_UP(height,16);
   format_out->es->video.crop.width = width;
   format_out->es->video.crop.height = height;
   
   
   if ( mmal_port_format_commit(jcoder->output[0]) != MMAL_SUCCESS ) {
     Fatal("failed to commit jpeg encoder output format");
   }
   
   //FIXME, should get from config
   if (mmal_port_parameter_set_uint32(jcoder->output[0], MMAL_PARAMETER_JPEG_Q_FACTOR, config.jpeg_file_quality) != MMAL_SUCCESS) {
   Fatal("failed to set jpeg quality for mmal jpeg encoder to %d",config.jpeg_file_quality);
   }   

   /* Display the input port format */
   display_format(&jcoder->input[0],&format_in);
   
   display_format(&jcoder->output[0],&format_out);
   

   /* The format of both ports is now set so we can get their buffer requirements and create
    * our buffer headers. We use the buffer pool API to create these. */
   jcoder->input[0]->buffer_num = jcoder->input[0]->buffer_num_min;
   jcoder->input[0]->buffer_size = jcoder->input[0]->buffer_size_min;
   jcoder->output[0]->buffer_num = jcoder->output[0]->buffer_num_min;
   jcoder->output[0]->buffer_size = jcoder->output[0]->buffer_size_min;
   
   pool_inj = mmal_port_pool_create(jcoder->input[0],jcoder->input[0]->buffer_num,
                              jcoder->input[0]->buffer_size);
   pool_outj = mmal_port_pool_create(jcoder->output[0],jcoder->output[0]->buffer_num,
                               jcoder->output[0]->buffer_size);
   /*                           
   pool_inj = mmal_pool_create(jcoder->input[0]->buffer_num,
                              jcoder->input[0]->buffer_size);
   pool_outj = mmal_pool_create(jcoder->output[0]->buffer_num,
                               jcoder->output[0]->buffer_size);                            
   */
   /* Create a queue to store our decoded video frames. The callback we will get when
    * a frame has been decoded will put the frame into this queue. */
   context.jqueue = mmal_queue_create();

   /* Store a reference to our context in each port (will be used during callbacks) */
   jcoder->input[0]->userdata = (MMAL_PORT_USERDATA_T *)&context;
   jcoder->output[0]->userdata = (MMAL_PORT_USERDATA_T *)&context;
   
   // Enable all the input port and the output port.
   if ( mmal_port_enable(jcoder->input[0], input_callback) != MMAL_SUCCESS ) {
     Fatal("failed to enable mmal jpeg encoder input port");
   }  
   
   if ( mmal_port_enable(jcoder->output[0], output_callbackj) != MMAL_SUCCESS ) {
     Fatal("failed to enable mmal jpeg encoder output port");
   }
   
   /* Component won't start processing data until it is enabled. */
   if ( mmal_component_enable(jcoder) != MMAL_SUCCESS ) {
     Fatal("failed to enable mmal jpeg encoder component");
   }  

   return 0;

}

Here is the code that passes the input and output buffers.

Code: Select all

int  FfmpegCamera::mmal_jpeg(uint8_t** jbuffer) {   //uses mFrame data
	MMAL_BUFFER_HEADER_T *buffer;
	if ((buffer = mmal_queue_get(pool_inj->queue)) != NULL) { 
		 
         av_image_copy_to_buffer(buffer->data, bufsize_r, (const uint8_t **)mFrame->data, mFrame->linesize,
                                 encoderPixFormat, mFrame->width, mFrame->height, 1);
         buffer->length=bufsize_r;
         
         buffer->pts = buffer->dts = MMAL_TIME_UNKNOWN;
         //buffer->flags=packet->flags;
         buffer->flags|=MMAL_BUFFER_HEADER_FLAG_FRAME_END;
         
         buffer->alloc_size = jcoder->input[0]->buffer_size;
            
         if (mmal_port_send_buffer(jcoder->input[0], buffer) != MMAL_SUCCESS) {
                 Warning("failed to send RGB buffer to jpeg encoder for frame %d\n", frameCount);
                  
         }
         
      }
      
      while ((buffer = mmal_queue_get(context.jqueue)) != NULL){
         if (buffer->length < jpeg_limit) {
           memcpy((*jbuffer),&buffer->length,4);
           memcpy((*jbuffer)+4,buffer->data,buffer->length);
         } else {
		   Info("JPEG buffer is too small at %d, while actual jpeg size is %d for quality %d", jpeg_limit, buffer->length, config.jpeg_file_quality);
	       int zero_size=0;
           memcpy((*jbuffer),&zero_size,4);
	     }
         mmal_buffer_header_release(buffer);
      }

     
      //if ((buffer = mmal_queue_get(pool_outr->queue)) != NULL) {
      while ((buffer = mmal_queue_get(pool_outj->queue)) != NULL) {
                   if (mmal_port_send_buffer(jcoder->output[0], buffer) != MMAL_SUCCESS) {
                      Warning("failed to send buffer to jpeg encoder output for frame %d\n", frameCount);
                   }
		  
      }
     return (0);    
}	

Here is the link to the source code for context.
https://raw.githubusercontent.com/cmisi ... camera.cpp

As far as I know, the code proceeds in step wise fashion. Packets are read from the input stream, decoded by mmal decoder, saved into an AVFrame mRawFrame, mRawFrame buffer copied into input buffer of vc.ril.isp which resizes it, output buffer of vc.ril.isp stored in an AVFrame mFrame, mFrame buffer copied into input buffer of jpeg encoder. The jpeg encoder should be getting full buffers.

Thanks,
Chris

[cmisip]

Here is a jpeg output that shows the problem. There seems to be a stair shaped seam in the middle of the images.

https://ibb.co/gKMxzU


https://ibb.co/bPNLKU

The second I think is when the IP camera is switching to or from IR mode so there is a big lighting difference between the "merged" frames.

Will mmal allow me to write to an input buffer while it is converting it in memory?

Thanks,
Chris

[6by9]

Will mmal allow me to write to an input buffer while it is converting it in memory?

There's no way to stop it without unmapping and remapping the buffer on every transition (very expensive in performance terms).
If you have the buffer address mmaped in your application then you can write to it any time you like. You need to ensure that once you have sent the buffer to the mmal component that you don't change the buffer. The same is true with v4l2 and many other apis.

I'd be surprised if this were a firmware bug rather than buffer management, but it is just possible.

[cmisip]

It seems the problem is a bit higher up in the code yet. Using libjpeg results in the same issue, so the RGB buffer is corrupted and the corrupted RGB buffer is sent to the jpeg block which therefore got blamed for the problem. The RGB buffer is created by vc.ril.isp from YUV420 packets provided by mmal decode of H264 packets. Using mmap only happens after the RGB buffer is created. I don't think the mmapped memory is being written to on the other end (analyse daemon).

Can you propose a countermeasure against this?

I think maybe successive loops of reading rtsp input -> decode -> vc.ril.isp convert to RGB -> jpeg block convert to JPEG,
causes the RGB buffer input to be stepped upon by the next loop iteration ( or maybe its decode that is stepping on a previous YUV buffer). vc.ril.isp might be sending input buffers back upon request, before it has finnished converting from YUV to RGB ( or maybe decode is). There must be something that could be used as a test to make sure that an input buffer is requested ( and therefore written to) only after the conversion in memory is complete. I hope it doesn't involve waiting or a hard coded delay as that will reduce the frame rate of the pipeline.

Do the mmal components provide an output buffer when requested ONLY when the conversion in memory is COMPLETE?

Thanks,
Chris

[6by9]

Do the mmal components provide an output buffer when requested ONLY when the conversion in memory is COMPLETE?

Yes, as with almost all pipeline frameworks (OpenMax, GStreamer, V4L2, etc), buffer ownership is the key to who is allowed to interact with the buffer.
The alternative mechanism some things use is fences, where the buffer is passed downstream before it is completed but with a fence handle that must be waited on before actually accessing the buffer.

I'm not aware of any issues in the ISP component. It really is set up based on picking one input and one output buffer off the queue, filling them, and then returning them in the appropriate directions.
Chromium's h/w accelerated video decode is doing almost exactly the same as you - video_decode -> isp -> RGBA buffers composed into X.