6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5657
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereoscopic camera capture - now implemented.

Mon Feb 19, 2018 9:37 am

Sorry, forgot to look until now.
Yes, I'm only notifying the first sensor that it should only ever use the chosen mode, the other one is either freely choosing for itself, or may be mismatching the chosen mode as array index values change.
I'll see if it is an easy fix (it should be), although I don't have a stereo rig set up at the moment to test it with.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

ethanol100
Posts: 538
Joined: Wed Oct 02, 2013 12:28 pm

Re: Stereoscopic camera capture - now implemented.

Mon Feb 19, 2018 10:22 am

Thank you for looking into this. It is not a big issue, the automatic mode selection usually does a good job, and most of the times I don't need to choose the sensor mode. I can provide some testing if you want to check if your fix will work. I'm using a Compute Module IO board v3, with a CM3L and two NOIR v2 cameras.

ethanol100
Posts: 538
Joined: Wed Oct 02, 2013 12:28 pm

Re: Stereoscopic camera capture - now implemented.

Sun Mar 04, 2018 8:22 pm

Thank you 6by9, your commit fixed my issue.

ethanol100
Posts: 538
Joined: Wed Oct 02, 2013 12:28 pm

Re: Stereoscopic camera capture - now implemented.

Tue Mar 13, 2018 12:09 pm

I have a follow up question. I tried to find a good combination of size/frame rate to get maximal possible picture size at reasonable frame rates.
I found SBS [email protected] using MJPEG(40MBit/s, modified upper limit in raspivid) working reliably when using mode 5, but dropping frames when using mode 6. This seems strange to me, as there is no scaling needed and I would have assumed, mode 6 should be faster. Do you have any recommendation to archive higher frame rates and/or resolutions in stereo mode. Setting other resolution sometimes end up with seconds missing in the video and about missing about each second frame. My storage is a usb drive which can write constantly >20MB/s on the compute module. I know I can get higher frame rates at lower resolutions, but 720p is kind of minimal requirement (looking at 3mm big objects in ~0.75m distance).

I'm happy with 720p at 25fps, but slighly higher frame rate and resolution would be nice, too.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5657
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereoscopic camera capture - now implemented.

Tue Mar 13, 2018 1:48 pm

I haven't run a stereo rig for quite a while, so my memory of details will be going.
I'm assuming you're using V2 sensors (IMX219) based on your comments of no scaling in mode 6.
The rescaling of 1640x922 to 1280x720 should be a minimal overhead compared to 1280x720 as input. The main warning flag to me is that mode 6 only permits 40 to 120 fps, so the minimum frame rate is 40fps rather than the 25fps you said you were asking for in mode 5.

MJPEG encode isn't nicely pipelined as H264 is. The JPEG block requires planar data, whilst H264 wants a proprietary format (referred to as SAND or YUVUV). MMAL and the video_encode component both assume H264 and YUVUV, therefore when the MJPEG codec finally gets an image it has to convert it back to planar. It does this as one big operation, and holds the vector unit for the whole period.
This came up on another thread recently. It may be more efficient to use the image_encode component and MMAL_ENCODING_I420 instead. It does mean you have to set a Q-factor instead of having bitrate control built in, but that may not be such a big deal to you. You also want to turn off EXIF, thumbnails, and any other fancy features of image_encode.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

ethanol100
Posts: 538
Joined: Wed Oct 02, 2013 12:28 pm

Re: Stereoscopic camera capture - now implemented.

Tue Mar 13, 2018 1:57 pm

Thanks 6by9, your comment is as always very helpful. Yes, I'm using 2 v2 cameras. I did not realise that their was a minimum frame rate in mode 6/7. I will need to test this and I will try to use the image_encode component.

ethanol100
Posts: 538
Joined: Wed Oct 02, 2013 12:28 pm

Re: Stereoscopic camera capture - now implemented.

Thu Mar 15, 2018 6:00 pm

I tried to use the image_encode component and it creates a stream of jpegs. However I can't check the timing, the pts of the callback is 0 all the time. I checked the number of recorded frames and frame number annotations instead, image_encode seems to be slightly slower then video_encode.

I don't know if I made some mistakes, or if I forget to set/modify something. I tried the following code, where I use the bitrate as quality value( raspivid ... -cd JPEG -b 10 ...).

Any ideas?

Code: Select all

diff --git a/host_applications/linux/apps/raspicam/RaspiVid.c b/host_applications/linux/apps/raspicam/RaspiVid.c
index 5cf0d66..68c0d94 100755
--- a/host_applications/linux/apps/raspicam/RaspiVid.c
+++ b/host_applications/linux/apps/raspicam/RaspiVid.c
@@ -233,6 +233,7 @@ struct RASPIVID_STATE_S
    int64_t lasttime;
 
    bool netListen;
+   bool imageEncode;
 };
 
 
@@ -440,7 +441,7 @@ static void default_status(RASPIVID_STATE *state)
    state->save_pts = 0;
 
    state->netListen = false;
-
+   state->imageEncode = false;
 
    // Setup preview window defaults
    raspipreview_set_defaults(&state->preview_parameters);
@@ -870,6 +871,8 @@ static int parse_cmdline(int argc, const char **argv, RASPIVID_STATE *state)
                state->encoding = MMAL_ENCODING_H264;
             else  if (len==5 && !strncmp("MJPEG", argv[i+1], 5))
                state->encoding = MMAL_ENCODING_MJPEG;
+            else if (len==4 && !strncmp("JPEG", argv[i+1], 4))
+               state->imageEncode = true;
             else
                valid = 0;
             i++;
@@ -1441,6 +1444,7 @@ static void encoder_buffer_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buf
                bytes_written = fwrite(buffer->data, 1, buffer->length, pData->file_handle);
                if(pData->flush_buffers) fflush(pData->file_handle);
 
+               fprintf(stderr, "pts=%lld flag=%d\n",buffer->pts,buffer->flags );
                if(pData->pstate->save_pts &&
                   (buffer->flags & MMAL_BUFFER_HEADER_FLAG_FRAME_END ||
                    buffer->flags == 0 ||
@@ -1733,8 +1737,14 @@ static MMAL_STATUS_T create_camera_component(RASPIVID_STATE *state)
                                                      { 167, 1000 }, {999, 1000}};
         mmal_port_parameter_set(video_port, &fps_range.hdr);
    }
-
-   format->encoding = MMAL_ENCODING_OPAQUE;
+   if(state->imageEncode)
+   {
+      format->encoding = MMAL_ENCODING_I420;
+   }
+   else
+   {
+      format->encoding = MMAL_ENCODING_OPAQUE;
+   }
    format->es->video.width = VCOS_ALIGN_UP(state->width, 32);
    format->es->video.height = VCOS_ALIGN_UP(state->height, 16);
    format->es->video.crop.x = 0;
@@ -1761,7 +1771,14 @@ static MMAL_STATUS_T create_camera_component(RASPIVID_STATE *state)
 
    format = still_port->format;
 
-   format->encoding = MMAL_ENCODING_OPAQUE;
+   if(state->imageEncode)
+   {
+      format->encoding = MMAL_ENCODING_I420;
+   }
+   else
+   {
+      format->encoding = MMAL_ENCODING_OPAQUE;
+   }
    format->encoding_variant = MMAL_ENCODING_I420;
 
    format->es->video.width = VCOS_ALIGN_UP(state->width, 32);
@@ -1999,14 +2016,26 @@ static MMAL_STATUS_T create_encoder_component(RASPIVID_STATE *state)
    MMAL_STATUS_T status;
    MMAL_POOL_T *pool;
 
-   status = mmal_component_create(MMAL_COMPONENT_DEFAULT_VIDEO_ENCODER, &encoder);
-
-   if (status != MMAL_SUCCESS)
+   if(state->imageEncode)
    {
-      vcos_log_error("Unable to create video encoder component");
-      goto error;
+      status = mmal_component_create(MMAL_COMPONENT_DEFAULT_IMAGE_ENCODER, &encoder);
+      if (status != MMAL_SUCCESS)
+      {
+         vcos_log_error("Unable to create video encoder component");
+         goto error;
+      }
    }
+   else
+   {
+      status = mmal_component_create(MMAL_COMPONENT_DEFAULT_VIDEO_ENCODER, &encoder);
 
+      if (status != MMAL_SUCCESS)
+      {
+         vcos_log_error("Unable to create video encoder component");
+         goto error;
+      }
+   }
+   
    if (!encoder->input_num || !encoder->output_num)
    {
       status = MMAL_ENOSYS;
@@ -2020,204 +2049,244 @@ static MMAL_STATUS_T create_encoder_component(RASPIVID_STATE *state)
    // We want same format on input and output
    mmal_format_copy(encoder_output->format, encoder_input->format);
 
-   // Only supporting H264 at the moment
-   encoder_output->format->encoding = state->encoding;
+   if(state->imageEncode)
+   {
+      // Specify out output format - Hardcoded to JPEG
+      encoder_output->format->encoding = MMAL_ENCODING_JPEG;
+
+      //encoder_output->buffer_size = encoder_output->buffer_size_recommended;
+      encoder_output->buffer_size = 2048<<10;
+      
+      if (encoder_output->buffer_size < encoder_output->buffer_size_min)
+         encoder_output->buffer_size = encoder_output->buffer_size_min;
+
+      encoder_output->buffer_num = encoder_output->buffer_num_recommended;
+
+      if (encoder_output->buffer_num < encoder_output->buffer_num_min)
+         encoder_output->buffer_num = encoder_output->buffer_num_min;
+
+      // Commit the port changes to the output port
+      status = mmal_port_format_commit(encoder_output);
+
+      if (status != MMAL_SUCCESS)
+      {
+         vcos_log_error("Unable to set format on video encoder output port");
+         goto error;
+      }
 
-   if(state->encoding == MMAL_ENCODING_H264)
+      // Set the JPEG quality level
+      status = mmal_port_parameter_set_uint32(encoder_output, MMAL_PARAMETER_JPEG_Q_FACTOR, state->bitrate);
+
+      if (status != MMAL_SUCCESS)
+      {
+         vcos_log_error("Unable to set JPEG quality");
+         goto error;
+      }
+      MMAL_PARAMETER_THUMBNAIL_CONFIG_T param_thumb = {{MMAL_PARAMETER_THUMBNAIL_CONFIGURATION, sizeof(MMAL_PARAMETER_THUMBNAIL_CONFIG_T)}, 0, 0, 0, 0};
+      status = mmal_port_parameter_set(encoder->control, &param_thumb.hdr);
+
+   }
+   else
    {
-      if(state->level == MMAL_VIDEO_LEVEL_H264_4)
+      // Only supporting H264 at the moment
+      encoder_output->format->encoding = state->encoding;
+
+      if(state->encoding == MMAL_ENCODING_H264)
       {
-         if(state->bitrate > MAX_BITRATE_LEVEL4)
+         if(state->level == MMAL_VIDEO_LEVEL_H264_4)
          {
-            fprintf(stderr, "Bitrate too high: Reducing to 25MBit/s\n");
-            state->bitrate = MAX_BITRATE_LEVEL4;
+            if(state->bitrate > MAX_BITRATE_LEVEL4)
+            {
+               fprintf(stderr, "Bitrate too high: Reducing to 25MBit/s\n");
+               state->bitrate = MAX_BITRATE_LEVEL4;
+            }
          }
-      }
-      else
-      {
-         if(state->bitrate > MAX_BITRATE_LEVEL42)
+         else
          {
-            fprintf(stderr, "Bitrate too high: Reducing to 62.5MBit/s\n");
-            state->bitrate = MAX_BITRATE_LEVEL42;
+            if(state->bitrate > MAX_BITRATE_LEVEL42)
+            {
+               fprintf(stderr, "Bitrate too high: Reducing to 62.5MBit/s\n");
+               state->bitrate = MAX_BITRATE_LEVEL42;
+            }
          }
       }
-   }
-   else if(state->encoding == MMAL_ENCODING_MJPEG)
-   {
-      if(state->bitrate > MAX_BITRATE_MJPEG)
+      else if(state->encoding == MMAL_ENCODING_MJPEG)
       {
-         fprintf(stderr, "Bitrate too high: Reducing to 25MBit/s\n");
-         state->bitrate = MAX_BITRATE_MJPEG;
+         if(state->bitrate > MAX_BITRATE_MJPEG)
+         {
+            fprintf(stderr, "Bitrate too high: Reducing to 25MBit/s\n");
+            state->bitrate = MAX_BITRATE_MJPEG;
+         }
       }
-   }
-   
-   encoder_output->format->bitrate = state->bitrate;
+      
+      encoder_output->format->bitrate = state->bitrate;
 
-   if (state->encoding == MMAL_ENCODING_H264)
-      encoder_output->buffer_size = encoder_output->buffer_size_recommended;
-   else
-      encoder_output->buffer_size = 256<<10;
+      if (state->encoding == MMAL_ENCODING_H264)
+         encoder_output->buffer_size = encoder_output->buffer_size_recommended;
+      else
+         encoder_output->buffer_size = 256<<10;
 
 
-   if (encoder_output->buffer_size < encoder_output->buffer_size_min)
-      encoder_output->buffer_size = encoder_output->buffer_size_min;
+      if (encoder_output->buffer_size < encoder_output->buffer_size_min)
+         encoder_output->buffer_size = encoder_output->buffer_size_min;
 
-   encoder_output->buffer_num = encoder_output->buffer_num_recommended;
+      encoder_output->buffer_num = encoder_output->buffer_num_recommended;
 
-   if (encoder_output->buffer_num < encoder_output->buffer_num_min)
-      encoder_output->buffer_num = encoder_output->buffer_num_min;
+      if (encoder_output->buffer_num < encoder_output->buffer_num_min)
+         encoder_output->buffer_num = encoder_output->buffer_num_min;
 
-   // We need to set the frame rate on output to 0, to ensure it gets
-   // updated correctly from the input framerate when port connected
-   encoder_output->format->es->video.frame_rate.num = 0;
-   encoder_output->format->es->video.frame_rate.den = 1;
+      // We need to set the frame rate on output to 0, to ensure it gets
+      // updated correctly from the input framerate when port connected
+      encoder_output->format->es->video.frame_rate.num = 0;
+      encoder_output->format->es->video.frame_rate.den = 1;
 
-   // Commit the port changes to the output port
-   status = mmal_port_format_commit(encoder_output);
+      // Commit the port changes to the output port
+      status = mmal_port_format_commit(encoder_output);
 
-   if (status != MMAL_SUCCESS)
-   {
-      vcos_log_error("Unable to set format on video encoder output port");
-      goto error;
-   }
-
-   // Set the rate control parameter
-   if (0)
-   {
-      MMAL_PARAMETER_VIDEO_RATECONTROL_T param = {{ MMAL_PARAMETER_RATECONTROL, sizeof(param)}, MMAL_VIDEO_RATECONTROL_DEFAULT};
-      status = mmal_port_parameter_set(encoder_output, &param.hdr);
       if (status != MMAL_SUCCESS)
       {
-         vcos_log_error("Unable to set ratecontrol");
+         vcos_log_error("Unable to set format on video encoder output port");
          goto error;
       }
 
-   }
-
-   if (state->encoding == MMAL_ENCODING_H264 &&
-       state->intraperiod != -1)
-   {
-      MMAL_PARAMETER_UINT32_T param = {{ MMAL_PARAMETER_INTRAPERIOD, sizeof(param)}, state->intraperiod};
-      status = mmal_port_parameter_set(encoder_output, &param.hdr);
-      if (status != MMAL_SUCCESS)
+      // Set the rate control parameter
+      if (0)
       {
-         vcos_log_error("Unable to set intraperiod");
-         goto error;
-      }
-   }
+         MMAL_PARAMETER_VIDEO_RATECONTROL_T param = {{ MMAL_PARAMETER_RATECONTROL, sizeof(param)}, MMAL_VIDEO_RATECONTROL_DEFAULT};
+         status = mmal_port_parameter_set(encoder_output, &param.hdr);
+         if (status != MMAL_SUCCESS)
+         {
+            vcos_log_error("Unable to set ratecontrol");
+            goto error;
+         }
 
-   if (state->encoding == MMAL_ENCODING_H264 &&
-       state->quantisationParameter)
-   {
-      MMAL_PARAMETER_UINT32_T param = {{ MMAL_PARAMETER_VIDEO_ENCODE_INITIAL_QUANT, sizeof(param)}, state->quantisationParameter};
-      status = mmal_port_parameter_set(encoder_output, &param.hdr);
-      if (status != MMAL_SUCCESS)
-      {
-         vcos_log_error("Unable to set initial QP");
-         goto error;
       }
 
-      MMAL_PARAMETER_UINT32_T param2 = {{ MMAL_PARAMETER_VIDEO_ENCODE_MIN_QUANT, sizeof(param)}, state->quantisationParameter};
-      status = mmal_port_parameter_set(encoder_output, &param2.hdr);
-      if (status != MMAL_SUCCESS)
+      if (state->encoding == MMAL_ENCODING_H264 &&
+          state->intraperiod != -1)
       {
-         vcos_log_error("Unable to set min QP");
-         goto error;
+         MMAL_PARAMETER_UINT32_T param = {{ MMAL_PARAMETER_INTRAPERIOD, sizeof(param)}, state->intraperiod};
+         status = mmal_port_parameter_set(encoder_output, &param.hdr);
+         if (status != MMAL_SUCCESS)
+         {
+            vcos_log_error("Unable to set intraperiod");
+            goto error;
+         }
       }
 
-      MMAL_PARAMETER_UINT32_T param3 = {{ MMAL_PARAMETER_VIDEO_ENCODE_MAX_QUANT, sizeof(param)}, state->quantisationParameter};
-      status = mmal_port_parameter_set(encoder_output, &param3.hdr);
-      if (status != MMAL_SUCCESS)
+      if (state->encoding == MMAL_ENCODING_H264 &&
+          state->quantisationParameter)
       {
-         vcos_log_error("Unable to set max QP");
-         goto error;
-      }
+         MMAL_PARAMETER_UINT32_T param = {{ MMAL_PARAMETER_VIDEO_ENCODE_INITIAL_QUANT, sizeof(param)}, state->quantisationParameter};
+         status = mmal_port_parameter_set(encoder_output, &param.hdr);
+         if (status != MMAL_SUCCESS)
+         {
+            vcos_log_error("Unable to set initial QP");
+            goto error;
+         }
 
-   }
+         MMAL_PARAMETER_UINT32_T param2 = {{ MMAL_PARAMETER_VIDEO_ENCODE_MIN_QUANT, sizeof(param)}, state->quantisationParameter};
+         status = mmal_port_parameter_set(encoder_output, &param2.hdr);
+         if (status != MMAL_SUCCESS)
+         {
+            vcos_log_error("Unable to set min QP");
+            goto error;
+         }
 
-   if (state->encoding == MMAL_ENCODING_H264)
-   {
-      MMAL_PARAMETER_VIDEO_PROFILE_T  param;
-      param.hdr.id = MMAL_PARAMETER_PROFILE;
-      param.hdr.size = sizeof(param);
+         MMAL_PARAMETER_UINT32_T param3 = {{ MMAL_PARAMETER_VIDEO_ENCODE_MAX_QUANT, sizeof(param)}, state->quantisationParameter};
+         status = mmal_port_parameter_set(encoder_output, &param3.hdr);
+         if (status != MMAL_SUCCESS)
+         {
+            vcos_log_error("Unable to set max QP");
+            goto error;
+         }
 
-      param.profile[0].profile = state->profile;
+      }
 
-      if((VCOS_ALIGN_UP(state->width,16) >> 4) * (VCOS_ALIGN_UP(state->height,16) >> 4) * state->framerate > 245760)
+      if (state->encoding == MMAL_ENCODING_H264)
       {
-         if((VCOS_ALIGN_UP(state->width,16) >> 4) * (VCOS_ALIGN_UP(state->height,16) >> 4) * state->framerate <= 522240)
+         MMAL_PARAMETER_VIDEO_PROFILE_T  param;
+         param.hdr.id = MMAL_PARAMETER_PROFILE;
+         param.hdr.size = sizeof(param);
+
+         param.profile[0].profile = state->profile;
+
+         if((VCOS_ALIGN_UP(state->width,16) >> 4) * (VCOS_ALIGN_UP(state->height,16) >> 4) * state->framerate > 245760)
          {
-            fprintf(stderr, "Too many macroblocks/s: Increasing H264 Level to 4.2\n");
-            state->level=MMAL_VIDEO_LEVEL_H264_42;
+            if((VCOS_ALIGN_UP(state->width,16) >> 4) * (VCOS_ALIGN_UP(state->height,16) >> 4) * state->framerate <= 522240)
+            {
+               fprintf(stderr, "Too many macroblocks/s: Increasing H264 Level to 4.2\n");
+               state->level=MMAL_VIDEO_LEVEL_H264_42;
+            }
+            else
+            {
+               vcos_log_error("Too many macroblocks/s requested");
+               goto error;
+            }
          }
-         else
+         
+         param.profile[0].level = state->level;
+
+         status = mmal_port_parameter_set(encoder_output, &param.hdr);
+         if (status != MMAL_SUCCESS)
          {
-            vcos_log_error("Too many macroblocks/s requested");
+            vcos_log_error("Unable to set H264 profile");
             goto error;
          }
       }
-      
-      param.profile[0].level = state->level;
 
-      status = mmal_port_parameter_set(encoder_output, &param.hdr);
-      if (status != MMAL_SUCCESS)
+      if (mmal_port_parameter_set_boolean(encoder_input, MMAL_PARAMETER_VIDEO_IMMUTABLE_INPUT, state->immutableInput) != MMAL_SUCCESS)
       {
-         vcos_log_error("Unable to set H264 profile");
-         goto error;
+         vcos_log_error("Unable to set immutable input flag");
+         // Continue rather than abort..
       }
-   }
 
-   if (mmal_port_parameter_set_boolean(encoder_input, MMAL_PARAMETER_VIDEO_IMMUTABLE_INPUT, state->immutableInput) != MMAL_SUCCESS)
-   {
-      vcos_log_error("Unable to set immutable input flag");
-      // Continue rather than abort..
-   }
-
-   //set INLINE HEADER flag to generate SPS and PPS for every IDR if requested
-   if (mmal_port_parameter_set_boolean(encoder_output, MMAL_PARAMETER_VIDEO_ENCODE_INLINE_HEADER, state->bInlineHeaders) != MMAL_SUCCESS)
-   {
-      vcos_log_error("failed to set INLINE HEADER FLAG parameters");
-      // Continue rather than abort..
-   }
-
-   //set INLINE VECTORS flag to request motion vector estimates
-   if (state->encoding == MMAL_ENCODING_H264 &&
-       mmal_port_parameter_set_boolean(encoder_output, MMAL_PARAMETER_VIDEO_ENCODE_INLINE_VECTORS, state->inlineMotionVectors) != MMAL_SUCCESS)
-   {
-      vcos_log_error("failed to set INLINE VECTORS parameters");
-      // Continue rather than abort..
-   }
-
-   // Adaptive intra refresh settings
-   if (state->encoding == MMAL_ENCODING_H264 &&
-       state->intra_refresh_type != -1)
-   {
-      MMAL_PARAMETER_VIDEO_INTRA_REFRESH_T  param;
-      param.hdr.id = MMAL_PARAMETER_VIDEO_INTRA_REFRESH;
-      param.hdr.size = sizeof(param);
+      //set INLINE HEADER flag to generate SPS and PPS for every IDR if requested
+      if (mmal_port_parameter_set_boolean(encoder_output, MMAL_PARAMETER_VIDEO_ENCODE_INLINE_HEADER, state->bInlineHeaders) != MMAL_SUCCESS)
+      {
+         vcos_log_error("failed to set INLINE HEADER FLAG parameters");
+         // Continue rather than abort..
+      }
 
-      // Get first so we don't overwrite anything unexpectedly
-      status = mmal_port_parameter_get(encoder_output, &param.hdr);
-      if (status != MMAL_SUCCESS)
+      //set INLINE VECTORS flag to request motion vector estimates
+      if (state->encoding == MMAL_ENCODING_H264 &&
+          mmal_port_parameter_set_boolean(encoder_output, MMAL_PARAMETER_VIDEO_ENCODE_INLINE_VECTORS, state->inlineMotionVectors) != MMAL_SUCCESS)
       {
-         vcos_log_warn("Unable to get existing H264 intra-refresh values. Please update your firmware");
-         // Set some defaults, don't just pass random stack data
-         param.air_mbs = param.air_ref = param.cir_mbs = param.pir_mbs = 0;
+         vcos_log_error("failed to set INLINE VECTORS parameters");
+         // Continue rather than abort..
       }
 
-      param.refresh_mode = state->intra_refresh_type;
+      // Adaptive intra refresh settings
+      if (state->encoding == MMAL_ENCODING_H264 &&
+          state->intra_refresh_type != -1)
+      {
+         MMAL_PARAMETER_VIDEO_INTRA_REFRESH_T  param;
+         param.hdr.id = MMAL_PARAMETER_VIDEO_INTRA_REFRESH;
+         param.hdr.size = sizeof(param);
 
-      //if (state->intra_refresh_type == MMAL_VIDEO_INTRA_REFRESH_CYCLIC_MROWS)
-      //   param.cir_mbs = 10;
+         // Get first so we don't overwrite anything unexpectedly
+         status = mmal_port_parameter_get(encoder_output, &param.hdr);
+         if (status != MMAL_SUCCESS)
+         {
+            vcos_log_warn("Unable to get existing H264 intra-refresh values. Please update your firmware");
+            // Set some defaults, don't just pass random stack data
+            param.air_mbs = param.air_ref = param.cir_mbs = param.pir_mbs = 0;
+         }
 
-      status = mmal_port_parameter_set(encoder_output, &param.hdr);
-      if (status != MMAL_SUCCESS)
-      {
-         vcos_log_error("Unable to set H264 intra-refresh values");
-         goto error;
+         param.refresh_mode = state->intra_refresh_type;
+
+         //if (state->intra_refresh_type == MMAL_VIDEO_INTRA_REFRESH_CYCLIC_MROWS)
+         //   param.cir_mbs = 10;
+
+         status = mmal_port_parameter_set(encoder_output, &param.hdr);
+         if (status != MMAL_SUCCESS)
+         {
+            vcos_log_error("Unable to set H264 intra-refresh values");
+            goto error;
+         }
       }
-   }
 
+   }
    //  Enable component
    status = mmal_component_enable(encoder);
 


MRV
Posts: 5
Joined: Thu Jul 19, 2018 3:09 am

Re: Stereoscopic camera capture - now implemented.

Tue Aug 28, 2018 6:43 am

I have ordered a CM3 + IO Board + 2 cameras w/ connector ribbons.
I have a few questions regarding the current state of stereoscopic support.

1) Is the -3d parameter now natively available in raspivid (Raspbian)?

2) As of today, August 28, 2018, what is the maximum width possible for the SBS option? Is 2048x1024 possible? Meaning, two 1024x1024 videos side-by-side? (divisible by 256)

3) Is there formal documentation (beyond the forums) regarding the raspivid -3d option?

4) What advantage, if any, does UV4L offer for stereoscopic acquisition that the raspivid -3d stereoscopic doesn't? (Besides the additional streaming server component)

5) Lastly I'm curious if raspivid can capture two camera inputs and output (pipe) them both to FFmpeg for additional processing? I know it can be done for one camera, like this: raspivid -t 10000 -w 1080 -h 1080 -fps 30 -b 25000000 -o - | ffmpeg -i - -codec copy -f mp4 "piped.mp4"

Can two cameras be simultaneously piped to ffmpeg as well?

Thanks.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5657
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereoscopic camera capture - now implemented.

Tue Aug 28, 2018 10:05 am

ethanol100 wrote:
Thu Mar 15, 2018 6:00 pm
I tried to use the image_encode component and it creates a stream of jpegs. However I can't check the timing, the pts of the callback is 0 all the time. I checked the number of recorded frames and frame number annotations instead, image_encode seems to be slightly slower then video_encode.

I don't know if I made some mistakes, or if I forget to set/modify something. I tried the following code, where I use the bitrate as quality value( raspivid ... -cd JPEG -b 10 ...).

Any ideas?
Sorry, totally missed this one. image_encode currently doesn't pass timestamps at all. I'll add it to my list of jobs as it's needed for the V4L2 driver.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5657
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereoscopic camera capture - now implemented.

Tue Aug 28, 2018 10:21 am

MRV wrote:
Tue Aug 28, 2018 6:43 am
I have ordered a CM3 + IO Board + 2 cameras w/ connector ribbons.
I have a few questions regarding the current state of stereoscopic support.

1) Is the -3d parameter now natively available in raspivid (Raspbian)?
Yes.
MRV wrote:2) As of today, August 28, 2018, what is the maximum width possible for the SBS option? Is 2048x1024 possible? Meaning, two 1024x1024 videos side-by-side? (divisible by 256)
IIRC the hardware block is limited at 1920, but I've never been able to track down documentation to confirm that. Trying just with a single camera appears to confirm that a width of 2048 fails.
A width of 2032 appears to be the limit (2034 fails) - I have a vague thought about why that may be happening, but it's not something I have the time to investigate. (2034 knocks across into requiring an additional macroblock, so I'd guess something is rounding up incorrectly.
MRV wrote:3) Is there formal documentation (beyond the forums) regarding the raspivid -3d option?
Not that I'm aware of. I didn't expend much effort on it as so few people are generally prepared to buy the CM and CMIO to be able to run it.
MRV wrote:4) What advantage, if any, does UV4L offer for stereoscopic acquisition that the raspivid -3d stereoscopic doesn't? (Besides the additional streaming server component)
It exposes the cameras as a V4L2 device. Is that a requirement for you?
I deliberately didn't try implementing stereoscopic via V4L2 as it is not defined in the API how stereoscopic should be handled.
MRV wrote:5) Lastly I'm curious if raspivid can capture two camera inputs and output (pipe) them both to FFmpeg for additional processing? I know it can be done for one camera, like this: raspivid -t 10000 -w 1080 -h 1080 -fps 30 -b 25000000 -o - | ffmpeg -i - -codec copy -f mp4 "piped.mp4"

Can two cameras be simultaneously piped to ffmpeg as well?
The two cameras can be run simultaneously but independently, and then the two resulting streams can be routed wherever you fancy. Please remember that you are still contending for the same hardware blocks, therefore don't expect to be able to capture dual 1080P30 streams as the codec block won't keep up (the camera side should do just about).

Code: Select all

raspivid -t 10000 -w 1080 -h 1080 -fps 30 -b 25000000  -cs 0 -o - | ffmpeg -i - -codec copy -f mp4 "piped.mp4"
raspivid -t 10000 -w 1080 -h 1080 -fps 30 -b 25000000  -cs 1 -o - | ffmpeg -i - -codec copy -f mp4 "piped2.mp4"
You can't use stdout for two independent streams, but you should be able to use named pipes instead.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MRV
Posts: 5
Joined: Thu Jul 19, 2018 3:09 am

Re: Stereoscopic camera capture - now implemented.

Tue Aug 28, 2018 1:12 pm

6by9 wrote:
Tue Aug 28, 2018 10:21 am
MRV wrote:
Tue Aug 28, 2018 6:43 am
I have ordered a CM3 + IO Board + 2 cameras w/ connector ribbons.
I have a few questions regarding the current state of stereoscopic support.

1) Is the -3d parameter now natively available in raspivid (Raspbian)?
Yes.
• Great!
MRV wrote:2) As of today, August 28, 2018, what is the maximum width possible for the SBS option? Is 2048x1024 possible? Meaning, two 1024x1024 videos side-by-side? (divisible by 256)
IIRC the hardware block is limited at 1920, but I've never been able to track down documentation to confirm that. Trying just with a single camera appears to confirm that a width of 2048 fails.
A width of 2032 appears to be the limit (2034 fails) - I have a vague thought about why that may be happening, but it's not something I have the time to investigate. (2034 knocks across into requiring an additional macroblock, so I'd guess something is rounding up incorrectly.

• Is 1920 or 2032 the limit? If the latter, then perhaps it's possible that recording two square videos at 1016x1016 may work?
MRV wrote:3) Is there formal documentation (beyond the forums) regarding the raspivid -3d option?
Not that I'm aware of. I didn't expend much effort on it as so few people are generally prepared to buy the CM and CMIO to be able to run it.
MRV wrote:4) What advantage, if any, does UV4L offer for stereoscopic acquisition that the raspivid -3d stereoscopic doesn't? (Besides the additional streaming server component)
It exposes the cameras as a V4L2 device. Is that a requirement for you?
I deliberately didn't try implementing stereoscopic via V4L2 as it is not defined in the API how stereoscopic should be handled.

• I don't think exposing the cameras as V4L2 is a requirement. At least not that I'm aware of.
MRV wrote:5) Lastly I'm curious if raspivid can capture two camera inputs and output (pipe) them both to FFmpeg for additional processing? I know it can be done for one camera, like this: raspivid -t 10000 -w 1080 -h 1080 -fps 30 -b 25000000 -o - | ffmpeg -i - -codec copy -f mp4 "piped.mp4"

Can two cameras be simultaneously piped to ffmpeg as well?
The two cameras can be run simultaneously but independently, and then the two resulting streams can be routed wherever you fancy. Please remember that you are still contending for the same hardware blocks, therefore don't expect to be able to capture dual 1080P30 streams as the codec block won't keep up (the camera side should do just about).

Code: Select all

raspivid -t 10000 -w 1080 -h 1080 -fps 30 -b 25000000  -cs 0 -o - | ffmpeg -i - -codec copy -f mp4 "piped.mp4"
raspivid -t 10000 -w 1080 -h 1080 -fps 30 -b 25000000  -cs 1 -o - | ffmpeg -i - -codec copy -f mp4 "piped2.mp4"
You can't use stdout for two independent streams, but you should be able to use named pipes instead.
• Alright. Thank you.

----

• So, let me just thank you for implementing this feature. One of the things I'm most excited to see is the camera slave system, whereby the second camera matches the settings of the first. I assume that's continual right not just at startup?

• Lastly, prior to using the Raspi MIPI cameras, I was using two USB cameras attached to the PI 3B+. On the desktop (Mac) I was able to simultaneously record two cameras and merge them using FFmpeg's

Code: Select all

-filter_complex framepack
command like this:

Code: Select all

ffmpeg \
-f avfoundation  -s 2592x1944 -r 15 -t 00:00:15 -i "0" \
-f avfoundation  -s 2592x1944 -r 15 -t 00:00:15 -i "2" \
-map 0 -b:v 8000k -bufsize 8000k -c:v libx264 -preset ultrafast -y  \
-map 1 -b:v 8000k -bufsize 8000k -c:v libx264 -preset ultrafast -y  \
-filter_complex "[0:v]crop=1944:1944:0:0[left],[1:v]crop=1944:1944:0:0[right], [left][right]framepack=sbs"
While this method didn't allow me to force both cameras to have the same settings (which is actually important), it did allow me to record each camera at a high resolution, such as1944x1944, and output a single file up to 3888 x 1944 @ 25fps. This method, however, wasn't working well on the PI 3B+ at high resolutions. Only something like 480x480 were working.

The

Code: Select all

-filter_complex pad 
method also records two simultaneous videos but double the background of the 1st and adds the 2nd as an overlay next to it.

Code: Select all

-filter_complex '[0:v]pad=iw*2:ih[int];[int][1:v]overlay=W/2:0[vid]' \
Essentially doing the same as the framepack sbs (side-by-side) methods.

Not sure if anything like these methods could be used in Raspivid to enable higher resolutions to be captured. As the MIPI connections are much faster, perhaps it's possible? Does Raspivid have some kind of -filter_complex methods and, if so, is there a way to force multiple cameras to mirror as designated primary camera?

Just a thought anyway.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5657
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereoscopic camera capture - now implemented.

Tue Aug 28, 2018 1:41 pm

Settings are slaved for the duration of the run, not just at startup.

FFmpeg is doing everything in software (with a couple of exceptions).
Your -filter_complex is resulting in copying all the pixels between buffers. At 5MP 15fps that's 226Mbytes/s (1.8Gbit/s) of data being thrown around for the two cameras. That's a fairly sizeable number and has a significant impact on small systems. If you can't queue up those transfers neatly then waiting on cache misses is going to increase the pain significantly.
There are some tricks that can sometimes be played to get the same behaviour but better optimised for the hardware, eg 2D DMA or similar. Typically those don't get merged into generic applications such as FFmpeg.

No, there is no easy way to make raspivid merge frames or simlar.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MRV
Posts: 5
Joined: Thu Jul 19, 2018 3:09 am

Re: Stereoscopic camera capture - now implemented.

Tue Aug 28, 2018 9:34 pm

• Am I able to acquire square, non-rectangular, video with the -3d function, such as

Code: Select all

raspivid -3d sbs -w 768 -h 768 -o test.h264

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5657
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereoscopic camera capture - now implemented.

Wed Aug 29, 2018 6:36 am

MRV wrote:
Tue Aug 28, 2018 9:34 pm
• Am I able to acquire square, non-rectangular, video with the -3d function, such as

Code: Select all

raspivid -3d sbs -w 768 -h 768 -o test.h264
That should work as 768 is a multiple of 256.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MRV
Posts: 5
Joined: Thu Jul 19, 2018 3:09 am

Re: Stereoscopic camera capture - now implemented.

Sun Sep 02, 2018 9:22 pm

The CM3 + IO Board should arrive this week.
In the meantime I'm trying to clarify a few things regarding 3D SBS mode.

In regards to the maximum width, I have the following:
• The "IIRC, the hardware block, is limited at 1920"
• "A width of 2032 appears to be the limit"
Question Is the maximum width 1920 or 2032 ?

Question What is the absolute largest width that will actually work?

Potential candidates:
1) 1920 (960 x 2) (not multiple of 128)
2) 2032 (1016 x 2) (not multiple of 128)
3) 1536 (768 x 2) ( multiple of 128 but too small for my purposes)
4) 1792 (896 x 2) (multiple of 128 but still too small)
5) 2048 (1024 x 2) (multiple of 128 but too large?)

Realizator
Posts: 8
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website

Re: Stereoscopic camera capture - now implemented.

Tue Oct 09, 2018 12:38 pm

6by9 wrote:
Fri Feb 20, 2015 11:31 pm
New options are
-3d [none|sbs|tb]
-dec
-3dswap
For the stereoscopic mode, decimation, and swapping the sensors respectively.
Looks like -3dswap does not work with raspivid. I noticed this a lot time ago, but now this question became important for my project. 6by9, could you please check if it works for you?
I use it this way:

Code: Select all

raspivid -3d sbs -w 1280 -h 480 -o 1.h264
and

Code: Select all

raspivid -3d sbs -3dswap -w 1280 -h 480 -o 1.h264

ethanol100
Posts: 538
Joined: Wed Oct 02, 2013 12:28 pm

Re: Stereoscopic camera capture - now implemented.

Tue Oct 09, 2018 1:02 pm

You can use the camera selection to swap the cameras. I don't know if -cs 1 or -cs 0 works, but one of them should swap the eyes. Don't have a stereo setup at the moment.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5657
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stereoscopic camera capture - now implemented.

Tue Oct 09, 2018 1:27 pm

Sorry, I don't have a stereoscopic rig set up at the moment.

When you say it doesn't work can you elaborate? Doesn't record anything, doesn't swap the eyes, or draws pink unicorns? If I know what you're looking for then I can look through the code even if I don't have the hardware set up.

After a quick code read and I suspect that you may be right that it isn't implemented. As ethanol100 says, selecting the other camera will result in the same effect.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Realizator
Posts: 8
Joined: Thu Jul 14, 2016 12:53 pm
Contact: Website

Re: Stereoscopic camera capture - now implemented.

Tue Oct 09, 2018 2:35 pm

6by9 wrote:
Tue Oct 09, 2018 1:27 pm
Sorry, I don't have a stereoscopic rig set up at the moment.

When you say it doesn't work can you elaborate? Doesn't record anything, doesn't swap the eyes, or draws pink unicorns? If I know what you're looking for then I can look through the code even if I don't have the hardware set up.

After a quick code read and I suspect that you may be right that it isn't implemented. As ethanol100 says, selecting the other camera will result in the same effect.
I mean that -3dswap does nothing - left image in stereo pair remains left, and right image remains right. All other things works Ok (capture, encoding, recording).
As for ethanol100 idea - I suggest -cs option is for single-camera mode only. Am I wrong? Is -cs 1 means that camera 1 became "main" and they will swap? Should I use -cs 1 or -cs 0 before starting raspivid?

UPD> Yes, -cs 1 works fine - it makes swap, thank you ethanol100 and 6by9!

Return to “Camera board”

Who is online

Users browsing this forum: No registered users and 12 guests