dropped frames

[RecDev]

The sensor is being driven from a 25MHz crystal when it should be a 24.8MHz one, hence it is a couple of percent fast.

What EXACTLY is the command line you are running?
Are you requesting a low bitrate? The rate control algo can then sometimes decide to not encode frames as it is over bit budget.
Have you tried redirecting the video data to /dev/null whilst recording the PTS values (see earlier in this thread) to see what the timestamps come out as?

First of all, thanks for replying to a lot of these Camera Board related threads, you helped me solve some issues in these last weeks.

I'm not running it directly from the commandline, the video is recorded with the picamera library in a python program. I figure that doesn't make much of a difference in the actual recording though.
I don't set the bitrate to anything, so it should be whatever the default is, with a resolution of 1296 x 730 @ 20 fps.

I haven't tried those methods you mentioned yet because I'm not sure how to get a custom raspivid to run with picamera and include everything together (Audio and video recording at the same time while sending both to an encryption script), which is necessary since the problem probably doesn't exist if data isn't written to two files at the same time on the same usb port.
I'll try to do that once I've really exhausted my options.

But either way - Let's assume such a test would show that the USB device can't handle the writes while formatted as EXT4 (it certainly does handle them on XFS, but new requirements made that not an option). What would I even do in that case? Play around with the bitrate and hope for the best? Buy a ton of USB drives and pray that one is more reliable with the Raspberry's USB ports?
Sadly testing any of this is very time consuming, as differences are only obvious after ~1-2 hours of recording per video and I basically need a day of (daylight) recording for every setting I try to be statistically sure that it works fine.

[6by9]

I'd first suggest you try encoding using raspivid with the pts flag and just see if the camera is dropping frames under ideal circumstances. If CRC errors are detected on the CSI-2 bus then I believe frames will get dropped, so best to eliminate hardware issues first before trying to track down the problem in your code.
Raspivid won't work at the same time as picamera as the camera only handles one client at a time, but it does allow you to break the problem down into bits.

What does "top" show when you're running your system? If the ARM is maxed out doing audio and encryption stuff, you probably need to look at overclocking as there's nothing the GPU/camera can do if you don't feed it buffers to fill fast enough.

[p4trykx]

I modified ethanol100's raspivid which counts frames. After few tests recording pts count of frames seem to match exactly what it is in the file.

For testing purposes I would like to have the frame number and system time on the video. Recently added annotation feature seems to be pefect for it. Picamera uses this feature to draw frame number however some frames have the same numbers.

So my question is where should I put
I assume somewhere in buffer_callback but where exactly?

I got the annotation code from
look at void cam_set_annotation ()
https://raw.githubusercontent.com/silva ... spiMJPEG.c

[6by9]

For frame number, setting show_frame_num within the MMAL_PARAMETER_CAMERA_ANNOTATE_V2_T structure is probably the best bet for adding frame number to the images. The sensor and ISP are running asynchronously to delivering the frames to the host, so trying to synchronise the request from the ARM side is likely to fail for some frames (as you've seen). Updating the time/date stamp once a second is less likely to be noticed if it is one frame out.

buffer_callback is probably OK. My reservation is that it is called from a different thread context to your app, but I seem to recall it is safe to call back down into MMAL from there.

[p4trykx]

I have another question about PTS. What is the source of the clock for PTS. How can I synchronise it with system time.

I tried to get zero based PTS but it doesn't seem to work.

Code: Select all

 MMAL_PARAMETER_CAMERA_CONFIG_T cam_config =
      {
         { MMAL_PARAMETER_CAMERA_CONFIG, sizeof(cam_config) },
         .max_stills_w = state->width,
         .max_stills_h = state->height,
         .stills_yuv422 = 0,
         .one_shot_stills = 0,
         .max_preview_video_w = state->width,
         .max_preview_video_h = state->height,
         .num_preview_video_frames = 10,
         .stills_capture_circular_buffer_height = 0,
         .fast_preview_resume = 0, 
         ///PTS
         //.use_stc_timestamp = MMAL_PARAM_TIMESTAMP_MODE_RAW_STC
         .use_stc_timestamp = MMAL_PARAM_TIMESTAMP_MODE_RESET_STC
      };

[6by9]

I have another question about PTS. What is the source of the clock for PTS. How can I synchronise it with system time.

It's the main SoC STC. Gets set to 0 at system power on, and is a 64bit counter (in nsec IIRC).
Not sure if it is directly accessible from the ARM, but you can do a mmal_port_parameter_get on MMAL_PARAMETER_SYSTEM_TIME to retrieve it (port can be any port as it is intercepted by the MMAL core framework on the GPU). That's how the V4L2 driver gets a baseline for its timestamps.

I tried to get zero based PTS but it doesn't seem to work.

Have you done an rpi-update recently? I pushed what I thought was a firmware fix for this about 10 days ago. Sounds like I might need to double check that one.

[p4trykx]

Have you done an rpi-update recently? I pushed what I thought was a firmware fix for this about 10 days ago. Sounds like I might need to double check that one.

I didn't upgrade my firmware. Now it's working. However I'm getting PTS with 33 ms for the first frame.

Also the MMAL_PARAMETER_SYSTEM_TIME works fine. Does this clock/counter run "at the same speed" as the system clock.

I was experimenting with annotations and I see that the frame counter does not start with 0. It's 3 or other values. Does it serve some purpose? I know that the camera have to have some time to adjust settings before recording but could this annotated frames start counting from 0?
I noticed that sometimes I get 2 frames in a row with the same frame number annotated but then the next frame has +2 number so the overall count does match.

[6by9]

Have you done an rpi-update recently? I pushed what I thought was a firmware fix for this about 10 days ago. Sounds like I might need to double check that one.

I didn't upgrade my firmware. Now it's working. However I'm getting PTS with 33 ms for the first frame.

Using RESET_STC? Odd. I'll have a check when I can.

Also the MMAL_PARAMETER_SYSTEM_TIME works fine. Does this clock/counter run "at the same speed" as the system clock.

It reads exactly the same STC as the camera uses for timestamping, so are you which clock are you worried about?
I suspect the Linux clock runs off the same STC (read only registers are easy to share between processors), but I don't know for certain.

I was experimenting with annotations and I see that the frame counter does not start with 0. It's 3 or other values. Does it serve some purpose? I know that the camera have to have some time to adjust settings before recording but could this annotated frames start counting from 0?
I noticed that sometimes I get 2 frames in a row with the same frame number annotated but then the next frame has +2 number so the overall count does match.

Frame counter is the number of frames from the sensor since the last change of mode. The sensor shouldn't need to switch mode when recording starts, so the count is not 0 when recording starts. It was written as a debug feature rather than production code, hence it is a bit rough around the edges. There's no easy way to reset the sensor frame count, so it's not a simple task to reset the counter on the image.
Duplicated frame counts sounds odd as if the metadata is incorrectly being transferred. Digging into that is not fun, and as it doesn't affect the majority of users I'm afraid I think I'm going to have to ignore it - sorry.

[nyanthiss]

Hi everyone,

Just chiming in to add a solution that has worked for me. (I'm a bit surprised nobody has suggested it - at least i didn't see it reading this thread..) You don't need an ultra fast SD card, nor an XFS filesystem, all you have to do is tune the kernel page cache a bit. Run the followig as root:

Code: Select all

echo 3 >/proc/sys/vm/dirty_background_ratio
echo 50 >/proc/sys/vm/dirty_ratio
echo 300 >/proc/sys/vm/dirty_writeback_centisecs
echo 300 >/proc/sys/vm/dirty_expire_centisecs

You can google what the values stand for, if you're interested in technical detail. In short, it makes the kernel write out to the SDcard (or USB stick) in small pieces very often - you'll see the led blink quite a lot. You can try and tune the values; the key is to keep dirty_background_ratio small and far away from dirty_ratio.

You'll still need a reasonably fast media - at least slightly faster than the stream you're recording. I have 10MB/s SD card (FAT filesystem) and record FullHD at 30fps (about 5MB/s of data), and it works perfectly fine

Hope this helps,
cheers

[jfd3]

Hi Everyone,

Thank you for the great information; I've run into this issue myself recently. Can anyone attest to this change below fixing the issue? It does make a lot of sense if you read up on the kernel page buffer; here is a related article:
https://lonesysadmin.net/2013/12/22/bet ... rty_ratio/

My plan is to add these changes and test this with the recording script I'm using now (as below):

#!/bin/bash

echo 3 >/proc/sys/vm/dirty_background_ratio
echo 50 >/proc/sys/vm/dirty_ratio
echo 300 >/proc/sys/vm/dirty_writeback_centisecs
echo 300 >/proc/sys/vm/dirty_expire_centisecs

raspivid -n -t 0 -w 1920 -h 1080 -b 15000000 -fps 30 -o video.h264 | arecord -f S16_LE -D hw:1,0 -r 44100 audio.wav &

You can see the effect of frame-dropping in the RocketPi recording in the rocket launch video below:

https://youtu.be/u1pNfOEDeac

Thank you
-James