dickh768
Posts: 6
Joined: Mon Jul 22, 2019 6:39 pm

Camera Dropped Frames

Fri Dec 13, 2019 12:04 pm

I'm putting together a streaming camera with video and audio using using the python mmal library.

I'm using a Pi3 and a 3rd party camera board which emulates the V2 camera. For the most part I have a working system with the camera feeding 1920 x 1080 video through the h264 encoder component. I am then multiplexing this with audio into mkv format before piping it to ffmpeg for streaming.

The main issue I have faced is synchronising the bit/frame rates of the audio and video. Initially I thought the only problem was the slight differences in clock rates of the video and audio modules, however I then found gaps in both the audio and video streams. I have now resolved the audio stream but there is a strange issue with the video stream.

My pipeline links the python mmal camera module and mmal encoder module to produce the h264 stream - nominally at 1920 x 1080, 30fps and with I frames inserted every 60 frames. The encoder module then triggers a callback whenever a frame is ready for output. This works correctly 95+% of the time with frames appearing at intervals of 1/30 sec (+/- a few mS). However my debugging has revealed that the gaps between some callbacks can be much greater - representing 2 - 69 of the normal frame periods over a recent 10 minute test run.

My initial thought was that there was a problem with the callback mechanism, however this is incorrect because the encoder is still encoding correctly with 60 intra frames between I frames. The problem must therefore lie either with the camera module itself or the pipeline between the camera and encoder.

I wondered originally if this was a problem with low light and long exposure times but since it only happens on a few percent of frames I think I can discount this theory,

Does anyone know if there is a solution to this problem, or do I have to find a way to live with the variation??

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7909
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Camera Dropped Frames

Fri Dec 13, 2019 3:15 pm

For a third party camera there is very little support that can be offered.

MMAL provides timestamps on all frames via the PTS field. It's always recommended to look at those values rather than assume anything.

The other concern I'd have is that memory says the CSI2 receiver will drop incoming frames which have too many CRC errors in the frame. If your 3rd party camera isn't producing clean CSI2 data, then frames will get dropped and you won't get a uniform 33ms between frames.
CSI2 is designed for use inside small devices such as phones and tablets, not for running long distances. The use of long cables will have a detrimental impact on signal integrity, and could cause this sort of issue.

Exposure time is clipped by the frame rate parameter, so that isn't an issue.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

dickh768
Posts: 6
Joined: Mon Jul 22, 2019 6:39 pm

Re: Camera Dropped Frames

Fri Dec 13, 2019 5:40 pm

Thanks for that suggestion - it certainly sounds very plausible, at least as far as errors are concerned. I've since come across an old post of yours in which you also say that 2 ECC errors will cause a frame discard.

My ribbon cable is actually quite short, although it was a bit tortuous and running close to other circuit tracks. However, having now done some disassembling to straighten the cable and trying some shielding it hasn't really made much difference. I suspect therefore that the camera module itself is generating errors - so the only resolution is probably to get hold of another camera and see if the problem goes away.

dickh768
Posts: 6
Joined: Mon Jul 22, 2019 6:39 pm

Re: Camera Dropped Frames

Mon Jan 06, 2020 9:34 am

Solved - I think. :)

This seems to be due to a couple of misunderstandings on my part. Firstly I assumed that a callback routine which only took a couple of mS would be fast enough to pull every frame from the h264 encoder. And secondly, the frame counter in the h264 encoder only counts successfully delivered frames rather than the frames received from the sensor.

What I think was happening therefore was that my callback routine was getting interrupted by the OS and preventing it from pulling all frames from the encoder.

I have now written a very minimalist callback routine which just pushes relevant data into a deque fifo to be processed in another thread. This seems error free although there must still be a small possibility of OS interrupt.

I see there is an asynchio package for python - has anyone tried this as a way of guaranteeing all encoder frames get captured?

Return to “Camera board”