techyian
Posts: 53
Joined: Mon Jan 22, 2018 11:40 am

Camera dropping frames, any advice?

Thu Apr 11, 2019 1:22 pm

Hi, I'm having some trouble with recording video using raspivid and it looks like I'm getting a lot of dropped frames. I have two Pi's (2B and 3B+) and also two camera modules (official 8MP and SUNNY NoIR). On each device and camera module the videos I'm taking are skipping large chunks of the video and I'm also seeing artifacts in the data that is recorded.

As I thought it might be a performance issue, I'm just recording 640 x 480 H.264 at 15 fps but that hasn't helped. I was just wondering if anyone might have any pointers as to what it could be? It may be the SD cards that I'm using (Sandisk Ultra Class 10) perhaps?

Thanks in advance!
MMALSharp - C# API for the Raspberry Pi camera module

https://github.com/techyian/MMALSharp

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22215
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera dropping frames, any advice?

Thu Apr 11, 2019 2:14 pm

What software are you using to record the streams, and what settings do you apply?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

techyian
Posts: 53
Joined: Mon Jan 22, 2018 11:40 am

Re: Camera dropping frames, any advice?

Thu Apr 11, 2019 2:45 pm

Sorry I should have said! I'm just using raspivid:

raspivid -o test.h264 -fps 15 -b 1300000 -w 640 -h 480 -t 15000

Also tried with MMALSharp and seeing same issue with that too.
MMALSharp - C# API for the Raspberry Pi camera module

https://github.com/techyian/MMALSharp

HermannSW
Posts: 1214
Joined: Fri Jul 22, 2016 9:09 pm

Re: Camera dropping frames, any advice?

Fri Apr 12, 2019 12:30 am

That command records default mode (1) 1920x1080 and the GPU scales to 640x480.
Please try 640x480 modes 6 (42.1-60fps) or 7 (60.1-90fps) for v1 camera, or mode 7 (40-200fps) for v2 camera:
https://www.raspberrypi.org/documentati ... /camera.md

This is a good command to start and is likely to see 0 frame skips for both cameras
("-w 640 -h 480" is needed because without GPU will scale to 1920x1080):

Code: Select all

raspivid --mode 7 -o test.h264 -fps 90 -w 640 -h 480 -t 15000 -pts test.pts
The timestamps file written can be analyzed with "ptsanalyze test.pts 0":
https://github.com/Hermann-SW/userland/ ... ptsanalyze
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

techyian
Posts: 53
Joined: Mon Jan 22, 2018 11:40 am

Re: Camera dropping frames, any advice?

Fri Apr 12, 2019 5:44 am

I've tried using the command you sent over but still seeing dropped frames and artefacts in the video. As more frames are now being recorded, the stream's length is now nearer to what I'd expect it to be, albeit slightly longer than 15 seconds.
I appreciate the GPU is having to scale down the video with my previous attempt, which is why I set a low framerate (just using standard raspivid settings also produces this problem). My thoughts are still pointing towards SD card write performance at this stage as that seems to be the commonality between both Pi's I'm using.
MMALSharp - C# API for the Raspberry Pi camera module

https://github.com/techyian/MMALSharp

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6692
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Camera dropping frames, any advice?

Fri Apr 12, 2019 6:57 am

Having - w 640 - h480 will choose the most appropriate mode, which will be the 2x2 binned full fov mode (1296x972 or 1640x1232). Modes 6 and 7 will be excluded as they can't meet the framerate requested.

If you suspect the sd card, then use - o /dev/null - pts timestamps.txt to discard the recording but capture the frame timestamps. Hermannsw has his tools for analysis, although I tend to throw them into a spreadsheet and do conditional formatting should the delta be bigger than expected.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

HermannSW
Posts: 1214
Joined: Fri Jul 22, 2016 9:09 pm

Re: Camera dropping frames, any advice?

Fri Apr 12, 2019 4:00 pm

techyian wrote:
Fri Apr 12, 2019 5:44 am
I've tried using the command you sent over but still seeing dropped frames and artefacts in the video.
I did run the command on Pi3B+ with v2 camera:

Code: Select all

[email protected]:~ $ camver
v2 camera found
[email protected]:~ $ raspivid --mode 7 -o test.h264 -fps 90 -w 640 -h 480 -t 15000 -pts test.pts
[email protected]:~ $ 
Zero frame skips, and 90fps. As I said the Pi SD card can easily keep up with .h264 stream coming out of GPU:

Code: Select all

[email protected]:~ $ ptsanalyze test.pts 0
creating tstamps.csv
1339 frames were captured at 90fps
frame delta time[us] distribution
      2 11059
      1 11063
      2 11064
      1 11065
      1 11066
      1 11068
      4 11071
      1 11072
      4 11074
      1 11075
      5 11076
     28 11077
     85 11078
    156 11079
    294 11080
    329 11081
    223 11082
    109 11083
     55 11084
     12 11085
      6 11086
      3 11087
      3 11090
      2 11091
      1 11093
      2 11094
      2 11095
      1 11096
      1 11102
      1 11103
after skip frame indices (middle column)
0 frame skips (0%)
[email protected]:~ $ 

I switched to v1 camera and rebooted, same results (0 frame skips, 90fps):

Code: Select all

[email protected]:~ $ camver
v1 camera found
[email protected]:~ $ raspivid --mode 7 -o test.h264 -fps 90 -w 640 -h 480 -t 15000 -pts test.pts
[email protected]:~ $ ptsanalyze test.pts 0
creating tstamps.csv
1343 frames were captured at 90fps
frame delta time[us] distribution
      1 11076
      1 11084
      1 11087
      4 11088
     78 11089
    280 11090
    667 11091
    223 11092
     76 11093
      6 11094
      1 11095
      1 11098
      1 11106
after skip frame indices (middle column)
0 frame skips (0%)
[email protected]rypi3B:~ $ 

Since you mention artefacts as well (which I don't see), maybe your camera modules are bad?
Please do "-o /dev/null" test as 6by9 suggested to exclude the SD card.
Generated video is not big:

Code: Select all

[email protected]:~ $ du -sm test.h264 
31	test.h264
[email protected]:~ $ 
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6692
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Camera dropping frames, any advice?

Fri Apr 12, 2019 6:15 pm

Iirc the gpu will drop frames if unrecoverable errors have been detected on the csi2 bus. I seem to recall it can cope with single bit errors per line, but not double bit ones.
What sort of artefacts are you seeing? If it is the colours suddenly going purple part way through a frame then it is almost certainly csi2 corruption.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

techyian
Posts: 53
Joined: Mon Jan 22, 2018 11:40 am

Re: Camera dropping frames, any advice?

Fri Apr 12, 2019 6:52 pm

I've uploaded a test video for you to have a look at (NoIR module). I did set the timeout value to 20 seconds for this stream but as you can see, only a few seconds are recorded with lots of artefacts seen. It'd be pretty annoying if I'm having CSI bus issues on both Pi's, different camera modules and CSI ribbons :(

https://techyian.github.io/downloads/test_640_480.h264

Thanks for your help so far.
MMALSharp - C# API for the Raspberry Pi camera module

https://github.com/techyian/MMALSharp

HermannSW
Posts: 1214
Joined: Fri Jul 22, 2016 9:09 pm

Re: Camera dropping frames, any advice?

Fri Apr 12, 2019 10:21 pm

.h264 does not contain timestamp data, please create another video like you did, but with "-pts test.pts" option and provide video as well as matching test.pts -- the timestamps will clarify what is going on.

P.S:

Code: Select all

$ mediainfo test_640_480.h264 
General
Complete name                            : test_640_480.h264
Format                                   : AVC
Format/Info                              : Advanced Video Codec
File size                                : 2.72 MiB

Video
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : [email protected]
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 1 frame
Format settings, GOP                     : M=1, N=60
Width                                    : 640 pixels
Height                                   : 480 pixels
Display aspect ratio                     : 4:3
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive


$
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

techyian
Posts: 53
Joined: Mon Jan 22, 2018 11:40 am

Re: Camera dropping frames, any advice?

Fri Jun 14, 2019 7:44 pm

Hi both,

Apologies for the delay in replying to you, I've not had much time recently to commit to the project. I have uploaded a new test video and also the timestamp file as requested, these can be downloaded from:

http://techyian.github.io/downloads/test_640_480.h264
http://techyian.github.io/downloads/timestamp.pts

I am not entirely sure what the timestamp values should be, the video is supposed to be 10 seconds long but as you can see, it's only approx 6. The dropped frames continue to make an even bigger impact the longer the videos are.

Many thanks for your help.
MMALSharp - C# API for the Raspberry Pi camera module

https://github.com/techyian/MMALSharp

HermannSW
Posts: 1214
Joined: Fri Jul 22, 2016 9:09 pm

Re: Camera dropping frames, any advice?

Fri Jun 14, 2019 9:27 pm

You don't say how you play the video.
When I play it it shows nearly 10s length.
And ptsanalyze shows 0 frameskips, and 291 frames at 30fps, that is 9.7s:

Code: Select all

$ ptsanalyze timestamp.pts 0
creating tstamps.csv
291 frames were captured at 30fps
frame delta time[us] distribution
      5 33324
     11 33325
     49 33326
    113 33327
     67 33328
     21 33329
     16 33330
      6 33331
after skip frame indices (middle column)
0 frame skips (0%)
$ 

These are first and last timestamps:

Code: Select all

$ head -3 timestamp.pts 
# timecode format v2
0.000
33.331
$ tail -2 timestamp.pts 
9631.603
9664.927
$ 
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

techyian
Posts: 53
Joined: Mon Jan 22, 2018 11:40 am

Re: Camera dropping frames, any advice?

Fri Jun 14, 2019 9:46 pm

I just use VLC to play it. I've tried directly on the Pi and also on my Windows PC, both show the same issue with the video - can you not see the artifacts that I've explained previously? I'd be very surprised if VLC was to blame here.
MMALSharp - C# API for the Raspberry Pi camera module

https://github.com/techyian/MMALSharp

HermannSW
Posts: 1214
Joined: Fri Jul 22, 2016 9:09 pm

Re: Camera dropping frames, any advice?

Sat Jun 15, 2019 12:49 am

I cannot see artifacts.
I played with Linux "Videos" as well as with totem video player, both times more than 9s on the incrementing time display.
Then I used vlc and indeed it completes faster, and does not show an incrementing seconds display.
So yes, vlc does not play this video right, use another video player and you will see.
You can upload to youtube as well, and will get either a 9.7s or a 291/25=11.64s video.

Two comments on your setup:
6by9 always states that any flat ribbon cable longer than 15cm can be a problem, you use 1m or longer.
You do record with 30fps while documentation states that minimal framerate for 640x480 is 42.1/40fps for v1/v2 camera:
https://www.raspberrypi.org/documentati ... /camera.md

Anyway, since I neither see artifacts nor a single frame skip both seem to have no impact.

P.S:
I did convert the .h264 plus timestamp.pts to Matroska video:

Code: Select all

mkvmerge test_640_480.h264 -o test_640_480.h264.mkv --timecodes 0:timestamp.pts
vlc does play that video nicely, with increasing time counter (but with more than 11s length), so vlc definitely has some problem.
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

techyian
Posts: 53
Joined: Mon Jan 22, 2018 11:40 am

Re: Camera dropping frames, any advice?

Sat Jun 15, 2019 6:08 am

Thank you HermannSW! Yes the issue does appear to be when using VLC when the file is in the containerless format. I've converted the file to MKV as you suggested using the timestamp file and it plays correctly in both omxplayer and VLC. I also tried playing the .h264 version of the video within omxplayer, but instead of showing artifacts, the video played at a rapid speed.
MMALSharp - C# API for the Raspberry Pi camera module

https://github.com/techyian/MMALSharp

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6692
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Camera dropping frames, any advice?

Sat Jun 15, 2019 9:41 am

techyian wrote:
Sat Jun 15, 2019 6:08 am
I also tried playing the .h264 version of the video within omxplayer, but instead of showing artifacts, the video played at a rapid speed.
When there are no timestamps, omxplayer plays back as fast as it can decode the frames. With a VGA resolution frame, that's pretty quick. 1080p would end up being around 40fps, mainly depending on bitrate (CABAC is generally the slowest step).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

techyian
Posts: 53
Joined: Mon Jan 22, 2018 11:40 am

Re: Camera dropping frames, any advice?

Sat Jun 15, 2019 6:08 pm

Thanks to you both again for the help, really appreciated, glad it's not a hardware issue.
MMALSharp - C# API for the Raspberry Pi camera module

https://github.com/techyian/MMALSharp

Return to “Camera board”