Emeryx
Posts: 3
Joined: Tue Jun 11, 2019 8:00 pm

Pi 3 B+, v2.1 camera abyssmal fps @1080p

Tue Jun 11, 2019 8:43 pm

Hello all,
I am quite new to the community, but I am facing one of those problems that seems like not to bother anyone but me. First let's start with the context.
I am trying to record a video through a nodejs server, then host this video for watching/downloading. I wrote the code and everything was okay until I started actually watching the videos. It was extremely sped up. I attributed that to raspivid not providing the promised fps for ffmpeg and since it has no timestamps ffmpeg has now way to adjust to the dropping fps.

I don't want to bore you with the full details of the trials I went through so I will just provide a simple reproduction for my issue, right now I have nothing heavy running on my pi the cpu load and memory usage are not crossing 10~15% and I am running this command:

Code: Select all

raspivid -md 1 -w 1920 -h 1080 -pts tst.pts -t 100000 -fps 20 -o tst.h264
then checking the number of captured frames and the result is quite shocking, almost 0.72 ~ 8 fps literally, when I was piping the output to ffmpeg before the fps started at the promised 25/30 fps and then kept dropping until it stabilized 18 fps in 1~2 mins, the videos looked to be around 30% sped up so I am not sure if this testing method is accurate but the issue is still there the only fps I could actually record at is 15fps with 1080p resolution and I didn't even test that for extended periods of time.

I tested this on 3 boards and 3 different cameras so far, 2 pi 3 B and 1 pi 3 B+, the results are the same a 30% sped up videos being recorded. the more extensive testing was done with the B+ model and v2.1 cam.

User avatar
HermannSW
Posts: 1235
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Pi 3 B+, v2.1 camera abyssmal fps @1080p

Wed Jun 12, 2019 12:56 pm

You did analysis of effective framerate alread, can you show the output of "ptsanalyze tst.pts 0" for more details?
https://github.com/Hermann-SW/userland/ ... ptsanalyze
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6874
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Pi 3 B+, v2.1 camera abyssmal fps @1080p

Wed Jun 12, 2019 1:04 pm

If you are analysing the timestamps file, please ensure you have picked up the latest version of userland.

A bug was fixed recently in https://github.com/raspberrypi/userland/pull/553 where should an encoded frame be be fragmented into multiple buffers then the timestamp wasn't written to the pts file. The higher the bitrate the more frequently this occurred. You're not setting the bitrate, so it'll be defaulting to 17Mbit/s (pretty high).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
HermannSW
Posts: 1235
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Pi 3 B+, v2.1 camera abyssmal fps @1080p

Wed Jun 12, 2019 9:10 pm

In addition to what 6by9 said, you can count the number of frames in tst.h264 by:
https://stackoverflow.com/questions/196 ... 6#19753646

Code: Select all

$ ffprobe -show_streams -count_frames -pretty tst.h264 2>/dev/null | grep nb_read_frames=
nb_read_frames=529
$
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

Emeryx
Posts: 3
Joined: Tue Jun 11, 2019 8:00 pm

Re: Pi 3 B+, v2.1 camera abyssmal fps @1080p

Wed Jun 12, 2019 9:51 pm

I have been experimenting as well, trying to figure out the bottleneck and it seems to be the HDMI signal output, When the pi is outputting 1080p HDMI signal, the camera performance is seriously degraded (almost 50%), it shows up as 30% lost frames, but actually the only way to get a stable video is recording at 15 fps for 1080p.

Here is the test results:
@1080p HDMI
raspivid -md 1 -w 1920 -h 1080 -pts tst.pts -t 10000 -b 5000000 -fps 30 -o tst.h264
77 frame skips (32%)

@720p HDMI
raspivid -md 1 -w 1920 -h 1080 -pts tst.pts -t 10000 -b 5000000 -fps 30 -o tst.h264
15 frame skips (5%)

I will attach the csv and the actual timestamp files for both trials
Attachments
framerate.zip
(4.98 KiB) Downloaded 9 times

User avatar
HermannSW
Posts: 1235
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Pi 3 B+, v2.1 camera abyssmal fps @1080p

Wed Jun 12, 2019 10:32 pm

I cannot recreate your findings, my Pi3B has connected HDMI monitor as well, and 0 frame skips!

Code: Select all

[email protected]:~ $ raspivid -md 1 -w 1920 -h 1080 -pts tst.pts -t 10000 -b 5000000 -fps 30 -o tst.h264
[email protected]:~ $ userland/tools/ptsanalyze tst.pts 0
creating tstamps.csv
294 frames were captured at 30fps
frame delta time[us] distribution
      4 33307
      3 33308
     17 33309
     29 33310
     81 33311
     95 33312
     37 33313
     15 33314
      6 33315
      4 33316
after skip frame indices (middle column)
0 frame skips (0%)
[email protected]:~ $
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

User avatar
HermannSW
Posts: 1235
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Pi 3 B+, v2.1 camera abyssmal fps @1080p

Wed Jun 12, 2019 10:38 pm

Oops, that was for v1 camera, I just replaced with v2 camera, rebooted and get same 0 skips results with v2 camera and HDMI monitor connected:

Code: Select all

[email protected]:~ $ fork-raspiraw/tools/camver 
v2 camera found
[email protected]:~ $ raspivid -md 1 -w 1920 -h 1080 -pts tst.pts -t 10000 -b 5000000 -fps 30 -o tst.h264
[email protected]:~ $ userland/tools/ptsanalyze tst.pts 0
creating tstamps.csv
288 frames were captured at 30fps
frame delta time[us] distribution
      1 33323
      1 33324
     17 33325
     24 33326
    111 33327
     87 33328
     13 33329
     22 33330
      8 33331
      1 33333
after skip frame indices (middle column)
0 frame skips (0%)
[email protected]:~ $ 

ffprobe confirms the findings:

Code: Select all

[email protected]:~ $ ffprobe -show_streams -count_frames -pretty tst.h264 2>/dev/null | grep nb_read_frames=
nb_read_frames=288
[email protected]:~ $ 
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22684
Joined: Sat Jul 30, 2011 7:41 pm

Re: Pi 3 B+, v2.1 camera abyssmal fps @1080p

Thu Jun 13, 2019 7:03 am

Emeryx wrote:
Wed Jun 12, 2019 9:51 pm
I have been experimenting as well, trying to figure out the bottleneck and it seems to be the HDMI signal output, When the pi is outputting 1080p HDMI signal, the camera performance is seriously degraded (almost 50%), it shows up as 30% lost frames, but actually the only way to get a stable video is recording at 15 fps for 1080p.

Here is the test results:
@1080p HDMI
raspivid -md 1 -w 1920 -h 1080 -pts tst.pts -t 10000 -b 5000000 -fps 30 -o tst.h264
77 frame skips (32%)

@720p HDMI
raspivid -md 1 -w 1920 -h 1080 -pts tst.pts -t 10000 -b 5000000 -fps 30 -o tst.h264
15 frame skips (5%)

I will attach the csv and the actual timestamp files for both trials
That doesn't seem likely. The HDMI output has little or no effect in this case, it's all hardware.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
RaTTuS
Posts: 10313
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: Pi 3 B+, v2.1 camera abyssmal fps @1080p

Thu Jun 13, 2019 7:27 am

what are you recording to ?
a very slow sdcard ? or ram, or spinning rust hd or fast sdcard , or usb stick
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

Emeryx
Posts: 3
Joined: Tue Jun 11, 2019 8:00 pm

Re: Pi 3 B+, v2.1 camera abyssmal fps @1080p

Thu Jun 13, 2019 7:07 pm

I was quite surprised by the cause as well, I can't make the connection. The only thing I though of is that both of them are going through the GPU somehow, which causes more strain on the GPU when trying to record and output 1080p as well, but that seems unlikely.

@HermannSW what is the resolution of the monitor you are testing on? is it 1080p?.
Wouldn't that be 12 frames skipped for your v2 result, being that 30fps and 10 seconds should equal 300 frames or is there some other way to calculate it like the camera actually didn't record for 10 seconds?

@RaTTuS I am recording to a class 10 SD card, but I doubt that's the issue because on either screen resolution the amount of data I am writing to disk is the same, so I don't think it should affect the frame skipping differently.

I have a delivery that I need to meet so I will use 15 fps but then I will come back to it and try to dig deeper into the root cause, hopefully I won't have to keep this 15 fps or go for a 720p screen instead, thanks for the support guys keep the suggestions coming :)

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22684
Joined: Sat Jul 30, 2011 7:41 pm

Re: Pi 3 B+, v2.1 camera abyssmal fps @1080p

Thu Jun 13, 2019 8:16 pm

Emeryx wrote:
Thu Jun 13, 2019 7:07 pm
I was quite surprised by the cause as well, I can't make the connection. The only thing I though of is that both of them are going through the GPU somehow, which causes more strain on the GPU when trying to record and output 1080p as well, but that seems unlikely.
The GPU can easily handle 1080p30 camera capture, H264 record and display at the same time, this is a standard use case.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
HermannSW
Posts: 1235
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Pi 3 B+, v2.1 camera abyssmal fps @1080p

Fri Jun 14, 2019 1:10 am

Emeryx wrote:
Thu Jun 13, 2019 7:07 pm
@HermannSW what is the resolution of the monitor you are testing on? is it 1080p?.
Wouldn't that be 12 frames skipped for your v2 result, being that 30fps and 10 seconds should equal 300 frames or is there some other way to calculate it like the camera actually didn't record for 10 seconds?
No, as you can see all deltas analyzed by ptsanalyse are in range 33307..33316.
A frame skip would be eg. some delta like 66014 or more.
So what happens is that although I specified "-t 10000" I only got 284 frames (as reported by ffprobe as well).
Run with "-t 20000" or "-t 30000" and you will get 300 more frames.
That is "no frame skip". If you need 300 frames just specify "-t X" with X slightly larger than 10000.
@6by9 or @jamesh might know why some frames less than you expect get recorded.
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6874
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Pi 3 B+, v2.1 camera abyssmal fps @1080p

Sat Jun 15, 2019 9:35 am

HermannSW wrote:
Fri Jun 14, 2019 1:10 am
@6by9 or @jamesh might know why some frames less than you expect get recorded.
When you request video capture the system has to switch modes. If currently streaming, that means:
- completing the current frame first (NB this may not be at the same frame rate as your recording)
- stopping the sensor
- reprogramming it
- reconfiguring the pipeline
- starting the sensor
- dropping the first frame as it is likely incorrectly exposed
- actually starting to pass frames across to the encoder.
12 frames would be approx 400ms. 2 frames worth are easily accounted for as completing the current frame and dropping a frame. I wouldn't expect the remaining steps to require as much as 300ms, but I haven't profiled it. I suspect it may partly be that you've only just kicked off preview, so actually completing the current frame includes having to start the sensor, drop a frame, receive and process the requested frame, and then stop the sensor, so that's 2-and-a-bit preview frame periods.

Setting the preview frame rate to being the same as the recording one may avoid having to switch sensor mode at all and only reconfigure the pipe, but I haven't checked that code path recently.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”