JustASquid
Posts: 4
Joined: Sat Jul 20, 2019 6:24 am

picamera - Actual framerate of V2 module is lower than expected

Sat Jul 20, 2019 6:56 am

I'm working on an image processing application using the v2 camera module with a pi model 3B+.
I need as high a framerate as possible with as high a resolution as possible.

Consulting the table of sensor modes, the v2 camera supports 1280x720 recording at 90fps on sensor mode 6.

However, I've noticed that the actual framerate is a lot lower when using camera.start_recording() with a custom output for image processing.
When the camera is initialized as PiCamera(sensor_mode=6, framerate=90), the actual framerate hovers around ~32. The framerate is measured as the number of complete jpeg images written to the file-like object per second.
When I use PiCamera(resolution=(1280, 720), framerate=90) instead, the framerate hovers around ~70. Interestingly this still isn't 90 but is an improvement on setting mode 6 directly - I suppose the camera is deciding to use a different mode and upscaling? Is there a way to figure out what mode the camera has chosen?.

If I manually specify a smaller resolution e.g. (1024, 576) then the framerate hits 90 no problem. Again, my guess would be that it's upscaling from mode 7 but I don't actually know how to find out if the camera is doing that.

There aren't any other applications or code using CPU cycles (this should be irrelevant anyway - I'm measuring frames coming directly out of the GPU here, right?)

Below is the entirety of the code.

Code: Select all

import io
import time
import picamera

class ProcessOutput(object):
    def __init__(self):
        self.frame_times = []
        self.done = False

    def write(self, buf):
        if buf.startswith(b'\xff\xd8'):
            # New frame!
                    
            now = time.perf_counter()
            self.frame_times.append(now)
            
    def get_fps_info(self):
        # Get FPS over the last 2 seconds by counting the number of frames that have been written
        
        FPS_PERIOD_SECONDS = 2
        now = time.perf_counter()
        self.frame_times = [t for t in self.frame_times if now - t < FPS_PERIOD_SECONDS]
        total_frame_count = len(self.frame_times)
        total_fps = total_frame_count / FPS_PERIOD_SECONDS
        return total_fps


#with picamera.PiCamera(resolution=(1280, 720), framerate=90) as camera:
with picamera.PiCamera(sensor_mode=6, framerate=90) as camera:
    camera.start_preview()
    time.sleep(2)
    output = ProcessOutput()
    camera.start_recording(output, format='mjpeg')
    while not output.done:
        camera.wait_recording(0.1)
        total_fps = output.get_fps_info()
        print(total_fps)
    camera.stop_recording()
    
Can anyone explain the discrepancies I'm seeing? Is there any way I can get real 90fps 1280x720 recording?

User avatar
HermannSW
Posts: 1392
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: picamera - Actual framerate of V2 module is lower than expected

Sun Jul 21, 2019 7:35 pm

Hi,

I tried doing same with raspivid and v2 camera, and the answer is "too many macroblocks".
6by9 will have to explain what that exactly means -- 60fps is good with level increase, 70fps and above fails:

Code: Select all

[email protected]:~ $ raspivid -md 6 -t 0 -p 22,50,960,540 -fps 50
Too many macroblocks/s: Increasing H264 Level to 4.2
^Cmmal: Aborting program

[email protected]:~ $ raspivid -md 6 -t 0 -p 22,50,960,540 -fps 60
Too many macroblocks/s: Increasing H264 Level to 4.2
^Cmmal: Aborting program

[email protected]:~ $ raspivid -md 6 -t 0 -p 22,50,960,540 -fps 70
mmal: Too many macroblocks/s requested
mmal: main: Failed to create encode component
mmal: Failed to run camera app. Please check for firmware updates

[email protected]:~ $ 

P.S:
The first time I did run that command with "-fps 90" I got the error, and camera did not work anymore afterwards (with any setting). I had to reboot the Pi, then all was good again. I just did it again getting the error message. But now I can run camera fine with lower framerates after that.
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://gitlab.freedesktop.org/HermannSW/gst-template
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

JustASquid
Posts: 4
Joined: Sat Jul 20, 2019 6:24 am

Re: picamera - Actual framerate of V2 module is lower than expected

Sun Jul 21, 2019 10:38 pm

HermannSW wrote:
Sun Jul 21, 2019 7:35 pm
Hi,

I tried doing same with raspivid and v2 camera, and the answer is "too many macroblocks".
6by9 will have to explain what that exactly means -- 60fps is good with level increase, 70fps and above fails:
Thanks for your response!

Maybe I'm missing something but that sounds like an issue specific to h.264 encoding? I've tried both mjpeg and raw YUV with similar results.
Also that doesn't explain why the documentation says sensor mode 6 can record up to 90fps...

ethanol100
Posts: 583
Joined: Wed Oct 02, 2013 12:28 pm

Re: picamera - Actual framerate of V2 module is lower than expected

Mon Jul 22, 2019 6:26 am

From your linked documentation:
Note

These are not the set of possible output resolutions or framerates. These are merely the set of resolutions and framerates that the sensor can output directly to the GPU. The GPU’s ISP block will resize to any requested resolution (within reason). Read on for details of mode selection.
The sensor can create, but the gpu can't consume.

User avatar
HermannSW
Posts: 1392
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: picamera - Actual framerate of V2 module is lower than expected

Mon Jul 22, 2019 3:37 pm

JustASquid wrote:
Sun Jul 21, 2019 10:38 pm
Maybe I'm missing something but that sounds like an issue specific to h.264 encoding? I've tried both mjpeg and raw YUV with similar results.
You are right, although my raspividyuv results are different than yours.
ethanol100 wrote:
Mon Jul 22, 2019 6:26 am
The sensor can create, but the gpu can't consume.
Seems to be true.

I identified the biggest bottleneck in storing the video on SD card.
After eliminating that things are better.
But still the GPU cannot keep up with what sensor creates.

I used raspividyuv with "-pts" option to write out timestamps for analysis.
Those get only written with "-o" option, so I used "-o /dev/null".
For timestamp analysis I used this tool:
https://raw.githubusercontent.com/Herma ... ptsanalyze

Running raspividyuv with 90fps show 33% frame skips, with 80fps 25%, with 70fps 15%, with 60fps 5 frame skips:

Code: Select all

[email protected]:~ $ raspividyuv -md 6 -p 22,50,960,540 -fps 60 -pts tst.pts -o /dev/null
[email protected]:~ $ ptsanalyze tst.pts 0 
creating tstamps.csv
283 frames were captured at 60fps
frame delta time[us] distribution
      1 16598
      2 16600
      3 16601
      9 16602
      3 16603
      2 16604
      1 16607
      2 16608
      4 16609
     27 16610
    107 16611
     57 16612
     26 16613
      7 16614
      2 16615
      2 16617
      2 16619
      6 16620
      7 16621
      3 16622
      2 16623
      1 33212
      1 33213
      2 33222
      1 33223
after skip frame indices (middle column)
> 33222,581397
> 33213,847179
> 33222,3073102
> 33223,3621277
> 33212,3887059
5 frame skips (1%)
[email protected]:~ $ 
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://gitlab.freedesktop.org/HermannSW/gst-template
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

User avatar
HermannSW
Posts: 1392
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: picamera - Actual framerate of V2 module is lower than expected

Mon Jul 22, 2019 4:15 pm

Please ignore my last numbers, the mode 6 capture will be scaled to 1920x1080 by those commands.

Correct command (with -w and -h) produces only 7% frame skips for 90fps:

Code: Select all

[email protected]:~ $ raspividyuv -md 6 -fps 90 -w 1280 -h 720 -pts tst.pts -o /dev/null
[email protected]:~ $ ptsanalyze tst.pts 0 
creating tstamps.csv
401 frames were captured at 90fps
frame delta time[us] distribution
      1 11054
      1 11055
      1 11056
      1 11059
      2 11061
      4 11062
      1 11063
      5 11064
      5 11065
      3 11066
      1 11067
      3 11068
      6 11069
      7 11070
     18 11071
     13 11072
      7 11073
      4 11074
      1 11076
      1 11078
      7 11079
     78 11080
     75 11081
     38 11082
      7 11083
      3 11085
      1 11086
      2 11087
      8 11088
     13 11089
     10 11090
      4 11091
      5 11092
      3 11093
      4 11094
      4 11095
      4 11096
      4 11097
      5 11098
      2 11099
      1 11100
      1 11102
      1 11106
      1 11109
      1 22160
      2 22161
     10 22162
      6 22163
      1 22164
      3 22169
      2 22170
      6 22171
      1 22172
after skip frame indices (middle column)
> 22160,642682
> 22171,675934
> 22169,742417
> 22162,908620
> 22171,941872
> 22162,1008347
> 22163,1096994
> 22169,1728603
> 22162,1795079
> 22172,1828332
> 22169,1894813
> 22162,1928058
> 22162,1994533
> 22171,2027784
> 22163,2182906
> 22163,2814509
> 22162,2880991
> 22171,2914243
> 22161,2980717
> 22163,3080445
> 22171,3113696
> 22162,3180171
> 22162,3268817
> 22170,3900428
> 22161,3966903
> 22171,4000155
> 22162,4066630
> 22164,4166358
> 22170,4199607
> 22162,4266083
> 22163,4354730
> 22163,4565264
32 frame skips (7%)
[email protected]:~ $ 

Only slightly more frame skips (10%) for raspivid for 90fps:

Code: Select all

[email protected]:~ $ raspivid -md 6 -fps 90 -w 1280 -h 720 -pts tst.pts -o /dev/null
Too many macroblocks/s: Increasing H264 Level to 4.2
[email protected]:~ $ ptsanalyze tst.pts 0 
creating tstamps.csv
382 frames were captured at 90fps
frame delta time[us] distribution
      5 11078
     20 11079
     73 11080
    139 11081
     75 11082
     18 11083
      3 11084
      1 11089
      1 11101
      1 22140
      1 22151
      1 22157
      5 22158
     11 22159
     17 22160
      8 22161
after skip frame indices (middle column)
> 22160,642682
> 22159,731328
> 22160,842135
> 22159,941862
> 22158,1030508
> 22160,1108074
> 22160,1207800
> 22161,1296447
> 22160,1396173
> 22159,1484818
> 22158,1595625
> 22159,1695352
> 22140,1783998
> 22160,1883725
> 22161,1972371
> 22160,2072097
> 22159,2160743
> 22151,2249389
> 22160,2349116
> 22157,2448842
> 22161,2526408
> 22160,2615053
> 22160,2714780
> 22159,2803425
> 22159,2903152
> 22160,2991798
> 22160,3102606
> 22159,3202332
> 22160,3290978
> 22158,3379625
> 22160,3468270
> 22161,3556916
> 22158,3656642
> 22159,3745288
> 22159,3856095
> 22159,3955822
> 22158,4044468
> 22160,4144195
> 22160,4232840
> 22161,4321487
> 22161,4399052
> 22161,4487699
> 22160,4609585
> 22161,4709313
44 frame skips (10%)
[email protected]:~ $ 

If you go to mode 7, you can get [email protected] with a single frame skip only for raspivid:

Code: Select all

[email protected]:~ $ raspivid -md 7 -fps 150 -w 640 -h 480 -pts tst.pts -o /dev/null
[email protected]:~ $ ptsanalyze tst.pts 0 
creating tstamps.csv
726 frames were captured at 150fps
frame delta time[us] distribution
      1 6635
      2 6636
      2 6637
      5 6638
     46 6639
    246 6640
    323 6641
     82 6642
      8 6643
      5 6644
      1 6646
      1 6647
      1 13282
after skip frame indices (middle column)
> 13282,617581
1 frame skips (0%)
[email protected]:~ $ 
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://gitlab.freedesktop.org/HermannSW/gst-template
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

User avatar
jbeale
Posts: 3476
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: picamera - Actual framerate of V2 module is lower than expected

Mon Jul 22, 2019 4:29 pm

I vaguely recall the maximum reported framerates involved some kind of overclocking.
See for example https://www.raspberrypi.org/forums/view ... 15#p960789
The H264 encoder will now allow the app to request level 4.2, but will only achieve any significant gain with overclocking. An update to raspivid is in order to allow > 720P68. 720P120 is almost working reliably but requires a significant overclock (achieved on a Pi3, but not on a Pi2 yet). Settings for how much overclocking, and any other trade-offs that can be made, are still being determined.

ethanol100
Posts: 583
Joined: Wed Oct 02, 2013 12:28 pm

Re: picamera - Actual framerate of V2 module is lower than expected

Mon Jul 22, 2019 6:50 pm

jbeale wrote: I vaguely recall the maximum reported framerates involved some kind of overclocking.
See for example https://www.raspberrypi.org/forums/view ... 15#p960789
Correct! This is a good example were the Pi4 slightly higher clock comes in handy. I tested recording 5 min to /dev/shm using

Code: Select all

raspivid -md 6 -t 300000 -w 1280 -h 720 -fps 90 -awb off -awbg 1,1 -ag 2 -dg 1 -ss 5000 -o test.h264 -pts test.pts -n -v
which did not drop a single frame. The frame rate is even slightly higher at 90.249fps.

For the macroblocks, have a look at wikipedia. The usually used Level 4 only allows for 245,760 macroblocks/s and this limit is reached at 1,280×[email protected] Level 4.2 increased this to 522,240 macroblocks/s which theoretically allows up to 1,280×[email protected] We have 16x16 macroblocks, 80x45 *90 frames = 324 000 macroblocks/s.

JustASquid
Posts: 4
Joined: Sat Jul 20, 2019 6:24 am

Re: picamera - Actual framerate of V2 module is lower than expected

Mon Jul 22, 2019 10:29 pm

HermannSW wrote:
Mon Jul 22, 2019 4:15 pm
Please ignore my last numbers, the mode 6 capture will be scaled to 1920x1080 by those commands.

Correct command (with -w and -h) produces only 7% frame skips for 90fps:
Can anyone explain why I'd be getting only 30fps in the Python script then?

User avatar
jbeale
Posts: 3476
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: picamera - Actual framerate of V2 module is lower than expected

Tue Jul 23, 2019 3:40 pm

if you copy ethanol100's command line and record to /dev/shm (NOT sd card) what frame rate do you get? That would give you an apples-to-apples comparison to start with. The maximum possible data rate you can record to SD card depends on the card and it is also not constant, sometimes cards will randomly require longer times to record the same amount of data, due to the underlying not-always-uniform properties of the flash memory cells.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7140
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: picamera - Actual framerate of V2 module is lower than expected

Tue Jul 23, 2019 3:53 pm

Moving data around in memory is not free - SDRAM bandwidth is finite.
raspivid etc are configured to do the minimum amount of work on the CPU, and the GPU is passing the images around as efficiently as possible. H264 is far more efficient than MJPEG.
Picamera has to interface with Python, and, from previous conversations with waveform80, there are inherent copies within Python that are tricky to avoid.

The ISP can normally cope with around 150MPix/s. 1280x720 @ 90fps is 82MPix/s, so fairly comfortably under the limit.

"PiCamera(sensor_mode=6, framerate=90)" has not set the output resolution, so it will be upscaling from the 1280x720 of mode 6 to the default resolution of the display's resolution (probably 1080p). Setting sensor_mode does not change the output resolution, only the sensor resolution.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

JustASquid
Posts: 4
Joined: Sat Jul 20, 2019 6:24 am

Re: picamera - Actual framerate of V2 module is lower than expected

Fri Jul 26, 2019 2:00 am

Thanks everyone,

This seems to explain the results I'm seeing.

Return to “Camera board”