DanL
Posts: 15
Joined: Mon Jan 26, 2015 3:02 pm

extending max frame rate

Sat Oct 24, 2015 1:50 pm

Hello,

I've been fiddling with the raspivid.c code and mmal functions over the past few days, trying to understand how things connect in there.

I was wondering how to change the maximum frame rate limits. I dont mind doing my pi any damages.

Can anyone point to me in the right direction? I have been searching for a limit in raspivid.c but it doesnt seem to exist. The code that says:
"if certain frame rate limit is passed -> use lower resolution"

Is there anything I can change in there? I am not an expert, so far, I just opened up the raspivid.c and changed it to have an extra raw output... seems to sometimes work :)
Last edited by DanL on Sat Oct 24, 2015 2:32 pm, edited 1 time in total.

User avatar
DougieLawson
Posts: 35789
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: extending max frame rate

Sat Oct 24, 2015 2:22 pm

You've got the source code, you can change it any which way you please. I doubt there's anything you can do that will damage anything in the hardware. Your main problem is that you're working with a closed source black box GPU that isn't fully documented.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

DanL
Posts: 15
Joined: Mon Jan 26, 2015 3:02 pm

Re: extending max frame rate

Sat Oct 24, 2015 2:36 pm

I found these obvious lines:

Code: Select all

if(state->camera_parameters.shutter_speed > 6000000)
   {
        MMAL_PARAMETER_FPS_RANGE_T fps_range = {{MMAL_PARAMETER_FPS_RANGE, sizeof(fps_range)},
                                                     { 50, 1000 }, {166, 1000}};
        mmal_port_parameter_set(video_port, &fps_range.hdr);
   }
   else if(state->camera_parameters.shutter_speed > 1000000)
   {
        MMAL_PARAMETER_FPS_RANGE_T fps_range = {{MMAL_PARAMETER_FPS_RANGE, sizeof(fps_range)},
                                                     { 167, 1000 }, {999, 1000}};
        mmal_port_parameter_set(video_port, &fps_range.hdr);
   }
I will try and set my own limit, but I have a feeling the resolution switch happens at a lower level.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7131
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: extending max frame rate

Sat Oct 24, 2015 4:51 pm

What are you actually trying to do?

MMAL_PARAMTETER_FPS_RANGE only configures the frame rate range if you specify an fps of 0 in the port format to indicate you want a variable frame rate.
For most video encoding applications you want a fixed frame rate, in which case specify it in the port format. Variable frame rate is more useful for stills preview where you want the best frame rate whilst still allowing AE to extend the exposure time. That is why the code you mention is in raspistill.c, not raspivid.c.

The GPU has a list of resolutions and frame rates for which it has register settings to program the sensor. Based on the parameters you feed it, it tries to select the most appropriate mode to optimise image quality. You can override that selection mechanism by using the -md option.

The maximum frame rate that we have register settings for is 90fps, but only at 640x480.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

DanL
Posts: 15
Joined: Mon Jan 26, 2015 3:02 pm

Re: extending max frame rate

Sat Oct 24, 2015 4:57 pm

I was actually hoping that I could record HD at 60 fps, but if fps is increased I get lower resolution

If necessary overclocking the GPU and H264 encoder is possible.. But I need to somehow override the fps limit

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7131
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: extending max frame rate

Sat Oct 24, 2015 5:45 pm

Not possible. 1296x730 @ 49fps was the best we got from the sensor, and it is the sensor that is the limiting factor here. I'd tried up the PLL settings, but failed to get reliable readout.
Feel free to take the raspiraw code and try to increase the rate - if it works then I'll happily take it into the firmware blob.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7131
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: extending max frame rate

Sat Oct 24, 2015 5:52 pm

FYI waveform80 has put together a nice table of the modes that we do have available at http://picamera.readthedocs.org/en/rele ... mera-modes
It's based on the info that I've posted on the forums as things got released, but he manages to find the time to clean those ramblings up into decent documentation.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

DanL
Posts: 15
Joined: Mon Jan 26, 2015 3:02 pm

Re: extending max frame rate

Sat Oct 24, 2015 11:54 pm

I took a quick look at the raspiraw and I think it is above my understanding.

What is quite nice is that I get my modified raspivid to run at ~47fps on HD.

I changed it to have a splitter with two outputs. One going to the H264 encoder and another to a converter that just stores the raw frame... That way I choose both a camera input format as well as a raw output format (I actually choose them to be the same) while maintaining the H264 output.

The raw output writes to file and "rewinds" the file descriptor to overwrite it (kind of a buffer). I am going to modify it to reside in RAM (shm_open might do it if I understand it correctly). And then I can have other processes reading the image data :D

DanL
Posts: 15
Joined: Mon Jan 26, 2015 3:02 pm

Re: extending max frame rate

Mon Oct 26, 2015 9:35 am

@6by9

The sensor specs I found on the internet indicate that it can do 60fps at 720p. I am more a physicist than a programmer so naturally I checked the voltages. I assume the problem is actually HW.

I am not sure how sensitive the hardware is to voltage changes, but you apply 3.31-3.32 volts on the camera connector (Pi side) when idle, you end up with 3.28-3.30 V (Camera side). This changes to 3.26-3.28 V when the camera is running (both sides). Actually seems stable to me.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7131
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: extending max frame rate

Mon Oct 26, 2015 11:10 am

DanL wrote:The sensor specs I found on the internet indicate that it can do 60fps at 720p.
Of course the internet is always right ;)
The limit is the ADC rate off the sensors, but actually crunching the numbers if we can do 2592x1944 @ 15 (we can), then 1280x720 @ 60 is the same rate. I'm not sure why I hit the limit at 49. I will double check the numbers.
I thought the register set that OV had given us for 720P was a cropped FOV - we avoid those at all costs as people don't understand when the FOV changes.
DanL wrote:I am more a physicist than a programmer so naturally I checked the voltages. I assume the problem is actually HW.

I am not sure how sensitive the hardware is to voltage changes, but you apply 3.31-3.32 volts on the camera connector (Pi side) when idle, you end up with 3.28-3.30 V (Camera side). This changes to 3.26-3.28 V when the camera is running (both sides). Actually seems stable to me.
Totally confused as to why you went that route of investigation. Voltage drops to the camera are not a known issue unless you're using a duff PSU. As stated above, the limit is the ADC reading the level off the CMOS sensor.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

DanL
Posts: 15
Joined: Mon Jan 26, 2015 3:02 pm

Re: extending max frame rate

Mon Oct 26, 2015 11:22 am

I tested voltages because I'm not really a programmer and I don't really have the knowledge to solve this issue :)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7131
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: extending max frame rate

Mon Oct 26, 2015 2:30 pm

Simple answer is that this is controlled by the GPU blob, so it's not one the community can solve (probably means me).

I've just checked my list of things to look into, and it does include checking why the minimum frame length on a couple of the camera modes are as high as they are.
As an example, the 1296x730 binned readout mode is configured to readout a minumum of 870 lines (ie 140 dummy lines of padding). There must have been a reason I set it so high, but if that can be dropped down to 730 then that should get us up to 59fps. I'll give it a try when I can.

There is a register set from OV for 720P - it is flagged as only being 720P30, but does use binning. It crops slightly though, so not so great (the numbers initially don't make sense to me - the limits appear to be off the edge of the array). TBH I'll try to tweak the numbers for the mode I do have working before looking at those in any more detail.
That crop would in theory be enough to get the frame rate up to 60.5fps, but cropping/FOV changed are such a nightmare I don't really want to go there.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jbeale
Posts: 3476
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: extending max frame rate

Mon Oct 26, 2015 9:11 pm

At the outset, everyone (including me) was clamoring for full-field-of-view video. Cropping and FOV changes may be undesirable in principal, but my experiments have satisfied me that this sensor produces significantly higher image quality in 1:1 pixel (cropped FOV) mode, than in 2x2 binned (full FOV) mode. It is not surprising, given how the sensor combines not nearest-neighbor, but nearest SAME-COLOR pixels, so given the 2x2 RGGB Bayer array this means you are giving up more than half the native resolution. Sometimes you do want the full FOV, but it's a significant quality tradeoff.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7131
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: extending max frame rate

Mon Oct 26, 2015 10:20 pm

The 720P60 mode would be binned AND cropped, so there's no gain there by cropping instead.
I will have a chat with my colleagues tomorrow. Binning means that the Bayer pattern is distorted from the typical 2x2 grid. I have a recollection that there is a way to enable some compensation for that, but comparing against other tuning files it looks like I may not be able to do a simple copy/paste.

Spare time is in short supply at the moment for me, but I will add looking into zoom cropping modes to my list of jobs - once you have zoomed past a certain point you can switch the sensor from binned to cropped mode instead. We already have a 30fps cropped mode (aka 1080P mode), so it should just be a signalling job at least for 16:9 formats. (Famous last words)
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

DanL
Posts: 15
Joined: Mon Jan 26, 2015 3:02 pm

Re: extending max frame rate

Tue Oct 27, 2015 9:41 am

I have a question regarding my raspivid:

When I push it to the limit at 49 fps, using both the encoder and a raw output to /run/shm, it runs fine. Usually it also quits fine when using CTRL+C. However, when I let it run a few minutes and then stop it, sometimes it just hangs after calling the signal handler. Do I need to add all the closures of shared memory and mutexes and stuff to the signal handler?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7131
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: extending max frame rate

Tue Oct 27, 2015 10:55 am

DanL wrote:I have a question regarding my raspivid:

When I push it to the limit at 49 fps, using both the encoder and a raw output to /run/shm, it runs fine. Usually it also quits fine when using CTRL+C. However, when I let it run a few minutes and then stop it, sometimes it just hangs after calling the signal handler. Do I need to add all the closures of shared memory and mutexes and stuff to the signal handler?
It should clean up as long as your callbacks do sensible things.
I'd suggest enabling the full MMAL trace to get a better idea of what is going on (there is an environment variable that you can set, but I can never remember it and always end up editing interface/mmal/core/mmal_logging.c, changing "mmal_log_level = VCOS_LOG_ERROR;" to "mmal_log_level = VCOS_LOG_TRACE;" and rebuild userland).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7131
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: extending max frame rate

Tue Oct 27, 2015 11:40 am

6by9 wrote:I will have a chat with my colleagues tomorrow. Binning means that the Bayer pattern is distorted from the typical 2x2 grid. I have a recollection that there is a way to enable some compensation for that, but comparing against other tuning files it looks like I may not be able to do a simple copy/paste.
I'd got the wrong block in my head which didn't help. The other advice is not to bother as it made very little real world difference - the distortion is minimal, though I may try it.
If you plot out what binning actually does, you effectively get missed pixels:

Code: Select all

Full sensor:
RGRGRGRGRGRGRGRG
GBGBGBGBGBGBGBGB
RGRGRGRGRGRGRGRG
GBGBGBGBGBGBGBGB
RGRGRGRGRGRGRGRG
GBGBGBGBGBGBGBGB
RGRGRGRGRGRGRGRG
GBGBGBGBGBGBGBGB

Binned
................
.RG..RG..RG..RG.
.GB..GB..GB..GB.
................
................
.RG..RG..RG..RG.
.GB..GB..GB..GB.
................
Note that the green pixels on a red row are treated as different to the green pixels on the blue row - sometimes referred to as Gr and Gb (why can't this forum do subscripts?!)
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

DanL
Posts: 15
Joined: Mon Jan 26, 2015 3:02 pm

Re: extending max frame rate

Tue Oct 27, 2015 3:25 pm

First you have binning then de-mosaic?

What are the bit depths after each stage? I am asking because the camera spec sheet suggests 10bit raw RGB data (RGGB whatever), but in a final file we get 12 bit per pixel. The odd thing is that this is almost equal 12/10 ~= 870/720 like padded data? Probably theres no connection between the two.

User avatar
jbeale
Posts: 3476
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: extending max frame rate

Tue Oct 27, 2015 3:46 pm

For what it's worth, the RAW portion of the output from 'raspistill --raw' gives you 10 bits per pixel in a packed format, see for example

viewtopic.php?p=357138#p357138

If the camera could deliver 12 bits per pixel, I'm pretty sure the OmniVision marketing department would have noted that fact :-)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7131
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: extending max frame rate

Tue Oct 27, 2015 3:57 pm

DanL wrote:First you have binning then de-mosaic?
Binning is done on the sensor in the analogue domain to get the ADC conversion rate within limits. Demosaic (and numerous other steps) are done in dedicated hardware blocks (known as the Image Sensor Pipeline, or ISP) within the BCM SoC.
DanL wrote:What are the bit depths after each stage? I am asking because the camera spec sheet suggests 10bit raw RGB data (RGGB whatever), but in a final file we get 12 bit per pixel. The odd thing is that this is almost equal 12/10 ~= 870/720 like padded data? Probably theres no connection between the two.
No connection whatsoever.
The raw data off the sensor is 10bit/pixel Bayer. IIRC The processing path within the ISP is at least 13bit to give some headroom, but the final stage formats it out into one of a number of formats (YUV444, YUV422, YUV420, or various RGB pixel interleaved formats).

The 870 / 720 difference is something I hope was just an oversight on my part - I'll see what I can find.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

DanL
Posts: 15
Joined: Mon Jan 26, 2015 3:02 pm

Re: extending max frame rate

Tue Oct 27, 2015 7:42 pm

jbeale, I am sure they would ;) I just thought there a simple mathematical conversion there to fit it into another format

This stuff, is above my head :) I should stop making stupid comments...

DanL
Posts: 15
Joined: Mon Jan 26, 2015 3:02 pm

Re: extending max frame rate

Wed Oct 28, 2015 11:55 am

I ran my code with the vcos logging, and there was no particular message before it hanged (if I just let it run the application stops after some time with no notice and I have to reboot because the camera stays on). I removed all my shared memory additions as well.

I added printf 's at the end and beginning of each callback to the encoder and converter. The execution stops when both finish writing to file.

To check if I'm managing the buffers incorrectly

Code: Select all

sudo /opt/vc/bin/vcdbg reloc | grep 'lock count 1'
[  93] 0x3bbd63e0: used 3.5M (refcount 1 lock count 1, size  3686400, align   32, data 0x3bbd6400, d1rual) 'ril mem video_convert-2'
[  95] 0x3c2de460: used 3.5M (refcount 1 lock count 1, size  3686400, align   32, data 0x3c2de480, d1rual) 'ril mem video_convert-2'
This tells me two converter callbacks are doing simultaneous writing to the same file, is that correct?

Reducing the buffer_num to 1 doesnt help, it still hangs

Code: Select all

sudo /opt/vc/bin/vcdbg reloc | grep 'lock count 1'
[  81] 0x3c2fe4e0: used 3.5M (refcount 1 lock count 1, size  3686400, align   32, data 0x3c2fe500, d1rual) 'ril mem video_convert-2'
[  65] 0x3ca73c20: used 705K (refcount 1 lock count 1, size   718000, align 4096, data 0x3ca74000, d3ruaL) 'venc_mem_alloc'
[  10] 0x3d895280: used 1.4M (refcount 1 lock count 1, size  1500352, align 4096, data 0x3d896000, d1ruaL) 'cameraRIL: hires vid pool'
Does the splitter also need a buffer pool?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7131
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: extending max frame rate

Wed Oct 28, 2015 12:58 pm

Straying onto a different topic here, but never mind.

No, the vcdgb reloc output doesn't imply that.
There is no MMU on the GPU, so it has what is termed the relocatable heap. Clients are given a handle to an object, and have to lock it to get a usable pointer to their data. They should unlock it when done. If an allocation fails, then it attempts to compact the heap by squashing all the unlocked blocks together (it can't move a locked block as that would invalidate the pointer the client has).
The number of clients that have locked the buffer is reference counted so that it can only move when all users have unlocked. "locked count 1" just means one client has it locked at that point in time.

What shared memory additions have you made? If you're using vcsm to share GPU memory (or the MMAL zero_copy option), then the allocations are always locked as the ARM has mmap'ed them memory, so it can't move. You imply /run/shm, so you're doing stuff solely on the ARM - in which case I don't know for certain as I'm not a Linux expert. When in doubt, clean up!

What function is your app in when it stalls? The MMAL trace should tell you what it is waiting for, if indeed it is MMAL that is stalling. Otherwise gdb should be able to tell you.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

DanL
Posts: 15
Joined: Mon Jan 26, 2015 3:02 pm

Re: extending max frame rate

Wed Oct 28, 2015 11:18 pm

Well my problem was solved anyway. I overclocked the GPU too much and it was simply unstable :)

paulhothersall
Posts: 28
Joined: Wed Jul 31, 2013 5:55 am

Re: extending max frame rate

Sat Oct 31, 2015 7:51 am

720P60 is a killer need for those of us in NTSC / 60Hz land! I think people would be totally cool with a working 720P60 version even if it was a 1x1 crop (I know I would)

The other item, which is the same for PAL / 50HZ land, is 50Hz is also a nice to have for any one of easy math and video playback reasons. Of course that logically seems "easier" to pull off in theory I am guessing masking the output is just that... so the reality may be different

the 870 lines puzzles me. I though the readout was strictly 2048x1104 (for 1920x1080P), 2048x984 (for 1280x960P), 2048x744 (for 1280x720P)


and heading off topic, it would be really cool to dial in some end user M12 / CS mount lens values. I use CS quite a bit and the colour hue curve is very obviously out (not just lens distortion with color bleed, if not the overall gain. Happy to be a tester to help dial this in!

Return to “Camera board”