FFAMax
Posts: 30
Joined: Fri Nov 09, 2012 10:30 am
Location: Moscow / GMT+4

Camera fps

Thu Aug 15, 2013 4:03 pm

Hi!

Today i try to use arg -fps 60 for raspivid and not found any effect.
Can anybody explain how i can get hight framerate?

From Camera characteristic i know that camera can give 60 fps at 720p and 90 fps at 480p.

Image

User avatar
RaTTuS
Posts: 10549
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: Camera fps

Thu Aug 15, 2013 4:05 pm

it's limited to 30fps currently
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

FFAMax
Posts: 30
Joined: Fri Nov 09, 2012 10:30 am
Location: Moscow / GMT+4

Re: Camera fps

Thu Aug 15, 2013 4:11 pm

RaTTuS wrote:it's limited to 30fps currently
Limitation in GPU firmware? Do You know any details where is this limitation?

User avatar
RaTTuS
Posts: 10549
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: Camera fps

Thu Aug 15, 2013 4:24 pm

http://www.raspberrypi.org/phpBB3/viewt ... 51#p399651
and other places
jamesh
will look at it when he can
it's hidden in the gpu stuff
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25999
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera fps

Thu Aug 15, 2013 4:32 pm

FFAMax wrote:
RaTTuS wrote:it's limited to 30fps currently
Limitation in GPU firmware? Do You know any details where is this limitation?
We have a set of register settings from Omnivision, but we need to match those up with registers settings for the ISP. This is not trivial. My first go produced black frames, so I need to take another look when I have time.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“My wife said to me `...you’re not even listening`.
I thought, that’s an odd way to start a conversation.."

User avatar
jansendup
Posts: 10
Joined: Mon Aug 19, 2013 9:02 am
Location: Stellenbosch, South Africa

Re: Camera fps

Mon Aug 19, 2013 9:27 am

so does setting MMAL_VIDEO_FORMAT_T.frame_rate > 30 simply not work?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25999
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera fps

Mon Aug 19, 2013 9:54 am

jansendup wrote:so does setting MMAL_VIDEO_FORMAT_T.frame_rate > 30 simply not work?
No.

Well, technically, yes, it does exactly what is needs to do to set up the frame rate. BUT, the programming on the camera and ISP itself has a problem, so even when asked for a higher frame rate, there are not any modes in the driver to support those higher frames rates so its drops back to the highest it can do. I'm unfortunately very busy, and about to go on holiday, so won't have a chance to look at fixing the modes for at least a few weeks.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“My wife said to me `...you’re not even listening`.
I thought, that’s an odd way to start a conversation.."

User avatar
jansendup
Posts: 10
Joined: Mon Aug 19, 2013 9:02 am
Location: Stellenbosch, South Africa

Re: Camera fps

Mon Aug 19, 2013 12:14 pm

i'm a total noob but would love to try and figure out what the problem might be. "the programming on the camera and ISP itself has a problem" I'm almost sure I miss interpreted that. Did you mean the problem might be in the code of vchiq, openmax and mmal? Enjoy your vacation :)

ghans
Posts: 7882
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: Camera fps

Mon Aug 19, 2013 12:49 pm

It is closed-source GPU code which needs to changed.
Only Broacom engineers (i.e. jamesh) can do that.

ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25999
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera fps

Mon Aug 19, 2013 1:07 pm

It's also horrible difficult to get right, involving all sorts of timing calculations etc. Each time we get a mode working, there is cause for beer.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“My wife said to me `...you’re not even listening`.
I thought, that’s an odd way to start a conversation.."

User avatar
jansendup
Posts: 10
Joined: Mon Aug 19, 2013 9:02 am
Location: Stellenbosch, South Africa

Re: Camera fps

Mon Aug 19, 2013 1:14 pm

I though it was open-sourced? https://github.com/raspberrypi/userland
It sounds difficult. Would the timing calculations involve internal GPU processes as well as calculated values to be written to ISP registers. If so I take it that the internal GPU process timings are most difficult to calculate?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25999
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera fps

Mon Aug 19, 2013 1:41 pm

jansendup wrote:I though it was open-sourced? https://github.com/raspberrypi/userland
It sounds difficult. Would the timing calculations involve internal GPU processes as well as calculated values to be written to ISP registers. If so I take it that the internal GPU process timings are most difficult to calculate?
The ARM side code is all open source. This stuff is all in the GPU which isn't. Basically, we have a bunch of register settings for the camera for 60 and 90fps modes. These need to be matched with the same sort of timings in the first stage of the ISP (Image system pipeline) which is the software/HW pipeline for processing images. It's that matching that is going wrong somewhere, and although everything appears to be working - the frames are black.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“My wife said to me `...you’re not even listening`.
I thought, that’s an odd way to start a conversation.."

User avatar
jansendup
Posts: 10
Joined: Mon Aug 19, 2013 9:02 am
Location: Stellenbosch, South Africa

Re: Camera fps

Mon Aug 19, 2013 2:16 pm

Thanks, that clarified a lot. Thought that by ISP you meant image sensor processor. So the compiled GPU code is then probably included in the raspberry firmware repository. Just for interest sake, when you write code to be loaded on the GPU and executed do you use an in-house language similar to to that of opencl?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25999
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera fps

Mon Aug 19, 2013 2:19 pm

jansendup wrote:Thanks, that clarified a lot. Thought that by ISP you meant image sensor processor. So the compiled GPU code is then probably included in the raspberry firmware repository. Just for interest sake, when you write code to be loaded on the GPU and executed do you use an in-house language similar to to that of opencl?
No, the GPU is all C, and assembler when we need to use the vector units (C compiler doesn't support vector coding). The compiled code is the start.elf file in the boot partition.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“My wife said to me `...you’re not even listening`.
I thought, that’s an odd way to start a conversation.."

User avatar
jansendup
Posts: 10
Joined: Mon Aug 19, 2013 9:02 am
Location: Stellenbosch, South Africa

Re: Camera fps

Tue Aug 27, 2013 9:38 pm

I'm still waiting for my raspberry pi and cam so haven't yet tested for myself. Does anyone know if the fps can be set to any value between 2-30 fps or is it only certain discrete values between 2 and 30 fps and also if it can be dynamically changed while capturing?

User avatar
RaTTuS
Posts: 10549
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: Camera fps

Wed Aug 28, 2013 7:55 am

2 to 30 fps works fine [in steps of 1]
and you cannot change the framerate with the current tools while it is running though it's possible you can rebuild them to do so ... not 100% sure
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25999
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera fps

Wed Aug 28, 2013 7:57 am

I think you could probably vary the rate as you go along. Probably. I've never tried it.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“My wife said to me `...you’re not even listening`.
I thought, that’s an odd way to start a conversation.."

User avatar
jansendup
Posts: 10
Joined: Mon Aug 19, 2013 9:02 am
Location: Stellenbosch, South Africa

Re: Camera fps

Wed Sep 18, 2013 10:03 pm

I tried changing the frame rate while streaming but it seems that after enabling of the port subsequent calls to set port format or setting the fps port parameter in the mmal api gets ignored. Would have to disable then re-enable to change frame rate :)

User avatar
jansendup
Posts: 10
Joined: Mon Aug 19, 2013 9:02 am
Location: Stellenbosch, South Africa

Re: Camera fps

Wed Sep 18, 2013 10:13 pm

Didn't measure this with an oscilloscope but it seems one have pretty fine control over the frame rate(measured with clock_gettime()). It seems very continues and not that discrete. James when you said "get a mode working" I though that a mode translates to a discrete frame rate? I'm a bit confused :? Would a mode be a frame rate range. Like for example all frame rates that could be generated with the same pixel clock.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25999
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera fps

Thu Sep 19, 2013 9:42 am

jansendup wrote:Didn't measure this with an oscilloscope but it seems one have pretty fine control over the frame rate(measured with clock_gettime()). It seems very continues and not that discrete. James when you said "get a mode working" I though that a mode translates to a discrete frame rate? I'm a bit confused :? Would a mode be a frame rate range. Like for example all frame rates that could be generated with the same pixel clock.
Modes means the set of timing parameters you program the camera and ISP with. So generally a mode has a max and min frame rate (max would be the max the HW could go using the settings, min is more a software limitation). So generally you could have a frame rate anywhere between the max and min.

Of course, you may be able to define a mode that has a max that actually exceeds the capabilities of the ISP, but that a different part of the pipeline.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“My wife said to me `...you’re not even listening`.
I thought, that’s an odd way to start a conversation.."

GauVeldt
Posts: 24
Joined: Thu Mar 14, 2013 4:46 pm
Contact: Website

Re: Camera fps (feasibility of NoIR IR tracking project)

Wed Mar 04, 2015 1:13 am

This looks to be about 18 months old topic now so it might be worth it to necro-bump it to query if @jamesh made any progress with the camera framerate logic. I want to consider the possbility of an IR tracking project using the NoIR camera. For this I would use the lowest-P mode at the highest framerate to allow cycling the LED sensors on-and-off in individual frames for correct tracking of individual LED points. Tracking all the points at once will not work for sensors close together as they may be interpreted incorrectly (joints may become swapped for the IR points since they cannot be uniquely identified if a frame contains more than one IR LED point in close proximity). To do so the framerate needs to be as high as possible to avoid flicker (of the LEDs as they cycle for their frame Edit: IR is invisible) and suitable overall recording period (of the entire LED cycling sequence) which ultimately yields the FPS of the motion recording as a whole. I'd probably want two NoIR cameras to track (and thus two Pis) to track the points in 3D using binocular vision algorithms. Ideally the goal would be a blender plugin for assigning LED position points to armature bones

Sketch: https://dl.dropboxusercontent.com/u/552 ... xample.png

ghans
Posts: 7882
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: Camera fps

Wed Mar 04, 2015 6:28 am

The camera now supports a maximum of 90fps at VGA resolution.

You won't get more.


ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

GauVeldt
Posts: 24
Joined: Thu Mar 14, 2013 4:46 pm
Contact: Website

Re: Camera fps

Wed Mar 04, 2015 7:09 am

ghans wrote:The camera now supports a maximum of 90fps at VGA resolution.

You won't get more.

ghans
I knew the part about max fps of 90 but there were firmware issues where requesting the >30fps lowres framerates didn't get them as the OP had mentioned so I'm wondering if @jamesh had fixed those. The last post from him didn't say he had fixed the problem.

If 90 fps is now possible (at the corresponding resolution) then that's the advertised max and the problem has been resolved.

so I could get four LED shifts within a 22.5 fps overall framerate using the 90 fps settings

one way I could get more LED's per shift is to center them on differently colored pads and have the code look for the neighbouring color around an IR (bright spot) LED point when detected (the neighbouring color differentiating multiple IR points within the same camera frame, RGB would differentiate three, adding CMY would differentiate six LED points per shfit for a theoritical maximum of 4*6=24 points in the overall 22.5 fps sensing rate). I could get more with more colors for pads but anything other than primaries (RGB) and the secondary mixes (CMY) runs the risk of interference from the actor's attire and other background in the frame). I think for an amateur motion tracking project (that doesn't cost $1000s of dollars) 24 3D tracking points would be lots.

Think I'm going to make a topic just for the feasibility of such a motion tracking project.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25999
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera fps

Wed Mar 04, 2015 9:45 am

I'm not aware of any outstanding issues with the current camera modes.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“My wife said to me `...you’re not even listening`.
I thought, that’s an odd way to start a conversation.."

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8439
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Camera fps

Wed Mar 04, 2015 12:42 pm

GauVeldt wrote:I knew the part about max fps of 90 but there were firmware issues where requesting the >30fps lowres framerates didn't get them as the OP had mentioned so I'm wondering if @jamesh had fixed those. The last post from him didn't say he had fixed the problem.

If 90 fps is now possible (at the corresponding resolution) then that's the advertised max and the problem has been resolved.

so I could get four LED shifts within a 22.5 fps overall framerate using the 90 fps settings

one way I could get more LED's per shift is to center them on differently colored pads and have the code look for the neighbouring color around an IR (bright spot) LED point when detected (the neighbouring color differentiating multiple IR points within the same camera frame, RGB would differentiate three, adding CMY would differentiate six LED points per shfit for a theoritical maximum of 4*6=24 points in the overall 22.5 fps sensing rate). I could get more with more colors for pads but anything other than primaries (RGB) and the secondary mixes (CMY) runs the risk of interference from the actor's attire and other background in the frame). I think for an amateur motion tracking project (that doesn't cost $1000s of dollars) 24 3D tracking points would be lots.

Think I'm going to make a topic just for the feasibility of such a motion tracking project.
You may need to watch out as the sensor is based around a rolling shutter (http://en.wikipedia.org/wiki/Rolling_shutter) - not all lines of the image are exposed at the same moment in time, so synchronising your LEDs to the sensor may not be feasible.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “General discussion”