User avatar
HermannSW
Posts: 2651
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: v2 camera can capture 1000fps videos ...

Wed Apr 17, 2019 9:22 pm

yt38 wrote:
Wed Apr 17, 2019 12:54 pm
Hi, would a black and white capture change anything in resolution, size, fps?
I don't know black and white cameras, only monochrome cameras besides the raw Bayer color cameras.
This thread is about capturing with raspiraw, and only few cameras connected to CSI-2 interface are supported right now:
ov5647 (v1 camera), imx219 (v2 camera) and adv7282m.

Which camera do you talk about?
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

yt38
Posts: 2
Joined: Wed Apr 17, 2019 12:50 pm

Re: v2 camera can capture 1000fps videos ...

Thu May 02, 2019 7:27 pm

Having a very limited background with image capture, I assumed the amount of information would be less if monochrome. I don't know if the picamera v2 would have this option before capturing. So my question was for a PiCamera V2.

User avatar
HermannSW
Posts: 2651
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: v2 camera can capture 1000fps videos ...

Fri May 03, 2019 3:51 pm

> I don't know if the picamera v2 would have this option before capturing.
>
To my knowledge not.

> So my question was for a PiCamera V2.
>
The v2 camera transmits raw Bayer data, either 8, 10, 12 or 16 bit. And the pixels receive one color only per 2x2 subarea (red/green/blue):
rg
gb

ov7251 monochrome global shutter camera does sends 8 or 10 bit grey pixels:
https://www.ovt.com/sensors/OV7251
gg
gg

So in case of same bit depth (8 or 10 for v2/ov7251) the transferred data size is identical.
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8918
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: v2 camera can capture 1000fps videos ...

Fri May 03, 2019 5:15 pm

As Hermann says, the sensor is not a full RGB array, it has a Bayer colour filter on the front (see https://en.wikipedia.org/wiki/Bayer_filter). Image processing can convert that into a full RGB image. They also generally produce a 10bit/pixel value, as the image processing requires a little more precision to avoid losing detail in the conversion. (Some more recent sensors produce 12 or 14bits/pixel for greater colour accuracy).

It is possible to get sensors without a Bayer CFA, but none of the standard Pi cameras work that way.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

hojo
Posts: 11
Joined: Mon Apr 18, 2016 3:54 pm

Re: v2 camera can capture 1000fps videos ...

Sun Jun 30, 2019 10:32 pm

This is all very exciting. I just spent quite some time looking for the I2C information for GoPro, wanting to figure out how to inject my own time code (there is an existing solution one can buy, of course).

But, since my solution doesn't need gopro or even that high of resolution (and the time code isn't for syncing multiple video sources), I am wondering how I can either code or embed absolute timestamps into 100fps video. Meaning, read in system time (synced with chrony against gps clock) and code unixEpoch.mills into each frame.

Any thoughts on this? I have never touched the camera stuff on the pi so this is brand new to me.

User avatar
HermannSW
Posts: 2651
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: v2 camera can capture 1000fps videos ...

Sun Jun 30, 2019 10:46 pm

You can capture Raspberry v2 camera with up to 120fps (limit is 200fps) as .h264 video with raspivid. Using -pts option you will get a file of microsecond reolution timestamps for each and every frame. Using mkvmerge you can merge the .h264 video and timestamps file into a video where each frame knows exactly its timestamp.Then you can use a gstreamer pipeline with timeoverlay to get what you want. This is just one example how you can do that.
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

User avatar
HermannSW
Posts: 2651
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: v2 camera can capture 1000fps videos ...

Tue Jul 02, 2019 10:11 am

I forgot that I posted the exact steps needed in this posting:
https://www.raspberrypi.org/forums/view ... e#p1294557

For timestamp 8467.329 from .pts file:
Image
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

hojo
Posts: 11
Joined: Mon Apr 18, 2016 3:54 pm

Re: v2 camera can capture 1000fps videos ...

Tue Jul 02, 2019 9:32 pm

Sweet. I'll have to look at that. This gives me a place to start as cameras and video are not my forte in any way.

In short, my goal is to build a system for race timing where the camera acts as a backup timer. Any issues with times and we can go back to the video and see who cross the finish line, in which order, and at what time. The main timing module is a GPS disiplined clock with thousandths resolution. 100 fps against this time source would be perfect.

This is not a unique solution as cameras for race timing are quite common in disciplines like Track where a beam isn't as usable. So I want to build my own solution for the challenge, education, and to fit low profile/power criteria. I time races in the middle of nowhere with low power, zero network connectivity, and I have to pack into/out of some challenging terrain.

Thanks for the info!

User avatar
HermannSW
Posts: 2651
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: v2 camera can capture 1000fps videos ...

Thu Jul 04, 2019 6:52 am

> The main timing module is a GPS disiplined clock with thousandths resolution. 100 fps against this time source would be perfect.
>
Be aware that while you could do gstreamer timeoverlay in pipeline while recording, it will drop raspivid's framerate significantly.
That was the reason that I did the recording step first separately, and then in post processing adding the time overlay.
That way the timestamps are precise at µs resolution (begin of receiving frame from camera over CSI-2 bus).

This is frame for timestamp for 8472.861 and shows (correct) rounding at millisecond, should be good enough for your application:
Image


This is animation created from youtube video "Spinning coin camera traversal at 200fps with millisecond gstreamer timeoverlay":
https://www.youtube.com/watch?v=OFzb4U6 ... &index=144
Image
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

Skeltek
Posts: 2
Joined: Wed Jul 17, 2019 11:03 pm

Re: v2 camera can capture 1000fps videos ...

Fri Jul 19, 2019 2:30 pm

This reminds me of my own hacks during university, when I wrote an arduino program outputting the first 5000 prime numbers within 0.27 seconds, when everyone else still believed 3,7 seconds was a nearly unbeatable record for the computation.
I will have a look at your work in more detail once I have time and maybe get to do similar things myself. I just bought my first Raspberry Cam few days ago and still got to get my knowledge up.
Just felt it being important to compliment you on your awesome work. Its rare to see someone else go so deep into some area of expertise just because its fun.

User avatar
HermannSW
Posts: 2651
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: v2 camera can capture 1000fps videos ...

Fri Sep 13, 2019 10:02 pm

1007fps is maximal Raspberry v2 camera framerate with raspiraw.
That is not the maximal framerate with Raspberry cameras, but for v2 camera.
Only v1 camera has global reset feature and allows for global (external) shutter capturing.

In this posting it is described how a 20000eps frame (exposures per second) was captured:
https://www.raspberrypi.org/forums/view ... 2#p1533932

Just completed to generate a 20000fps video from new 20000eps frame with white marker (7µs flashes at 20KHz frequency).
Will post description, tools, .... tomorrow in the other thread.
The animation here just demonstrates that much more than 1007fps can be captured with Raspberry v1 camera (and post processing).

20000fps video, played at 20fps, 1000× slower than real. White marker at blade tip rotates with 49.7m/s or 179km/h:
Image


P.S:
A tool is used to find a cover of the fast moving part, stamp the different white marker positions out and overlay them onto an "empty" image of the scene (created from 20000eps frame with gimp's "Filters->Enhance->Heal Selection"):
Image
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

chita
Posts: 3
Joined: Sun Jan 21, 2018 6:22 am

Re: v2 camera can capture 1000fps videos ...

Thu Apr 30, 2020 5:13 pm

Great Work. Just came across New Pi Camera. Excited to know it s capability in terms of high speed photography.

New product: Raspberry Pi High Quality Camera on sale now at $50
https://www.raspberrypi.org/blog/new-pr ... now-at-50/

hk.kim
Posts: 7
Joined: Tue May 19, 2020 8:32 am

Re: v2 camera can capture 1000fps videos ...

Wed May 20, 2020 12:39 pm

Hello.

It sounds very interesting for me.
Can I ask something further?
Can we capture videos with 2000fps using smaller region of images? such as 320*75 pixels?

I'm looking forward to your answer.

User avatar
HermannSW
Posts: 2651
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: v2 camera can capture 1000fps videos ...

Wed May 20, 2020 12:43 pm

hk.kim wrote:
Wed May 20, 2020 12:39 pm
Can we capture videos with 2000fps using smaller region of images? such as 320*75 pixels?
Definitely not with v1 camera.
I have not looked into raspiraw wrt HQ camera yet.

And with v2 camera I did always hit 1007fps barrier, regardless how small the frame was.
For going above 1007fps some other registers need to be changed, but I was not able to figure out how.

Two postings ago I did show a 20,000fps video:
viewtopic.php?f=43&t=212518&p=1663668#p1535779
Those are only possible with flash strobe technique and post processing as described.
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

carbotaniuman
Posts: 4
Joined: Wed May 20, 2020 10:26 pm

Re: v2 camera can capture 1000fps videos ...

Wed May 20, 2020 10:39 pm

How can I do ~200 fps on Pi Camera v2? What is the limitation for going higher?

carbotaniuman
Posts: 4
Joined: Wed May 20, 2020 10:26 pm

Re: v2 camera can capture 1000fps videos ...

Thu May 21, 2020 1:13 am

It looks like there's several raspicam forks for doing what I want, but the official docs also appear to say I can run at 200fps? Are the forks merged into upstream now?/

User avatar
HermannSW
Posts: 2651
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: v2 camera can capture 1000fps videos ...

Thu May 21, 2020 7:05 am

carbotaniuman wrote:
Thu May 21, 2020 1:13 am
but the official docs also appear to say I can run at 200fps? Are the forks merged into upstream now?/
raspivid is able to capture with 40-200fps since even before it has been documented 2 years ago:
viewtopic.php?f=43&t=206047&p=1430747&h ... s#p1336355

While you can ask for 200fps, since imx219 sensor is only capable for ≈160fps on average in mode7 (640x480), you will have frameskips with requesting more than that. Important for you to know what happened is to use "-pts tst.pts" option of raspivid. That writes millisecond timestamp for each frame captured, with microsecond precision ! You can use ptsanalyze tool for frameskip and delay analysis. It will report majority as well as average framerate (they are the same in case of no frameskips):
https://github.com/Hermann-SW/userland/ ... ptsanalyze

I experienced that you can safely capture with up to 150fps with zero frameskips typically, see this thread:
viewtopic.php?f=43&t=262032&p=1595689&h ... s#p1595689

In the posting I experienced a scene I would not have classified as that complex.
It did only allow for 100fps with 0 frameskips, so it all depends:
Image
I created slowmo animation playing at 25fps, 4× slower than real from the recorded .h264 file using ffmpeg.

You can find a v2 200fps recording slowmo of spinning coin in "coins" section of my "Raspberry camera gallery", and much much more, all with links to how the capturings were done:
https://github.com/Hermann-SW/Raspberry ... ra-gallery
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

hk.kim
Posts: 7
Joined: Tue May 19, 2020 8:32 am

Re: v2 camera can capture 1000fps videos ...

Fri May 22, 2020 7:37 am

HermannSW wrote:
Sat Jan 12, 2019 8:48 pm
I still have found no way to get more than 1007fps with v2 camera.

Today I checked v1 camera. As the diagram showed v1 camera allows for 640x64@665fps. 640x32 tool allows for 750fps.
I went down with height captured as possible. Height 16 or below does not work anymore. So I used new 640x17 tool derived from 640x32 by just changing height and fps.

Requesting 1200fps is ridiculous because there are more frameskips than frames at target framerate. But what this little experiment shows is that v1 camera has no just above 1000fps hard framerate limit like the v2 camera (1007fps). v1 "maximal framerate" (reported wrong by script) is 1,000,000/825=1212fps! Not really useful because of way too many frameskips, but maybe doing 320xH captures instead of 640xH can get rid of the many frame skips:

Code: Select all

pi@raspberrypi3BplusX:~/fork-raspiraw/c $ ./640x17 4000
removing /dev/shm/out.*.raw
capturing frames for 4000ms with 1200fps requested
2667 frames were captured at 605fps
frame delta time[us] distribution
      1 
      1 823
     33 824
    317 825
    288 826
     31 827
      1 828
      1 1647
     40 1649
    238 1650
   1332 1651
    168 1652
...
It is very interesting for me that we can get more than 1000fps using V2 cameras.
I just wonder why more than 1007 fps can not be obtained, why lots of frameskips occured, and how did you calculated "maximum framerate".
Can I ask you about that?

User avatar
HermannSW
Posts: 2651
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: v2 camera can capture 1000fps videos ...

Fri May 22, 2020 9:49 am

hk.kim wrote:
Fri May 22, 2020 7:37 am
It is very interesting for me that we can get more than 1000fps using V2 cameras.
1007fps, not more.
I just wonder why more than 1007 fps can not be obtained, why lots of frameskips occured, and how did you calculated "maximum framerate".
Can I ask you about that?
I did reduce the height further in the tools (640x64, ...) having the potential for more than 1007fps seen with 640x75, but always the reported (majority) framerate was 1007fps. Majority framerate takes frame delta that is maximally seen, and divides 1,000,000µs by that frame delta.

See the formulas in this posting:
viewtopic.php?f=43&t=274599&p=1663713#p1663713
If the 1007fps barrier could be overcome, 640x64 would only capture at 1255fps, not that much more.
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

carbotaniuman
Posts: 4
Joined: Wed May 20, 2020 10:26 pm

Re: v2 camera can capture 1000fps videos ...

Fri May 22, 2020 2:45 pm

HermannSW wrote:
Fri May 22, 2020 9:49 am
hk.kim wrote:
Fri May 22, 2020 7:37 am
It is very interesting for me that we can get more than 1000fps using V2 cameras.
1007fps, not more.
I just wonder why more than 1007 fps can not be obtained, why lots of frameskips occured, and how did you calculated "maximum framerate".
Can I ask you about that?
I did reduce the height further in the tools (640x64, ...) having the potential for more than 1007fps seen with 640x75, but always the reported (majority) framerate was 1007fps. Majority framerate takes frame delta that is maximally seen, and divides 1,000,000µs by that frame delta.

See the formulas in this posting:
viewtopic.php?f=43&t=274599&p=1663713#p1663713
If the 1007fps barrier could be overcome, 640x64 would only capture at 1255fps, not that much more.
Is the 640x480 (realtime) limitation because of the 2x MIPI lanes, or because of a GPU limitation?

User avatar
HermannSW
Posts: 2651
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: v2 camera can capture 1000fps videos ...

Fri May 22, 2020 4:29 pm

carbotaniuman wrote:
Fri May 22, 2020 2:45 pm
Is the 640x480 (realtime) limitation because of the 2x MIPI lanes, or because of a GPU limitation?
The deal is:
half vertical resolution, get (nearly) double the framerate
So the bandwidth for all the 640xH modes is roughly the same and easily fits into the 2 CSI2 lanes.

With more than 200fps the only solutions is raspiraw, and that does not use the GPU at all.
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

carbotaniuman
Posts: 4
Joined: Wed May 20, 2020 10:26 pm

Re: v2 camera can capture 1000fps videos ...

Sun May 24, 2020 7:02 am

HermannSW wrote:
Fri May 22, 2020 4:29 pm
carbotaniuman wrote:
Fri May 22, 2020 2:45 pm
Is the 640x480 (realtime) limitation because of the 2x MIPI lanes, or because of a GPU limitation?
The deal is:
half vertical resolution, get (nearly) double the framerate
So the bandwidth for all the 640xH modes is roughly the same and easily fits into the 2 CSI2 lanes.

With more than 200fps the only solutions is raspiraw, and that does not use the GPU at all.
What is the reason for you saying frameskips occur at 150fps, when the datasheet says the camera can go up to 180fps at higher reses?

I'm getting the same behavior, but it's not matching the datasheet at all and I'm kind of stumped.

User avatar
HermannSW
Posts: 2651
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: v2 camera can capture 1000fps videos ...

Sun May 24, 2020 8:13 am

carbotaniuman wrote:
Sun May 24, 2020 7:02 am
What is the reason for you saying frameskips occur at 150fps,
If you request 160fps with raspivid and use "-pts tst.pts", then analysis with ptsanalyze always shows frameskips:
https://github.com/Hermann-SW/userland/ ... ptsanalyze
when the datasheet says the camera can go up to 180fps at higher reses?

I'm getting the same behavior, but it's not matching the datasheet at all and I'm kind of stumped.
Page 82 of imx219 datasheet says 1408x792@180fps is only with 4 lanes, Pi has only two lanes:
https://github.com/rellimmot/Sony-IMX21 ... et_XXX.PDF
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

chita
Posts: 3
Joined: Sun Jan 21, 2018 6:22 am

Re: v2 camera can capture 1000fps videos ...

Wed Jun 03, 2020 12:18 pm

New product: Raspberry Pi High-Quality Camera.

Will it be possible to achieve a better resolution?

Are you guys excited?
https://www.raspberrypi.org/blog/new-pr ... now-at-50/

User avatar
HermannSW
Posts: 2651
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: v2 camera can capture 1000fps videos ...

Wed Jun 03, 2020 4:28 pm

chita wrote:
Wed Jun 03, 2020 12:18 pm
New product: Raspberry Pi High-Quality Camera.

Will it be possible to achieve a better resolution?
Yes, but that camera is not made for high framerate capturing:
viewtopic.php?f=43&t=109137&p=1666035#p1666035
Capturing 1012x128 was possible with majority framerate 358fps, but average framerate was only 190fps.
HQ camera cannot compete with v1/v2 cameras wrt high framerates (665fps/1007fps).

But HQ camera can do long exposures up to 239s(!).
Either by building userland yourself and using its raspistiil:
viewtopic.php?f=43&t=273358&p=1672332#p1672332
Or with raspiraw, completely without GPU:
viewtopic.php?f=43&t=109137&p=1671859#p1670818

Here exactly one 8x8 matrix display led lights for 4s, from top row to bottom, from left to right.
That loops after 8*8*4s=256s. The animation was created by 3 consecutive 237s long exposures.
256s-237s>16s, so always 4 leds do not light (room was dark, only the tiny leds provided light):
Image
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

Return to “Camera board”