HermannSW
Posts: 436
Joined: Fri Jul 22, 2016 9:09 pm

High framerate raspivid (initially 640x480 at 180fps)

Wed Feb 07, 2018 9:47 am

raspivid is the default tool many people use to capture videos with Raspberry.

In thread raspivid: fpsₘₐₓ(v₂)=120, fpsₘₐₓ(v₁)=90 / 800x760: fps(v₂)=120, fps(v₁)=90 it was demonstrated, that while Raspberry documentation says v2 camera can do 90fps maximal, raspivid "-fps" option works fine until 120fps(!).

You can provide higher values, but the framerate gets capped at 120fps. @6by9 confirmed that this capping happens in closed source GPU firmware:
viewtopic.php?f=43&t=109137&p=1269504#p1269165

In very long raspiraw thread, this posting stated that @6by9 did merge all high framerate work into raspiraw master branch:
viewtopic.php?f=43&t=109137&start=375#p1266319

The high framerate work included capturing 640x128_s frames at 665fps without frame skips for v1 camera, and all the standard v2 camera modes after I captured and decoded "raspivid" I2C commands from Pi to v2 camera (under the guidance of @6by9). Although not all high framerate features work for v2 camera yet, it was possible to capture 640x480 frames at 240fps with raspiraw.


Yesterday I wanted to try out with how high framerate 640x480 frames can be captured with v2 camera and raspivid.
Because the relevant I2C commands to v2 camera are sent from GPU closed source code this was a challenge.
So I did first time "logic analyzer assisted programing":
Image

I started by syncing @6by9's userland repo and inserted "sleep(1)" in various places in "host_applications/linux/apps/raspicam/RaspiVid.c" and comipled (without overwriting installed raspivid et al, initial build takes 6mins, this is incremental build):

Code: Select all

$ time ( ./buildme dummy )
...
-- Up-to-date: dummy/opt/vc/bin/dtmerge
-- Up-to-date: dummy/opt/vc/lib/libdtovl.so
~/userland/t/userland

real	0m49.253s
user	1m19.500s
sys	0m20.780s
$

Unfortunately the location I need to emit my own I2C command to set framerate (v2 FRM_LENGTH_A register) was inside main loop of raspivid. Even worse, I have not found a signal from GPU code on when GPU is done with sending I2C commands to camera and started capturing video. Because of that I had to usleep(200000) to get into the right spot in time.

Yes, this is a (initial) cruel hack of raspivid -- but with it I can capture 640x480 at 180fps(!!).
More importantly, this RaspiVid.c code location allows to port other raspiraw high framerate features to raspivid (--height, --vinc) allowing for 640x240 capturing at 360(?)fps and 640x120 capturing at 720(?)fps ... with full GPU support.
Because this is a hack, I wanted to have everything in just one place (including needed additional includes and defines):

Code: Select all

pi@raspberrypi2B:~/userland/t/userland $ !diff
diff -c host_applications/linux/apps/raspicam/RaspiVid.c*
*** host_applications/linux/apps/raspicam/RaspiVid.c	2018-02-07 02:27:36.751556483 +0100
--- host_applications/linux/apps/raspicam/RaspiVid.c.orig	2018-02-07 00:32:11.142714109 +0100
***************
*** 2908,2971 ****
                       // How to handle?
                    }
  
- /**
-  * raspiraw high framerate stuff here
-  *
-  * initial cruel hack, but raspivid 640x480 up to 180fps(!) w/o frameskips
-  *
-  * for now
-  * - only for v2 camera
-  * - only fps option
-  * - no need to do anything up to 120fps, works already
-  */
- if (state.bCapturing && (state.framerate > 120))
- {
-   #include <fcntl.h>
-   #include <sys/ioctl.h>
-   #include <unistd.h>
- 
-   #define I2C_DEVICE_NAME_LEN 13  // "/dev/i2c-XXX"+NULL
-   #define I2C_SLAVE_FORCE 0x0706
- 
-   #define WRITE_I2C(fd, str)  (write(fd, str, sizeof(str)) != sizeof(str))
- 
-   int fd;
-   char i2c_device_name[I2C_DEVICE_NAME_LEN];
- 
-   if (state.verbose)  fprintf(stderr,"Now raspiraw fps ...\n");
- 
-   // I have no better idea yet how to get past GPU emitted I2C commands
-   usleep(200000);
- 
-   snprintf(i2c_device_name, sizeof(i2c_device_name), "/dev/i2c-%d", state.cameraNum);
- 
-   fd = open(i2c_device_name, O_RDWR);
-   if (!fd)
-   {
-     vcos_log_error("Couldn't open I2C device");
-     goto error;
-   }
-   if (ioctl(fd, I2C_SLAVE_FORCE, 0x10/*imx219*/) < 0)
-   {
-     vcos_log_error("Failed to set I2C address");
-     goto error;
-   }
- 
-   // values taken from imx219_modes[]
-   {
-     unsigned line_time_ns = (state.sensor_mode < 6) ? 18904 : 19517;
-     unsigned val = 1000000000 / (line_time_ns * state.framerate);
-     unsigned char msg[] = {0x01, 0x60, val>>8, val&0xFF};
-     if ( WRITE_I2C(fd, msg) )
-     {
-       vcos_log_error("Failed to write register FRM_LENGTH_A\n");
-       goto error;
-     }
-   }
-   close(fd);
-   if (state.verbose)  fprintf(stderr,"... raspiraw fps done.\n");
- }
- 
                    // In circular buffer mode, exit and save the buffer (make sure we do this after having paused the capture
                    if(state.bCircularBuffer && !state.bCapturing)
                    {
--- 2908,2913 ----
pi@raspberrypi2B:~/userland/t/userland $ 

So what does this code do?
Here is tail of decoded I2C capture from the execution of raspivid command shown further below:

Code: Select all

$ tail 180.csv 
1.954188500000000,I2C,0x00 + ACK
1.985222500000000,I2C,Setup Write to [0x20] + ACK
1.985312500000000,I2C,0x01 + ACK
1.985402500000000,I2C,0x60 + ACK
1.985492500000000,I2C,0x01 + ACK
1.985582500000000,I2C,0x1C + ACK
5.032943500000000,I2C,Setup Write to [0x20] + ACK
5.033033500000000,I2C,0x01 + ACK
5.033123500000000,I2C,0x00 + ACK
5.033213500000000,I2C,0x00 + ACK
$ 
Because of the usleep(200000) my inserted I2C command "reg(0x0160)=0x011C" happens at timestamp 1.9852225s, a bit after last of the last GPU emitted I2C command at 1.9541885s, and 3s before video capturing stops at 5.0329435s.

Here is the raspivid command I used, with verbose output, so that you can see the messages emitted from above new code piece as well:

Code: Select all

$ ./build/bin/raspivid -v -md 7 -fps 180 -pts 180.pts -w 640 -h 480 -o t.h264 -t 3000 -ex off -ss 1000 -ag 1.0 -dg 1.0

raspivid Camera App v1.3.12

Width 640, Height 480, filename t.h264
bitrate 17000000, framerate 180, time delay 3000
H264 Profile high
H264 Level 4
H264 Quantisation level 0, Inline headers No
H264 Intra refresh type (null), period -1
Wait method : Simple capture
Initial state 'record'


Preview Yes, Full screen Yes
Preview window 0,0,1024,768
Opacity 255
Sharpness 0, Contrast 0, Brightness 50
Saturation 0, ISO 0, Video Stabilisation No, Exposure compensation 0
Exposure Mode 'off', AWB Mode 'auto', Image Effect 'none'
Flicker Avoid Mode 'off'
Metering Mode 'average', Colour Effect Enabled No with U = 128, V = 128
Rotation 0, hflip No, vflip No
ROI x 0.000000, y 0.000000, w 1.000000 h 1.000000
Camera component done
Encoder component done
Starting component connection stage
Connecting camera preview port to preview input port
Starting video preview
Connecting camera video port to encoder input port
Opening output file "t.h264"
Opening output file "180.pts"
Enabling encoder output port
Now raspiraw fps ...
... raspiraw fps done.
Starting video capture
Finished capture
Closing down
Close down completed, all components disconnected, disabled and destroyed

$

Before we look at the video captured, lets see frame skip+delata analysis.

First the bottom part tells us that there were 6 frames with deltas not matching 180fps. But that is OK, those are frames 2-7 of the video, taken before the new I2C command was executed!

Code: Select all

...
after skip frame indices (middle column)
8296,8296
8296,16592
8296,24888
8296,33184
8296,41480
8295,49775
1% frame skips
$ 



The initial part of the analysis shows that frame deltas are 5531μs ± 13μs (180fps):

Code: Select all

$ echo "1000000/5531" | bc -ql
180.79913216416561200506
$ ./ptsanalyze 180.pts 0-46-9
creating tstamp.csv
545 tstamps.csv frames were captured at 180fps
frame delta time[us] distribution
      1 
      1 5518
      2 5521
      1 5525
      1 5526
      1 5527
      3 5528
     30 5529
    213 5530
    236 5531
     39 5532
      5 5533
      1 5534
      1 5535
      1 5536
      2 5540
      1 5544
      1 8295
      5 8296
after skip frame indices (middle column)
...



Summary:
Not looking at the initial 7 frames no frame skips at 180fps!
Here you can find the taken t.h264, uploaded to youtube:
https://www.youtube.com/watch?v=9XdcxUD ... e=youtu.be

Since youtube default framerate is 25fps, the video is played 180/25=7.2 times slowed down. Therefore video length becomes 21s on youtube although only 3s got captured (at 180fps).
I used an eggbeater as "high framerate gadget" this time:
Image
--> Raspberry camera / gstreamer / raspivid / raspiraw (bookmark list):
https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/raspiraw      https://github.com/Hermann-SW/userland      https://twitter.com/HermannSW

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4817
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: High framerate raspivid (initially 640x480 at 180fps)

Wed Feb 07, 2018 10:15 am

If AE/AGC is running, then there is never a safe time to inject additional I2C commands.

A quick test has given me 200fps from raspivid, but with many frame drops. That's mainly as the tuners have to complete within one frame period, but reality is that they're somewhere around 7-10ms depending on exactly what is enabled.

Things lock up above 202 fps for unclear reasons - the sensor produces no data. Initial guess would be that some calculation has overflowed and stuffed a daft number into a register somewhere and slowed everything down. The other possibility is that the codec is throwing a wobbler as it has to kick up to level 4.2 to cope with > 62,914,560 pixels/sec, but that doesn't happen until 204.8fps.

I really haven't got time at the moment to investigate. I'll bump the limit up to 200fps, and leave myself an open issue to investigate why we can't go above 202fps.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4817
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: High framerate raspivid (initially 640x480 at 180fps)

Wed Feb 07, 2018 10:23 am

Just tried 180fps, and I'm getting a drop every 4-6 frames. That's probably why the person who integrated the driver played it safe with 120fps as the maximum.
You could try overclocking the VPU to see if you can get reliable results at these higher rates.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

HermannSW
Posts: 436
Joined: Fri Jul 22, 2016 9:09 pm

Re: High framerate raspivid (initially 640x480 at 180fps)

Wed Feb 07, 2018 10:28 am

6by9 wrote:
Wed Feb 07, 2018 10:23 am
Just tried 180fps, and I'm getting a drop every 4-6 frames. That's probably why the person who integrated the driver played it safe with 120fps as the maximum.
You could try overclocking the VPU to see if you can get reliable results at these higher rates.
That is the reason I copied my exact command line into above posting:

Code: Select all

$ ./build/bin/raspivid -v -md 7 -fps 180 -pts 180.pts -w 640 -h 480 -o t.h264 -t 3000 -ex off -ss 1000 -ag 1.0 -dg 1.0
I did try "$ ./build/bin/raspivid -v -md 7 -fps 180 -pts 180.pts -w 640 -h 480 -o t.h264 -t 3000" as well, and it worked after I increased usleep() time to 1.4s.

So some of the "-ex off -ss 1000 -ag 1.0 -dg 1.0" options I have from previous postings of you have a positive effect on no frame skips at 180fps ...

P.S:
Different to raspiraw high framerate work where storing into ramdisk was needed to achieve high frame rates, here storing t.h264 on SD card is no problem because auf GPU support(!)
--> Raspberry camera / gstreamer / raspivid / raspiraw (bookmark list):
https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/raspiraw      https://github.com/Hermann-SW/userland      https://twitter.com/HermannSW

HermannSW
Posts: 436
Joined: Fri Jul 22, 2016 9:09 pm

Re: High framerate raspivid (initially 640x480 at 180fps)

Thu Feb 08, 2018 1:18 am

I did another eggbeater high framerate video, this time with shutter time 5000μs instead of 1000μs last time, much brighter video!
https://www.youtube.com/watch?v=T8VvjJJ ... e=youtu.be
Image
Played with youtube default framerate 25fps, the 180fps video is played 7.2 times slower than real. Frame deltas are 5531μs ± 13μs (see analysis below).

And I did change the location in code where the raspiraw high framerate I2C injection happens now, in the callback that writes each buffer retrieved from GPU to disk. That is a much better place.

I did a fork from @6by9's userland repo and comitted my changes. In addition to changes to RaspiVid.c, I added selfcontained and adapting frame delay+skip analysis tool as well. If you want to redo below commands, then just clone my userland fork, then "cd userland", and finally "time (./buildme dummy)" to build all. I am only interested in "raspvid" and don't want to overwrite existing raspivid, therefore the "dummy" argument for destination directory. This is the fork:
https://github.com/Hermann-SW/userland


This is the command I executed to create above video, and the analysis command detailing the frame delta distribution and no skips:

Code: Select all

$ build/bin/raspivid -md 7 -fps 180 -stf 5 -pts 180.pts -w 640 -h 480 -o t.h264 -t 4000  -ex off -ss 5000
$ tools/ptsanalyze 180.pts 5
creating tstamps.csv
684 frames were captured at 180fps
frame delta time[us] distribution
      2 5518
      1 5524
      1 5525
      1 5527
      4 5528
     38 5529
    257 5530
    334 5531
     34 5532
      3 5533
      1 5534
      1 5535
      1 5536
      1 5537
      1 5543
      1 5544
after skip frame indices (middle column)
0 frame skips (0%)
$ 

Now if I let away both "-ex" and "-ss" options, only 121fps get achieved although 180fps were requested. As you can see the analysis tool does analysis for the identified framerate:

Code: Select all

$ build/bin/raspivid -md 7 -fps 180 -stf 5 -pts 180.pts -w 640 -h 480 -o t.h264 -t 4000 
$ tools/ptsanalyze 180.pts 5
creating tstamps.csv
461 frames were captured at 121fps
frame delta time[us] distribution
      1 8215
     11 8216
     88 8217
    278 8218
     77 8219
      2 8220
      1 8221
after skip frame indices (middle column)
0 frame skips (0%)
$ 

The new raspivid option "-stf" defines the startframe, that is the frame number of the frame, where the I2C command gets send. This start frame has to be passed as 2nd argument to analysis tool as well. It takes another two frames after start frame until the new framerate set by I2C command takes effect. The analysis is done excluding those initial frame deltas where requested framerate is not effective.
--> Raspberry camera / gstreamer / raspivid / raspiraw (bookmark list):
https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/raspiraw      https://github.com/Hermann-SW/userland      https://twitter.com/HermannSW

HermannSW
Posts: 436
Joined: Fri Jul 22, 2016 9:09 pm

Re: High framerate raspivid (initially 640x480 at 180fps)

Sat Feb 10, 2018 10:46 pm

I went a step further and tried to evaluate whether the other raspiraw high framerate options can be ported to raspivid as well, and how this all plays together with the GPU.

Today I did a long train ride and started with my mobile raspivid dev environment, see this thread for details:
Image
(Android smartphone is wireless access point, as well as activity generator with its stopwatch).

Despite the new option name (-vi) I exemplarily tried to restrict the vertical lines captured by v1 camera in order to increase framerate. For raspiraw going from 640x480 to x240 and x120 increased framerate from 90fps to 180fps to 360fps.

I did send 2nd I2C command besides manipulating framerate at 5th frame captured, which is after all GPU emitted I2C commdands are done. And raspivid did achieve 195fps that way. The restriction is because of the bug @6by9 mentioned (more than 202fps break raspivid capturing, otherwise it should be going direction 360fps as with raspiraw). This frame is from youtube video of that run:
https://www.youtube.com/watch?v=DJ03B1dtA6M
Image

Since the injected I2C commands get not sent before 5th frame, some full 640x480 frames get captured. What I learned is that v1 camera seems to have 3 planes/storage areas that get written cyclically. You can see this when single stepping youtube video (stop, then "."/"," for fore/back). The bottom 3 quartes of the frame repeat every 3rd frame. Only the top quarter gets updated.

The view is in train direction out of the window. The two beer glasses do not belong to me but the person on window seat [I admit that I did drink beer last evening ;-)].


Here you see top half (240 lines) updated only, also 640x480, this time at 150fps. This is much closer to 240 lines raspiraw framerate of 180fps.:
https://www.youtube.com/watch?v=cENEbxUe590
Image

Again the bottom half repeats every 3rd frame (the last three frames before injected I2C commands took effect and only top half gets captured).


From an earlier raspivid video with only updating top half at only 100fps, this is frame 4 before I2C command injection:
Image


This is frame 10 some frames after I2C commands injected took effect:
Image


As you can see there is train ceiling light at top left of frame. With higher framerate GPU seems to mix up colors. On the other hand the previous two youtube videos don't look that wrong from color perspective.

(both videos are played with youtube standard framerate 25fps, 7.8 times or 6 times slower than real)


So this prototypical high framerate vertical line count restriction feature works quit well.
But further work on this is blocked until the >202fps framerate bug in closed source GPU code has been fixed by @6by9.

Halving vertical lines made even 665fps possible for 640x64 with raspiraw and v1 camera.
The charme of porting raspiraw high framerate features to raspivid is, that you get a (.h264) video without need for any postprocessing as with raspiraw -.- postprocessing will be done by GPU!


P.S
This is command for capturing 2nd video with frame delay and skip analysi, not a single frane skip:

Code: Select all

pi@raspberrypi2B:~/userland $ build/bin/raspivid -md 7 -stf 5 -fps 150 -vi 240 -pts /dev/shm/180.pts -w 640 -h 480 -o /dev/shm/y.h264 -t 3000  -ex off -ss 6000
I2C injection done.
pi@raspberrypi2B:~/userland $ tools/ptsanalyze /dev/shm/180.pts 5
creating tstamps.csv
426 frames were captured at 150fps
frame delta time[us] distribution
      1 6640
      2 6641
     17 6644
    101 6645
    160 6646
     94 6647
     36 6648
      9 6649
      1 6650
      2 6652
after skip frame indices (middle column)
0 frame skips (0%)
pi@raspberrypi2B:~/userland $ 
P.P.S:
The GPU should be able to do this kind of high framerate processing. Every time number of lines captured is halved, framrate is at most doubled. Therefore GPU needs to be able to deal with VGA bitrate or lower only.
--> Raspberry camera / gstreamer / raspivid / raspiraw (bookmark list):
https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/raspiraw      https://github.com/Hermann-SW/userland      https://twitter.com/HermannSW

HermannSW
Posts: 436
Joined: Fri Jul 22, 2016 9:09 pm

Re: High framerate raspivid (initially 640x480 at 180fps)

Sun Feb 11, 2018 10:19 am

I did submit the prototyping changes for being able to recreate what I did/showed:
https://github.com/Hermann-SW/userland

This is from a raspivid capture of top half at 150fps today. It demonstrates better the the color changes that happen, best at the border inside red Lego piece. The bottom half of the frame was captured with "normal" framerate for mode 7, which is 60fps, and that has 16666 shutter time:
Image

The video was only possible by using flash light from Android smartphone, most likely because of shutter speed of only 6000. Normal call to raspivid with websocket browser tool works with normal room light, and shows the real colors of Lego pieces for comparison:
Image
--> Raspberry camera / gstreamer / raspivid / raspiraw (bookmark list):
https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/raspiraw      https://github.com/Hermann-SW/userland      https://twitter.com/HermannSW

HermannSW
Posts: 436
Joined: Fri Jul 22, 2016 9:09 pm

Re: High framerate raspivid (initially 640x480 at 180fps)

Wed Feb 14, 2018 8:52 am

Made progress on long train ride on Monday, convinced Raspberry GPU to update 3 quarters of 640x240 video frames at 150fps with v1 camera and modified raspivid ! (on left of frame is my arm, on the right lower half you can see window border, video taken while train was driving, through window, dev setup here)
Image

Was able to get full display update, but framerate dropped to 90fps. Yesterday I used logic analyzer to see what is going on, and it showed completely different I2C capture, although only one value of new "-ispy" option (for setting ov5647 register TIMING_ISP_Y_WIN) changed from 307 to 310. Since the code for this (and the I2C command sent) gets executed on frame 5 after GPU has started recording, I currently have no explanation how its value change can be responsible for a change in all previous I2C commands sent by GPU, needs more debugging ...
--> Raspberry camera / gstreamer / raspivid / raspiraw (bookmark list):
https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/raspiraw      https://github.com/Hermann-SW/userland      https://twitter.com/HermannSW

Return to “Camera board”

Who is online

Users browsing this forum: No registered users and 11 guests