drmatt
Posts: 14
Joined: Sat Sep 07, 2019 3:47 am

higher frame rates from v2 camera?

Sat Oct 19, 2019 4:14 am

Hello,

I'm wondering whether it's possible to capture more quickly than 40 fps with 2x2 binning using the v2 camera. Since 15 fps can be achieved at full resolution, then if CSI2 bandwidth is the limiting factor, shouldn't 60 fps (4x speedup) be achievable at half resolution? My understanding is that 1/4 of the full-frame pixels need to be transferred over CSI2 in mode 4 (1640x1232) vs. mode 2 (3280x2464).

There is a table in one of the IMX219 datasheets that gives example frame rates with binning, and it states 30 fps for full frame and 120 fps for x2binning. Presumably this is referring to the full 4-lane speed, but nonetheless it seems to imply that there should be a 4x speedup over full frame with x2binning in both X and Y.

(I originally posted to this thread but that topic is slightly different, as I'm interested in streaming video rather than stills).

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9001
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: higher frame rates from v2 camera?

Sat Oct 19, 2019 8:24 am

Binning modes are often limited by the electronics required to read the pixels as much as the csi2 link rate. Register sets are often provided by the sensor manufacturer with little supporting information for exactly what the limits are.

You could take raspiraw and try tweaking the register set to see if you can get a reliable higher frame rate. If so then report back and we can investigate amending the firmware accordingly. At present we are not intending to update the drivers at all.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
HermannSW
Posts: 2680
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: higher frame rates from v2 camera?

Sat Oct 19, 2019 11:17 am

6by9 wrote:
Sat Oct 19, 2019 8:24 am
If so then report back and we can investigate amending the firmware accordingly.
That offer of 6by9 is real, for a long time raspivid framerate was capped at 120fps for mode7 640x480. After I did show that up to 200fps were possible (with I2C command injection, just to overcome the GPU capping), 6by9 did increase v2 mode7 framerate max to 200fps. Official documentation was adjusted for that as well:
https://www.raspberrypi.org/forums/view ... ?p=1285069

So if you can enhance the framerate for mode4 (eg. with raspiraw), then please do.
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

drmatt
Posts: 14
Joined: Sat Sep 07, 2019 3:47 am

Re: higher frame rates from v2 camera?

Sat Oct 19, 2019 5:26 pm

HermannSW wrote:
Sat Oct 19, 2019 11:17 am
That offer of 6by9 is real
I believe it. I've followed many forum posts from the two of you on related topics, you're doing fantastic work.

I've started tinkering with raspiraw and studying the datasheets that I've been able to find for exactly this. My particular use case is likely to require the 10-bit raw pixel data, so I may have to stay in the raspiraw domain, but it would be great if migrating speed improvements into the firmware could benefit others. I'll continue work and will be sure to report back if I'm able to make any headway - it's a fairly steep learning curve and documentation is spotty, so it may take some time. Just wanted to see first if there was a fundamental reason that would prevent faster rates.

Thanks!

User avatar
HermannSW
Posts: 2680
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: higher frame rates from v2 camera?

Sat Oct 19, 2019 8:36 pm

For getting started with raspiraw Robert Elder did a good blog post and associated youtube videos:
https://blog.robertelder.org/recording- ... pi-camera/
He did not use nor mention the tools in raspiraw/tools directory.

In addition to the mostly v1 camera tools, you can find many v2 camera tools in this posting's code drop:
https://www.raspberrypi.org/forums/view ... 4#p1320044
They were used to generate these framerate curves for 640xH frames:
Image
https://stamm-wilbrandt.de/en/Raspberry_camera.html
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/raspiraw
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264

drmatt
Posts: 14
Joined: Sat Sep 07, 2019 3:47 am

Re: higher frame rates from v2 camera?

Sun Oct 20, 2019 3:45 am

Great, thank you, those tools will be useful for analysis. Fortunately I'm up and running with raspiraw itself and I'm comfortable with building/using it and associated utilities. The main task ahead is better understanding the IMX219 timings and corresponding register settings.

drmatt
Posts: 14
Joined: Sat Sep 07, 2019 3:47 am

Re: higher frame rates from v2 camera?

Sun Oct 20, 2019 8:00 am

Quick update: I started by searching for any other imx219 driver source code I could find and studied the differences between their register sets and those of raspiraw. For the most part, they were identical. However, several of the other drivers differed in registers 0x0307 (PLL_VT_MPY) and 0x030d (PLL_OP_MPY), the multipliers on PLL video timing system and PLL output system, respectively.

Interestingly, the PLL_VT_MPY value differed by a factor of about 1.5, which is also the mystery factor of 60 fps to 40 fps. I modified mode4 in raspiraw/imx219_modes.h to change the values of those two registers and the corresponding line_time_ns, and was able to achieve just over 60 fps with no frame skips (according to tstamps.csv).

Here's the diff:

Code: Select all

diff --git a/imx219_modes.h b/imx219_modes.h
index fffbb80..bda6682 100644
--- a/imx219_modes.h
+++ b/imx219_modes.h
@@ -346,11 +346,13 @@ struct sensor_regs imx219_mode4[] =
       {0x0304, 0x03},
       {0x0305, 0x03},
       {0x0306, 0x00},
-      {0x0307, 0x39},
+      {0x0307, 0x57},  // this is from the other lib
+      //{0x0307, 0x39},  // this is the original
       {0x0309, 0x0a},
       {0x030b, 0x01},
       {0x030c, 0x00},
-      {0x030d, 0x72},
+      {0x030d, 0x5a},  // this is from the other lib
+      //{0x030d, 0x72},  // this is the original
       {0x455e, 0x00},
       {0x471e, 0x4b},
       {0x4767, 0x0f},
@@ -643,7 +645,8 @@ struct mode_def imx219_modes[] = {
       .image_id      = 0x2B,
       .data_lanes    = 2,
       .min_vts       = 1236,
-      .line_time_ns  = 18904,
+      .line_time_ns  = 12385,  // this is from the other lib
+      //.line_time_ns  = 18904,  // this is the original
       .timing        = {0, 0, 0, 0, 0},
       .term          = {0, 0},
       .black_level   = 66,
I also looked at a few of the 10-bit images that were written with these changes and compared them to full-resolution images, generated with unmodified registers and the same exposure time. The images looked good visually, and the overall pixel brightness was very similar across the two, which gives some hope that the line_time_ns setting is close to correct.

Unfortunately I don't have any insight into the implications of these changes, nor whether this is the right way to achieve the predicted frame rate. The other drivers use these multipliers in all modes, not just half-res mode, but when I tried them in raspicam with mode2, the full-frame images were garbled. Then again, those other drivers use 4x CSI2 data lanes (register 0x0114), while we only have 2x lanes available on the stock pi, so maybe that's a factor.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9001
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: higher frame rates from v2 camera?

Sun Oct 20, 2019 1:16 pm

Sorry, pll updates aren't something that cam realisitcally be merged into the firmware. Resetting pll settings on mode switches will cause issues whilst they resync, and those settings have been checked by Sony.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

drmatt
Posts: 14
Joined: Sat Sep 07, 2019 3:47 am

Re: higher frame rates from v2 camera?

Sun Oct 20, 2019 4:14 pm

6by9 wrote:
Sun Oct 20, 2019 1:16 pm
Sorry, pll updates aren't something that cam realisitcally be merged into the firmware. Resetting pll settings on mode switches will cause issues whilst they resync, and those settings have been checked by Sony.
I see, thanks. To clarify, is the objection more with having different pll settings across modes, or with changing any of the pll settings at all? (or perhaps both?) Can you provide any guidance into which registers / settings are off limits for this exercise, and any other constraints that we should be aware of?

drmatt
Posts: 14
Joined: Sat Sep 07, 2019 3:47 am

Re: higher frame rates from v2 camera?

Mon Oct 21, 2019 6:58 pm

Following up with some rough calculations and observations.

To achieve 60 fps in mode 4, we would require a line rate of at least 60*1232 lines per second, which amounts to a maximum of ~13.5 us per line, as compared with the current setting of 18.9 us per line.

According to the datasheet I've seen, the time per line can be calculated as

line_time_sec = line_length_pix / (2*pix_clock_freq_hz)

This makes intuitive sense: to decrease the line time, we can either increase the pixel clock frequency, which is what I described in my previous post, or we can decrease the line length.

However, there are some inconsistencies in the documentation as to how "line length" and "line time" are used in various contexts. The datasheet states - and reading actual register values confirms - that the "line_length_pck" in registers 0x0162 and 0x0163 is set to its fixed minimum value of 3448, so there is no way to decrease that. But evidently that value is not always the one that determines the overall line rate.

I can see how a partial / cropped fov could decrease the effective line length because fewer pixels need to be touched per line, which perhaps is how much higher frame rates can be achieved in the lower-resolution sensor modes. Similarly, I had thought that the effective line length in mode 4 would be halved due to the horizontal binning, but so far it appears that any mode using the full fov width, even binned, still operates at the same line rate.

Also, the raspiraw code computes the number of lines (vts or frame_length_lines for registers 0x0160 and 0x0161) as:

Code: Select all

int n = 1000000000 / (sensor_mode->line_time_ns * cfg.fps);
This seems a bit strange for the faster modes, because for instance in mode 6, which has 720 lines and a published max rate of 90 fps, this calculation implies that only about 70 fps is achievable; plugging in 90 fps above, frame_length_lines would only be 570 rather than 720.

In any case, it appears that options are limited for which registers can be modified and actually make an impact in mode 4. Perhaps the only means is to change clock multipliers, and if so, it sounds like a non-starter as far as the firmware goes. But any insight is appreciated.

Return to “Camera board”