Raw sensor access / CSI-2 receiver peripheral

[PierreG]

Hi everybody!
I'm thinking about interfacing another sony imx sensor (224) with Rpi3.
What is the requirements to get raw image with raspiraw?
1) Design/use a sensor board
2) Connect i2c/clock/CSI between sensor board and rpi
3) Define the sensor registers and mode in *_modes.h file
4) run raspiraw, debug and pray

Did someone try to connect another sensor than imx219 and ov5647?

Thanks a lot!

[6by9]

....
To get 8 bit data through you need to either set the sensor to produce only 8 bits (and reset rawcam etc to expect the new data type value and packing), or you could run 10 bit and then set expsoure time and/or analogue gain to result in only the bottom 8 bits being used (likely to give worse image quality as all the analogue stages are then closer to the noise floor).

I tried to go along the 1st way by setting following IMX219 registers: 0x0155 COMP_ENABLE_A(compression 10 to 8 mode), 0x018C 0x018D CSI_DATA_FORMAT_A

Code: Select all

 
{0x0155, 0x01}
{0x018C, 0x08}
{0x018D, 0x08}

+
rx_cfg.unpack = MMAL_CAMERA_RX_CONFIG_UNPACK_NONE;
rx_cfg.pack = MMAL_CAMERA_RX_CONFIG_PACK_NONE;
output->format->encoding = MMAL_ENCODING_BAYER_SBGGR8

But no luck. Output data is little bit messy and it looks like pixels were shifted. I suspect that it is due to different output method used by the sensor for 8 bit mode

0x0155 enables DPCM 10 to 8 compression which is not what you want - see Annex E "Data Compression for RAW Data Types" in the CSI2 spec for details of what that is doing. The hardware supports unpacking that back out to 10bpp data, but that doesn't really help you.
I'm expecting there to be a register that switches to sending just the 8 most significant bits of the 10 bit value - 0x018C and 0x018D do look plausible, but don't enable compression with 0x0155. You need to alter the CSI data type to be 0x2A to match the sensor for it to work.
Whilst I'd expect that to work, it doesn't for me

Is there any chance 8 bit mode will be correctly supported by the rawcam? ignoring 5th byte of RGRGrgrgr sequence will be perfect, as well as clipping last last 2 bits instead of top 2 of 10 bit data? (Sorry for the naive question, not sure how the vc.ril.rawcam component handles the data internally)

rawcam is just programming up the hardware (it doesn't touch the pixel data itself) and it's the hardware that isn't shifting the data up and down as I was expecting.
Likelihood of changes in the hardware? Sadly zero.

I trying to process rawcam's output in OpenGL as fast as I can, but 10->8 bit conversion on CPU side takes tooo long which slows down overall performance.
rawcam in 16 bit mode seems to have the same clipping problem. I think 10 bit data get clipped to last 8 bits and then get promoted to 16 bit. At least this is what I see from the raw output.
Playing with exposure and gain is not an option for me as quality of the image will be poor(((

Last time I looked the unpacking to 16 bit mode was correct, although it is leaving the 10 bits of data in the least significant bits of the 16 bit word.
Testing again does show that I have a bug in my version of dcraw for raw16, but the actual data is good. It's LSB first, then MSB, which is not how an ARM will interpret it if cast as a uint16. I've pushed a fix to my version of dcraw (must split that off as a new repo).

[6by9]

I've moved the modified version of dcraw out of my hacky RPiTest repo into https://github.com/6by9/dcraw

[HermannSW]

I noticed in the video the rest of the image going from grey to black.
Is auto contrast/white balance or something on?
What is causing this?

I don't know, it is not an effect of animated .gif creation, because you can see that on the youtube video as well.

I was told as child not to take photos direction of sun, maybe this can be an explanation (flame == sun)?

I took another 600fps video, intentionally "into another sun"
In Germany we have 50Hz power network, wich should have a repeating pattern all 600/(50*2)=6 frames in 600fps video.
And it has!
I modified the conversion script to only take the last 24 frames, and to create a video with display framerate of 1fps for easy inspection:
(I did capture with "-t 120" only 120ms of video, and got 71 frames at 600fps. Then I did set index=48 in modified conversion script)

[HermannSW]

Back at our house I connected Pi 2B to HDMI monitor.
After I installed "mplayer" ...
https://github.com/notro/fbtft/wiki/Fra ... se#mplayer

... I was able to watch the generated "bulk.600fps.1ps.ogg" video on HDMI monitor looping by this command:

Code: Select all

$ mplayer -loop 0 -nolirc -vo fbdev2:/dev/fb0  bulb.600fps.1ps.ogg

I tid take two more "into the sun" 600fps videos, animated .gif from video consists of 24 frames with display speed 1fps.

The halogen lamp does not indicate any changes ...


... but the halogen bulbs at room ceiling 4m away show the same 6 frame periodicity because of 50Hz power netwok in Germany: Left to the bulbs is my head. In this video the same overall brightness changes that Gavinmc42 asked about for the candle light video happen. But the frequency of that effect is here much higher.

[HermannSW]

I got new gear motors that should do 2000rpm.
I calculated that less than 20 frames per motor axis rotation are needed at 600fps video.
Here you see animated .gif generated from video, this time at 2fps:


I did record with "-t 100" and got 56 frames.
The number of frames for 1 rotation of gear motor axis can be conted as between 16 and 17.
Motor was powered with 12.03V. The rpm (rotations per minute) range matches:

Code: Select all

$ bc -ql
1/(17/(56/0.1))*60
1976.47058823529411783260
1/(16/(56/0.1))*60
2100.00000000000000063000

Measurement could be done much more precise.
Running raspiraw with "2>/dev/shm/err" captures all frame timestamps:

Code: Select all

...
mal: Buffer 0x7a12a0 returned, filled 51200, timestamp 8943100358, flags 0084
mmal: Buffer 0x7a0d18 returned, filled 51200, timestamp 8943102009, flags 0004
mmal: Buffer 0x7a0ef0 returned, filled 51200, timestamp 8943102009, flags 0084
mmal: Buffer 0x7a10c8 returned, filled 51200, timestamp 8943103660, flags 0004
mmal: Buffer 0x7a12a0 returned, filled 51200, timestamp 8943103660, flags 0084
mmal: Buffer 0x7a0d18 returned, filled 51200, timestamp 8943105311, flags 0004
...

It is important for such application to store standard error output "err" on ramdisk as well.

Btw, less than 60 frames for 0.1s is caused by skipped frames, timestamp deltas show 605fps:

Code: Select all

$ bc -ql
1000000/(8943102009-8943100358)
605.69351907934585099939
1000000/(8943103660-8943102009)
605.69351907934585099939
1000000/(8943105311-8943103660)
605.69351907934585099939

Then just step through the converted frames, find two just one rotation apart, lookup both frame's microsecond timestamps t0 and t1, and get rpm as 60,000,000/(t1-t0).

[HermannSW]

You might get better performance if you just dump the output of raspiraw into a black hole. (I don't have a suitable setup, so I can't check this), unless you really need that output.

My previous response that I tested that and it makes no difference was based on modes with 180fps or 360fps. For these modes the statement remains true.

But for >=600fps modes it definitely makes a difference. The standard output does only contain 1 line and can be redirected to /dev/null. Standard error output can be directed to /dev/null as well if you do not need frame microsecond timestamps. If timestamps are needed, redirecting to /dev/shm/err is better.

In 460fps range only with err output to SD card:

Code: Select all

$ rm /dev/shm/out*raw
$ time ( raspiraw -hd -md 9 -t 1000 -sr 1 -o /dev/shm/out.%04d.raw 2>err >/dev/null )

real	0m1.091s
user	0m0.090s
sys	0m0.470s
$ ll /dev/shm/out*raw | wc --lines
463
$ 

In 580-590fps range with "2>/dev/shm/err":

Code: Select all

$ rm /dev/shm/out*raw
$ time ( raspiraw -hd -md 9 -t 1000 -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/shm/err >/dev/null )

real	0m1.082s
user	0m0.140s
sys	0m0.480s
$ ll /dev/shm/out*raw | wc --lines
589
$ 

600fps range with "2>/dev/null":

Code: Select all

$ rm /dev/shm/out*raw
$ time ( raspiraw -hd -md 9 -t 1000 -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null >/dev/null )

real	0m1.079s
user	0m0.110s
sys	0m0.490s
$ ll /dev/shm/out*raw | wc --lines
600
$

[HermannSW]

The main thing on the firmware is supporting a sensible set of modes. FOV is more important to most people than very high speed capture - far too many memories of all the complaints early on when video was the cropped 1080P mode and stills were the full 5MPix. I'm afraid you are the exception, which is part of the reason we're trying to open up the drivers where possible, but all settings are then your responsiblity!

Agreed, good news is that there will be no new modes besides 1-7, high framerate options will just modify those existing. Later after lot of testing by you, me and others maybe new modes that make sense may arise.

Fixed and rawcam branch updated. My bad, I was stating what it sounded like and that needed to be the first thing to check.
The default format was VGA raw10, resulting in a 384000 byte buffer. The framework would increase that should the buffer size requirements increase, but would never decrease it. raspiraw didn't copy the values from buffer_size_recommended across, therefore you ended up with a minimum of 384000.

Thank you for fixing, I verified that the fix is good for all temporary new modes.

All your new modes should be possible from the sensor, but 480fps may be pushing the GPU in the current configuration - suck it and see time. If you wanted to create a pull request then I would review and merge.

I am just learning how to use github. I learned already that I have to create my own fork of your userland for doing a pull request, and did that. I added raspiraw documentation (new raspiraw folder with just README.md), and many new options. Will test more, and check and extend documentation.

These are the new command line options I added -- they allow to run all high framerate modes I ever created sofar:

Code: Select all

-r, --regs	: Change regs of current mode
-hi, --hinc	: Set horizontal odd/even inc
-vi, --vinc	: Set vertical odd/even inc
-f, --fps	: Set framerate
-w, --width	: Set width
-h, --height	: Set height
-v, --vts	: Set min_vts
-l, --line	: Set line_time_ns
-hd0, --header0	: Write the BRCM header to output file 0
-0, --zero	: Write empty output files

Cool is that eg. "--fps" works for all modes, here you can see speeding up 1296x720 mode5 from 30fps to 49fps:

Code: Select all

$ rm /dev/shm/out.*.raw
$ raspiraw -md 5 -t 1000 -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null
Using i2C device /dev/i2c-0
$ ls -l /dev/shm/out.*.raw | wc --lines
30
$ rm /dev/shm/out.*.raw
$ raspiraw -md 5 -t 1000 --fps 49 -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null
Using i2C device /dev/i2c-0
$ ls -l /dev/shm/out.*.raw | wc --lines
49
$ 

And here is 600fps 640x128 stretched ("-vinc 3D" replaces 35) temporary mode 9 with just command line options!

Code: Select all

$ rm /dev/shm/out.*.raw
$ raspiraw -md 7 -t 1000 -h 64 -v 65 -l 10000 --vinc 3D --fps 600 -r "380A,0040;3802,78;3806,05FF" -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null
Using i2C device /dev/i2c-0
$ ls -l /dev/shm/out.*.raw | wc --lines
604
$ 

This mode is my current favorite mode, 600fps is super high framerate, while 640x128 is not too small framesize.

"git diff" works well, will see how "git add", "git commit" and "git push" will work soon on Hermann-SW userland repo.

[6by9]

Hi everybody!
I'm thinking about interfacing another sony imx sensor (224) with Rpi3.
What is the requirements to get raw image with raspiraw?
1) Design/use a sensor board
2) Connect i2c/clock/CSI between sensor board and rpi
3) Define the sensor registers and mode in *_modes.h file
4) run raspiraw, debug and pray

Did someone try to connect another sensor than imx219 and ov5647?

Sorry, missed this one before.

You're right on what's required. The only bit I think you've missed is adding the include and sensor entry into raspiraw.c. You may need to tweak termination settings depending on how your register set configures csi low to high speed transitions.
Getting the register set right is half the battle, although I seem to recall imx224 has a few drivers kicking around if you search a bit.
Do remember that the csi clock and data lanes are pretty high speed (100-500MHz typically) and are meant to be impedance matched.

I know one other has had an fpga solution interfaced to raspiraw.
There are settings in raspiraw for adv7282m (analogue video to csi bridge), but I never got them to work reliably. I now know they keep the clock lane in high speed mode, so I might revisit that as hopefully it will be about a two line change.

[6by9]

Raspiraw app is now split out from userland to https://github.com/6by9/raspiraw.
It should have the required source, camera_i2c script, buildme (I probably should have done a makefile - never mind), and a copy of rpi3-gpiovirtbuf. One bonus is it builds faster, however it's not set up for cross compiling.

I've added deprecration notices to my userland rawcam branch, but won't delete it yet as HermannSW is learning git against it

[6by9]

There are settings in raspiraw for adv7282m (analogue video to csi bridge), but I never got them to work reliably. I now know they keep the clock lane in high speed mode, so I might revisit that as hopefully it will be about a two line change.

ADV7282M setup now fixed on https://github.com/6by9/raspiraw (although it seems to have the potential for memory corruption - adding an extra few lines of padding to the buffer seems to fix it).

And I forgot that the TC358743 HDMI to CSI2 bridge chip is also working with the same setup, either via https://github.com/6by9/userland/tree/hdmi_in_hack, or UV4L is also using rawcam to talk to it

[HermannSW]

I saw bulb I used last week in secondary living place and did a test picture. Then I realized that I had two bulbs in photo taken, from reading lamp and from bulb on room ceiling. It took quite some time to adjust the camera position in order to show both bulbs in top quarter of 640x480 frame. That way they would show up in stretched 640x128 600fps video together. Since they share same power network I expected both to show the 600/(2*50)=6 frames repeat pattern together (because we have 50Hz power line frequency in Germany). And they do show synchronous pattern repeating every 6 frames.

This is how the video was taken with new raspiraw command line options:

Code: Select all

$ rm /dev/shm/out.*.raw
$ raspiraw -md 7 -t 1000 -hd0 -h 64 -v 65 -l 10000 --vinc 3D --fps 600 -r "380A,0040;3802,78;3806,05FF" -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null
Using i2C device /dev/i2c-0
$ ls -l /dev/shm/out.*.raw | wc --lines
604
$

And using modified raw2ogg2anim script shown below this animated .gif playing with 1fps was created:


For comparison the 600fps animated .gif from same bulb created last week (6 frames take 10ms) :


Only the top part of the script was modified, and "index" in "gst-launch-1.0" was changed to match the start of loop frames. As you can see "out.0000.raw" needs to be prepended to each frame taken because it holds the BCRM header needed for decoding with dcraw. Size of the header is 32768, size of the raw frames without header is 51200. Even when storing in ramdisk, using normal "-hd" option would skip some frames and drop overall framerate to 580-590fps range. With "-hd0" option used we get more than 600fps reliably:

Code: Select all

$ cat raw2ogg2anim 
#!/bin/bash
echo "copying /dev/shm/out.????.raw files"
# cp /dev/shm/out.????.raw .
for((i=300; i<324; ++i))
do
  cat /dev/shm/out.0000.raw /dev/shm/out.0$i.raw > out.0$i.raw
done
echo "dcraw each .raw file (to .ppm)"
for f in *.raw
do
  dcraw $f
  echo -en "$f     \r"
done
echo
echo ".ppm -> .ppm.d"
for f in *.ppm
do
  double $f > $f.d 
  echo -en "$f     \r"
done
echo
echo ".ppm.d -> .ppm.d.png"
for f in *.ppm.d
do
  pnmtopng $f > $f.png 
  echo -en "$f     \r"
done
echo
echo "now creating $1.ogg"
gst-launch-1.0 multifilesrc location="out.%04d.ppm.d.png" index=300 caps="image/png,framerate=\(fraction\)1/1" ! pngdec ! videorate ! videoconvert ! videorate ! theoraenc ! oggmux ! filesink location="$1.ogg"
echo "now creating $1.anim.gif"
gifenc.sh $1.ogg $1.anim.gif
echo "done"
$ 

[HermannSW]

Raspiraw app is now split out from userland to https://github.com/6by9/raspiraw.
It should have the required source, camera_i2c script, buildme (I probably should have done a makefile - never mind), and a copy of rpi3-gpiovirtbuf.

Thank you for that. raspiraw is such a cool piece of software -- definitely deserving its own repository!

One bonus is it builds faster, however it's not set up for cross compiling.

That's true, under userland I did compile changes with "make -C ~/userland/build/raspberry/release raspiraw/fast" in less than 10 seconds on a Pi 2B. On same Pi 2B "./buildme" takes just 3 seconds.

I've added deprecration notices to my userland rawcam branch, but won't delete it yet as HermannSW is learning git against it

You can delete the repository, I did so with my userland fork from you.

I forked https://github.com/6by9/raspiraw, cloned it and brought my changes there.
Then I did 5 adds, my first commit and finally my first git push.

Before I will do pull request I will have to correct typos in README.md documentation where I added all the new high framerate options and examples and thoughts on ramdisk usage for high framerate raspiraw video capture. And I need to implement the "--timestps" option, until now it is only documentation. All other new options are implemented and work. You can lookup new documentation here
https://github.com/Hermann-SW/raspiraw

And clone that repo if you want to play with the new options now.

[HermannSW]

I just implemented the easy part of "--tstamps" option, recording timestamps for each callback call into a singly-linked list and storing into file. After storing timestamps only (big numbers because of microsecond resolution) I thought that storing deltas as well might be interesting for human reader of timstamp file. I ended up with writing "delta,index,timestamp" per line because that allows to easily determine frame indices where frame skips occured.

As shown before these are the 3 commands I used for a single capture:

Code: Select all

$ rm /dev/shm/out*raw
$ raspiraw -md 7 -t 1000 -ts -hd0 -h 64 -v 65 -l 10000 --vinc 3D --fps 600 -r "380A,0040;3802,78;3806,05FF" -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null
Using i2C device /dev/i2c-0
$ ls -l /dev/shm/out.*.raw | wc --lines
    607
$ 

This means that 605 frames got captured, /dev/shm/out.0000.raw contains BCRM header (because of -hd0) and /dev/shm/out.-001.raw contains the timestamps (because of -ts).

Looking at the file I realized that I do not need to write any analysis code in raspiraw.c, "cut", "sort" and "uniq" are friends here.

Here is a first run without frame skips:

Code: Select all

$ cat /dev/shm/out.-001.raw | cut -f1 -d, | sort -n | uniq -c
      1 
      3 1648
      2 1649
     61 1650
    532 1651
      1 1652
      3 1653
      2 1654
$ 

As you can see we have a really small range of deltas (1648us-1654us) with majority exactly in center 1651us, which corresponds to 605.7fps:

Code: Select all

$ echo "1000000/1651" | bc -ql
605.69351907934585099939
$ 

Here is another run with frameskips (despite storing everything into /dev/shm ramdisk):

Code: Select all

$ cat /dev/shm/out.-001.raw | cut -f1 -d, | sort -n | uniq -c
      1 
     65 1650
    531 1651
      4 1652
      2 3302
$ 

Having delta column as 1st has the benefit that identifying frame indices of skips is easy:

Code: Select all

$ grep "^3" /dev/shm/out.-001.raw
3302,2,14461261624
3302,91,14461410205
$ 

So we have frame skips before frame 2 and before frame 91. But frames 91-605 do not have any frame skip.

I did already git push the "--tstamps" changes and updated its documentation.

Have I said that I really like raspiraw?

[HermannSW]

New command line option "-hd0" for storing BCRM header only once pays.

My favorite mode 640x128 stretched maximal framerate was 605fps.
Now I tested with "-hd0" and reached 665fps for same 640x128 stretched!

This is 8s recording:

Code: Select all

$ rm /dev/shm/out*raw
$ raspiraw -md 7 -t 8000 -ts -hd0 -h 64 -v 65 -l 10000 --vinc 3D --fps 660 -r "380A,0040;3802,78;3806,0603" -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null
Using i2C device /dev/i2c-0
$ ls -l /dev/shm/out*raw | wc --lines
5317
$

Analysis shows that only 6 frame skips happen during the 8s phase!
Mean frame delta is 1503us ± 5us, corresponding to 1,000,000/1503 = 665.3fps:

Code: Select all

$ cat /dev/shm/out.-001.raw | cut -f1 -d, | sort -n | uniq -c
      1 
     22 1498
    251 1499
    312 1500
    274 1501
   1061 1502
   2388 1503
    277 1504
    282 1505
    309 1506
    132 1507
      5 3005
      1 3006
$ 

Here are the frame skips:

Code: Select all

$ grep ^3 /dev/shm/out.-001.raw 
3005,2,27911765057
3005,84,27911889785
3006,1089,27913401545
3005,1691,27914307698
3005,1823,27914507563
3005,2292,27915213852
$ 

So we got frames 2292-5315 without frame skips.

I created animated .gif from 65 frames (2635-2699) worth less than 0.1s real time, playing with 19fps or 35 times slower than real:



I still have no explanation why it seems my fast moving finger is transparent:



Here is the same scene taken with "raspistill -w 640 -h 480 -o test.jpg" initially for calibration:



I used Lego pieces of many colors to see what effect the 665fps framerate has on colors.
As can be seen the video is very pale/colorless compared to the raspistill image.
It seems that 1.503ms is not enough time to sum up light.
For this framesize time has come to play with other raspiraw settings and see the effects on frames captured


P.S:
Stepped back to 640x240 stretched video (same scene, again lighted with smartphone flash, no moving finger):

Code: Select all

$ raspiraw -md 7 -t 8000 -ts -hd0 -h 120 -v 121 -l 10000 --vinc 3D --fps 360 -r "380A,0078;3802,78;3806,063B" -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null

No frame skip in 8s, played with 10fps 36 times slower than real.
Mean frame delta is 2773us ± 4us, corresponding to 1,000,000/2773 = 360.6fps:



P.P.S:
Ouch!
Had that before, same command line with just "-vinc 1F" instead of "-vinc 3D", and the colors come back!

No frame skip in 8s, played with 10fps 36 times slower than real.
Mean frame delta is 2773us ± 3us, corresponding to 1,000,000/2773 = 360.6fps:

[HermannSW]

Hi 6by9,

as requested by you I created pull requst for the new command line options:
https://github.com/6by9/raspiraw/pull/1

Please let me know if I need to change anything before you can accept the request.

I plan to create new tools directory for sample scripts later

• gstreamer pipeline for conversion of captured frame set into .ogg video
• gifenc.sh for conversion of .ogg video into animated .gif
• sample script for doing capture into /dev/shm ramdisk with frame delta analysis (rm ...; raspiraw ...; ls -l ...; cut -f1 ...)

Btw, I plan to capture missing register sets for imx219 sensor modes 2-7 after Christmas ...

[mab333]

Dear everyone,

Seeking your guidance and recommendations. I am new at raspberry pi development, so please forgive naivety in my questions.

I am using a pi 3 with a sony imx219 camera sensor. For my project I need to capture still images with raw output, at maximum resolution and frame rate that is larger than 2. Capturing raw bayer data should also be fine. As I understand, the raspiraw package enables capturing the raw bayer data from the camera's buffers and outputting them directly onto an SD card, thereby expediting the image capture process.

So, my questions are:

1. Would using raspiraw enable me to capture raw bayer pattern images with full resolution and at 2+fps?
2. Could you please point me at either a tutorial or a demo code for setting up raspiraw on pi and running it with various settings?
3. Would I need any additional packages for data manipulation?

Thank you very much and appreciate any feedback.

[HermannSW]

Hi,

I don't have a v2 camera yet, but full v2 resolution is roughly 8/5 of full v1 resolution:

Code: Select all

$ echo "3280*2464/(2592*1944)" | bc -ql
1.60392216633643245440
$ 

You don't need the new high framerate stuff I posted on earlier, but the scripts allow to show what is possible.

> 1. Would using raspiraw enable me to capture raw bayer pattern images with full resolution and at 2+fps?
>
Yes.

> 2. Could you please point me at either a tutorial or a demo code for setting up raspiraw on pi and running it with various settings?
>
Take this repo:
https://github.com/6by9/raspiraw

Or this if you want "--fps" command line option:
https://github.com/Hermann-SW/raspiraw

Follow these instructions:
viewtopic.php?f=43&t=189661&p=1232058#p1218763

> 3. Would I need any additional packages for data manipulation?
>
Yes, this repo, has its own build instructions:
https://github.com/6by9/dcraw


You will be able to capture at 15fps for just processing, or storing into /dev/shm ramdisk.
On Pi Zero you can capture 2s that way:

Code: Select all

$ ./md2 2000
removing /dev/shm/out.*.raw
capturing frames for 2000ms with 15 requested
30 frames were captured at 15fps
frame delta time[us] distribution
      1 
      1 66663
     16 66664
     12 66665
after skip frame indices (middle column)
0% frame skips
$ 
$ df /dev/shm
Filesystem     1K-blocks   Used Available Use% Mounted on
tmpfs             189784 187680      2104  99% /dev/shm
$ 

So on Pi Zero you will be able to capture 1.25s of 3280*2464 at 15fps without frameskips (on Pi3 more than double of that).

I tested with 3 fps and storing on SD card and get no frameskips for up to 4s.
Pi3 will allow for more, and the quality of your SD card will decide what you can achieve (md2.sd attached):

Code: Select all

$ ./md2.sd 4000
removing out.*.raw
capturing frames for 4000ms with 3 requested
12 frames were captured at 3fps
frame delta time[us] distribution
      1 
      2 333321
      8 333322
      1 333323
after skip frame indices (middle column)
0% frame skips
$

This is your command line (remove redirects to null for debugging):

Code: Select all

raspiraw -md 2 -t 4000 -hd --fps 3 -sr 1 -o out.%04d.raw 2>/dev/null >/dev/null

Raspiraw documentation and instructions on how to convert to .ogg video here:
https://github.com/Hermann-SW/raspiraw

[mab333]

HermannSW, thank you very much for an expedient and detailed answer - I will look into your recommendations.

A couple of quick follow up questions:

1. If I want to capture a sequence of images for 15 seconds or so, it sounds like there will be some frames skipped (with Pi Zero more often, than with Pi 3). However, would it be the case that only 1 frame is skipped and then you get another sequence of frames without interruption?

2. Finally, I assume it is also possible to apply a timestamp to the saved file, so you can identify the time difference between the consecutive and skipped frames. My project is focusing on tracking micro-objects in the image and characterizing their speed and trajectories. That is why having a relatively high FPS and an ability to precisely know the timing of the images is important.

Thank you for your help, once again!

[6by9]

HermannSW, thank you very much for an expedient and detailed answer - I will look into your recommendations.

A couple of quick follow up questions:

1. If I want to capture a sequence of images for 15 seconds or so, it sounds like there will be some frames skipped (with Pi Zero more often, than with Pi 3). However, would it be the case that only 1 frame is skipped and then you get another sequence of frames without interruption?

It's mainly a question of how fast your storage is. raspiraw as it stands will be running the camera at 15fps 8MPix 10bits/pixel. That's 144Mbytes/s. Almost no storage will cope with that.
The default setup is to save one frame every N frames (N set on the command line with "-sr N", default of 20), and therefore drop the capture rate to something more manageable. (It is however blocking the capture thread, so it isn't totally even).
HermannSW has just added frame rate control to raspiraw for the V1 OV5647 sensor (I need to review the pull request again) as the better option is to tell the sensor to run at your desired 2fps than drop frames in a potentially unpredictable manner. That change hasn't been done for the V2 IMX219 sensor yet, and I'm constrained due to NDAs as to what I can say.

2. Finally, I assume it is also possible to apply a timestamp to the saved file, so you can identify the time difference between the consecutive and skipped frames. My project is focusing on tracking micro-objects in the image and characterizing their speed and trajectories. That is why having a relatively high FPS and an ability to precisely know the timing of the images is important.

So every buffer that comes back is timestamped via the pts field in the MMAL_BUFFER_HEADER_T. It's based off the GPU's microsecond accuracy clock at the point the Frame Start interrupt fires.
The current code doesn't do anything with that timestamp, but you have the full source code so can make changes as you require.

[mab333]

Thank you for clarification 6by9.

It's mainly a question of how fast your storage is. raspiraw as it stands will be running the camera at 15fps 8MPix 10bits/pixel. That's 144Mbytes/s. Almost no storage will cope with that.
The default setup is to save one frame every N frames (N set on the command line with "-sr N", default of 20), and therefore drop the capture rate to something more manageable. (It is however blocking the capture thread, so it isn't totally even).
HermannSW has just added frame rate control to raspiraw for the V1 OV5647 sensor (I need to review the pull request again) as the better option is to tell the sensor to run at your desired 2fps than drop frames in a potentially unpredictable manner. That change hasn't been done for the V2 IMX219 sensor yet, and I'm constrained due to NDAs as to what I can say.

So in this case if, as you say, currently raspiraw is running the sensor at 15fps 8Mpix 10bits/pixel = 144Mbytes/s, then if I set it to, say, capture every 4th frame (giving me effective fps of ~3.7), it would end up being roughly 40Mbytes/s. As I understand, if I opt for a, say, UHS-1 sd card and enable sd card overclocking on PI, it should be able to support that capture rate. Is my thinking correct?

Thank you!

[6by9]

So in this case if, as you say, currently raspiraw is running the sensor at 15fps 8Mpix 10bits/pixel = 144Mbytes/s, then if I set it to, say, capture every 4th frame (giving me effective fps of ~3.7), it would end up being roughly 40Mbytes/s. As I understand, if I opt for a, say, UHS-1 sd card and enable sd card overclocking on PI, it should be able to support that capture rate. Is my thinking correct?

As long as they are using MB/s as MegaBytes/s and not MegaBits/s, then you've got a fighting chance.
Do be aware that Linux buffers numerous file activities though, so it won't be as uniform as you might at first expect.

[HermannSW]

mab333 asked for a Pi3, wouldn't use of a harddisk be a better storage choice than SD card (enterprise HDDs can do 140MB/s) or a SDD (can to up to 600MB/s write speed)?

[6by9]

mab333 asked for a Pi3, wouldn't use of a harddisk be a better storage choice than SD card (enterprise HDDs can do 140MB/s) or a SDD (can to up to 600MB/s write speed)?

You're limited by USB2 bandwidth (480Mbit/s, or 60MBytes/s, if my maths/memory are correct)

[HermannSW]

I made huge progress with raspiraw high framerate changes. First I fixed the limitation for vertically maxmial 416 lines, then I added new --top option, then changed all tools to replace --regs "...." completely with --height ... --top ..., made 640x480 and 640x480_s work and added "640x480_B*" modes capturing the lower half of sensor fov! I did all that on long train ride back home today,frames below taken there.

new 640x480 at 90fps:


new 640x480_s at 180fps:


640x240 at 180fps, top half:


new 640x240_B at 180fps, bottom half:


640x240_s at 360fps, top half:


new 640x240_B_s at 360fps, bottom half:



All is available with latest commit in case you want to play with:
https://github.com/Hermann-SW/raspiraw

New modes added to framerate table: