Camera trap

[christopherpm]

Hi,

I have an RPi 3, and a 1.3 camera module. What I would like to do is set up a camera trap at the finish line of a race so that the Pi captures the highest possible resolution images when the camera detects motion.

What would be the best way to achieve this please?

Thanks in advance.

[jbeale]

The question you have to answer is, will video be good enough (1080p) or do you require the full 8 Mpixel still image? If you take 30 fps video, you can be sure to get a frame within 1/30 sec of the actual finish. If you need to take a still image, there is a variable shutter lag which may be more than 1/2 second. I have been using the RPi camera for some time with several different software approaches, and I have never managed to get the still image mode shutter lag low enough to work well for the "photo-finish" application you suggest.

It may be possible to do some "burst mode" rapid capture (in theory up to 15 fps) recording to RAM which you trigger a short time before the actual finish and then select the best frame afterwards. However I think the full-frame 15 fps figure doesn't account for the extra processing normally done in the RPI still image mode. 6by9 could tell you more about it, probably.

See also: viewtopic.php?f=43&t=145248&p=958182&hi ... de#p958182

Update: I see an answer to this question viewtopic.php?f=43&t=145248&p=958182&hi ... de#p958182 here http://raspberrypi.stackexchange.com/qu ... raspistill claims a 40 msec lag time. Frankly I do not believe it, but try and see what you get! (The posted method sets a 1-msec timeout so there will be no autoexposure, and the image will likely be poorly exposed; would be better to have the preview run continually to get exposure set, then take the still based on a process signal instead).

UPDATE: I tried it, having made this shell alias:

Code: Select all

alias shot='SHOTTIME=$(date +"%Y-%m-%d_%H%M") && raspistill -o shot-$SHOTTIME.jpg --nopreview --exposure sports --timeout 1'

and with that code, on my system the actual image is taken about 1/2 second after pressing enter, so I don't know why they claimed 40 msec.

[jbeale]

Using the -k option, for example

Code: Select all

 raspistill -ex sports -k -o test%04d.jpg

and pressing "enter" to take the image, has less delay than the code example in my last post. Not sure how much delay exactly but some experiments with a stopwatch suggests about 220 msec.

[gregeric]

Is a finish line camera of use to you, like the ones that produce photo-finish images from sports events?

In a proof of concept quick hack, I modified the callback in raspividyuv.c to append one row of pixel luminance data from the centre of the camera view to the output file.

A greyscale picture is then assembled.

5 seconds @ around 100 slices per second:


10 seconds at around 100 slices per second:


In the above, the right hand of the imaged is at t=0, the left hand at t=5 or 10s, and you can see which finger passed over the finish line first. Timings could be taken from the image too.

[gordon77]

This looks interesting. Could you give more details how you did it?

Gordon

[gregeric]

Cheers Gordon - it's something I've been meaning to look at for a good while.

At line 688 in raspividyuv.c, modify with this:

Code: Select all

      if (buffer->length)
      {
         mmal_buffer_header_mem_lock(buffer);
         //write one row number width of greyscale pixels from centre of view
         //do we need to round up to strides/multiples of 32/16 on width/height?
         bytes_written = fwrite(buffer->data+width*(height>>1), 1, width, pData->file_handle);
         mmal_buffer_header_mem_unlock(buffer);

         if (bytes_written != width)
         {
            vcos_log_error("Failed to write buffer data (%d from %d)- aborting", bytes_written, buffer->length);
            pData->abort = 1;
         }
      }

Here, the firmware gives us a buffer in memory with planar YUV 420 data at the requested dimensions. The luminance data for the whole image is first, followed by 2 x chroma blocks. For speed & ease of hacking, we can ignore the colour info & take a slice of grayscale pixels. I took a row from the centre of the image, hence buffer->data+width*(height>>1)

That's all the mods to raspividyuv - I assemble a valid image file afterwards.

Capture some data:

Code: Select all

/opt/vc/bin/raspividyuv -w 320 -h 240 -fps 100 -t 10000 -o ydata.bin

Ideally I'd convert the .bin into at valid image file before exiting within raspividyuv. For testing, I used Mathematica instead:

Code: Select all

imgdat = Import["~/ydata.bin", "Byte"]
height = 320
slices = Length[imgdat]/height
im = Array[0 &, {slices, height}]
Do[im[[i, j]] = imgdat[[(slices - i)*height + j]]/255, {i, slices}, {j, height}]
im2 = Transpose[im]
Export["~/ypic.png", Image[im2]]

If you use the above, place the camera on its side with the ribbon on the right or left as you look at it, and arrange for the motion to be left<->right in front of the camera.

[R61zt]

How about linking two rPi with camera units, using GND and one digital pin on the GPIO to carry trigger information from either an external trigger or from one to the other.

[jbeale]

In a proof of concept quick hack, I modified the callback in raspividyuv.c to append one row of pixel luminance data from the centre of the camera view to the output file. A greyscale picture is then assembled.

This is great! I really like this idea, a genuinely different use of the camera from normal imaging. It also completely separates the moving object from the static background, useful for some applications and can be a hard problem otherwise.

Another point: if you are trying to use the camera at night (eg. security camera), it is always a challenge to get enough light. But if you can rely on your subject moving by at some fairly constant speed (almost always true, for cars), you can focus the light down to just that single vertical line you are using, giving you a much brighter image for the same total amount of light. With more effective light, you can also use a shorter shutter speed, so there is less motion blur... this is all very interesting

A long time ago I asked about a line-scan option (only reading out a single row of pixels, possibly very fast) but I think the sensor did not support that mode. But at 100 fps it seems you can get usable results even reading out everything and just throwing away all but one line. I guess you could also use 2 or more lines at once, depending on requirements. To assemble the image with the correct aspect ratio you do need to know the horizontal speed of the object in pixels per second, but there are various ways to get that.

[tenochtitlanuk]

Thanks Gregeric for your contribution- it's what I was trying to do back in 2013 in the early days of pi...

[jbeale]

At 100 fps, this does work best for rather slow speeds. For example, if you want a 640x480 image, that means your object should take 6.4 seconds to cross from one side of the frame to the other. If it took 3 seconds to pass by, then only 1/2 of the object would be imaged in your line of pixels. I would imagine for a finish-line type camera you'd want at least 10x that frame rate (line rate?) Maybe 100x.

If the Sony sensor can read out a full 3280x2464 frame at 15 fps, that means it can read 2464x15 = 36960 lines per second (!) on average when reading the entire frame. Unfortunately, I guess there is no mode to just repeatedly read one line only.

[6by9]

If the Sony sensor can read out a full 3280x2464 frame at 15 fps, that means it can read 2464x15 = 36960 lines per second (!) on average when reading the entire frame. Unfortunately, I guess there is no mode to just repeatedly read one line only.

Would an exposure time of 27usecs give you anything of use? That's the corollary of reading out one line repeatedly at 36960 lines/second.

[PiGraham]

Using the -k option, for example

Code: Select all

 raspistill -ex sports -k -o test%04d.jpg

and pressing "enter" to take the image, has less delay than the code example in my last post. Not sure how much delay exactly but some experiments with a stopwatch suggests about 220 msec.

I would think the signal mode is the better option for this might be the signal mode that could be initiated by a photo-sensor into GPIO. I don't know what the lag might be.

The strip camera mode is a great solution.

[PiGraham]

Another point: if you are trying to use the camera at night (eg. security camera), it is always a challenge to get enough light. But if you can rely on your subject moving by at some fairly constant speed (almost always true, for cars), you can focus the light down to just that single vertical line you are using, giving you a much brighter image for the same total amount of light. With more effective light, you can also use a shorter shutter speed, so there is less motion blur... this is all very interesting

I'm afraid that won't work. You need an image-forming lens. If you used a odd shaped lens (cylindrical concave lens?) to collect light to either side of the scan line you would have distorted the image so that it blurred everything together in the centre. You would lose the ability to determine where things were horizontally or when a line was crossed. This works by having a normal focused image and just sampling a line from it.

[PiGraham]

At 100 fps, this does work best for rather slow speeds. For example, if you want a 640x480 image, that means your object should take 6.4 seconds to cross from one side of the frame to the other. If it took 3 seconds to pass by, then only 1/2 of the object would be imaged in your line of pixels. I would imagine for a finish-line type camera you'd want at least 10x that frame rate (line rate?) Maybe 100x.

If you record 100 lines/s you have a temporal resolution of 10ms, which I should think was quite useful for various sports. You don't much care how long it takes for an object to fill the frame. You want to know when it crosses the line. Two lines can tell you that.

[jbeale]

If the Sony sensor can read out a full 3280x2464 frame at 15 fps, that means it can read 2464x15 = 36960 lines per second (!) on average when reading the entire frame. Unfortunately, I guess there is no mode to just repeatedly read one line only.

Would an exposure time of 27usecs give you anything of use? That's the corollary of reading out one line repeatedly at 36960 lines/second.

Yes of course that's pretty short! Probably not, but for example, I could absolutely use a 1 msec exposure (~ 1000 lines per second, minus read+reset overhead) if it was somehow possible. If you only have to illuminate a single row of pixels, you can use a laser line generator, and at least over a small area, those can easily be bright enough to allow a 1 msec exposure. Probably the first limit you hit is eye safety[1], rather than any technical issues.

@PiGraham: The camera does use a normal image-forming lens. By focusing the light, I mean optics directing the external light source, like a spotlight that makes a line instead of a spot. The illumination can be just a line that is aligned to the active image region, since that's the only part of the scene you are using at any given moment, and the system does work. This is the way desktop scanners work, for example.
And yes, maybe for sports, 100 fps is OK. In my application I would prefer something close to a normal 2D image of the passing object, so I want more dense samples along the axis of motion, not just a few lines.

[1] The FDA laser MPE for "eye-safe" for a 850 nm CW laser is around 2 mW/cm^2. The sun on a clear day delivers around 1 kW/m^2 which is 100 mW/cm^2 so the sun is a factor of 50 brighter than "laser eye-safe". Clearly, the FDA should put some warning labels on the sun

[PiGraham]

By focusing the light, I mean optics directing the external light source, like a spotlight that makes a line instead of a spot.Absolutely, you only need to illuminate the part of the scene that corresponds to the sensor line

[PiGraham]

I see the Sony chip in the camera can run at 240fps.

High-Speed Video Imaging Function

Not only the simultaneous still picture and video imaging at 30 frame/s in all pixels (no shutter lag and simultaneous recording), but also 4 times faster imaging became possible by 2 × 2 analog binning mode. This enables HD video recording with 2M pixels at full field of view and 120 frame/s, or 1280 × 720 pixels of 16:9 aspect ratio at 180 frame/s. (Using MIPI 4Lane)
Slow-motion recording using this function reproduces dynamic movements into a smooth video (See moving image 1.) in sports scene recording, and sensitive movements or expressions beyond human vision can be captured. (See moving image 2.)
The IMX219PQ fulfills camera features and also provides an exciting new experience for all of customers, enabling Sony and the IMX219PQ to contribute to the functional differentiation of smartphone cameras.

...

960 × 540 (QHD, cropping) 240 frame/s

http://www.sony.net/Products/SC-HP/new_ ... 219_e.html

That requires four MIPI lanes, which the Pi doesn't have.

[gregeric]

A faster line capture rate would be very useful. In this usage the camera is fixed and we can arrange for illumination as required. I added a time scale top & bottom to allow timings to be taken of that all important egg and spoon race. The microsecond callback time is embedded at the bottom for each slice too:

10 seconds worth @ 640x480x90


20 seconds worth 640x480x90

Those are at walking pace, back & forth in front of the camera. No matter which way you pass through, you always come out pointing the same way - the arrow of time

[6by9]

I see the Sony chip in the camera can run at 240fps.

High-Speed Video Imaging Function

Not only the simultaneous still picture and video imaging at 30 frame/s in all pixels (no shutter lag and simultaneous recording), but also 4 times faster imaging became possible by 2 × 2 analog binning mode. This enables HD video recording with 2M pixels at full field of view and 120 frame/s, or 1280 × 720 pixels of 16:9 aspect ratio at 180 frame/s. (Using MIPI 4Lane)
Slow-motion recording using this function reproduces dynamic movements into a smooth video (See moving image 1.) in sports scene recording, and sensitive movements or expressions beyond human vision can be captured. (See moving image 2.)
The IMX219PQ fulfills camera features and also provides an exciting new experience for all of customers, enabling Sony and the IMX219PQ to contribute to the functional differentiation of smartphone cameras.

...

960 × 540 (QHD, cropping) 240 frame/s

http://www.sony.net/Products/SC-HP/new_ ... 219_e.html

That requires four MIPI lanes, which the Pi doesn't have.

The Compute Module does have 4 lanes on one interface.
960x540 @ 240fps doesn't necessarily need 4 lanes - 960 * 540 * 240 * 10(bpp) = 1,244,160,000 bits/s, and each CSI2 lane can carry a nominal 1Gbit/s. There are overheads, and sometimes sensor manufacturers use extra lanes for simplicity.

Reading the line

This enables HD video recording with 2M pixels at full field of view and 120 frame/s, or 1280 × 720 pixels of 16:9 aspect ratio at 180 frame/s. (Using MIPI 4Lane)

1280 * 720 * 180 * 10(bpp) = 1,658,880,000bits/s, so significantly closer to the capacity of 2 lanes such that overheads may make it impossible. CSI2 lanes are almost always used in powers of 2, hence stepping up to wanting 4 lanes.

Big fly in the ointment though is that the ISP throughput is somewhere between 100 and 150MPix/s, so processing that data is going to be pushing it.
720P180 would also exceed even H264 level 4.2 which is a max of 133,693,440 luma samples/sec (ie pixels), or 720P145.1.

Once we get V4L2 drivers sorted for the raw sensor, you can all play to your heart's content with modes, and if you get any really funky stuff working then it could be merged into the firmware. Got to get there first (see request for help at viewtopic.php?f=43&t=109137&start=125#p992986)

[jbeale]

No matter which way you pass through, you always come out pointing the same way - the arrow of time

Amazing, informative, and wonderfully strange-looking images- reminiscent of Salvador Dali. I think every science & technology museum should have an exhibit with this type (maybe they already do?). I can imagine my kids spending hours creating artwork by moving past this kind of camera.

This does motivate me to think about what kind of hacking might be possible to get a highly asymmetric crop with a higher frame rate. I don't know anything about the Sony chip on the v2.1 board, since I've seen no datasheet. If you look at the 140-page preliminary datasheet for the older sensor OV5647, at http://cdn.sparkfun.com/datasheets/Dev/ ... 7_full.pdf it is very complex, and seems highly configurable. Also thinly documented, but in "Timing Control" Table 6-5 (p.61-62) and also "System Timing Registers" Table 7-5 (p. 84-86) it shows names like TIMING_X_ADDR_START, TIMING_X_ADDR_END, TIMING_X_OUTPUT_SIZE, (plus same for Y) and TIMING_HTS (total horizontal size), TIMING_VTS (total vertical size). Maybe not all values are valid, but it seems at least possible that a high-aspect-ratio frame might be configured.

The OV5647 pixel clock runs at 68 MHz in the 1920x1080,30fps (cropped) mode, so if that was the only limit, a lot faster frame rate seems possible with an extreme crop like 1920 x 64. I suspect there is likely an additional limit on total frame rate and/or minimum line exposure time, but I am not sure.

[gregeric]

There is some fun to be had with the images that's for sure. May be of use in the classroom for science experiments too - timing a pendulum swing, estimating the speed of passing vehicles, timing free fall objects etc

[jbeale]

Just in case anyone doubted a 1 msec exposure could ever be useful. Below close-up image is taken by RPi Camera (the old one, OV5647) using a simple consumer laser-level with line generator, marked "<5mW", shining across my fingers. EXIF says 1/1013 sec exposure. True, it is dim, but then again brighter lasers exist.

Code: Select all

raspistill -ISO 800 -ss 1000 -awb off -awbg 1.5,1.5 -t 10000 -o t2.jpg