pmbdk
Posts: 8
Joined: Thu Jan 02, 2020 2:09 pm

Low-latency machine vision with RPi4 and PiCam 2.1

Thu Jan 02, 2020 2:29 pm

Hi all

I'm trying to port some old machine vision code to RPI4 with picam 2.1. raspistill gives great images so the basic stuff works and the machine vision algorithms in c runs super-fast (on images loaded to /dev/shm).

The main goal is to have as low average latency as possible; essentially this then boils down to having high framerate/low exposure time. Image resolution is not super-important; 640x480 would be nice but smaller can be accepted. There are no realtime requirements, so the RPi4 seems like a good candidate. I have full control of lighting conditions so exposure time can be low.

But I have a hard time figuring out what the best way is to go from here. raspistill is super-easy to use, but I am not able to run more than ~10 Hz @640x480 using timelapse, which is probably also a stupid way to do it (but convenient!). I don't really need auto gain, so I tried using the option exposure off and burst (and setting shutter speed, digital gain and analog gain manually) but that does not seem to work for some reason (I get all black images).

Looking at 6by9's raspiraw I know that super-fast speeds can be obtained; is that the way to go, even though it then requires debayering the data afterwards? I have also seen mention of reading /dev/fb0 and some say using v4l.

So my question is really: Do anyone have experience in low-latency machine vision on the RPi and can recommend a way forward so I don't have to test all possible ways of doing it :-)?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24948
Joined: Sat Jul 30, 2011 7:41 pm

Re: Low-latency machine vision with RPi4 and PiCam 2.1

Thu Jan 02, 2020 4:17 pm

So what sort of framerates are you looking for?

You could try running raspivid and output MJPEG frames, not sure what sort of rate you will get, depends on resolution of course. But you ought to be able to get 30fps at VGA, with one ARM code decoding the MJPEG as they come back from the GPU. Or use raspividyuv and get YUV frames back, more data to transfer so a bit slower, but that don't require decoding.

Number of options, depends on your needs and capabilities.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

User avatar
HermannSW
Posts: 1879
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Low-latency machine vision with RPi4 and PiCam 2.1

Thu Jan 02, 2020 5:11 pm

jamesh wrote:
Thu Jan 02, 2020 4:17 pm
You could try running raspivid and output MJPEG frames, not sure what sort of rate you will get, depends on resolution of course. But you ought to be able to get 30fps at VGA,
For 640x480 mode 7 with v1 camera you can get 90fps with 0 frame skips, with v2 camera even 150fps.

camver script:
https://www.raspberrypi.org/forums/view ... 9#p1277731
Needs raspiraw cloned:
https://github.com/6by9/raspiraw

ptsanalyze tool analyses microsecond resolution timestamp file created by "-pts" option:
https://github.com/Hermann-SW/userland/ ... ptsanalyze
[email protected]:~ $ export PS1=$\
$ ./camver
v1 camera found
$ raspivid -md 7 -w 640 -h 480 -cd MJPEG -pts tst.pts -o tst.mjpeg -t 8000 -fps 90
$ du -sm tst.mjpeg
17 tst.mjpeg
$ ~/userland/tools/ptsanalyze tst.pts 0
creating tstamps.csv
720 frames were captured at 90fps
frame delta time[us] distribution
43 11089
113 11090
410 11091
113 11092
37 11093
1 11094
after skip frame indices (middle column)
0 frame skips (0%)
$

P.S:
You might have a look at my i420toh264 framework for analyzing and/or processing raspividyuv frames while recording.
I need this to control stepper motors of PT camera system for always centered on airplane recording of approach for landing:
https://github.com/Hermann-SW2/userland ... i420toh264
Image
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

gordon77
Posts: 4498
Joined: Sun Aug 05, 2012 3:12 pm

Re: Low-latency machine vision with RPi4 and PiCam 2.1

Thu Jan 02, 2020 5:24 pm

This with opencv should do upto 40fps

Code: Select all

 import cv2
import time
import os

invert_mask = 0

# mouse action
def point(event, x, y, flags, param):
    global invert_mask
    # Left mouse click to change mask
    if event == cv2.EVENT_LBUTTONDOWN:
        if invert_mask == 0:
            invert_mask = 1
        else:
            invert_mask = 0
    # Right mouse click to save screenshot
    if event == cv2.EVENT_RBUTTONDOWN:
        path = 'scrot screenshot.jpg'
        os.system (path)

cam = cv2.VideoCapture(0)

# set video size
cam.set(3,640)
cam.set(4,480)

if os.path.exists('/dev/video0') == False:
  path = 'sudo modprobe bcm2835-v4l2'
  os.system (path)
  time.sleep(1)

FPS          = 40
AutoExp      = 0   # 0 = auto ON , 1 = auto OFF
AEB          = 12  # range 0 - 24
ExpTime      = 1000 
brightness   = 60
ISO          = 0   # 0 = auto, 1 = 100, 2 = 200 etc
contrast     = 50
scene        = 8   # 0 = normal, 8 = night, 11 = sports
saturation   = 25
red_balance  = 1188
blue_balance = 1228

winName = "Scope"
cv2.namedWindow(winName)#
#cv2.namedWindow(winName,cv2.WINDOW_GUI_NORMAL)
#cv2.setWindowProperty(winName,cv2.WND_PROP_FULLSCREEN, cv2.WINDOW_FULLSCREEN)

# setup mouse callback               
cv2.setMouseCallback('Scope',point)

# setup camera

path = 'v4l2-ctl --set-ctrl=brightness=' + str(brightness)
os.system (path)
path = 'v4l2-ctl --set-ctrl=contrast=' + str(contrast)
os.system (path)
if AutoExp == 0:
  path = 'v4l2-ctl --set-ctrl=auto_exposure=0'
  os.system (path)
  path = 'v4l2-ctl --set-ctrl=auto_exposure_bias=' + str(AEB)
  os.system (path)
else:
  path = 'v4l2-ctl --set-ctrl=auto_exposure=1'
  os.system (path)
  if AutoExp == 1:
    path = 'v4l2-ctl --set-ctrl=exposure_time_absolute=' + str(ExpTime)
    os.system (path)
if ISO == 0:
   path = 'v4l2-ctl --set-ctrl=iso_sensitivity_auto=1'
   os.system (path)
else:
   path = 'v4l2-ctl --set-ctrl=iso_sensitivity_auto=0'
   os.system (path)
path = 'v4l2-ctl --set-ctrl=iso_sensitivity=' + str(ISO)
os.system (path)
path = 'v4l2-ctl --set-ctrl=auto_exposure_bias=' + str(AEB)
os.system (path)
path = 'v4l2-ctl -p ' + str(FPS)
os.system (path)
path = 'v4l2-ctl --set-ctrl=scene_mode=' + str(scene)
os.system (path)
path = "v4l2-ctl -c white_balance_auto_preset=0"
os.system (path)
path = "v4l2-ctl -c red_balance=" + str(red_balance) 
os.system (path)
path = "v4l2-ctl -c blue_balance=" + str(blue_balance) 
os.system (path)
path = "v4l2-ctl -c sharpness=50" 
os.system (path)
path = "v4l2-ctl -c saturation=" + str(saturation) 
os.system (path)

# read foreground image
img2 = cv2.imread('Reticle2.jpg')
img3 = (255-img2)
img2gray = cv2.cvtColor(img2,cv2.COLOR_BGR2GRAY)
ret, mask = cv2.threshold(img2gray, 10, 255, cv2.THRESH_BINARY)

while True:
        
     ok, img = cam.read()
     if invert_mask == 0:
        added_image = cv2.bitwise_and(img,img,mask = mask)
     else:
        added_image = cv2.addWeighted(img,0.99,img3,0.7,0.2)
     cv2.imshow( winName,added_image)
     key = cv2.waitKey(10)
     if key == 27:
        cv2.destroyWindow(winName)
        break
 
print ("Goodbye")

pmbdk
Posts: 8
Joined: Thu Jan 02, 2020 2:09 pm

Re: Low-latency machine vision with RPi4 and PiCam 2.1

Thu Jan 02, 2020 9:34 pm

This is some serious excellent feedback, thank you guys!!!

@jamesh: I was hoping for 10 ms exposure time @ 640x480 px (i.e. 100 fps), preferably un-encoded. This gives 92+ MB/s, which is probably pushing it a bit (I have no idea though!). The main point with decoding in my application is that I need to be able to extract data from small portions of the image without actually decoding the entire image. I guess YUV420 encoding would allow me to do that though; but more complex encodings is no-go.

@HermannSW: Those are some really encouraging numbers!!! I will definitely take a look at your i420toh264 tool; this seems exactly what I need! :-) Probably a stupid question though: Does the yuv420 data from raspividyuv contain some form of header or sync mechanism? The format seems almost too simple... ;-)

@gordon77: Thanks for the script; this is really well written and even readable for a python-noob as myself... :-)

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24948
Joined: Sat Jul 30, 2011 7:41 pm

Re: Low-latency machine vision with RPi4 and PiCam 2.1

Thu Jan 02, 2020 10:10 pm

pmbdk wrote:
Thu Jan 02, 2020 9:34 pm
This is some serious excellent feedback, thank you guys!!!

@jamesh: I was hoping for 10 ms exposure time @ 640x480 px (i.e. 100 fps), preferably un-encoded. This gives 92+ MB/s, which is probably pushing it a bit (I have no idea though!). The main point with decoding in my application is that I need to be able to extract data from small portions of the image without actually decoding the entire image. I guess YUV420 encoding would allow me to do that though; but more complex encodings is no-go.

@HermannSW: Those are some really encouraging numbers!!! I will definitely take a look at your i420toh264 tool; this seems exactly what I need! :-) Probably a stupid question though: Does the yuv420 data from raspividyuv contain some form of header or sync mechanism? The format seems almost too simple... ;-)

@gordon77: Thanks for the script; this is really well written and even readable for a python-noob as myself... :-)
No header, raw data, but be aware of the packing and possible gaps between Y U and V planes if things are not divisible by 16 (IIRC)
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

User avatar
HermannSW
Posts: 1879
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Low-latency machine vision with RPi4 and PiCam 2.1

Fri Jan 03, 2020 8:31 am

jamesh wrote:
Thu Jan 02, 2020 10:10 pm
pmbdk wrote:
Thu Jan 02, 2020 9:34 pm
@HermannSW: Those are some really encouraging numbers!!! I will definitely take a look at your i420toh264 tool; this seems exactly what I need! :-) Probably a stupid question though: Does the yuv420 data from raspividyuv contain some form of header or sync mechanism? The format seems almost too simple... ;-)
No header, raw data, but be aware of the packing and possible gaps between Y U and V planes if things are not divisible by 16 (IIRC)
That is exactly what align_up() function is for.
The implementation only works for powers of 2, so "align_up(width, 5)" and "align_up(height, 4)" is needed to process raspividyuv generated i420 YUV frames:
https://github.com/Hermann-SW2/userland ... 2grey.c#L6
sample_yuv2grey.c is simple demo converting raspividyuv stream of i420 frames to grey.

Yes, the format is that simple (fixed size and YUV arrangement per frame):
Image
On the other hand you have to address the pixel value split on full width/height Y and half width/height U and V frames.

sample_yuv_airplane.c demonstrates how to identify a (dark) airplane position against blue sky and mark the found position with a white 2x2 marker, described in section "Locating an airplane":
https://github.com/Hermann-SW2/userland ... g-airplane
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

pmbdk
Posts: 8
Joined: Thu Jan 02, 2020 2:09 pm

Re: Low-latency machine vision with RPi4 and PiCam 2.1

Fri Jan 03, 2020 7:59 pm

And the results are in! :-)

Since I didn't bother decoding yuv (I know it's easy, but this is just prototyping), I just used raspivid -rf rgb. I'm lazy like that, so shoot me... ;-)

There are probably a bunch of pitfalls to this, but is seems to run nicely up to 50 fps @ 640x480 px with less than 0.1 skipped frame per second (I can live with skipped frames in any case). Using yuv instead of rgb makes it run nicely up to around 90 fps.

I first tested with -pts and tried to correlate the number of timestamps with the actual number of received frames. For some reason the output does not match when frames are skipped; even when the timestamps states that no frames has been skipped there can still be a significant number of skipped frames in the actual frame data. This may be actual skipped frames or simply due to cut-off when ending or starting or something like that. However even with skipped frames, the number of bytes in the received data is always a multiple of the pixel-data in a frame, which was actually my greatest worry, as it will be very difficult to re-syncronize the datastream if sync is lost (or even just detect it).

I wrote a small c-program going through the stream during frame reception to calculate average red, green and blue (just as a worst case example of processing); during operation the number of extracted pixels will be much smaller.

I then used it like this:

Code: Select all

sudo raspivid -md 7 -w 640 -h 480 -fps 50 -t 10000 -n -pts /dev/shm/pts.txt -v -rf rgb -r -  -o /dev/null | camlatency
Since low latency is what I am after, I then hooked up a red LED to the GPIO and put it in front of the pi camera. Everytime the red average came above a certain threshold I then turned off the LED and vice versa, The number of frames it stays high (or low) is then the latency.

To my surprise the result was exceptional low! Only two frames latency! In rare cases three frames, but usually two. I don't know anything about the sensor (rolling shutter/global shutter/readout frequency) but two frames is way faster than I had expected, taking into account that the frame needs to be exposed, read out, pre-processed by the GPU, buffered by Linux, processed by my own program and then out through GPIO (ok, the last one is obviously fast). I could of course have made a mistake somewhere, because just frame exposure and readout is probably the first two frames worth of latency, but honestly the LED blinks at the correct frequency and everything looks fine.

Of course this may change once I go to yuv and 90 fps, but already now it looks good! :D

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Low-latency machine vision with RPi4 and PiCam 2.1

Fri Jan 03, 2020 8:26 pm

Very interesting to see these results! I have not tried this, but I am also surprised you can achieve two frames of latency at 50 fps.

I haven't used raspivid in a while. Without fixing the analog and digital gain ('-ag xxx -dg yyy' or '-ex off' ), is there still some AGC action going on?
I read that ' -awb off -awbg x.x,y.y -ss xxxx ' is also used for full manual control, since otherwise auto-white balance also comes into play.

I notice raspivid has the option "-fl, --flush : Flush buffers in order to decrease latency" does that have any effect here?

pmbdk
Posts: 8
Joined: Thu Jan 02, 2020 2:09 pm

Re: Low-latency machine vision with RPi4 and PiCam 2.1

Sat Jan 04, 2020 3:33 am

I am almost sure the AGC is still running in the background; it is clear when I manually light up the scene. For me though, the AGC and white-balance does not really matter that much; the scene is pretty well-lit and the contrasts does not change much.

I completely forgot about flush; I tried it during tests with raspistill, but there it (obviously?) did not work. I have just tried it with raspivid but it does not seem to change anything significantly (I have not made any statistics though).

pmbdk
Posts: 8
Joined: Thu Jan 02, 2020 2:09 pm

Re: Low-latency machine vision with RPi4 and PiCam 2.1

Sat Jan 04, 2020 3:44 am

This is the code, you are welcome to test it; it would be interesting to get data from other Pi's as well:

Code: Select all

// camlatency.c
//
// After installing bcm2835, you can build this 
// with something like:
// gcc -o camlatency camlatency.c -l bcm2835
// sudo raspivid -md 7 -w 640 -h 480 -fps 80 -t 100000 -n -pts /dev/shm/pts.txt -v -rf rgb -r -  -o /dev/null | camlatency
//
#include <bcm2835.h>
#include <stdio.h>
#include <stdio.h>   
#include <unistd.h>  
#include <stdlib.h>
#include <limits.h>
#include <stdint.h>
#include <stdbool.h>

// RGB
#define MAX_READ_LEN ( 640 * 480 * 3 )
// YUV
// #define MAX_READ_LEN ( 640 * 480 * 3 / 2 )

#define NUMBER_OF_BASELINE_IMAGES ( 50 )

// Blinks on RPi Plug P1 pin 40 (which is GPIO pin 21)
#define PIN ( RPI_V2_GPIO_P1_40 )

int main(int argc, char **argv)
{
  int numberOfImages = 0;
  uint32_t redAverage, greenAverage, blueAverage, threshold;
  size_t numberOfReadBytes;
  char buffer[ MAX_READ_LEN ];

  // If you call this, it will not actually access the GPIO
  // Use for testing
  //    bcm2835_set_debug(1);
  if (!bcm2835_init()) {
    return 1;
  }
  // Set the pin to be an output
  bcm2835_gpio_fsel( PIN, BCM2835_GPIO_FSEL_OUTP );
  // Turn it off
  bcm2835_gpio_write( PIN, LOW );
  int index, toggles = 0;
  int prevLEDState = LOW;
  // Read from rgb or yuv stream
  while ( true )
  {
    numberOfReadBytes = fread( buffer, sizeof(char), MAX_READ_LEN, stdin );
    if ( numberOfReadBytes < MAX_READ_LEN ) {
      if ( feof( stdin ) ) {
        printf( "EoF reached after %d images\n", numberOfImages );
        break;
      } else {
        printf( "%d bytes read\n", MAX_READ_LEN );
        exit( 2 );
      }
    }
    numberOfImages++;
    redAverage = 0;
    greenAverage = 0;
    blueAverage = 0;
    index = 0;
    for ( int i = 0; i < 640 * 480; i++ ) {
      redAverage += buffer[ index ]; index++;
      greenAverage += buffer[ index ]; index++;
      blueAverage += buffer[ index ]; index++;
    }
    redAverage /= ( 480 * 640 );
    greenAverage /= ( 480 * 640 );
    blueAverage /= ( 480 * 640 );
    // Use the first N images to get a red baseline
    if ( numberOfImages < NUMBER_OF_BASELINE_IMAGES ) {
      threshold = redAverage * 1.1;
    } else {
      if ( redAverage > threshold ) {
        // Turn it off
        bcm2835_gpio_write( PIN, LOW );
        if ( prevLEDState != LOW ) toggles++;
        prevLEDState = LOW;
      } else {
        // Turn it on
        bcm2835_gpio_write( PIN, HIGH );
        if ( prevLEDState != HIGH ) toggles++;
        prevLEDState = HIGH;
      }
    }
    printf( "%d,%d\n", redAverage,prevLEDState );
  }
  // Turn it off
  bcm2835_gpio_write( PIN, LOW );
  bcm2835_close();
  printf( "Toggles:%d\n", toggles );
  printf( "Number of frames:%d\n", numberOfImages - NUMBER_OF_BASELINE_IMAGES );
  printf( "Average latency:%4.2f frames\n", 1.0 * ( numberOfImages - NUMBER_OF_BASELINE_IMAGES ) / toggles );
  return 0;
}
Last edited by pmbdk on Sat Jan 04, 2020 3:57 am, edited 1 time in total.

pmbdk
Posts: 8
Joined: Thu Jan 02, 2020 2:09 pm

Re: Low-latency machine vision with RPi4 and PiCam 2.1

Sat Jan 04, 2020 3:48 am

This is the data I get from a run (leaving out the first 50 samples where I get the threshold for the LED on/off. I was wrong in stating two frames. It seems to be more like 2.5 frames on average which seems quite plausible.

Code: Select all

# Average red, LED state
98,1
100,1
189,0
191,0
112,0
98,1
103,1
190,0
192,0
111,0
98,1
99,1
188,0
192,0
110,0
99,1
99,1
189,0
192,0
150,0
98,1
98,1
167,0
192,0
150,0
98,1
98,1
171,0
192,0
120,0
98,1
100,1
189,0
192,0
111,0
99,1
103,1
191,0
192,0
112,0
98,1
100,1
189,0
192,0
111,0
99,1
104,1
191,0
192,0
113,0
98,1
100,1


Return to “Camera board”