How to sense gestures like right, left, front, back by using pi camera.

[Ravi Kiran]

Very good morning to all.

I'm doing a project in that i want to operate my vehicle diversion indication lights to glow by sense ing some gestures like left, right using pi camera from the driver.

I'm new to raspberry pi, so please help me for this project, thanks in advance.

[topguy]

https://www.intorobotics.com/9-opencv-t ... cognition/

https://www.youtube.com/watch?v=W4Afc9qcj4I


There is also alternative sensors than camera around.
https://www.pi-supply.com/flick-raspber ... ol-boards/

[scotty101]

Rather than a camera, would you consider something like this?
https://shop.pimoroni.com/products/skywriter-hat

A critical thing to consider with this project is that people actually like the positive feedback they get from moving the indicator lever. If you move to a gesture based system, will they KNOW that the indicators have been activiated/cancelled without becoming distracted from the road.
This is a big consideration for the human factors in aircraft with more and more touchscreens. The pilots like to feel a button being pressed, hear the click and instantly see a change.

[cpunk]

To get gesture information, you might ask the "raspivid" command to output motion vectors from the video encoding process (there are other possibilities too). If you pick that route, you can do that with the -x command line option.

The output will be one motion vector per each 16 x 16 pixel macroblock. I have found the following handy (but I cannot guarantee the correctness of any code below) to describe the data block that gets output for each video frame (sample case: 320 x 320 pixels, snippet from Java code):

Code: Select all

int W = 320;
int H = 320;
int MB_SIZE = 16;
int MB_COLS = W / MB_SIZE;
int MB_ROWS = H / MB_SIZE;
int MB_COUNT = ((MB_COLS * MB_ROWS) + MB_COLS);
int VECSIZE = (MB_COUNT * 4);

Then, your program would have to read the output stream of "raspivid" and analyze the vectors... and decide if the criteria for a gesture command have been met.

In this step, you would find that simple gestures are easy to detect (e.g. "is the average of all vectors strongly biased to the left"), but that more complicated patterns are very hard to catch. :)

A snippet from my Java code to decode the vectors from a byte array... maybe useful, maybe not:

Code: Select all

import java.awt.Point;
public class MotionVector {
	public Point center;
	public Point dir;
	public MotionVector(int centerX, int centerY, int dirX, int dirY) {
		center = new Point(centerX, centerY);
		dir = new Point(dirX, dirY);
	}
}


ArrayList<MotionVector> vectors = new ArrayList<MotionVector>();


private void getVectors(byte[] v) {
    	
    	// Prepare array for consumption
    	ByteBuffer bb = ByteBuffer.wrap(v);
    	
    	int mby = 0;
    	int mbx = 0;
    	
    	vectors.clear();
    	
    	for (int i = 0; i < MB_COUNT; i++) {

    		byte x = bb.get();
    		byte y = bb.get();

    		// SAD is generally useless for motion analysis, but read the bytes anyway.
    		byte sadLo = bb.get();
    		byte sadHi = bb.get();
    		int sad = ((int) sadHi << 8) | ((int) sadLo & 0xFF);
    		
    		int centerX = mbx * VideoGrabber.MB_SIZE;
		int centerY = mby * VideoGrabber.MB_SIZE;

    		vectors.add(new MotionVector(centerX, centerY, x, y));
    		
    		// Row and column incrementer.
    		mbx++;
    		if (mbx > MB_COLS) {
    			mbx = 0;
    			mby++;
    		}
    	}
   }