Direction of Motion Vectors from h264 stream

[cmisip]

Hello again,
I have been working on trying to visualize the motion vectors including their location and direction as displayed in an RGB buffer by modifying the buffer to draw pixels in the patterns of arrows going towards 8 directions. For some reason, I am only seeing arrows that are directed eastward or southeast ward or southward. This to me indicates that the x_vector and y_vector are only positive values. I assumed that x_vector that is positive means macroblock motion to the Right and y_vector positive means macroblock motion downward. Maybe somebody can take a look at a code snippet or explain what I should be seeing in the macroblock x_vector and y_vector.

The code snippets here are based on the assumption of motion vector struct as having the following design. I have seen in other posts that it lists the sad before the x_vector and y_vector.

Code: Select all

    struct mmal_motion_vector {
     char x_vector;
     char y_vector; 
     short sad;
    };

Here is the code that iterates through all the motion vector blocks, determining direction of each block and setting bits in a byte.

Code: Select all

for (int j=0;j < numblocks ; j++) {
							
							uint8_t block_direction=0;
							 
							 //[nw][w][sw][s][se][e][ne][n]
							 //[0] [1][2] [3][4] [5][6] [7]
							 
							 
							 //[n][ne][e][se][s][sw][w][nw]
							 //[7][6] [5][4] [3][2] [1][0] 
                        mmal_motion_vector *mvarray=(mmal_motion_vector *)buffer->data;
                        (....)
                            if ((abs(mvarray[j].x_vector) + abs(mvarray[j].y_vector)) > min_vector_distance) { 
                               registers = registers | (0x80000000 >> count);
                            
                            
 /*     N      */           if (mvarray[j].x_vector >0) { //to e
                                if (mvarray[j].y_vector >0) { //to se
 /* W       E  */                  block_direction |= 1<<4;
                                } else if (mvarray[j].y_vector <0) { //to ne
 /*     S      */                  block_direction |= 1<<6;
                                } else { //e
                                   block_direction |= 1<<5;
							    }
                            } else if (mvarray[j].x_vector <0) {  //to w 
								if (mvarray[j].y_vector >0) { //to sw
                                   block_direction |= 1<<2; 
                                } else if (mvarray[j].y_vector <0) { //to nw
                                   block_direction |= 1<<0;
                                } else { //w
                                   block_direction |= 1<<1;
							    }
								     
                            } else { //dx=0
								if (mvarray[j].y_vector >0) { //to s
                                   block_direction |= 1<<3; 
                                } else if (mvarray[j].y_vector <0) { //to n
                                   block_direction |= 1<<7;
                                } else { //dy=0
                                
							    }  
							}
							
							//each Block save to the zones direction buffer on the jth position
							memcpy(direction[i]+j,&block_direction,sizeof(block_direction));
							
						    }

If the algorithm to read the directions of the macroblocks here is correct, then I know the problem is elsewhere.

Also, what would be a recommended value for SAD to determine if this macroblock should be considered for motion scoring?

Thanks,
Chris

[cmisip]

I changed the mmal_motion_vector struct into :

Code: Select all

    struct mmal_motion_vector {
     int8_t x_vector;
     int8_t y_vector;
     int16_t sad; 
    };

And that change resulted in motion vectors with arrows in the display pointing in the direction opposite the motion (pointing to where the macroblock moved FROM). I am now seeing vectors in all 8 directions. Maybe the devs can confirm that this is how the mmal motion vector should be defined as there seems to be conflicting info from various sources. I even tried putting the sad value before the x_vectors and y_vectors but that did not result in any motion detection.

Thanks,
Chris

[6by9]

You've been round this loop before and confirmed you had it working - https://www.raspberrypi.org/forums/view ... 0#p1184359
Use of char and short is slightly dangerous when casting areas of memory as they only guarantee the minimum size of the element, not the absolute size (see https://en.wikipedia.org/wiki/C_data_types#Basic_types).
On a Pi they should be equivalent, but int8_t / int16_t are guaranteed.

For reference, the datasheet says that the data written out is:

Code: Select all

Bit 31 INVALID Set for macroblocks not searched, because they were outside VLIMIT or HLIMIT. In this case, all the other fields will be zero.
Bits 30:16 SAD Best SAD value, including lambda bias R x
Bits 15:8 VOFF Vertical offset (whole pel), signed. Note that bit 8 will always be clear.
Bits 7:0 HOFF Horizontal offset (whole pel), signed. Note that bit 0 will always be clear

I would expect it to reflect where the image has moved from. Please remember that this is part of the video encoder, and it will be finding the best reference macroblock to be used to construct the new frame.

[cmisip]

Thanks for the reply 6by9.

I stopped working on this project last year and decided to pick it up again this year. The farthest I got then was visualizing the location of the blocks. Now I can see the direction. I am having a lot of fun working on it. This opens up new possibilities such as direction triggered events like detecting garage opening but not closing or maybe excluding things like falling snow, or excluding events that seem to have a very small overall composite vector averaged over time like maybe grass being blown by the wind.

I appreciate even the more basic advice since as you can tell I am learning cpp as I go along.

Chris