CME -x postponed?


57 posts   Page 1 of 3   1, 2, 3
by peepo » Sat May 10, 2014 2:08 pm
did I miss the main event?

Gordon: "Will get it pushed in tomorrow, promise!"
24th April 2014

could we please have an update?
raspivid -x

as they say 2 weeks is a long time in politics

over-anxious parent

~:"

Vectors from coarse motion estimation
http://www.raspberrypi.org/vectors-from ... /#comments
User avatar
Posts: 304
Joined: Sun Oct 21, 2012 9:36 am
by jamesh » Sat May 10, 2014 2:50 pm
I htought it had already been pushed. There was a firmware update yesterday.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16724
Joined: Sat Jul 30, 2011 7:41 pm
by steve_gulick » Sun May 11, 2014 2:21 am
No, the latest raspivid.c has no -x in its menu options - no mention of motion vectors in the code.
If there is some problem integrating Gordon's motion vector extraction code with the rest of raspivid.c, could Gordon publish his version that has the motion vector code as a separate program, maybe "raspividX.c"? And maybe also the example code he used to produce the 2 GIFs (walking man and juggler) that demonstrate its use. It would be a great help to the many of us interested in motion detection. Using the GPU to provide the motion vectors seems a brilliant use of the capabilities BCM8235 and probably unique to the Raspberry Pi platform.
Thank you!
Steve
Posts: 31
Joined: Wed Jul 18, 2012 12:06 pm
by gsh » Sun May 11, 2014 9:36 am
Sorry,

I lost the code so will have to do it again, it's very easy to implement though, all the MMAL options have been implemented and pushed, you just have to set the INLINE_VECTORS parameter for the encode component. The callback function is then hit with buffers with a new flag in them (if you print out the flags you'll see the difference between the two).

This is the flag value you get
MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO

This is the parameter you need
MMAL_PARAMETER_VIDEO_ENCODE_INLINE_VECTORS

Just add it much like the INLINE_HEADERS option is already added...

Then you should get the data...

Note you get one integer per macroblock (16x16 block) with the MS word the minimum absolute sum of differences found and the LS word the 8 bit vector in pixels (signed)

So 0x0030fe08

Would be a minimum SAD of 0x0030 with a vector of (-2, 8)

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1141
Joined: Sat Sep 10, 2011 11:43 am
by ethanol100 » Sun May 11, 2014 11:47 am
I would like to ask some questions. Does the buffer contains a header? From your blog post I got that the data is stored in (mbx+1)*mby 4byte blocks, these blocks are written line by line. This would be 32912 bytes for a 1920x1080 image. The buffer size is exactly the same, but if I write these buffer to file the first line has an offset of about 19pixels. And the vectors are stored as "backward" pointing vector to the last known position, so we need to invert them to get "velocity" vectors? If it has no header, but I would like to produce a valid h264 stream including the motion vectors, could we add a Nal header of type sei to the buffer?

Edit: my fault, there is no header, I just have some problems to use memcpy to copy the buffer to an array of structs.
Edit 2: I am so stupid, have memcpy buffer instead of buffer->data...
Edit 3: If anyone wants to play you can find my try at https://github.com/ethanol100/userland/tree/imv
Last edited by ethanol100 on Sun May 11, 2014 4:00 pm, edited 3 times in total.
Posts: 448
Joined: Wed Oct 02, 2013 12:28 pm
by peepo » Sun May 11, 2014 12:32 pm
gsh,

what a hassle,
well unfortunately this is all a bit mandarin for me,
so will wait...

~:"
User avatar
Posts: 304
Joined: Sun Oct 21, 2012 9:36 am
by kaos » Sun May 11, 2014 1:59 pm
gsh wrote:with the MS word the minimum absolute


For one moment I wondered what Microsoft's word processor had to do with this :shock: , and then realized that MS meant most significant in this case :lol: .

gsh wrote:vector of (-2, 8)


Sorry if it's a stupid question, but what kind of vector are we talking about here? The geometric kind that has a heading and velocity? If so, what are the units?

--
Best regards, Kári.
Posts: 108
Joined: Mon Mar 26, 2012 8:14 pm
by ethanol100 » Sun May 11, 2014 2:20 pm
kaos wrote:Sorry if it's a stupid question, but what kind of vector are we talking about here? The geometric kind that has a heading and velocity? If so, what are the units?


The units are pixel. It is the displacement of this macro block, to the most similar part in the last image. Say it says x=16 y=8, you cut crop these macroblock and shift it 16 pixel to the right and 8 pixel down. and compare this with the old image, then there would be the minimum difference. These units are now in the range from -128 to 127 pixel in x and y direction. If you move your image to far the algorithm would not find it. This helps the encoder, which basically only stores differences, to store the minimum amount of differences if he takes these shift values into account.
Posts: 448
Joined: Wed Oct 02, 2013 12:28 pm
by gsh » Sun May 11, 2014 6:28 pm
@ethanol100

If you've got it working then please submit a pull request. I'm fairly sure it should be a very simple PR!

@peepo

Sorry, if you think this is hassle, but it's not going to get any easier to actually process the data!

Be very interested if anyone has been able to get something useful out of it, basically if you compile a histogram of the vector values then you should be able to implement a very simple motion type thing in Raspivid and trigger the video recording / playback very simply...

It'd be very interesting for someone to hook it up to the circular buffer code I wrote previously because then you can record the period before and after the trigger period. I'd like to use this as a cycling camera... I'd set up a number of Pi's (with batteries) running the camera app then when someone cycles past it records each person. Then you could automatically stitch the video together to create video recordings of a mountainbike race for example! (It's something that I do!)

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1141
Joined: Sat Sep 10, 2011 11:43 am
by ethanol100 » Sun May 11, 2014 7:57 pm
gsh wrote:@ethanol100

If you've got it working then please submit a pull request. I'm fairly sure it should be a very simple PR!

Yes, if we want simply to output the raw buffer to file it is really easy. When I want to do some processing with these I have some problems.

I'm not yet happy with it, using memcpy to copy the vectors into a structure takes too long, so I start to drop frames. I want to use a structure, to easy access the data and to output it in a more user-friendly way. Additionaly I need to test it, because I am not sure if I break many things, i.e. the circular buffer, segmentation...

Be very interested if anyone has been able to get something useful out of it, basically if you compile a histogram of the vector values then you should be able to implement a very simple motion type thing in Raspivid and trigger the video recording / playback very simply...

Nice idea, but I guess it will be to slow, will try to play with that.

It'd be very interesting for someone to hook it up to the circular buffer code I wrote previously because then you can record the period before and after the trigger period. I'd like to use this as a cycling camera... I'd set up a number of Pi's (with batteries) running the camera app then when someone cycles past it records each person. Then you could automatically stitch the video together to create video recordings of a mountainbike race for example! (It's something that I do!)

That was exactly what I was trying to archive the last hour after I noticed that I need to edit the the circular buffer to keep the h264 stream clean, if someone tries to use circular buffer and the inline motion vectors together. Now I just take one velocity vector to check for change. So very simple and basic. I can now use "raspivid -trigger 960,540,80,0.2 -o test.h264" to monitor the macro block containing the coordinate x=960 and y=540 and trigger the recording, if the velocity magnitude exceeded 80. The last 0.2 is the part of the timeout before trigger, i.e., setting it to 0.2 will record 0.2*timeout before and 0.8*timeout seconds after the trigger event. Here it is getting dark and I can't test it today. Now I have changed to much for a pull request, because I need some "internal" trigger and need to change some other parts and need to include math.h for "sqrt". And I'm not sure if the velocities are stored "[unsigned char vx][unsigned char vy][short sam]" or the other way around.

But it is really easy to use in raspivid. Will try to continue tomorrow. Have pushed my changes to https://github.com/ethanol100/userland/tree/imv
Posts: 448
Joined: Wed Oct 02, 2013 12:28 pm
by algorithm » Sun May 11, 2014 8:04 pm
gsh wrote:@peepo

Sorry, if you think this is hassle, but it's not going to get any easier to actually process the data!

Oh, I thought he meant a hassle for you, that you lost the code and would have to rewrite it.
User avatar
Posts: 134
Joined: Mon Nov 25, 2013 9:09 pm
Location: Flatland
by peepo » Mon May 12, 2014 6:13 am
indeed,

though as gsh indicates, I don't expect it to get any easier....

~:"
User avatar
Posts: 304
Joined: Sun Oct 21, 2012 9:36 am
by Rayden » Mon May 12, 2014 12:06 pm
gsh wrote:Note you get one integer per macroblock (16x16 block) with the MS word the minimum absolute sum of differences found and the LS word the 8 bit vector in pixels (signed)


Hi Gordon,
Just quick question, vector is in whole pixels not quarter pixels? Does the pi encoder produce quarter-pixel vectors internally?
Posts: 15
Joined: Fri Jan 03, 2014 4:23 am
by 6by9 » Mon May 12, 2014 12:51 pm
Rayden wrote:Hi Gordon,
Just quick question, vector is in whole pixels not quarter pixels? Does the pi encoder produce quarter-pixel vectors internally?

The H264 hardware has both a Coarse Motion Estimation (CME) and a Fine Motion Estimation (FME) block - there are power savings if the search range is reduced on the fine search.
Gordon's code is spitting out the results of the CME only, and that is whole pixels vectors only.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 3823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.
by ethanol100 » Tue May 13, 2014 8:18 am
With the rpi-update today you will be able to save the inline motion vectors to a file. It really only dumps the raw buffer to a file. Hopefully I have not broken anything.

It can be used with, i.e.:
Code: Select all
 raspivid -x test.imv -o test.h264

The file test.imv will have a size of "number of frames * (120+1)*68*4 bytes".
You can split these buffer to individual files using split:
Code: Select all
split -a 4 -d -b $(((120+1)*68*4)) test.imv frame-

From this you will get frame-0000, frame-0001,... each containing one frame of motion vectors.

For some post processing I have added the source for two very simple programs:
https://github.com/raspberrypi/userland/tree/master/host_applications/linux/apps/raspicam/imv_examples
With these you can create images of the velocity magnitude and export the buffer to a more user-friendly text format. These text-format files can be used to plot the velocity vectors with some plotting software(i.e., xmgrace, which is in the debian package "grace").

If you want to work with the inline vectors in the source of raspivid. You can create an array of structures similar to the imv2txt.c example and memcpy "&buffer->data[0]" to that structure(Instead of the line "bytes_written = fwrite(buffer->data, 1, buffer->length, pData->imv_file_handle);"). Or you can access single elements (i,j) with:
Code: Select all
signed char vx=0;
signed char vy=0;
short sam=0;
memcpy(&vx, buffer->data+4*(i+(pData->pstate->mbx+1)*j), sizeof(signed char));
memcpy(&vy, buffer->data+4*(i+(pData->pstate->mbx+1)*j)+1, sizeof(signed char));
memcpy(&sam, buffer->data+4*(i+(pData->pstate->mbx+1)*j)+2, sizeof(short));


Hope that helps to get you started to play with these motion vectors.

A small note: you need to specify an outputfile with -o filename, else no h264 encoding is done and no motion vectors are created, but you can use /dev/null as output file to send the encoded data to nowhere. The output of motion vectors is disabled for the circular buffer mode but you can play with it and add some trigger code or something. Now there is only a place holder in the encoder callback.
Posts: 448
Joined: Wed Oct 02, 2013 12:28 pm
by jamesh » Tue May 13, 2014 8:28 am
Thanks for that code update Ethanol100 - you have joined a select bunch of 'people who have contributed to raspicam'!
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16724
Joined: Sat Jul 30, 2011 7:41 pm
by steve_gulick » Tue May 13, 2014 9:02 am
My thanks also to Ethanol100. Just what I needed. Great job!
Steve
Posts: 31
Joined: Wed Jul 18, 2012 12:06 pm
by ethanol100 » Tue May 13, 2014 9:14 am
It was only a small pull request and Gorden has told exactly what to do. So the thanks should go to Gorden. For the velocity vectors I would like to get some feedback, I'm not sure if I have swapped the x and y component.
Posts: 448
Joined: Wed Oct 02, 2013 12:28 pm
by peepo » Tue May 13, 2014 1:01 pm
I am currently concerned that CME might be already breaking OMX
thread:
rpi-update: encode -> black
viewtopic.php?f=43&t=77066
and
encode_OGL encoding to black, intermittent?
https://github.com/peepo/openGL-RPi-tutorial/issues/3

the issue being that recent images with latest userland encode to black.
ie after openGL shaders, not from raspivid.

would appreciate some testing and feedback

cant wait to get started on -x and add a tutorial

~:"
User avatar
Posts: 304
Joined: Sun Oct 21, 2012 9:36 am
by ethanol100 » Tue May 13, 2014 1:20 pm
The firmware CMV support has been added to the firmware on Apr 11, 2014 with this commit:
https://github.com/Hexxeh/rpi-firmware/commit/5c5ed9537e93371ebcf99f64466ee3a1eae28de0
Which kernel are you running? Have you rebooted the raspberrypi after rpi-update?
Have you tested to downgrade your firmware with rpi-update to the version from yesterday? And one before Apr 11?

I have only tested raspivid. No idea how this should influence omx or ogl.

Just for reference:
Firmware from May 09, 2014: "rpi-update 3f722be8d8de6740b32d76ef938a05744bf0d34f"
Firmware form Apr 07, 2014: "rpi-update f0eeb5b9431a87906ee77bc70a5baa9203c361e2"
Posts: 448
Joined: Wed Oct 02, 2013 12:28 pm
by peepo » Tue May 13, 2014 3:02 pm
ok tx for that iiuc

rpi-update f0eeb5b9431a87906ee77bc70a5baa9203c361e2
to regress to April 07
reboot
then try encode_OGL
if it then encodes as expected, we may surmise 'bug' introduced subsequently.

am enacting as above

did not resolve issue, still remains black, this rpi-update was backwards,
next one will be forwards, from a fresh install 140107.
Last edited by peepo on Tue May 13, 2014 4:53 pm, edited 2 times in total.
User avatar
Posts: 304
Joined: Sun Oct 21, 2012 9:36 am
by jamesh » Tue May 13, 2014 3:20 pm
Not necessarily an introduced bug in the firmware. It may be a pre-existing issue in your code that only now exhibits with a change to the firmware. But not able to make that judgement just yet.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16724
Joined: Sat Jul 30, 2011 7:41 pm
by peepo » Tue May 13, 2014 3:48 pm
absolutely agree, encode_OGL is not a great reduced test case,

I have taken some more openGL stuff out, mainly the 3d aspect.

It would be great to have a minimal test case tutorial, both for testing and for teaching

feel this thread is being poached

~:"
User avatar
Posts: 304
Joined: Sun Oct 21, 2012 9:36 am
by ethanol100 » Tue May 13, 2014 4:04 pm
I would guess, that the code breaks when the new full FOV modes where introduced. You are requesting a resolution of 640x640 which will chose one of the new modes. Somewhere there has been reported a similar problem with the "RPi Cam Web Interface" http://www.raspberrypi.org/forums/viewtopic.php?p=542523#p542523. So I guess you need to update your code somehow.
Posts: 448
Joined: Wed Oct 02, 2013 12:28 pm
by 6by9 » Tue May 13, 2014 4:31 pm
ethanol100 wrote:I would guess, that the code breaks when the new full FOV modes where introduced. You are requesting a resolution of 640x640 which will chose one of the new modes. Somewhere there has been reported a similar problem with the "RPi Cam Web Interface" http://www.raspberrypi.org/forums/viewtopic.php?p=542523#p542523. So I guess you need to update your code somehow.

Full FOV firmware changes went in on Mar 14, 2014, so those are present in f0eeb5b9431a87906ee77bc70a5baa9203c361e2 April 07 that peepo says works.
CME / inline motion vectors code was released in the Apr 11, 2014, but only touches the video encode component which isn't being used in that app. The changes to MMAL shouldn't affect anything else (it adds the new parameter enum and a mapping function for it)

What exactly is the problem in "RPI Cam Web Interface"? I'm not trawling through 29 pages of forum post trying to extract the info when there will be many issues discussed and many fixed. Some comments seem to imply MJPEG, but that can't be the issue here as we're not encoding.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 3823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.