6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4703
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Image stabilization

Fri Feb 27, 2015 2:04 pm

DougieLawson wrote:Perhaps the answer is to only ever publish the "bleeding egde" untested stuff on BRANCH=next and only have camera code and fully tested kernel/bootcode/"firmware" stuff on the master branch. That way I have to do something special to attempt to render my RPi as a non-bootable zombie.
Isn't that the equivalent of people generally doing sudo apt-get dist-upgrade vs those on the bleeding edge doing sudo rpi-update?
It's even documented that way at http://www.raspberrypi.org/documentatio ... pdating.md
Updating the kernel and firmware

The kernel and firmware are installed as a Debian package, and so will also get updates when using the procedure above. These packages are updated infrequently (after extensive testing); if you want to try more recent experimental software, it's also easy to update to the latest available version using the rpi-update tool.
So you've already been warned that rpi-update is giving you the bleeding edge which has had less testing.

Perhaps someone who actually has to maintain these things would care to comment (though this is a slightly hidden discussion given the thread title), but otherwise I'm dropping out as I have no direct input on releasing.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

ethanol100
Posts: 481
Joined: Wed Oct 02, 2013 12:28 pm

Re: Image stabilization

Fri Feb 27, 2015 6:41 pm

Back to the original topic, the recorded video using -vs now is zoomed in. But which type of movements should be counteracted by the stabilization?
I have dangled the camera over the edge of the table and it performs an oscillating motion.
http://www2.geo.uni-bonn.de/members/kro ... byside.mkv
On the left is the "-vs" and on the right without the stabilization.
There is no obvious stabilization. Do I need some other motion, higher frequencies, lower amplitudes?

To the other topic, I would prefer unstable and stable firmware packages supplied trough apt-get repositories. At the moment applying an dist-upgrade will sometime overwrite the firmware which was received via rpi-update. But generally I like the idea of a bleeding edge firmware, where everybody can help with "beta" testing. A pure "debian" approach would be cleaner than a "manual" change of the boot files.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4703
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Image stabilization

Fri Feb 27, 2015 7:56 pm

ethanol100 wrote:Back to the original topic, the recorded video using -vs now is zoomed in. But which type of movements should be counteracted by the stabilization?
I have dangled the camera over the edge of the table and it performs an oscillating motion.
http://www2.geo.uni-bonn.de/members/kro ... byside.mkv
On the left is the "-vs" and on the right without the stabilization.
There is no obvious stabilization. Do I need some other motion, higher frequencies, lower amplitudes?
Enable the motion vectors bit from the annotate stage - that should show the motion that it is detecting (or not). I'd suggest you don't turn on any of the other data as I think it tries to put it in the correct place on the image.
I did say that I'd just blindly copied settings. I don't believe any of those who really understood the stabilisation algo are playing with Pi so asking them is a little cheeky. (I do work with one of them, so I'll see if she's in a good mood on Monday).
It was meant to reduce small amplitude shake, but had to be careful not to interfere if you start panning - not an easy task to differentiate. It only has a fairly small window to play with (that 5-10% zoom), so can't do anything about large amplitudes.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

redovan
Posts: 4
Joined: Tue Oct 07, 2014 2:24 am

Re: Image stabilization

Mon Mar 02, 2015 11:12 pm

I pulled in the latest, and I see the reduced FOV when I enable stabilization - sweet!

Unfortunately I do not see any compensation being performed after running a few shake tests (enable / disable stabilization with the camera on a simple shaker jig). I still need to confirm this with a more advanced test, but from an initial visual inspection I don't see any stabilization happening - just the zoomed in FOV :(.

Regardless I really appreciate you guys taking some time to look at this, and it definitely seems like it's closer to working! I'll play around with it some more and double check my setup.

ethanol100
Posts: 481
Joined: Wed Oct 02, 2013 12:28 pm

Re: Image stabilization

Tue Mar 03, 2015 5:02 pm

6by9 wrote: Enable the motion vectors bit from the annotate stage - that should show the motion that it is detecting (or not). I'd suggest you don't turn on any of the other data as I think it tries to put it in the correct place on the image.
It has taken a while to figure out a way to create a "software" video stabilization tests. I will display a picture on a monitor which will be randomly shifted in the range of -5 to 5 pixel in "x" and "y" direction. It will contain 9 dots overlayed fixed with the monitor. The camera captures the monitor and should try to correct the movement...
This is the captured video:
VidStab.mp4 (20MB,google drive)
The motion is captured but the image is static, the nine dots still keep there original position.
I did say that I'd just blindly copied settings. I don't believe any of those who really understood the stabilisation algo are playing with Pi so asking them is a little cheeky. (I do work with one of them, so I'll see if she's in a good mood on Monday).
It was meant to reduce small amplitude shake, but had to be careful not to interfere if you start panning - not an easy task to differentiate. It only has a fairly small window to play with (that 5-10% zoom), so can't do anything about large amplitudes.
Thank you for investing your time into this feature. I think it seems to almost work... ;)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4703
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Image stabilization

Tue Mar 03, 2015 5:10 pm

Does enabling show_motion in the annotate structure not show up anything? It should draw a set of red and blue lines on the image for raw (red) and integrated (blue) motion vectors detected in the image.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

ethanol100
Posts: 481
Joined: Wed Oct 02, 2013 12:28 pm

Re: Image stabilization

Tue Mar 03, 2015 5:48 pm

6by9 wrote:Does enabling show_motion in the annotate structure not show up anything? It should draw a set of red and blue lines on the image for raw (red) and integrated (blue) motion vectors detected in the image.
Yes it does, but it is a bit hard to see. I have copied the 12 vectors form this 1280x720 image frame, the text on the top says:
"Motion ( -4 , 3 ) Integrated Offset( -4 , 3)"
Vectors_-4_3.png
Vectors_-4_3.png (12.56 KiB) Viewed 1637 times
And the estimated motion vectors could fit, when manually comparing this frame to the previous one. I am a bit confused on the direction, does -4, 3 means the image has moved 4 px to the left and 3 px down? And I would need to shift the image 4px to the right and 3 px up? Because I would need to move the image down instead of up to get a minimum difference.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4703
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Image stabilization

Tue Mar 03, 2015 5:58 pm

ethanol100 wrote:Yes it does, but it is a bit hard to see. I have copied the 12 vectors form this 1280x720 image frame, the text on the top says:
"Motion ( -4 , 3 ) Integrated Offset( -4 , 3)"
Vectors_-4_3.png
And the estimated motion vectors could fit, when manually comparing this frame to the previous one. I am a bit confused on the direction, does -4, 3 means the image has moved 4 px to the left and 3 px down? And I would need to shift the image 4px to the right and 3 px up? Because I would need to move the image down instead of up to get a minimum difference.
Image stabilisation is in two bits - a software stage that detects the motion from the Bayer image (and has to compute the motion vectors before the ISP can start on the image), and then some calcs that work out if and how to compensate for that motion.
It sounds like the first bit is working (assuming those values are changing in a sensible manner as the sensor is moved), but the second bit isn't. It'll be a case of digging into the settings that are applied and ensuring that they make sense/actually do anything.

Thanks (as always) for doing some of the legwork for me. I'm unlikely to get a chance to look at it tonight, but potentially soonish.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lwi
Posts: 3
Joined: Thu Sep 17, 2015 8:51 pm

Re: Image stabilization

Wed Sep 23, 2015 9:50 pm

Sorry to revive an older thread, but are those motion vectors accessible via mmal or omx? Or are they hidden in the firmware?

Because if they are, we could implement video stabilisation in a shader (in a new gl scene).

thanks

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4703
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Image stabilization

Sat Sep 26, 2015 8:14 pm

lwi wrote:Sorry to revive an older thread, but are those motion vectors accessible via mmal or omx? Or are they hidden in the firmware?

Because if they are, we could implement video stabilisation in a shader (in a new gl scene).
Hidden in the firmware I'm afraid.
The actual stabilisation is for free once the amended crop region is computed, whereas GL would have a significant overhead and not be recordable by the video codec.
ethanol100's tests imply that the motion vectors are all computed correctly, so it is only the computation of the new crop window that is wrong. It is on the list to look at, and I'm hopeful I may get a day or two of Pi time soon :)
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

czar1na
Posts: 129
Joined: Wed Dec 14, 2016 9:46 am

Re: Image stabilization

Fri Jan 20, 2017 1:49 pm

6by9 wrote:
lwi wrote:Sorry to revive an older thread, but are those motion vectors accessible via mmal or omx? Or are they hidden in the firmware?

Because if they are, we could implement video stabilisation in a shader (in a new gl scene).
Hidden in the firmware I'm afraid.
The actual stabilisation is for free once the amended crop region is computed, whereas GL would have a significant overhead and not be recordable by the video codec.
ethanol100's tests imply that the motion vectors are all computed correctly, so it is only the computation of the new crop window that is wrong. It is on the list to look at, and I'm hopeful I may get a day or two of Pi time soon :)
Sorry to revive an old thread, but I'm also interested in using video stabilization and wondering the status of the stabilization setting, since the above posts make it sound like it never fully worked?

To be specific, I'd ideally like to counteract the oscillating motion of a camera being worn on a walking person. But if this is asking for too much, correction of jitteriness is useful too :D

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4703
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Image stabilization

Fri Jan 20, 2017 1:57 pm

It's still on the to-do list (it's a long list).
IIRC it still doesn't compensate for rolling shutter effects, so you are likely to still see some distortion if the person is walking and you are needing even moderately long exposure times.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Piper984
Posts: 10
Joined: Tue Oct 07, 2014 7:37 pm

Re: Image stabilization

Thu Dec 07, 2017 5:06 pm

Hello folks, sorry to post to an older thread, but I thought this would be the best place to capture a learning regarding raspivid --vstab.

Using a B+ with the very latest Raspbian (2017-11-29), I ran ...

Code: Select all

sudo rpi-update
... to make sure I had the latest firmware installed.

I have found that when using the V1 camera modules, running the command ...

Code: Select all

raspivid -t 0 --vstab -a 256
... I will see the Motion and Integrated Offset numbers change along with the motion dots/lines in the video preview when I wiggle the camera module. When I use the newer V2 camera module, then Motion and Integrated Offset numbers are all set to zero and no motion indicators in the preview video.

I popped that same SD card into a Pi 3 and saw the same behavior.

Any comment from RPi Foundation on this? Maybe something simple like an #ifdef looking for the old Omnivision chip in the firmware somewhere?

-Piper

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4703
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Image stabilization

Thu Dec 07, 2017 5:41 pm

Piper984 wrote:
Thu Dec 07, 2017 5:06 pm
I have found that when using the V1 camera modules, running the command ...

Code: Select all

raspivid -t 0 --vstab -a 256
... I will see the Motion and Integrated Offset numbers change along with the motion dots/lines in the video preview when I wiggle the camera module. When I use the newer V2 camera module, then Motion and Integrated Offset numbers are all set to zero and no motion indicators in the preview video.

I popped that same SD card into a Pi 3 and saw the same behavior.

Any comment from RPi Foundation on this? Maybe something simple like an #ifdef looking for the old Omnivision chip in the firmware somewhere?
Indeed, the stabilisation analysis was never enabled on the IMX219 for some reason. (Not quite an ifdef, but not far off).
I can't see any harm in enabling it, although the stabilisation algorithm was always a little temperamental so the results often aren't stellar. I'll push a change that at least enables users to try it out.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4703
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Image stabilization

Tue Jan 09, 2018 9:36 am

6by9 wrote:
Thu Dec 07, 2017 5:41 pm
Indeed, the stabilisation analysis was never enabled on the IMX219 for some reason. (Not quite an ifdef, but not far off).
I can't see any harm in enabling it, although the stabilisation algorithm was always a little temperamental so the results often aren't stellar. I'll push a change that at least enables users to try it out.
Sorry it's taken a while - Christmas and other priorities meant that there weren't many firmware releases for a while.
The latest rpi-update firmware has now enabled stabilisation on IMX219. Normal warnings apply over using rpi-update and backing up first.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”

Who is online

Users browsing this forum: No registered users and 20 guests