Sight for the Blind for <100$

[mikey11]

Hello all,

I crosschecked the Raspberry PI specs's against running the software designed by Peter Meijer who runs this site:

http://www.seeingwithsound.com/

He made software that takes webcam input and converts it into a soundscape, providing sight to the blind at a resolution that is more than competitive with surgical methods costing many orders of magnitude more.

In the drive to bring the cost of sight for the blind down, I think the rpi is the cheapest possible hardware platform to run this software on with the lowest power consumption (longest battery life, cheaper batteries), and I can't find any information on it being done yet. Current implementations use hardware that costs $400-500, and I believe with the rpi you can do it for $100.

They have produced a version as an android app, so I think it should be possible to make it run on the rpi, but I think you can do it at a lower level, and more effectively on the rpi than how the android app is operating. The android app has a lower resolution than the original program, probably because of performance issues, or maybe intellectual property issues.

I think this is a worthy goal, and I've already emailed him to make him aware if he isn't already. The more people who get in touch with him on the subject, the more likely he would be to set some code free for this purpose. I notice he has some free versions. Perhaps if we had rpi developers who would code not to exceed the capabilities of the free versions he might be interested

If you have coding skills, and think you could link up image processing with sound generation, then you should definitely get ahold of this guy!

I don't have low level coding skills, but these may not even be required. It looks like he is using high level languages on windows machines, and recommends processing power that is less than the rpi's specs.

If we could get a lower level implementation going, there should be no performance issues.

I like the rpi for silly projects, but this kind of project carries a lot of meaning for everyone. If you have the skills, please consider it. If you don't have the skills, I think politely pestering Peter about it might help spur it on.

Thanks, Mike

edit:

Peter is aware of this thread and has been posting. Therefore, its not necessary to pester him!

I got a reply from Peter as follows:

Hi Mike,

Thank you for your interest. Some people on the seeingwithsound list
http://www.seeingwithsound.com/subscribe.htm have been working towards
a version of The vOICe for Raspberry Pi, based on my elementary C code
from http://www.seeingwithsound.com/im2sound ... ial_scenes
and OpenCV. So you could join the seeingwithsound list and ask around
about the current status.

Best regards,

Peter Meijer

I am going to start sifting through that and see where it takes me.

I found this so far (and duh? it's on this site. haha)
http://www.raspberrypi.org/archives/tag/opencv
http://www.seeingwithsound.com/subscribe.htm

[seeingwithsound]

Hi Mike,

There is now a modified version of my hificode.c soundscape generation program that you can use with OpenCV under Windows or Linux for sounding the live camera view:

http://www.seeingwithsound.com/hificode_OpenCV.cpp

This is quite unpolished code that is not optimized in any way, so it is just to get you started. For some context on the hard-coded images (now disabled by default but left in for debugging purposes), visit

http://www.seeingwithsound.com/im2sound.htm

Any chance that Android will be ported to Raspberry Pi "soonish"? Then one could just run The vOICe for Android on Raspberry Pi. However, it looks like progress in this area has slowed down lately, judging from the recent discussion at http://forum.xda-developers.com/showthr ... ?t=2321557

Best regards,

Peter


Seeing with Sound - The vOICe
http://www.seeingwithsound.com

[mikey11]

I thought about using the android port to the RPI, but I don't think that is a great solution. I think it's a bad mismatch of the resources of the pi to the task at hand (other than being able to re-use code that has been proven to be functional).

I've tried using numerous distributions on the RPI, and I have found that any of them that use a graphical interface just barely crawl along. The reasons behind this have been that the RPI's GPU has not been engaged to handle the video.

Unless this is different in the android port, then having this type of interface completely eats of the CPU to render the graphical interface, rendering other tasks slow and buggy.

I consider the vOICe to be the ultimate in graphical user interfaces, but it no doubt is going to eat up CPU cycles to process images and create the audio. I doubt the CPU could do both the android environment, and efficiently do the rest.

The great power of the RPI in my opinion is to run as a headless device, and maximize it's computational power on other tasks.

At least, thats my philosophy behind things.

From my cursory reading of the openCV information for the RPI, it looks like using the RPI camera is a computationally efficient method of image capture when compared to a USB webcam because it has access to the GPU functions (where a USB camera would not), so I would probably want to focus on using the native camera and avoiding USB webcams, except maybe as a way to make early development easier (I don't have an rpi camera module yet).

To me, using the camera module will offload some of the image processing to the GPU, leaving the CPU more responsive to taking that input and generating the sound output.

For example:

http://hackaday.com/2012/05/30/gpu-prog ... rocessing/

This example does edge detection via GPU, so I think you might be able to generate optimized sharpened images without putting that work onto the CPU. It looked like sharpened images would be easier to cognitively understand from my looks at Peters site.

I of course may be mistaken on some of this information, as I am relatively new to the RPI, and have never implemented vOICe code on any device.

[ghans]

Just FYI , this is the last update from Broadcoms side on
Android :
http://www.raspberrypi.org/phpBB3/viewt ... 39#p344518

To be honest , that was not unexpected for me.

I don't know why the foundation opened pandoras box with
the post last year after all ...

ghans

[PranavLal]

Hi all,

I am one of the people working on porting the vOICe to the pi. I have made a couple of minor modifications to Peter's code and have sent them to him for inclusion. Basically, the includes needed fixing and I commented out the lines referencing the display since they required the GTK which is resource inrtensive at least as far as I know.

The reason for a USB camera is because we have commertially available video glasses for the vOICe.
Please see the thread at
http://www.raspberrypi.org/phpBB3/viewt ... 13#p387013

which deals with the porting of the video glasses to the pi. We need head mounted cameras for the vOICe to be useful.

The camera board is an option but I see no way of mounting it on my head. I am a user of the vOICe and I need something functional.

As of now, the code runs very slowly on the pi. By slowly I mean that there is a significant time lag between soundscapes. This may be fixable with code optimization but I do not fully understand what the code is doing so cannot optimize. My development so far has been almost like playing with Lego blocks and sticking the code together. I did specify the compiler optimization in my compilation commandline but don't know if it is helping.

Another issue with the current code is that it is writing to a file which is a bad idea but that is something I cannot fix for the reasons specified above. I have tried to modify Peter's code as little as is possible. There is a another developer helping with this on the opencv forum called Berak (I hope I am spelling his handle correctly) who has tried a few things too. I wish there was a way to cc people across forums since this discussion is fascinating.

Pranav

[seeingwithsound]

Thanks Pranav, I'll fix the missing include file(s) for Linux once I receive your update. I could only quickly test my code with OpenCV under Windows. Concerning the camera glasses driver issue, I do not know which Syntek drivers other than the Windows STK03N drivers are compatible with the camera glasses that we use, but you could check out the website http://sourceforge.net/projects/syntekdriver/ which contains recent work on Linux/Unix drivers for some Syntek devices.

Peter


USB Camera Glasses for the Blind
http://www.seeingwithsound.com/camera_glasses.htm

[PranavLal]

Hi Peter and all,
Yes, I am referring to http://sourceforge.net/projects/syntekdriver/

It is not possible to compile the drivers due to a missing make file. See the following shell script.
http://kaz.dl.sourceforge.net/project/s ... /syntek.sh

I have written to the author of the above script but have not got a response until now.


Pranav

[seeingwithsound]

Sure. That shell script refers to a Makefile-syntekdriver that you find in the Internet archive at the URL http://web.archive.org/web/201002171114 ... ntekdriver

Peter


USB Camera Glasses for the Blind
http://www.seeingwithsound.com/camera_glasses.htm

[mikey11]

Hi all,

I am one of the people working on porting the vOICe to the pi. I have made a couple of minor modifications to Peter's code and have sent them to him for inclusion. Basically, the includes needed fixing and I commented out the lines referencing the display since they required the GTK which is resource inrtensive at least as far as I know.

The reason for a USB camera is because we have commertially available video glasses for the vOICe.
Please see the thread at
http://www.raspberrypi.org/phpBB3/viewt ... 13#p387013

which deals with the porting of the video glasses to the pi. We need head mounted cameras for the vOICe to be useful.

The camera board is an option but I see no way of mounting it on my head. I am a user of the vOICe and I need something functional.

As of now, the code runs very slowly on the pi. By slowly I mean that there is a significant time lag between soundscapes. This may be fixable with code optimization but I do not fully understand what the code is doing so cannot optimize. My development so far has been almost like playing with Lego blocks and sticking the code together. I did specify the compiler optimization in my compilation commandline but don't know if it is helping.

Another issue with the current code is that it is writing to a file which is a bad idea but that is something I cannot fix for the reasons specified above. I have tried to modify Peter's code as little as is possible. There is a another developer helping with this on the opencv forum called Berak (I hope I am spelling his handle correctly) who has tried a few things too. I wish there was a way to cc people across forums since this discussion is fascinating.

Pranav

Pranav,

I stressed that I thought using the rpi's camera board was part of an overall performance increase solution. The reasoning behind this was that the rpi camera board can access GPU functions.

You have indicated that code you have cobbled together seems to be quite slow. Offloading the image acquisition and processing functions to the GPU will solve that problem to a large degree.

Another thing you mention is saving images to files, and this is also a bad idea. Most implementations of the rpi are using the SD card which is slow to begin with, and has problems with persistent rewrites over time.

The best way to handle the vOICe would be to only store the images in memory (RAM), and do any transformations there. This would solve numerous problems. Staying in RAM is faster (and the small resolution virtually guarantees that this is possble).

As for how to mount the rpi plus it's camera board, I don't see this as being a huge issue. I could easily mount it on the front of a ballcap. It might not be 'fashion forward', but I am really jazzed at the idea of seeing a device to help vision impaired people that has a price tag of $100.

I would likely worry about the aesthetics later.

I haven't had time to really launch into this yet, and so much of it is new territory for me, but when I get caught up, I will absolutely be looking at how to tie openCV to the rpi camera board, and keep image processing in memory.

I believe that if this software is approached that way, it makes the best use of resources, and increases the chances for success.

[mikey11]

Some things also occurred to me over the weekend while thinking about this software.

In the example software on windows I notice you can do things like invert the grayscale/sound, and change the playback rate to give a faster audio framerate, amongst quite a few other effects.

With the rpi, I think taking the most often used functions, and putting them on a handheld control with 5 buttons or so, and tying those buttons to the rpi's GPIO header could make for a more functionally useful device.

I don't use vOICe personally despite my interest in it, so I don't know what features are particularly useful for people that do use it.

Pranav indicated he uses this software, care to chime in and let us know which of those features seem to be the most useful?

[seeingwithsound]

> Offloading the image acquisition and processing functions to the GPU will solve that problem to a large degree.

Image acquisition, color conversion and scaling is done via OpenCV, so proper use of associated resources is the responsibility of the OpenCV library implementation for RPI.

> Another thing you mention is saving images to files, and this is also a bad idea.

Images are not saved to file. Soundscapes are, for lack of a platform independent interface for playing audio output. In the early days of Android there was no support for in-memory sound synthesis, so I had to write soundscapes to SD card before playing them (on Android phones). Although ugly, this did not give a large performance penalty, and SD card wear is limited through automatic low-level reallocation of memory cells on the SD card. In other words, one should first measure and check where time is really spent on the RPI rather than making assumptions about it. Next one can focus on the main RPI performance bottlenecks, tackling them by order of severity.

Peter


USB Camera Glasses for the Blind
http://www.seeingwithsound.com/camera_glasses.htm

[PranavLal]

Hi all,

In terms of features we need, here goes.
1. Speed control.
2. Foveal mapping. I will right about this in the latter part of this message.
3. Color filters.
4. Low light boost.
5. Collision detection.
6. Blinders.
7. Zoom.


I also want to address the points made about esthetics and the use of the camera board. The core philosophy of the vOICe is to use off-the-shelf hardware. When we first started in the early 2000s (the vOICe has been around for much longer) we started with hatcams. Webcams were attached to baseball caps etc. We then got affordable glasses with a camera such that we could now see at eye-level. This helps in positioning and matching vision as close to the sighted world as possible. In addition, the current video glasses we use are the only way we have of having wide angle vision which requires foveal mapping. This is exactly how the human eye works. I have been using the vOICe since about 2001. I find that the use of a wide angle lens helps me more closely match what I am seeing to what I experience with my other senses as well as my mental map of a place. This is why I am so interested in using the existing USB glasses. It is not looks alone. <smile

I appreciate the need for reducing the load on the CPU of the pi. I did look at using opencv on the GPU. You may find the following link interesting.
http://opencv.willowgarage.com/wiki/OpenCV_GPU

Our experience with the use of the GPU has not been too good though. A developer tried implementing the vOICe on the iPhone. This application is called voice vision and is available from the apple store. He used the GPU and the black pixels are noisy rendering the application very difficult to use. You can see the thread here.
http://www.applevis.com/forum/ios-ios-a ... -it-useful
Warning: link may wrap.

I am not saying that we will have the same experience with the pi but I am trying to give some background and provide perspective.

Pranav

[ghans]

The point of the Pi Camera using the GPU is not doing
the analysis there , but completely avoiding the Pis *limited*
USB system. USB on the Pi can take a lot of CPU , while
the Pi Camera does not suffer this drawback.

Of course , you could do the analysis on the GPU too , but
i myself can't comment on possible limitations.


ghans

[PranavLal]

Hi ghans,

Many thanks for the clarification and I do hear you. <smile

Pranav

[mikey11]

Thanks for the replies guys, and also for the clarification.

It is the case that the rpi's CPU is easily overwhelmed by having to take care of tasks other than light processing.

I believe it will not be possible to make a useful device without using the GPU, and the camera made specifically for the rpi. This will be a key to being successful on this hardware.

I understand that the vOICe was meant to be able to use off the shelf hardware to make it more accessible, but I think that by using the rpi with it's specialized hardware is OK, and the reason why is that the total cost for the rpi plus rpi camera is so much lower than the competing alternatives. This alone justifies making a step away from supporting other hardware including random webcams (and unfotunately, the purpose made glasses).

The advantages of restricting hardware are to improve performance to the point where vOICe is useable on the rpi, not to exclude hardware manufacturers, or to set barriers that prevent people from using the code generated on the rpi. I just want to start down the path that will most likely lead to success on proving that the rpi is capable of providing the processing performance. The low price point of ~$100 is also an incentive to restrict hardware. The low price combined with less programming complexity will result in a more robust, reliable product. Very similar to how Apple limited hardware for their systems. It's not a bad lesson to take.

I understand from Pranav that there are functional aspects to the other off the shelf glasses that the rpi camera would lack. The wide angle, and not being easily located to a pair of glasses to provide proper positioning. I feel that these issues are secondary to proving that the rpi can even act as a function vOICe device on the processing side of things. If that area can be proven, and provide the level of performance required, then the other changes are likely to be made possible as well.

Thanks,

[mikey11]

Also, regarding the use of files as part of the overall processing bottlenecks as mentioned by Peter.

http://www.raspberrypi.org/phpBB3/viewtopic.php?t=24685

I see that ram drives are definitely an option. This avoids use of SD cards as potential speed bottlenecks, and avoids the rewrite problem (which although it is managed through a re-write location balancing algorithm, will lead to earlier product death. Imagine a device that lasts 5 years instead of just a few).

The ram drive may be a solution that makes numerous things easier. In that libraries for video/audio functions that are dependent on file creation/reading can be sped up dramatically without having to delve into more complicated programming techniques that involve memory addressing. So for a coder like me who more likes to stitch other peoples work together, I would likely approach things this way.

The issue of the camera not meeting useable specifications in terms of viewing angle may not be a large problem either:

http://www.raspberrypi-spy.co.uk/2013/0 ... ra-module/

I also found some evidence that extension cords for the camera of over 1 meter have been devised and used successfully. With how small the rpi is though, this may not be necessary. It is almost a 'wearable' computer. Granted, it's ugly on its own, but imagine a custom printed case that incorporates it into a headwear design.

I just ordered my rpi camera today, so once it arrives I will be able to proceed in earnest.

[mikey11]

Pranav:

Can you please help me fully understand what you mean when you listed your features just so its clear:

In terms of features we need, here goes.

1. Speed control.
2. Foveal mapping. I will right about this in the latter part of this message.
3. Color filters.
4. Low light boost.
5. Collision detection.
6. Blinders.
7. Zoom.

1. Speed Control : I assume you mean the number of video frames converted to sound over a given time period. Such that the sound file is played back at a faster rate to give more audio frames/time period.

2. Foveal mapping: I assume that this somewhat has to do with the use of the wide angle lens, but also to the perspective of the image, in that the center of the image has more detail than the periphery

3. colour filters: I don't understand what these would achieve, are you looking to strip away colours/ see only one colour? What does this do that is advantageous?

4. Low light boost: I believe this is possible by changing camera sensor sensitivity settings, or by increasing exposure time to light, and this may impact frame rates negatively

5. Collision Detection: I don't understand what you mean here. Is this to preemptively detect objects that the wearer may collide with? I can see the value in that, but I have no idea currently how one would implement that.

6. Blinders: I assume this is to block the periphery from view to allow more dedicated focus at the center of the image. Would this be OK to shorten the length of the sound file to match the smaller resolution being generated? Can vOICe users dynamically resize the area they are hearing?

7. Zoom: This is pretty clear, I assume you would want to take the central area of the view, and double its size. Would you want the zoomed area to have an increase in quality that occurs in step with the zoom? or simply a doubling of the size of the existing resolution? I can see how if you would want to improve the quality of the zoomed area, you would have to change the resolution the camera is capturing at, and process only the area used for the zoom. I can see how using a zoom to increase the size and quality like this could cause a performance hit. Also: what zoom multipliers would be useful. If you could switch through 0, 2, and 4 times multipliers, is that enough?

[PranavLal]

Hi Mike,
My answers are inline.



1. Speed Control : I assume you mean the number of video frames converted to sound over a given time period. Such that the sound file is played back at a faster rate to give more audio frames/time period.
PL] Correct.
2. Foveal mapping: I assume that this somewhat has to do with the use of the wide angle lens, but also to the perspective of the image, in that the center of the image has more detail than the periphery
PL] Yes.
3. colour filters: I don't understand what these would achieve, are you looking to strip away colours/ see only one colour? What does this do that is advantageous?
PL] Yes. The advantage of a color filter is that there are times when we are tracking specific objects that have specific colors so we use the color filters for that. For example, I sit in an office which is visually uniform. To get from where I sit to the exit, I have to walk past about 4 cupboards and then turn right. The cupboards are visually of the same color and counting them is tricky since many times, my colleagues lean against the cupboards etc. The carpet in the office however has a pattern of red squares. The trick is to count the red squares using the vOICe. This is one instance where the color filter is useful. Similarly, if I am told to pickup a red box from a table, I will set the vOICe to filter out everything except red objects and then try to locate the box.

4. Low light boost: I believe this is possible by changing camera sensor sensitivity settings, or by increasing exposure time to light, and this may impact frame rates negatively
PL] I am not sure how it has been implemented in the pc version. It increases the brightness of the pixels in low light conditions which makes viewed objects clearer.

5. Collision Detection: I don't understand what you mean here. Is this to preemptively detect objects that the wearer may collide with? I can see the value in that, but I have no idea currently how one would implement that.
PL] Yes. See the feature in the pc version of the vOICe. If we had stereo cameras, this would not be a problem but I have seen research papers dealing with this with a single camera. The object the user is going to collide with is hashed and sounds really horrible but it gets the message across. <smile
6. Blinders: I assume this is to block the periphery from view to allow more dedicated focus at the center of the image. Would this be OK to shorten the length of the sound file to match the smaller resolution being generated? Can vOICe users dynamically resize the area they are hearing?
PL] Yes, users of the vOICe can set the area they are hearing. It is not shortening of the sound file but a restriction of the viewing angle from what I can tell. You can specify how much of the x y plain you want to see.
7. Zoom: This is pretty clear, I assume you would want to take the central area of the view, and double its size. Would you want the zoomed area to have an increase in quality that occurs in step with the zoom? or simply a doubling of the size of the existing resolution? I can see how if you would want to improve the quality of the zoomed area, you would have to change the resolution the camera is capturing at, and process only the area used for the zoom. I can see how using a zoom to increase the size and quality like this could cause a performance hit. Also: what zoom multipliers would be useful. If you could switch through 0, 2, and 4 times multipliers, is that enough?
PL] I have not noticed any changes in quality but Dr. Meijer is the best one to comment. In terms of multipliers, we have 2, 4, 8 and 16. In realtime applications, I don't think anything beyond 4 times is useful. It is the feature with the lowest development priority since during immersive usage, I can walk closer to an object and see it.

Coming to the GPU, cameras etc, I agree that we should get a working version first. Opencv appears to be camera agnostic. However, before we do anything, we should run some performance tests on the code and determine where the problem is. I do not know what tools I can use for this so am no use. I am all for ram drives. <smile I thought they had gone out of usage. <chuckle Once we have a working version, users will only use it if it is easy-to-assemble which is another factor behind the glasses. You plug them in and they work. Still, let us get something usable first.

Pranav

[seeingwithsound]

> I have not noticed any changes in quality but Dr. Meijer is the best one to comment.

In principle the camera resolution setting should increase in tandem with zoom level in order to maintain constant perceptual quality (no blocking artefacts at high zoom levels). This is why you noticed no changes in quality. However, trade-offs with what the hardware can handle may sometimes be necessary: taking snapshots at a high resolution will slow down the system and use a lot of memory because with normal digital cameras you can only change resolution of the entire camera view - no matter if you need only a small slice of it. High zoom levels (say > 4) also correspond to severe tunnel vision and even small camera movements can then easily make you lose track of what you were looking at.

Peter


USB Camera Glasses for the Blind
http://www.seeingwithsound.com/camera_glasses.htm

[johnf]

Referring back to the matter of "blinders"(horse blinkers to Brits); there is a very interesting variation to produce a wide horizontal slot perhaps 20 or 30 pixels high.
Go to the Windows vOICe Help menu.
Then choose Find Executable.
Type in the following command. Note the space after exe .
voicetest.exe -blinders 0 10 176 40 0
40 is top line of pxl height,
10 is bottom level of slot, lower ¼ of pic. 40>10 therefore slot is 30 pxls deep.
340 is full width of screen
Leading 0 is offset from LHS
Final 0 is background level ie Black.
C:\Users\johnf\Downloads\voice.exe -blinders 5 10 340 15 0 is a typical example .
This discards most of the picture. Most of this is extraneous to the blind person, just as the white cane is preferably used low down ,mostly to seek out obstacles .
The sound scape is nearly monotonic, according to the chosen pixel slot. However, there is considerable "tweet" modulation if a slanted edge is encountered.
The reduced amount of sound detail, especially the loud, high pitched sounds from the Sun and lights, windows, etc; helps the beginner to get a focus upon the important stuff, with small victories at the earliest stage of experience.
It is immediately possible to detect door openings , and fairly easy to scan for and pick up small objects on a contrasting surface.
Recognition of some of the alphabet (capitals) is possible, if they are marked with black ink on white paper- and the Inverse Video option is chosen from the menu. Listen for the double "slope" of the letter V, compare it with "A" and "W" and so forth.
With Colour options, the correct flavour of M&M can be picked, every time. With a hand held webcam, a seated person can pick one off the floor without looking.
It seems to me that there is an analytical, calculating, etc. process involved in parsing out the meaning of a limited soundscape such as this, as contrasted with the full-screen, full dynamics, full tonal spectrum of a "real" vOICe outdoors experience.
The slot can be widened by steps as progress is made towards the full-screen sonification.
In the home and in the street, most important objects are vertical , eg pillars and wall openings. A head mounted camera will work fine with slots because the vertical elevation has little effect on the detection of such obstacles.
Many blind users prefer a wide angle lens system which gives better awareness all round,but must inevitably bring in a huge amount of brain traffic.
There is much more to The vOICe than meets the ear. If it can be emulated on the Raspberry-Pi, with a decent (button) control interface,it will have great potential with a huge market, world-wide.

[johnf]

The question of Pi camera cables came up in a recent post. In the forum under "replacement camera cables" it is indeed proven that up to 4 metre length of camera cables can be used.
This is the flat ribbon type as originally supplied, not very cosmetically attractive as a headcam connector, and very flimsy. . No doubt a workaround may be achieved in time , meanwhile it is very encouraging news. Certainly, a robust cable , preferably with a quick-detach link to protect against snatching, is a design imperative.

[PranavLal]

Hi all,

I now have the camera board for the raspberry pi. How do I know that ithas been detected?

I am ready to test further.
Pranav

[johnf]

http://www.farnell.com/datasheets/1722614.pdf
is a beginners level step by step guide.... please forgive if I am under-estimating your guru status.
johnf

[mikey11]

Thanks for the guide. It makes getting started look pretty easy. I am just waiting for my camera module to arrive. It's been a few weeks now, and I'm really excited to get going on this!

[mikey11]

Well, after quite the wait, I've gotten the camera.

I followed the install guide to get the rpi firmware up-to-date, and to devote 128mb to the GPU (although I think in the long run this can be shaved down dramatically because the resolutions in use are quite tiny).

I read a little about the default programs that are meant for the cam. It appears they already access the GPU, so it may not be necessary to use the openCV.

I am going to try to rough up a program that grabs frames with raspistill, writes them to a ram drive, and then see about have a looping vOICe interpretation of whatever file is present in the ram drive.

It looks like some of the desired features for processing can be done by changing the capture parameters on raspistill for instance, inversion, resolution changes.. I believe that by tweaking the flags a very useable program can be made quite quickly.
I will upload my code later on.

One note: When I did the configuration as supplied in this thread, I got no command called raspicam. I did get raspistill and raspivid. So things may have changed since that guide was current. No matter, I still think there is enough to get cracking.