mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Wed Apr 01, 2015 5:05 am

Hi all,

after the long hiatus, the rpi 2 has arrived.

I reiterate the challenge:

get the existing code to work, and I've got 1K$ for you. I'm trying to drag my cousin in on this as I now have 2 kids under 2 (been busy you see...)

If I get the time (not likely) I will recompile on the new hardware. The 6X rate increase should kill all problems.

The last task would be to port the python code to c++ and have a single routine to manage the whole thing.

I already have a 3D printer now, so I could build custom cases for the final product, and then the only hurdle would be a cable modification for the camera.

Come on! Take the challenge, make $1000

get the code to go at least 1/sec!

Everything is here to make it work!

If we have the code going 1/sec I will start a crowd funded campaign to do a first run of the devices. for vOICe for under $100.

Ares
Posts: 12
Joined: Sun Apr 19, 2015 7:57 pm

Re: Sight for the Blind for <100$

Sun Apr 19, 2015 8:21 pm

Hi Mikey11 and others,

I've been looking into getting the code from http://www.seeingwithsound.com/hificode_OpenCV.cpp working on a raspberry pi the past few days, then I stumbled upon this thread. I might be able to help, is the challenge still current? Did the Raspberry Pi 2 solve the performance issues?

If the challenge is still up, could you please define again what functionality is needed? I'm sorry, the length of this thread makes it a litte bit unclear for someone who was not following from day one...

These are the features that I have been working on / thinking about:
- choose between Raspberry Pi camera and USB camera / camera goggles (are there any that have a working linux driver?)
- do sound output in-memory
- include some image processing features like edge and contrast enhancement
- make options accessible without recompiling, at least as command line parameters
- get speed fast enough so delay between frames is at least shorter than the audio duration per frame (~1sec), without sacrificing too much audio quality

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Sun Apr 19, 2015 10:13 pm

It is current.

I haven't compiled on the rpi2 yet, but somewhere in the history of the posts I provided a python script which activates the camera, looks for keystrokes and then changes the camera mode accordingly.

I made small modifications to the code provided by Peter Meijer the hifi cpp file. I instructed it to look for the image being generated by the python script, convert to a soundscape, and playback.

There is a post with both files that work together properly. I think getting the responsiveness up will be a matter of recompiling the cpp file on the new rpi2, to take advantage of the generation next generation cpu. When you compile, you will likely have to change some of the compiler flags to reflect this.

Ares
Posts: 12
Joined: Sun Apr 19, 2015 7:57 pm

Re: Sight for the Blind for <100$

Mon Apr 20, 2015 12:42 am

Ok, I'll look into this. I have a RPI2 ordered already, so I should be able to test this soon.

Right now I am wrapping up some major rewriting of the original code which allows for choosable cameras and image processing options (should be able to share this soon). A C++ API for the Raspberry Pi camera is included: http://www.uco.es/investiga/grupos/ava/node/40
Apparently some of the features of the python script like image effects (which ones are important, anyway?) are not accessible with this library, however, brightness, contrast, gain and exposure time are.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Mon Apr 20, 2015 1:49 am

You need to use the rpi camera module because it is processed by the gpu. this frees up the CPU for the audio file generation, and the USB system on the rpi is dreadfully slow (maybe the rpi2 is better, but I haven't heard that this is the case)

all the picture modes are part of the built in features for the rpi camera module software provided by the rpi foundation.

which modes are the most important? read through the long thread, and focus on the info provided by pranav lal, as he is actually a user of this software.

the camera module is triggered to constantly take pictures, and takes them at a rate of 150-200 ms/shot. the frame is then read, processed, and played back, all existing frames are erased, and the next available frame is read, processed, and played back.

The playback needs to happen at least once per second, and in an ideal world, the audio sample can be sped up to provide two playbacks per second as an option.

Ares
Posts: 12
Joined: Sun Apr 19, 2015 7:57 pm

Re: Sight for the Blind for <100$

Mon Apr 20, 2015 8:04 pm

Thanks for the info!

Quite a few things of what I have tried have worked out, so I uploaded my code to github to make it easier to share, you can get it from the raspberry pi command line this way:

Code: Select all

[email protected] ~ $ git clone https://github.com/Ar-es/raspivoice
This should download all source files into a folder called raspivoice, check out the INSTALL file there how to compile. Should work on any current Raspbian on Rpi1 and 2. So far theres not much documentation except for the install file, so use raspivoice --help to check out the features already implemented. A path to an input image file name can be used instead of a camera, so in principle it should work in conjunction with the python script mentioned above. However, I have had problems getting the python script to run, so I don't know yet if it really works or not, need to check this out later.

I found a few ways to make the whole image processing a lot faster (~400ms for the algorithm on Rpi1, ~120ms on Rpi2 with 176x64 resolution, 16bit), hopefully without breaking the algorithm - would be cool if anyone who has experience with the vOICe could try it, so I know if its usable or not. To me, the audio output sounds reasonable. I can give a more detailed update later, if you are interested.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Mon Apr 20, 2015 8:17 pm

I have to work dedicated for the next 4-5 days.

Following that time I will try to make the time to check your results out.

In order to evaluate, check with Pranav Lal, as he actually uses the software. I am familiar enough that I can probably give a half informed opinion.

If I check, and it looks/sounds good, I will either see if pranav can reproduce it, or I will send him a compiled version with the hardware (he may need an rpi 2)

If he is happy, we can make arrangements for a funds transfer. I do expect that this code stays 100% open with no restrictions.

Perhaps when I get to testing we could exchange phone numbers so that I can more rapidly evaluate? PM me if that's ok.

Ares
Posts: 12
Joined: Sun Apr 19, 2015 7:57 pm

Re: Sight for the Blind for <100$

Tue Apr 21, 2015 9:35 pm

Great, I am looking forward to seeing if this works for any of you.

The code is released under the same unrestrictive license as the hifi-code it is based on, so no need to worry.

I had a look at useful the features mentioned by Pranav. Speed control is possible by setting the total time of each frame. Also, I implemented some rough foveal mapping by means of a generic barrel distortion (needs 320x240 intput image resolution for decent quality), exposure control to set effective brightness, blinders at the sides and zoom options.

Color filters are still missing. For this, the images have to be read in full color instead of gray scale, so I do not know yet if this will be a hit on performance.

Reliable collision detection / close-object-warning probably needs additional hardware, so I am not looking into this for now.

A serious problem i noticed with the Raspberry Pi (both versions) is noise on the headphone jack, probably caused by interference with other circuitry on the board. I cannot hear this noise with USB headphones, so probably a cheap memory-stick sized USB sound card can alleviate this, if a 3.mm jack is needed.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Tue Apr 21, 2015 11:31 pm

I have experienced the audio noise issues as well. I thought this was fixed on the rpi2.

No matter, I believe you can get clean audio from the hdmi port in addition to the usb. For this to be sub 100$ though, every piece of hardware that can be avoided is gold.

Ares
Posts: 12
Joined: Sun Apr 19, 2015 7:57 pm

Re: Sight for the Blind for <100$

Wed Apr 22, 2015 9:13 pm

Ah, forgot about the HDMI output, good idea. I tested again on the Rpi2 using aplay (linux audio palyer) and a "silent" wav file, using a usb power pack battery to avoid power supply noise. Apparently the 3.5mm jack is still quite noisy. I checked the output of the optimized image-to-soundscape-algorithm, a black input image gives an all-zero wav file (except for headers and the click at the beginning), so this should sound perfectly silent.

I did another update to the code, adding some kind of interactive mode where some of the processing settings can be cycled by key presses like in the python script, so combining both programs might not be neccessary anymore, unless I missed some important features.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Wed Apr 22, 2015 10:59 pm

I can't wait to test it out. Providing it is all going well I will try hard to get input from pranav on the best vision mode settings, and I would like to focus on the top 5.

I envision a headless pi with a bower button and a jog wheel/button that flips through modes to keep the parts count low.

PranavLal
Posts: 124
Joined: Fri Jun 28, 2013 4:49 pm

Re: Sight for the Blind for <100$

Thu Apr 23, 2015 1:42 am

Hi Mikey11,

The vision mode settings are quite easy. Set the vOICe at normal speed that is sounding one frame per second with the foveal view enabled assuming we have that in the existing implementation. Settings like brightness etc do not come much into play because the vOICe uses grayscale images. I am still monitoring this thread so please feel free to ask any questions.

Thanks for your effort and commitment.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Sun Apr 26, 2015 6:21 pm

As for collision detection, I built an ultrasonic rangefinder for the rpi once. It would definitely increase the cost, but also be worth it I think.

https://www.youtube.com/watch?v=aPnvxh_UWuc

You could easily tie the distance to a haptic feedback such as a variable speed vibration motor. The sensor I bought was limited to 5 meters range with a minimum distance of 30cm.

It was about $25, and would push the goal over $100 at unit prices.

If the software side gets finished for the vOICe, I will do a bulk sourcing for components just for fun to see where things end up. It would be interesting to see what could happen.

For now though, the focus is on the vOICe implementation.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Sun Apr 26, 2015 7:28 pm

Well everyone,

I have great news:

Ares implementation appears to work flawlessly for me. I'm more than happy to provide him the reward.

Next steps?

1. get a single button jog wheel integrated using the GPIO (my local electronics store has none, so I will have to order some)
This control device will allow the user to rotate between options, then push the button to select an option.
2. get a lithium polymer battery plus charging system installed
3. Design case for all components for 3D printers
4. Assemble and give out ten units for feedback from actual vOICe users

Optional but fun steps that can go parallel while waiting to get feedback from steps 1-4
5. Install ultrasonic rangefinder on GPIO (serial tx/rx pins)
6. Install haptic feedback vibration motor
7. New case design for 3D printers.
8. Assemble and give out ten more units for feedback on both systems operating together.

9. Start a dedicated website for this effort.

Software development requirements: If people are interested anyways.

I can do the rangefinder and haptic feedback portions as a separate process from the vOICe program by modifying the code I used in the youtube video I showed earlier.

but, the integration of the jog wheel/button needs to be done through the raspivoice process, so if Ares is amiable, it shouldn't be too hard. The relevant library to easily integrate the GPIO into cpp is called wiringpi. I can help with this by identifying and providing the hardware to test on.

One other item that would be extremely handy is an automatic update feature, such that if you plug your pi into an ethernet port, it will update to the latest version of the software(s).

Also usage statistics gathering could be helpful. like how often a vision mode is engaged, and for what duration. This info could be sent whenever the unit is plugged in for automatic updates.

A guy can dream anyways.

Step 10. Assuming everything works, and gets good feedback from actual users, launch a crowdfunded initiative to get 100-1000 units assembled and distributed.

As I live in Canada, at some point I might want to approach our national blindness association to see about distribution and feedback.

User avatar
seeingwithsound
Posts: 165
Joined: Sun Aug 28, 2011 6:07 am
Contact: Website

Re: Sight for the Blind for <100$

Sun Apr 26, 2015 7:49 pm

Congrats to Ares! I do not have a Raspberry Pi to test with, but have been pleased to observe the perseverance in this challenge over a period of almost 2 years. I also wish to congratulate mikey11 with bringing his initiative to fruition, and of course several others have made valuable contributions.

Best regards,

Peter Meijer

Seeing with Sound - The vOICe

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Sun Apr 26, 2015 8:41 pm

So, in looking at the next steps, I have selected the following components:

rotary encoder/button: https://www.adafruit.com/products/377
range finder: https://www.adafruit.com/product/172
battery: https://www.adafruit.com/products/328
Charger: https://www.adafruit.com/products/1944

I am going to source resistors/capacitors and the vibration motor locally for now.

I will begin 3D case design once the batteries and charger units arrive.
I will start with the case for the camera/battery/charger/rotary encoder/pi.
I will then modify that to include the rangefinder and it's associated components.

Once I have the encoder sorted out, I will post here with an example of sample code indicating how to use it, or I may even get adventurous and try to modify Ares implementation to take the text menu to the encoder. My plan to indicate menu selections would either be tones of a given frequency for 1 second, or maybe even a wav file saying each settings name once selected.

Got to admit, I'm pretty excited now that there are tasks I feel capable of completing again.

PranavLal
Posts: 124
Joined: Fri Jun 28, 2013 4:49 pm

Re: Sight for the Blind for <100$

Sun Apr 26, 2015 11:58 pm

Hi Mikey11,

This is indeed very exciting! As for the menu, look at the speak up screen reader. It would be better if you mapped that screen reader's controls to your rotery dial etc. That way, you need not build a separate solution. Moreover, it could handle errors without any extra work. Alternatively, use the espeak synthesizer from the commandline and send whatever text output you like.

Lets test the implementation before we build the interface.

Ares
Posts: 12
Joined: Sun Apr 19, 2015 7:57 pm

Re: Sight for the Blind for <100$

Mon Apr 27, 2015 9:49 pm

Hallo all,

Great to hear it is working for you!
Integrating the GPIOs should be no big problem, I will have a look at this when time permits.

Some features I added (not yet on git, need to make them a bit more stable):
- daemon mode (independent of terminal), so the program can be run automatically when booting the device.
- for this to work, a keyboard needs to be connected to (or something like a keyboard, I used a usb numerical keypad, but also usb remote controls or even usb headphone buttons should work, if the keys are added to the key list)
- I also did a quick (but apparently working) hack to integrate espeak, so there is spoken feedback when options are changed at runtime.

PranavLal
Posts: 124
Joined: Fri Jun 28, 2013 4:49 pm

Re: Sight for the Blind for <100$

Mon Apr 27, 2015 11:54 pm

Hi Ares,

That is excellent news! It is important for the program to start automatically. Connecting a USB numeric pad should not be a problem. Ideally though the program should start without any external user interface and begin sounding the camera view.

How are we planning to head mount the camera?

Pranav

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Tue Apr 28, 2015 3:13 am

I am planning on adapting an rpi case model from thingiverse that uses a friction ball mount to aim the camera.

I am waiting on the battery and power board before I do that though.

Something like this:
http://www.thingiverse.com/thing:256960

The case will be larger and I am looking to add mounts for a chest strap or something.

Eventually I will find a better cable for the camera.

PranavLal
Posts: 124
Joined: Fri Jun 28, 2013 4:49 pm

Re: Sight for the Blind for <100$

Wed Apr 29, 2015 12:31 am

Hi,

Hmm. The pi could be chest or belt mounted. The camera cable will have to be very strong. You would need to mount the camera at eye-level. I would have suggested the gopro head strap which is what I am using my ODROID setup but that keeps the camera above eye-level which is not the best solution.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Sat May 02, 2015 3:56 am

regarding the eventual camera mount location/strap:

I'm not going to worry about this right now. When I get to the stage of looking at sourcing parts in bulk, I hope that some more room in the budget can be used to address that problem.

In the meantime, I got my rotary encoder/battery/charging board/ultrasonic rangefinder

My first goal is to integrate the rotary encoder into the C code of Ares. My goal will be to rotate through his numbered menu, and have espeak say the current option (but not describe current setting of that option), and the button press of the encoder will increment the current option. Once the encoder is done, the code will be ready to make the device and distribute. I have touched base with a fellow I know who lost sight at a very young age who can help me with feedback locally, though of course with pranav having far more experience, I will definitely defer to his judgement.

I have located a probable candidate for useable source for the rotary encoder at:

http://theatticlight.net/posts/Reading- ... pberry-Pi/

and I'm currently just trying to get signs of life from it using another toolset called pigpio and piscope which should let me observe whether or not the encoder is working properly.

I have gotten signs of life from the rangefinder, but I have to do a signal inversion to go from the default output to something the pi will read on the rxd pin. I have already built the power filter required (a capacitor and resistor) as described here:

http://maxbotix.com/documents/MB7961_Datasheet.pdf

I have not built the signal inverter for this yet as described here:

https://github.com/simonmonk/raspirobot ... er-Adapter

But I did buy the parts. I also got an acceptable power transistor to isolate the haptic feedback vibration motor from the power circuits on the raspberry pi board, and will tie it directly to the lipo battery.

I am planning on writing a parallel process (probably in python as I struggle in c) that operates the rangefinder and translates the distance into a change in vibration output.

First though, the rotary encoder.

When I know more, I will update here.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Sat May 02, 2015 6:08 am

So the good news is that I sort of got the encoder to work using the code above.

using the following:

Code: Select all

#include "stdio.h"
#include "rotaryencoder.h"
#include <unistd.h>

unsigned int microseconds;

int main()
{
printf("Hello!\n");
wiringPiSetup () ;
/*using pins 23/24*/
struct encoder *encoder = setupencoder(4,5);
long value;
while (1)
 {
 updateEncoders();
usleep(100000);
 long l = encoder->value;
   if(l!=value)
   {
     printf("value: %d\n", (void *)l);
     value = l;
   }
 }
return(0);
}
I can reliably get the output to either go up or down depending on the direction I turn the encoder. It doesn't increment or decrement by 1, but rather by 4, and only if I turn carefully. however, simply knowing if the value goes up or down should be "good enough" to rotate through the options menu. I haven't got the button press figured out, but that should be relatively easy in comparison. the big usleep was for a bit of debouncing (physical switches are noisy) the encoder as it rotated. I used values as low as 5000 and got similar results to the higher choices.
The code seems useful as it has been prepared to be a library added to other projects already. My snippet above uses it in this fashion. It does require the installation of wiringPi.


I did try to take a look at Ares code, but I don't know enough to integrate this code into his, and still have it compile after. He is a real coder, I am not.

The meta code for it would be something like this (I realize some of this is already in there for the keyboard input in raspivoicemain.cpp):

create options list
numbered 0-9
named mute, negative, zoom, blinders, edge detect, threshold, brightness, contrast, foveal mapping, restore defaults
with individual option states as appropriate
start at option 0

update encoder value
if encoder value > old_encoder_value then option = option + 1
if encoder value < old_encoder_value then option = option - 1
espeak option

update button press
if button press = true then option.state = option.state + 1
if option.state > max option.state then option.state = 0
/* ex. for mute, 0 and 1 are possible, for zoom, 0,1,2,3 are possible representing none, 25, 50, and 75% */
run raspivoice with new option

Ares:

If you are willing to work on this, I can send you the prototype stage hardware to get the encoder functional. I don't know how you feel about hardware, but what I would send would just plug onto the gpio header on the rpi board. No need for soldering.

While this continues, I would commence the work on the ultrasonic rangefinder and vibration motor feedback. I would make this a separate process programmed in python. I feel confident that I can take this from start to finish on my own in short order as I find python far more forgiving.

Ares
Posts: 12
Joined: Sun Apr 19, 2015 7:57 pm

Re: Sight for the Blind for <100$

Sat May 02, 2015 9:58 am

Nice progress!

I prepared an implementation of the rotary encoder code (as well as some of the other features I promised), it ended up in the file KeyboardInput.cpp which is responsible for some kind of menu structure as you suggested. Right now the menu has no visuals, but should work with asdw keys for testing. Current limitation: If the rotary encoder or usb keyboard are chosen as input, the normal keyboard from the control terminal is disabled.

You can update the source code from the folder where the source files are with

Code: Select all

git pull
Then follow the INSTALL again starting with

Code: Select all

make -j 4 CONFIG=release_rpi2
(Assuming you have wiringPi installed already).

Some of the short-hand notations of the command line options have changed, as well as the way the sound card is addressed (I implemented a way to change the system volume).

I can get the rotary encoder myself, need to order some electronics anyways.
This should be the one: http://www.mouser.com/ProductDetail/Bou ... 0wzQ%3D%3D
Did you connect it like in http://theatticlight.net/posts/Reading- ... pberry-Pi/?

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Sat May 02, 2015 3:47 pm

I attached it similarly, I just used other pins, but for the sake of keeping things easy, I can use whichever you choose. Just have care that the wiringPi pin definition is not the same as the broadcom SOC pin definition, and also differs from the raspberry pi gpio #.

I certainly got confused the first time I ran into wiringPi on another project.

http://wiringpi.com/pins/

I won't have the chance to test out your new additions till later tonight as I am on double kid watch duty solo today.

The mouser part looks exactly like what I have, and the specs match up as well.

Return to “Assistive technology and accessibility”