Hospitals no fun, not talking worse.

[DavidS]

Sorry for my momentary absence. I spent some time in a hospital, do to something that happened to me.

Without going into any detail, the result is that I am unable to form spoken language, or sign language, I am weak, and I have lost much of my dexterity (which makes typing very slow, and writing impossible).

So now I am getting myself together enough to get back into some things, and am looking at how I can use my new limits to help others that have similar problems. So I am looking at what speech synthesis options are available, and what front ends for all common platforms (Win32/ReactOS, ARM Linux (eg Raspbian, Debian, Ubuntu, etc), Android (such as the ones I am using now), etc.

I am also looking at a way to simply implement a front end for Android as well as ARM Linux to simplify the flow of conversation as apposed to the traditional text to speech front ends. What is needed quick categorical navigation to user created pre-set phrases, that evolve as the user learns what they commonly use in conversation, with a quickly accessible text to speech to say things that are not common or they wish to add to a category. The current common methods of having long monolithic lists of saved files that are only categorized by directory structure is not fast enough to keep up with conversation, especially once you have lost a significant amount of dexterity.

To this point it has taken me 20 minutes to type this short post do to my partial loss of dexterity.

This means I will have to learn Java (a language that I had avoided like the plague in the past).

If someone wishes to take the idea and run with it, please do. My goal is only to help people that are in a similar situation to myself at this point. I do not care whose name is on the resulting program, so long as it is open source (NO GPL, open source like BSD or MIT licenses).

Thank you for reading this anyone that chooses to read it. I hope it may lead to some form of aid for some others that are in a similar situation of being unable to form spoken language, unable to form sign language, though still able to type if with restriction of being very slow if they also lost some dexterity.

Yes this is a bit long winded, and has taken me a long time to type though it is worth it.

[DavidS]

Currently I am using an Android Tablet to communicate, as it is what I had laying in storage for which there is free understandable speech synthesis and usable front ends available, as well is an easily portable device. Though with my limited dexterity I am looking for any old smaller laptop or netbook that I could use, and has a regular keyboard. It is even slower and more difficult for me to type on a touch screen than it is like this.

I wish to figure out a solution that can make it possible to keep up with conversation, is intuitive enough for the 'average' end user, even if permanently impaired from a similar event.

For me it is more important to find something that will help other people than it is to figure out how to improve my ability. Though this is one of those projects that will do both at the same time.

[zog]

I am sorry a great misfortune has befallen you. Have you investigated if there are any charitable organisations that can help you with your disability?
An occupational therapist might have some good ideas with regards to mobility aids.
There are various bluetooth keyboards and mice which can be linked to tablets, and computers easily. I think the first thing to do is to find out what is already available before starting a new project.

A good charity that might be able to help you is e-nable.

I agree with you about java though. However the nice thing is that the dev tools for android tablets are free and android tablets are very reasonably priced compared to the apple equivalents.

Jack

[hippy]

I am very sorry to hear of your misfortune David.

I was also going to suggest an Android tablet or phone. PicoTTS seems to be of very good quality to me and it is fairly easy to create simple apps which use that using MIT App Inventor 2 using their Blockly-style programming paradigm. That can be handy for speaking text entered into a text box or for dedicated word or phrase buttons.

It might be possible to create an Android app which can be controlled via Bluetooth or WiFi so it can be commanded by a Pi or something else better suited for input and sentence construction.

Microsoft Agent used to be reasonably good when used for text to speech but the last time I played with that was on Windows 98/XP and am not sure what has happened with that since.

An AIY voice hat might be a solution. Haven't used that myself and had not found any decent speech engines for the Pi, though I only tried the ones which are well know, did not try to seek others out. Most sounded robotic and are difficult to understand, but anything may be better than nothing. There did appear to be some paid for speech systems which claimed to have good quality.

Someone did produce one with decent natural speech quality but never revealed how to create additional words to add to its dictionary.

I recall there have been others on the forum looking to create solutions for people in similar positions as yourself and it may be worth reaching out to them.

I would personally suggest not being be too concerned over how open it is to start with; choose a design which partitions control from speech generation and allows you to move to open source when and if possible, use whatever is available to achieve a practical and useful solution first then look at making it more philosophically suitable later.

[gregeric]

So sorry to hear of your misfortune David and best of luck with recovery.

As hippy has suggested, the Google AIY project may be worth looking at. You don't need the Voice HAT to use it - it will work with any playback & capture device. There's a "say" action which repeats whatever comes after "say". I guess you could hack that for keyboard entry in place of voice.

[zog]

Oh yeah I forgot to mention, I wrote a websocket server for communicating with the pi, written in pascal. This means that you can create a button on a html page and then make something happen on the pi, when it's pushed. This might be helpful as you wouldn't have to write an app at all. You can simply create an apache server on the pi, that serves web pages to your device. The javascript on the web pages can then send commands to another computer, and you are just using the tablet as a terminal device. The program was written using lazarus which means that it is a cinch to port to windows.
Using a web interface might be a preferable way to do things because you can get round all the licensing problems associated with tablets and the same interface can be displayed on any tablet / computer.
The same program can also take inputs from a bluetooth serial device and an infrared sensor, and it can also output infrared signals via lirc.
I meant to write a program to generate the HTML for various button layouts, but I didn't get round to it due to lack of time / enthusiasm.
I find that I have very little time to code these days due to the demands of being a Dad, and having to go to work everyday, which I find incredibly frustrating at times.

There is also a new technology out for the web called WebAssembly which lets you compile C++ code for use on a web page. I have had a cursory look at this technology and I got a small example to compile and run.

[DavidS]

WOW, lot of typing for me.

Replies in order of posts being replied to.

@zog
I am not looking for Help for myself, I am looking for a way to use what happened to me to help others.

Though Thank you.

@Hippy:
Thank you for the input. If not woried about how open it is I could use the one that comes with Mac OS 7.5 through Mac OS 9.2.2, (MacinTalk I think) in Basillisk II or SheepShaver, as it produces good natural speech if setup correctly. Though for people that do not have a real Macintosh in storage there is the ROM licensing issue.

So I do worry about license a little. Thank you much for your input.

Google AIY is not an option as there are frequent visits to medical campus, and many medical campus block all signals for Cell and WiFi thus making something that does not need Internet a must.

@gregeric:
Google AIY is not an option as there are frequent visits to medical campus, and many medical campus block all signals for Cell and WiFi thus making something that does not need Internet a must.

Though thank you.

@zog:
Because of having to use whatever solution on medical campus, for anyone in this situation, and most medical campus blocking WiFi and Cellular anything requiring internet is off of the table.

[DavidS]

It seems that Festival Lite is a decent speech synthesis engine, and fairly simple to use. The only difficulty is figuring out how to directly pass in phonemes to flite. I am a bit slower about getting research done now, unfortunately, so not picking this up as quickly as I would have before this happened.

[DavidS]

To help to understand what I am going for I am working on a prototype version in FreeBASIC that is using Festival Lite for speech output, and I have enough of the initial comments written (after working most of the morning just writing comments) to give an idea where I am going with this:

Code: Select all

'Program to provide a decent front end for speaking in a way
'  in a way that allows for conversational use of speech.
'  The hope is that this may help those that have lost there
'  ability to form speech, though I can not provide any
'  assurance, as each person will be different.

'I would like to do a portable version in Java, this is
'  just a concept prototype implementation.

'Category labels are limited to 10 characters to reduce
'  screen space, as the two level category list is always
'  visible on the screen for quick access.
'The UI is setup for a maximum of 52 characters width for
'  the list display, including both category lists, and
'  the final items list.  This is done so that it should fit
'  comfortably on most displays, including most portable
'  devices, without any sideways scrolling.

'The order of items in the list, as well as of categories
'  can be changed by the user at any time by dragging. The
'  order of any single list can be locked to prevent further
'  changes until unlocked.

If anyone else is working on this idea, I figured it would be a good idea to provide some concept of where I am going with it.

[gregeric]

Hi David. I haven't dug deep, but it appears Google AIY has both a locally synthesised voice, and the more refined online one. Both are female, so may not suit anyways!

[DavidS]

Hi David. I haven't dug deep, but it appears Google AIY has both a locally synthesised voice, and the more refined online one. Both are female, so may not suit anyways!

Well that is good to know. I would like to find something along the lines of the speech synthesizer built into Android, as it is adjustable and very clear and natural sounding. though that is likely asking to much on any system other than Android.

[DavidS]

I am attempting to type as much as I can, to improve my dexterity all over again (relearning like a child ). As such I am going to give a little more about my particular situation, and frustrations that lead me to the realization of the importance of doing something to help others.

I have spent more time inside of medical institutions recently than I really wish to think about. I have had 6 MRIs, 8 CT scans, a huge number of ECGs, 3 EEGs, and more blood stolen than I want to guess at (to many vampires in those places ).

Through all this the most difficult thing in dealing with others has been the limited nature of existing speech synthesis front ends. The speech quality is good with what is available, though the ability to use pre-recorded phrasing quickly is greatly lacking, as they just use normal file management listings for accessing pre-recorded speech. I have learned since this happened to me that it is something that a lot of people have to deal with.

As some of what I am experiencing, that is apparently fairly common for this kind of problem includes:
+Weakness, full body, more on Left side.
+Loss of much base Coordination.
+Loss of much of dexterity.
+Frequent Loss of balance.
As well as the problem in forming spoken and signed words/phrases. It seems the ability to communicate is the one thing that can make the biggest positive difference, as there is already enough to deal with without not being able to easily communicate.

There are many questions that the medical staff ask at those places, many of which are redundant to absurdity. Thus pre-recorded answers that are well categorized and quick to access would be helpful.

I think that they are striving to get rid of, eliminate, and abolish, superfluous, extraneous, redundancy .

Then there is the problem with normal daily conversation, something you really do not give much thought until you can not do it. This is another area where having quickly navigable categorized pre-recorded synthesized speech can help a lot. If it done well it would be possible to keep up with normal conversation most of the time, as we tend to reuse the same sub-phrases probably 90% of the time in normal conversation.

So it is that I set out on this project, I am hoping I can actually get somewhere with it, and that just maybe it can be of aid to others having this experience.

SECONDARY NOTE:
This is truly something that can not be understood by anyone whom has not personally experienced it. This is so different from any other human experience that there is not a way to communicate what it is like, no matter what. I would never have thought that anything can be so far out of the normal experience had I never experienced the situation I now find myself in.

On the Positive:
This is just another loss of ability, so nothing to be down about.

I shall enjoy every day of my life, and use this set of limits to help as many as I am able to. This is a lesson I learned when I first lost my ability to walk, and a lesson that has made dealing with this much easier than it could be otherwise.

For everything we may lose in life, there is something we gain. At very least an understanding of more that may be used to help other people, and provide a bit more joy to this world we live in.

[Tzarls]

What are your limits rigbt now regarding physical abilities? I'm asking because I've seen lots of advice on the output part (the speech synthesizer) but not much regarding the input system, the interface to allow you to compose text as fast as possible. You say you have difficulties typing... Is that because you have lost control of your fingers, your whole hand, only move a few fingers? Depending on that an appropriate interface should be designed. This might help or not - long time ago I saw a system for composing text.... The idea was that you entered a word and then many words appeared flying closd to that first word. You had to move your pointer to the word you wanted to be the next one on your phrase. As soon as you did that some more words would appear so you could choose the next one and so on. The idea is that for a certain word there is a limited amount of words that could follow... You get the idea. I don't remember the name of the system /algorithm, but based on my description someone here could identify it.

[DavidS]

What are your limits rigbt now regarding physical abilities? I'm asking because I've seen lots of advice on the output part (the speech synthesizer) but not much regarding the input system, the interface to allow you to compose text as fast as possible. You say you have difficulties typing... Is that because you have lost control of your fingers, your whole hand, only move a few fingers? Depending on that an appropriate interface should be designed. This might help or not - long time ago I saw a system for composing text.... The idea was that you entered a word and then many words appeared flying closd to that first word. You had to move your pointer to the word you wanted to be the next one on your phrase. As soon as you did that some more words would appear so you could choose the next one and so on. The idea is that for a certain word there is a limited amount of words that could follow... You get the idea. I don't remember the name of the system /algorithm, but based on my description someone here could identify it.

I do see your point.

My physical limitation that has lead to my difficulty typing (and complete inability to write) is a loss of dexterity, the ability to articulate the hands and limbs in a way that is intended being lost.

Though I suppose that many would have more physical limits in that area, and require a different user interface system to compensate.

To more completely answer your question, my physical limitations that are new as a result of what happened to me are (generalizing to keep short):
+A loss of a large portion of my dexterity.
+Extreme weakness compared to normal (maybe one eighth of normal strenghth), with more loss on left side.
+Lack of sensation in left fingers, and bottom of left foot.
+Slow in moving.
+The inability to turn thought into spoken or signed language.
That is only a partial list, though it speaks to the directly physical limitations. There are other limitations, that are not as directly physical.

Though for me it is at least reasonable to type on an ordinary keyboard (albeit slow and difficult). For others this may not be the case, so it is something that has to be taken into consideration.

Thank you very much. That added layer of thought is very helpful in attempting to make the result as helpful to as many people as possible.

[stubright]

This might help or not - long time ago I saw a system for composing text.... The idea was that you entered a word and then many words appeared flying closd to that first word.

Hi,
I remember this system, it was called Dasher https://en.wikipedia.org/wiki/Dasher_(software), it was a predictive text input.
You basically steer your pointer towards the letters, as you start to spell a word the most likely letters will be the closest to your cursor. There's an Android app for it https://play.google.com/store/apps/deta ... er.android and a github https://github.com/ipomoena/dasher/releases and here's a video demo https://www.youtube.com/watch?v=nr3s4613DX8
Unfortunately the Cambridge University Website for it doesn't appear to work http://www.inference.phy.cam.ac.uk/dasher/
Stu

[DavidS]

This might help or not - long time ago I saw a system for composing text.... The idea was that you entered a word and then many words appeared flying closd to that first word.

Hi,
I remember this system, it was called Dasher https://en.wikipedia.org/wiki/Dasher_(software), it was a predictive text input.
You basically steer your pointer towards the letters, as you start to spell a word the most likely letters will be the closest to your cursor. There's an Android app for it https://play.google.com/store/apps/deta ... er.android and a github https://github.com/ipomoena/dasher/releases and here's a video demo https://www.youtube.com/watch?v=nr3s4613DX8
Unfortunately the Cambridge University Website for it doesn't appear to work http://www.inference.phy.cam.ac.uk/dasher/
Stu

Kool thank you.

I will have to do some research on that.

[DavidS]

In an interesting twist of irony, not being able to speak, I find myself writing a lecture as part of attempting to type enough to get my dexterity back. I am attempting to write a lecture on these kinds of problems, from the perspective of a person whom is experiencing them, and looking to find ways to help others with similar problems.

Writing a lecture also gives me something to occupy my time in between bouts of coding on the front end.

I hope to have the front end, designed specifically to be of aid to people in this kind of situation, and the lecture complete around the same time. This way I can give the lecture and provide one piece of aid at the same time (even if I have to give the lecture by speech synthesis, I would not be the first).

In reality the lecture borders on a thesis, just in lecture form. So it will be interesting (I just hope that no university seats attempt to nominate me for another degree, I am running out of places to hide these unwanted degrees, while keeping the promise to have them displayed).

[Gavinmc42]

Hi David,

Hope you get better before you need a solution full time.
I was actually googling text to speech stuff when I noticed your post.
Was specifically looking for Pascal source code, but had lots of tabs open.

http://www.fmxuniversity.com/build-text ... n-android/

But I guess for your application anything that works would help.
The key word to google is accessibility

https://www.microsoft.com/en-us/Accessibility/windows
Windows based solutions have been around for a long time.
Text to speech interface bring Windows to the blind etc.
Then there is the predictive keyboard text input devices.

To get you up and talking so to speak a Windows solution might be fastest.

Most or all of these features are available in Linux.
And hence for Pi's, espeak, festival, ....
http://www.cepstral.com/en/personal/linux

Eye gaze sensors are for the most serious cases of mobility impairment.
Joystick, trackball, maybe even one of those gesture sensors would help.
There is a Gesture hat for Pi's
https://www.adafruit.com/product/2325

Laser and IR led based gesture sensors are getting built into colour, light sensors.
Some mobiles have them, so table/phone based solutions might be possible, now/soon.
https://play.google.com/store/apps/deta ... mity&hl=en

Most of these proximity/gesture/light sensors have i2c interfaces, easy Pi interfacing.
https://www.kickstarter.com/projects/gr ... ing-module
https://www.parallax.com/sites/default/ ... r-v1.2.pdf
http://www.microchip.com/design-centers ... y/overview

Phone cameras used for eyegaze detection?

Hand mounted accelerometers, always thought they would work.
Hack some fitbit type bluetooth gadgets?
Cordic keyboards like the old Microwriter, means the hand can stay still and only fingers move.

Not being a talker myself by choice, stutter, speech impediment, lisp, dyslexic word retrieval etc.
I have always thought speaking is over rated, thinking has always been more important to me.
I know quite a few politicians now who need to do less talking and more thinking and action.

It might seem insensitive but loss of mobility has never scared me, that would just challenge me to cyborg myself.
What worries me now is loosing my memory and thinking skills and the signs seem to be there for me now re memory.

[Gavinmc42]

Festival comes form here.
http://www.cstr.ed.ac.uk/research/

Flite is the light version
http://www.speech.cs.cmu.edu/flite/

[DavidS]

@Gavenmc42:
Thank you for the input. You have my mind working.

The only difficulty with my limits now is that I can think and think, though not so easy to share what I figure out.

I am looking at much of what you linked, to see what is what. For the time being I am using Pico TTS for the text to speech part, as it appears to be available for every platform that I want to support with the target software (except for Mac OS).

There is a 50/50 chance that this will or will not be permanent, at least some of it.

[gkreidl]

Hi,
I remember this system, it was called Dasher https://en.wikipedia.org/wiki/Dasher_(software), it was a predictive text input.
You basically steer your pointer towards the letters, as you start to spell a word the most likely letters will be the closest to your cursor. There's an Android app for it https://play.google.com/store/apps/deta ... er.android and a github https://github.com/ipomoena/dasher/releases and here's a video demo https://www.youtube.com/watch?v=nr3s4613DX8
Unfortunately the Cambridge University Website for it doesn't appear to work http://www.inference.phy.cam.ac.uk/dasher/
Stu

Dasher is in the Raspbian repository.

[Gavinmc42]

espeak next gen?
https://en.wikipedia.org/wiki/ESpeakNG
https://github.com/espeak-ng/espeak-ng/
MBROLA? been a few years since I ran across that.

And you have no requirement to share especially if it takes so much effort now.
Save your energies for what matters.
I also think and think, which is ok unless I forget (easy these days) that it is the doing that needs doing too.

[DavidS]

espeak next gen?
https://en.wikipedia.org/wiki/ESpeakNG
https://github.com/espeak-ng/espeak-ng/
MBROLA? been a few years since I ran across that.

And you have no requirement to share especially if it takes so much effort now.
Save your energies for what matters.
I also think and think, which is ok unless I forget (easy these days) that it is the doing that needs doing too.

Thank you.

Now giving eSpeak a closer look, at least eSpeakNG. Though it looks like I will have to build it, as I can not find a ARMhf package yet, though I would expect the Win32 version to run on ReactOS.

Now I really need a net-book, or smaller x86 laptop.

[DavidS]

I am looking into eSpeakNG as a possibility for a backend. I am already implementing use of Pico TTS, shelling out to the pico2wave program.

I would like to be able to directly call Pico TTS by its API, though despite searching can not find any information on its API (other than some android specific Java stuff). Any one know where I need to look?

[DavidS]

For pre-recorded speech I Want to keep category lists to 16 main categories with 16 sub categories each, to allow the full main category list on screen, as well as the full current sub category list at all times. This leads to a question, as I wish to keep the final item lists limited to aid quick navigation.

So would 8192 entries total maximum be enough to cover what would likely be used (32 entries per item list maximum), or should I go with 16384 entries (64 entries per item list maximum)?

The advantage of keeping to 32 entries per item list is improved navigation speed, as the list can usually be mostly or completely on the screen (depending on resolution of screen in use), so it would be preferable. Though I am having difficulty figuring out if 8192 prerecorded speech items would be enough to cover the needs of the most common speech plus the answers to the oft repeated medical questions. It is difficult to figure out how many entries are really needed, as I have been dealing with the more limited listing methods, and thus having to type many things that could be in quick lists.

Also there are likely to be some entries used by users for things that are kind of specific, like for me jokes, and statements related to 3D printing or programming and algorithms, or Solar power, etc that I use a lot.

So in the opinion of those that are looking at this thread:
Do you think that a total of 8192 quickly navigable entries is enough, or should I sacrifice some speed of navigation in favor of having 16384 entries total maximum (the user would still be able to set the most commonly used entries to the top half of each list).

On implementation:
I am beginning the rewrite in Java. I figure I should start this before I get to far along, as I have to learn Java while implementing the application. While obviously in syntax based on C/C++ Java is quite a bit different in structure.