sispheor
Posts: 11
Joined: Thu Aug 01, 2013 7:48 am

Kalliope (Jarvis like assistant)

Tue Mar 21, 2017 2:04 pm

Hi!

We made a little bot assistant for the Rpi !
It's more like a framework where you you can build your own assistant like Jarvis in the language of your choice.
Some modules (neurons) are already availlable to make your brain(eg: gmail, gcalendar, facebook, date,...).
And you are free to create your own and propose them to the community.

Demo video
The website
The source code
Already available modules

YCN-
Posts: 246
Joined: Fri Jun 10, 2016 3:18 pm

Re: Kalliope (Jarvis like assistant)

Tue Mar 21, 2017 3:33 pm

Wonderfull I like this kind of project !
What did you use for voice recognition ?
I like the way you did the "synapses" stuff. I'm not really sure that I exactly get what you meant. Is it like plug-in ?
Is one built for the use of relays ?

YCN-

sispheor
Posts: 11
Joined: Thu Aug 01, 2013 7:48 am

Re: Kalliope (Jarvis like assistant)

Tue Mar 21, 2017 4:19 pm

For the voice recognition you can select a STT engine. There is also a self hosted one.

Yes a synapse is the link between input and output actions.
The input is your order, the output is a neuron (a plugin).

There is no neuron for controling GPIO direclty. But you can use the neuron script
or shellto do it.

bacardi55
Posts: 7
Joined: Tue Mar 21, 2017 4:25 pm
Contact: Website

Re: Kalliope (Jarvis like assistant)

Tue Mar 21, 2017 4:41 pm

Multiple STT and TSS can be setup (with multiple languages), I'm mainly using google STT for now as on a single RPI, it can become a bit slow to install too much stuff on it (already have Kalliopé and Mopidy) ^^.

Sispheor has explained the synapses, but the idea is really simple:
Links your voice orders (input) and neurons (functionality and message templates - the output).
So you can create orders and templates in the language you need and just reuse data sent back by the neurons. Readme of neurons can give you more ideas of what I mean :).

For the GPIO connection or other domotic devices, I'm using domoticz to manage them, and created a neuron for Kalliopé to interact with domoticz.

If you are interested, I wrote some stuff about my (heavy?) kalliopé usage here
That being said, Kalliopé is a cool tool and have now some cool neurons

YCN-
Posts: 246
Joined: Fri Jun 10, 2016 3:18 pm

Re: Kalliope (Jarvis like assistant)

Wed Mar 22, 2017 9:28 am

Wow !

I love it !! I was thinking about doing this kind of assistant myself, but I would have done it with a low level approach, using C and my I/O would have been very user-specific, but I like your style, it's better and more user-friendly.

I think that I'll soon get involved in the project, did you consider adding an http server so that you can drive kalliope in a silent mode? May be with a web site for the user where you'll only see something as the facebook messenger app. I'm not really into web devellopment but that could be something I'd like to have.

What do you think ? What's your current approach for voice recognition ? Do you translate words into strings and then annalyse them ? If this is how you do it, the server might be pretty straight forward, I'll just have to forward to kalliope the http string data.

YCN-

sispheor
Posts: 11
Joined: Thu Aug 01, 2013 7:48 am

Re: Kalliope (Jarvis like assistant)

Wed Mar 22, 2017 9:37 am

Currently Kalliope has an API.
A projecthas been started for controlling Kalliope via web interface.
And a Phone app should be available soon.

And yes we convert voice into string and then we analyse them to match a declared order in the conf.

bacardi55
Posts: 7
Joined: Tue Mar 21, 2017 4:25 pm
Contact: Website

Re: Kalliope (Jarvis like assistant)

Wed Mar 22, 2017 9:41 am

YCN- wrote:Wow !

I think that I'll soon get involved in the project, did you consider adding an http server so that you can drive kalliope in a silent mode? May be with a web site for the user where you'll only see something as the facebook messenger app. I'm not really into web devellopment but that could be something I'd like to have.
I'm not sure what you mean by "silent mode" ^^.
Kalliopé provide an API so you can start synapse (or order) via API. You can even send an audio file via API and then Kalliopé will use the TTS as usual.
That being said, when you use the API, the synapse is executed on the rpi, so if your synapse has an audio return, it will be said on the rpi that leverage kalliopé. I have created an issue to maybe modify this behavior, but it's not started yet. Issue can be found here

Also, I have started a web remote app (using emberjs) that leverage the API to run synapse. I have stopped for some time to contribute to more neurons but I'll restart working on it soon. It will be able to send free text (orders) or an audio file, so you won't need a mic or being in the same room as your Kalliopé
App can be found here
YCN- wrote: What do you think ? What's your current approach for voice recognition ? Do you translate words into strings and then annalyse them ? If this is how you do it, the server might be pretty straight forward, I'll just have to forward to kalliope the http string data.
YCN-
Yes, Voice recognition works by transforming audio into strings that are then analyse by kalliopé.

YCN-
Posts: 246
Joined: Fri Jun 10, 2016 3:18 pm

Re: Kalliope (Jarvis like assistant)

Wed Mar 22, 2017 9:57 am

Well I was meaning instead of talking to kalliope finding a way to write to kalliope. For instance with a chat interface such as the messenger app. I don't really like the idea of talking to a robot. That why I'd like to have a web interface where I would be able to write what I would have said.

You'll go to a web page and then just write and have responce in real time with kalliope. Just like the FaceBook messenger app or whatever messenger you use (whatsapp...).

I'll give it a try soon I don't think it will be too complicated.

YCN-

bacardi55
Posts: 7
Joined: Tue Mar 21, 2017 4:25 pm
Contact: Website

Re: Kalliope (Jarvis like assistant)

Wed Mar 22, 2017 10:00 am

YCN- wrote:Well I was meaning instead of talking to kalliope finding a way to write to kalliope. For instance with a chat interface such as the messenger app. I don't really like the idea of talking to a robot. That why I'd like to have a web interface where I would be able to write what I would have said.

You'll go to a web page and then just write and have responce in real time with kalliope. Just like the FaceBook messenger app or whatever messenger you use (whatsapp...).

I'll give it a try soon I don't think it will be too complicated.

YCN-
That is the goal of the web app i've done. So far, it's only listing available synapse and you can click to start them. I had in mind this ability to just run free text so it's in the roadmap, just had no time to tackle it yet.

Otherwise, I started a small CLI tool to do so, where you have a prompt and can simply type "order my great order" and it will send the order via API ("my great order"). You can find it here but it is just started (but this feature works).
I had in mind other features like installing new neurons or even configuring kalliopé via this tool.

*edit* : the link is: https://github.com/bacardi55/kalliope-cli

But all this is still yound and need more work :)

morecoffee
Posts: 4
Joined: Thu Feb 05, 2015 3:06 pm

Re: Kalliope (Jarvis like assistant)

Sun Apr 02, 2017 2:50 am

bacardi5,

I'm trying to run you CLI interface on a raspberry pi B to get an idea of how things works. I am using the v0.4.3 image. I'm having some trouble getting started. All I have done is setup the image and download your CLI program and changed it to executable.

When I run the command

Code: Select all

kalliope start
I get a long message that ends with:
in swig_import_helper
import _snowboydetect
ImportError: No module named _snowboydetect
When I try

Code: Select all

./kalliope-cli.py
The results were:
Traceback (most recent call last):
File "./kalliope-cli.py", line 10, in <module>
from cmd2 import Cmd
ImportError: No module named cmd2

sispheor
Posts: 11
Joined: Thu Aug 01, 2013 7:48 am

Re: Kalliope (Jarvis like assistant)

Thu Apr 06, 2017 9:16 am

Is that ok for you now @morecoffee?

morecoffee
Posts: 4
Joined: Thu Feb 05, 2015 3:06 pm

Re: Kalliope (Jarvis like assistant)

Thu Apr 06, 2017 1:43 pm

I'm fine for now. I plan on getting a Pi 3B with a mic and try again later. Thanks for the help on the gitter chat. I look forward seeing how this project continues to develop.

sispheor
Posts: 11
Joined: Thu Aug 01, 2013 7:48 am

Re: Kalliope (Jarvis like assistant)

Wed Jun 28, 2017 10:00 am

Version 0.4.4 online! https://github.com/kalliope-project/kal ... tag/v0.4.4

Better performance on Rpi !

sispheor
Posts: 11
Joined: Thu Aug 01, 2013 7:48 am

Re: Kalliope (Jarvis like assistant)

Sun Jan 14, 2018 10:11 pm

V0.5.0 online !!
New text to speech, hooks support and more. See the complete changelog here.

sispheor
Posts: 11
Joined: Thu Aug 01, 2013 7:48 am

Re: Kalliope (Jarvis like assistant)

Thu May 31, 2018 2:03 pm

v0.5.1 online.
You can now chain your orders!

New doc: https://kalliope-project.github.io/kalliope/

NOsen
Posts: 9
Joined: Wed Feb 06, 2013 11:08 pm

Re: Kalliope (Jarvis like assistant)

Sun Jul 15, 2018 6:49 pm

First of all thank you for your work on this.

Question i got it running on my raspberry it talks i can chat with it from my phone
however when i use the hotword "snowboy" it do not respond at all im using google speech to text paid version with a proper API key

If i record from the mic it works and it sounds good when i play up it but it just wont respond to any spoken word.

my asoundrc looks like this

Code: Select all

 cat .asoundrc
pcm.!default {
    type hw
    card 1
}
ctl.!default {
    type hw
    card 1
}

and like i wrote it do record my voice when i run arecord.


My settings file

Code: Select all

default_speech_to_text: "google"

speech_to_text:
  - google:
      language: "en-EN"
      key: "APICODE"

it seems like its not even trying to connect to google at all my google account has 0 hits and im running the API unrestricted so it should work

Kalliope log (there is no errors in the eror log

Code: Select all

Starting REST API Listening port: 5000
Starting Kalliope
Press Ctrl+C for stopping
zy loading)
Starting order signal
 * Environment: production
I'm ready
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: on
Any suggestions would be appreciated!

Return to “Automation, sensing and robotics”

Who is online

Users browsing this forum: No registered users and 9 guests