Within the conversational AI project we are developing a chatbot solution that uses a 3D chatbot to help with the coaching of students. The virtual avatar integrates with the learning management system and uses elements of cognitive based therapy to help students overcome typical problems that plague our students such as procrastination, lack of study planning and communicative problems (with peers and staff). The chatbot should be intelligent enough to forward students to the correct persons if the system cannot determine the correct course of action.

The development of a 3D chatbot poses many problems and contains many moving parts. The technical implementation has support for 3D animation, face and emotion recognition, language understanding, speech recognition and text to speech.

The chatbot uses a simple pattern recognition system as a first step to recognize the intent of the user and extract entities. If no pattern is recognized, the user input is handed over to a third party language understanding service (e.g. Microsoft LUIS). The intent is then processed by our system which delivers the correct response. The subjects of the discussion are maintained in a DAG (directed acyclic graph) to make it possible to maintain a working state of the conversation. It is therefore possible for the student to reference an earlier response of the chatbot.

For the speech recognition and text to speech we also rely on third party services (i.e. Google Cloud speech). For the response model a programming language was developed that makes it possible to gives responses to the user (student) via multiple channels. The primary channel is off course text, which can be converted to speech, but it also possible to for example start a video in response to a request of the user. The coach bot can also receive cues about which animation to play as part of a response, or for example the facial expression that should be assumed.

Finally, the 3D chatbot can be hosted into any number of platforms including mobile which makes it possible for students to access the chatbot whenever needed.

Status: project ongoing, pwo funding

Related topics: AI academy