Northeastern Researchers Develop AI App to Help Speech-Impaired

Northeastern University researchers are developing an artificial intelligence-powered app that will give speech-impaired users access to speech recognition and personalized text-to-speech synthesis.

Computer science professors Aanchan Mohan's and Mirjana Prpa's app, which is being called Speak Ease, that will give speech-impaired users access to a range of communication tools on their phones: speech recognition, text, whole-word selection, emojis, and personalized text-to-speech synthesis.

"People either use speech recognition in isolation, or they use text-to-speech in isolation, or they type in isolation," Mohan said "Nobody had put all three together."

Using large language models to predict users' next phrases, the app will help people with communication disorders converse in real time and in their own voices and moods.

Transcriptions can be edited to correct errors, and the app suggests contextually relevant phrases with an emotional tone suggested by AI.

Prpa, whose research focuses on human-computer interactions, and Mohan, who works on natural language processing, are based on Northeastern's Vancouver campus. They are developing the app with help from speech language pathologists and a partner agency in British Columbia, Communication Assistance for Youth and Adults, whose speech and language pathologists provided input in the app's development.

Using samples of user voices, the app will eventually be able to convert atypical speech to a more intelligible version. The app's speak mode will create a transcription that users can edit and play back in their own voice using text-to-speech software.

The app's large language model features will use past conversations to suggest relevant words and phrases.

Preserved speech samples would make the app useful for someone with a degenerative condition, Prpa said. As their capacity deteriorates, they can use the app to continue speaking as they intend to. The same feature could be used in the opposite context, for someone recovering from a stroke. Speak Ease could support a person as they gain the capacity to speak again.