A Talking Head Lends a Helping Hand

Two professors at the University of California-Santa Cruz received a $100,000 grant from Microsoft to fund their work on a virtual speech therapy tutoring system to help stroke victims regain their powers of speech.

The project, funded by Microsoft's "Cell Phone as a Platform for Healthcare" program, is first being deployed in Malaysia and targets 40,000 people there who suffer a stroke every year. Rather than launching the program at speech therapy centers, the project's team has made the technology available via mobile phones. Because Malaysia has a lack of speech therapy professionals, and travel to these specialists is difficult for most, the talking head avatar will provide patients with an easy-to-access means of instruction. Using text-to-speech (TTS), the virtual tutor will imitate the mouth and lip movements of a native Malaysian speaker.

Project leaders Dominic Massaro, a psychology professor, and Sri Kurniawan, a computer engineering professor, are building off one another's areas of specialty and using ground work from previous projects as a starting point. When Kuriawan arrived at UCSC in September she began a remote, phone-in speech therapy program for Malaysian stroke victims, but found shortcomings throughout the pilot. And while she says the phone-based therapy was a better option than having patients travel to speech therapists, it did not solve every problem. With too many patients and not enough providers, Kurniawan needed an easily accessible program. She had ideas about using cell phones in therapy, but was unsure of her approach. Then, she met Massaro, who had completed vast amounts of research and work using a virtual aide to teach deaf and autistic children.

"At the time, I was also having a call about using a cell phone for healthcare platforms," Kurniawan says. "[Massaro] showed me a talking head on his iPhone, in English. My head was spinning, because I suddenly had this idea and everything clicked together."

Having seen the virtual talking head, Kurniawan envisioned a similar tool for stroke survivors: a virtual spech therapist who taught patients using synthesized speech. From there, she went on to investigate available TTS technologies that would support the Malay language. She says the team is close to choosing their software, and that the process of assigning proper lip and mouth movements with Malay phonemes was not as difficult as expected.

"If someone speaks Arabic, it's quite easy to translate into Malay using lip and jaw movement," Kurniawan states, noting that the talking head can already accommodate the Arabic language.

The program will go on a test run with 10 patients this year, and functions as part of a two-tier program. First, patients receive an evaluation and are given a tailored speech therapy from an industry professional. Then they receive regular, timed calls to their cell phone from the virtual tutor, who then conducts the lesson over the user's cell phone screen. Patients continue to check in with their actual speech therapist every two weeks and chart their progress. Though the project has begun in Malaysia, Kurniawan -- who is a native Indonesian -- says she thinks it can expand to other developing countries.

"The Malay language or a variation is spoken as well in parts of Singapore, but also more importantly, by the whole Indonesian country," she says. "Malaysian and Indonesian are 90 percent similar, so once we develop a talking head that speaks Malay, it's a stone's throw away from having an Indonesian talking head. I really hope this [program] could be as widely available as possible."