When Bad IVRs Are Good Enough
I recently ran a usability test on an IVR system that left me wondering about the value of such tests. The system was, in a word, awful. The call started with a long list of infrequently used options, each explained in exhaustive detail; for example, "If you're calling to talk about or make changes to an existing account or discuss opening a new account, say, 'Account Services.'" The system identified calls via ANI without informing callers that ANI was being used or asking them to confirm their identity, and then launched into a status readout that had not been introduced or requested. The readout contained a great deal of information presented in a mix of recorded prompts and poorly tuned text-to-speech, which failed to include pauses that would have made the information far easier to comprehend. I was sure that the test was going to produce hours of scathing commentary.
I was quite wrong. Participants didn't hate the system; on the contrary, some said it was easy enough to use and understand. To be fair, the majority did successfully complete their tasks. Participants had some complaints, but overall, they were satisfied.
I asked a colleague, "Why can't they see how much better it could be?" And then the light bulb went on: Maybe they can't picture a system that is designed better. If participants can't imagine anything better, and they're able to complete the tasks in front of them, maybe this bad system doesn't seem so bad.
The role of user testing versus the role of design is a longstanding debate in the larger (that is, non-VUI) design community. Even a casual Google search will return articles and blog posts about how user testing is worthless because it fails to produce new ideas. The thinking is that no amount of usability testing will lead us to the next big thing or the next-generation user experience. In a literal sense, I agree: User testing does not directly produce innovation. But this is a fundamental misunderstanding of the purpose of user testing. We test to understand the interaction from users' perspectives and find instances in which the existing design fails, not to devise solutions to the problems. Solving interaction problems isn't the user's job.
That's not to say users don't try to solve the problems they discover in a system. Usability participants often share what they think you should do to fix the problems they discover, but we should be cautious in taking these suggestions at face value. Participants often recommend solutions based on other systems that they are accustomed to. There's no way to judge the quality of these systems, whether the solution used there will also work here, or if the participant is even faithfully representing the interaction. We need to listen past the specific suggestions and hear what users really desire. When participants in my test said they didn't want the company to use their phone number to identify their call, I sensed their discomfort with ANI being used without their knowledge, but the solution was not to discontinue ANI identification as participants suggested. A better solution is to add prompting that explains what's occurring and puts callers at ease about the company using information it already has to provide a more streamlined experience.
We can't expect usability results to reveal how to fix the system we're testing, only what needs to be fixed. Usability testing is an excellent method for finding flaws, but what we should do to eliminate them is usually not revealed. Imagining a better interaction is the role of the designer. Looking back at my own experience, I realize I had confused my role and that of the user. First, I made the rookie mistake of thinking I knew what the usability results would be before listening to the users. Then, I was confused when the users didn't see the solutions to problems that seemed so clear to me. What looked like a failure of imagination on the part of the user was really a reminder to this long-time usability tester to approach each test with a beginner's mind.
Susan Hura, Ph.D., is a principal and founder of SpeechUsability, a VUI design consulting firm. She can be reached at susan@speechusability.com.
Related Articles
Consider this solution to reel in elusive answers.
28 Jul 2014
When project teams lose sight of the big picture, users lose too.
10 May 2013
It's a fine line between love-hate and love-to-hate.
01 May 2013
This valuable tool offers unique insights into caller motivations.
10 Jul 2012