In Praise of User Research
Most research projects involve looking for the answer to a particular question, such as why are so many customers coming into the call center without being identified? Or, should we make technical support the first option on the menu? Such questions are often too broad to test directly, so the research actually begins when we hypothesize about the factors that might be contributing to our question. If we hypothesize that low recognition rates for account numbers might be why customers are failing to be identified, we measure recognition rate at that state in the application. If we think technical support is more important than other menu options, we can measure the percentage of customers who choose each option at the relevant menu.
The accepted procedure is to ask a question, hypothesize about the factors at play, take some measurements, then tweak things to see if we can do better. For those working with speech applications, we might add or remove items from the grammar or adjust parameters, then measure recognition rates again. We make similar changes to user interface elements (prompts, menus) based on usability test results.
Many of us in the speech community were trained in the scientific method, which means that we tend to devalue research that is more exploratory in nature. Any research effort that seems insufficiently focused risks being labeled a "fishing expedition." I'd like to suggest that we would benefit by broadening our horizons and adopting more loosely defined research techniques. Specifically, I'm thinking about field methods of user research in which you observe users in their own environment as a way of more fully understanding their needs.
One of the biggest objections to this type of research is that it is not methodologically rigorous, but the lack of a tightly defined hypothesis does not have to mean you abandon methodological controls altogether. User research maintains controls over participant selection, and is often more rigorous in selecting participants than a typical usability test because you invest so much time and effort in each participant. The rigor of user research is also boosted by including a formal test protocol, as well as qualitative and quantitative methods that allow for a greater breadth of measurement and types of analyses.
User research methods are often quite straightforward. Working with a client building a mobile application for business travelers, I accompanied several travelers as they took off for a business trip. These people allowed me to tag along during the last 30 minutes or so before they left for the airport, gathering their laptops, keys, and suitcases; made their way through security; and then boarded their flights. My protocol was mostly just to follow and observe, allowing the participants to volunteer any information they wanted, and asking a limited number of questions about the mobile apps they used during the session.
Logistically, this was a challenging study, and it resulted in what at first seemed like an overwhelming mountain of data. But we soon realized that the breadth of the data allowed us to mine it for many purposes. Based on the research, a key product feature was scrapped because customers had no need or desire for it. Instead, the product team discovered a previously unknown issue that could be easily addressed by adding a new feature to the application. The reason we were so confident making these big decisions is that this research was grounded in reality. We were careful not to draw overly broad conclusions, which is important for data collected in varied environments with fewer methodological controls between participants. In spite of this, the data helped my client understand behavioral and attitudinal trends among its customer base, and validate its assumptions in a real-world setting.
User research can be a hard sell to some clients—if you don't know what you're looking for, it's hard to guarantee what you'll find. On the other hand, there are often times when organizations have hard-to-diagnose issues that are a perfect fit for user research. Unexplained patterns of customer behavior cry out for user and field research methods because nothing you're currently testing explains the issue. User research is also among the best methods of identifying emerging trends in customer behavior and attitudes.
What are the questions that you don't know to ask? What hidden information about your customers might allow you to offer service that differentiates you from the competition? Take a fishing expedition and you might find out.
Susan Hura, Ph.D., is a principal and founder of SpeechUsability, a VUI design consulting firm. She can be reached at susan@speechusability.com.
Related Articles
When serving customers, a little change can be a good thing.
10 Feb 2014
Users accustomed to substandard systems can't imagine any better.
10 Feb 2013