Usability Scorecard
This is the fourth in a series of "interactive columns" where our readership can participate in auditing self-service systems across nine industry categories. Here we provide highlights on companies who score in the upper quartile of the Sterling Audits Usability Index. The index is a standard methodology for rating the overall efficacy of both Web sites and Voice Response Systems. In this issue, we concentrate on voice response systems in the government sector. The Internal Revenue Service Teletax Information Line edged out the other government-operated voice response systems based on the surveys submitted.
IRS Teletax Scores Highest on Usability Index
The Internal Revenue Service Teletax information line is a combination Audiotex and transactional system. You can check the status of your refund - if you're fortunate enough to have one coming to you. You can also get audio information on general topics. Frankly, those of you who scored this one in our survey were way too generous in my opinion, but more on that later.
We chose the government segment randomly for this issue and tabulated the scores for this industry using the Sterling Audits Usability Index. The top metrics are: Navigation, Content, Usability, Interactivity and Credibility. We figured the Upper Quartile; Median; and Lower Quartile scores for each of the top metrics individually. Then, we add these for a grand total, which is used to figure the top performer. We do this to ensure that the top overall performer can be recognized even if they are not the best in a single category.
Based on all of the surveys Speech Technology Magazine readers and our own researchers submitted, the IRS came out on top (See the usability scorecard sidebar). It is important to note that I did not participate in establishing the scores. What I do here is add my own interpretation and observations after having audited the system myself — after the scores were tabulated.
Navigation
The Teletax system’s Navigation score is 10 out of 20, which establishes the system as being mediocre in this part of the index. The scores of the upper quartile, median and lower quartile are all within two points of each other here. This means that government systems score low on the Navigation metric across the board. With this I can agree.
The most glaring issue is that this system absolutely cries out for speech recognition. The fact that it scored the highest and doesn’t even use speech says a lot about the state of government sector systems. This one uses three-digit "topic codes" for general Audiotex information. Each of these top-level codes yields yet another verbal menu of three-digit topic sub-codes. It’s a nightmare. Something a simple directed dialog ASR could fix in a jiffy. (Won’t someone please put our hard-earned tax dollars to work and sell the IRS an upgrade? I’ll do the design part at a patriotic discount.)
The main greeting, language choice and main menu take 40 seconds. And the pacing is about 105 words per minute. That’s a bit too fast considering all the content they have you listen to before your menu choices. The bottom line is the greeting and menu are way too long. For anyone calling not seeking refund status, they would have to repeat the menu because it has phone numbers in it. The main menu choices themselves are straightforward: to check the status of your refund, press one; for recorded tax topics information, press two. It’s unfortunate that it takes so long to get there, because the simplicity of choices (two) in the main menu is sublime.
The Teletax system uses non-standard navigation keys. For example, in some places, it tells you to use "9" to repeat ("9" is the IVR standard for hang-up). In one place, it tells you to use the "2" key to exit and yet another, to use a "3." The use of "8" to repeat a menu was established by Intervoice and others in the late 1980s. Likewise, the use of "*" for the previous menu and "**" for main menu was established by Genesis Electronics, an early voice messaging maker back in the 1980s.
Forget about getting to an operator. If you hit "0" you’ll either get a menu repeat or get disconnected. This is odd, considering they give you a phone number to call for live assistance. Did it occur to them to simply route callers to that number rather than hang up on them?
Content
The IRS Teletax system scores 20 out of 20 from a content standpoint. This is where I think our survey participants are being way too generous. Yes, you can get your refund status and that’s good content. But if you’re calling to get general tax information with the Audiotex part of the system, it’s pretty shallow stuff. Slogging through the three-digit topic codes is difficult because if you want to use "9" to repeat – you have to do it during the prompt you want repeated. If you do it afterwards, it repeats the next prompt. (OK, so that’s navigation, but boy is it frustrating). Anyway, after you finally get to the content, it’s pretty weak. It seems like they simply read the first paragraph from the tax return instructions for each general topic. Unless your question is very general, there’s no help here.
This would be a perfect place for the system to offer alternate access to content through other media such as mail or fax. Ironically, the IRS does have a fax-back system. But it is referenced only tangentially in the Teletax system. You have to call another 800 number and then that system refers you to yet a third 800 number for the fax-back alternative. What a hassle!
The system also tends to be a bit wordy. As is the (bad) practice of many voice response systems, the availability of the IRS Web site is mentioned not only in the main greeting, but in other places in the dialog. It’s a real human memory-clogger.
In addition to being wordy, the phrases are too formal. Regular people don’t speak the language of this machine. For example, the system says: "Please enter the social security number for which you are calling." How about: "Please enter the social security number you’re calling about."
Getting help was confusing. The "repeat" and "exit" commands were different depending on what part of the system I was in. There was no main menu escape (* or ** usually) and if you did not hit "9" to repeat a menu during the menu – it would skip to the next topic. This means you’d have to wait until getting to the end of the next topic to "exit" and start over again from the sub-menu to listen to the whole thing over again.
Usability
The Teletax system scored higher in the Usability department compared to other government systems. The score was 16.67 here, with the bottom of the barrel being 8.90. This just goes to show that the government sector is not so great in the usability area. Most self-service experts think this is due to the lack of competition in this sector. I mean, where else are you going to go to pay taxes? We’re all kind of a captive audience…
You can overdial prompts, but not at the initial greeting. But considering this is the kind of system you only call a few times a year, this is not a big detractor.
The system was intelligible, but the pacing was too fast in places (105 words per minute). Fast pacing affects usability because it causes too many repeats. The recommended 55-60 wpm sounds really slow – but consider the medium. Since voice response transactions are conducted in the temporal domain, there are no spatial cues such as gaze or body language to help you along in the dialog. So, if there’s little to say - as there should be in audio menus - you won’t even realize the dialog is slow-paced. Therefore, it’s a best practice to make the pacing slower than a face-to-face dialog so there are not so many repeats.
The Teletax system has pretty straightforward turn-taking. The indicators are chiefly based on syntax, although there are a few prosodic cues. The sentences were well organized so I had no problem anticipating my choices. The grammatical structure of the sentences, albeit a bit too formal, left little question about when it was my turn.
There were five steps to task completion for refund status. This is good if you don’t make any mistakes, because the system uses implicit verification. But if you make one mistake (your social security number, filing status or refund amount), the system reverts to explicit verification for each item – and therefore doubles the number of steps. You can tell that some thought was put into this. Strangely, this takes some solid design know-how but they missed some of the simpler things like standard navigation keys.
Interactivity
The Teletax system scores higher than other performance quartiles in the Interactivity area, but there is still room for significant improvement. The system scored 14.11 out of 20 here, which distances the lower quartile score of 8.03 by 6.08 points. The median was 10.65. Besides Navigation, this is the second weakest area for the IRS in the overall Sterling Audits Usability Index.
Fortunately, the system does well in task completion. I believe we owe this to the simplicity of the tasks; however, and not to a superior design. If the tasks were more difficult, this system would really suffer here.
Error recovery routines are either simplistic or virtually non-existent. For example, on the second pass of a turn, after five seconds, the system drops you in a menu with referrals to four toll-free numbers and tells you to call back when you have all the information you need to use the system. It then hangs up on you. I guess hanging up on you is a form of error recovery? I guess your third try is to call back? As you can see, the system is a bit unforgiving, so that’s why it’s a blessing that the choices are pretty simple.
If you hit the wrong combination of digits, let’s say, to repeat a menu, you will hear: "We're sorry, due to technical difficulties, your call cannot be completed at this time. We apologize for any inconvenience. Please try your call at another time. Thank you for calling the internal revenue service [hangs up]."
The dialog style is direct with dominant use of the active voice. Most choices are cast in the imperative active voice. For example: "Please enter the social security number for which you are calling." Yes, it’s wordy and way too formal for most callers, but it’s better than: "Your social security number is needed."
Timeouts for starting an entry are generous. For example, on the first pass, the system waits for five seconds before repeating your choices. Timeouts between multiple digits (for social security number, topic number, etc.) are set at seven seconds, which is ample time for fumbling with the keys.
Generally, the system allows for "two strikes and you’re out" on silence and the same for errors. And when I say "you’re out" this system just hangs up on you after a brief message: "Thank you for calling the Internal Revenue Service." The overall observation here is that the designers spent very little time on error recovery, and considering there is no operator to help callers this was just plain lazy.
Credibility
The Teletax system scored slightly lower than the upper quartile in Credibility with a 17.59 out of 20. The lower quartile scores in the government sector were 11.38 and the median was 15.25.
Most respondents characterize Teletax as a "professional-sounding machine." It is clear that the designers were not attempting to personify the system. But the speech talent and recordings are well done and professional throughout. The diction is articulate, using a well-coached female voice as the talent. The terms the system used to describe choices were clear and easily understood. And the concatenation of digits was pretty good and not as "choppy" as most systems.
One of the biggest reasons why the system did not score higher in the Credibility area was because of how disjointed some of the referral information is. The application was fairly consistent throughout, but it referred you to many other systems. In numerous places, you are referred to other telephone numbers to get information that should have been linked directly into this system. These phone numbers are barked-out in the middle of menus. At the very least, you should be directed to submenus where the numbers are kept. Instead, you go scrambling for a pencil to jot down a number for something that should be in the very system you called. This disparate mélange of systems really gets under the users’ skin. At least they seem to be keeping the system current, as it gave me tax refund data based on the 2003 tax year.
We urge you to participate in this column by doing your own surveys of systems you regularly use. See the "Usability Scorecard" sidebar for instructions.
Usability Scorecard - Government Segment
Based on all of the surveys completed, the Internal Revenue Service Teletax Information Line was tops in all categories with the exception of Navigation and Credibility. I know there are better voice response systems in the government segment, but no one submitted them or scored them as part of our ongoing survey. Want your voice heard? Log on to the research portal at http://www.sterlingaudits.com/research.html. Sign up as one of our researchers. Submit a few of the companies you do business with as projects. Once approved, you’ll get a notice to go ahead with the survey the next time you log on. There are over 200 in-depth questions, so be prepared to spend an hour on the first one. We pay the paltry sum of $10 for your trouble. An on-line dictionary explains "shop talk."
Edwin Margulies is co-founder of Sterling Audits, a firm dedicated to quality improvements in customer service automation and contact centers. The company specializes in benchmarking the usability of self-service systems. As EVP and chief of research, Margulies is responsible for research projects including the Web Site Usability Almanac and the Voice Response Usability Almanac. He is also on the board of directors of AVIOS (Applied Voice Input/Output Society), where he participates on the marketing and conferencing committees. He can be reached at 702-341-0314 or ed@sterlingaudits.com .