Almost everyone has heard of application tuning; however few people can point to specific examples of how tuning improves completion rates and customer satisfaction. Here are several real-world examples of how tuning identified and rectified an application problem.
Tuning Case 1 – Utility customers got what they asked for, not what they needed.
One utility company deployed a call routing application (“billing”, “repairs”, etc.) but more than half of their callers were still requesting the operator. This company was very customer service oriented, so they had specified an initial prompt that offered “operator” as the final option. When callers used the speech system, they were successful, but their polite callers were patiently waiting through the entire prompt and then asking for the operator because that was what they were used to doing. Removing that option from the first prompt increased their completion rate by 50% with no decrease in caller satisfaction.
Tuning Case 2 — Reducing confusability in a banking application
A banking customer was prompting for the command “payment”, but the speech recognition system would hear “agent” 10% of the time, and transfer them unnecessarily. This was corrected by improving word pronunciations and adding appropriate grammar probabilities. Another system (or two) would often hear “help” when the caller actually said “no”, and then would give a help message to the confused caller who couldn’t have been more clear.
Tuning Case 3 — Don’t use your speech app to re-train callers
Another customer had callers who were used to speaking with agents and giving their account number and PIN in one sentence. Rather than rejecting the whole utterance and forcing the caller to start over, the system was changed to allow this, smoothing a negative first interaction that was being experienced by 25% of their callers during the switch to automation.
Tuning Case 4 — A newspaper’s use of barge-in catches callers wrong footed
It can be hard for callers to get in sync with the system, especially when barge-in is enabled. One newspaper application that asked for the caller’s phone number was failing because many callers would pause after the area code and the system would cut them off. The system was changed to accept an area code, and then just go on and ask for the rest of it. This increased the automation rate by 20 percentage points.
Tuning Case 5 — Ubiquitous name and address modules need pronunciation tuning
Systems that take a person’s name, or a street address, usually depend upon automatically generated pronunciations. Results can vary widely. One system went from 77% accuracy on 30,000 cities to 85% by hand tuning pronunciations, especially for the most common cities. Another went from 81% accuracy to 83% by simply asking the system to make up fewer alternate pronunciations.
Tuning Case 6 — Eliminating impossible choices increases odds in a wagering application
A wagering application was built that could sell bets on races at 400 different tracks. There were a lot of similar names, and first attempt recognition accuracy was only about 85%. Demanding bettors needed speed and accuracy above all, and this was not good enough. It turned out however, that while there were 400 different tracks, the system would actually only be selling bets on 10-40 of them at any one time. A dynamic grammar, updated once daily, was used to restrict the choices to those actually available and recognition accuracy shot up to 97%.
Tuning Case 7 – Listen carefully when a caller says nothing
One common problem VA looks for is when the callers don’t say anything. If the no input rate is high after a particular prompt, there could be three problems.The prompt may be confusing or difficult to understand. The caller may have been misrecognized in the previous state and ended up getting a prompt they weren’t expecting. Or, if the no input rate is high across the entire application, there may be a problem with the audio tuning parameters or even the audio signal. In one application, callers were being routed in with “in-band” touchtones played to signal point of origin. The system often started listening to the call while the touchtones were still being played, and adjusted its audio accordingly, making it deaf to the caller when they tried to talk to it.
Tuning Case 8 — A major retailer’s grammar expanded to reflect what callers were actually saying
Another common flag for problems is rejection – when the recognition system isn’t matching on anything at all. This is usually because callers are saying something out of grammar, something the system hasn’t been programmed to listen for. The completion rate for a retail call routing application was increased from 70% to 77% by adding the 100 most common things people were saying that were out of grammar.