Ada's Predictive Suggestions
About the project
Ada Support is used by companies like Zoom, Square, AirAsia, Medium, and many more. In 2019, I was a Senior Product Designer working on Ada's new feature called Predictive Suggestions. My team consisted of Machine Learning engineers and a Product Manager.
The problem
When the bot isn’t able to answer a question, it serves a standard “Not Understood” answer to the chatter. 10% of all conversations that receive a “Not Understood” response have a high abandonment and low satisfaction rate. We reviewed 500 such conversations and found that:
A single “Not Understood” response could reduce chatter confidence.
Chatters are quick to ask for an agent (sometimes the first thing asked).
The bot’s large knowledge base is not apparent to the chatter.
How might we…
Reduce the chances of “Not Understood” responses?
Surface results faster and more accurately?
Let chatters discover more of the bot’s knowledge base?
The design challenge
Design an experience that allows the bot to auto-suggest relevant answers as soon as chatters start typing. In addition, design the experience for enabling the feature in the admin tool.
Chat requirements:
Show up to 3 suggestions
Allow users to show and hide suggestions.
Suggestions should appear quickly and refresh automatically.
Admin Tool requirements:
Show how suggestions differ from clarifications.
Select how many suggestions are shown to chatters at a time.
Control which answers are included in suggestions.
How predictive suggestions work in chat
The chatter sees suggestion as they type. The suggested answers do not appear until at least two words (excluding stop words) are typed. As the chatter continues to type, the suggested answers continue to update. We experimented with how fast the suggestions should update to avoid being too distracting. When the chatter reduces what they have typed to less than two words, the suggestion are no longer shown.
Selecting a suggestion in chat
Testing with real chatters
Here were some of the questions we asked and the results.
Is it obvious that suggestions can be clicked on?
All testers found it obvious to know suggestion can be clicked on.
When do you expect predictions to be shown?
Users expected predictions to show up after typing 2 words.
Are labels clear enough to help you make a decision?
Testers gave label clarity a score of 4.9/5
Is it easy to hide and show suggestions?
All testers found it very intuitive and easy.
Does it significantly improve your chat experience?
Suggestions did improve the chat experience (4.6/5 rating)
Final experience with Suggestions
UX for Bot Managers
Settings View
Bot managers use Ada’s Dashboard to build the bot’s knowledge base and control how the bot functions. Here we included some quick settings that allow the Bot manager to choose how many suggestions are served to chatters.
Animated Tooltips in Settings
I also designed two new animated tooltips that make it easy to understand the difference between clarifications and suggestions. This was one of the delightful moments for Bot managers to see!
Answer Settings
Bot admins can control whether answers can be presented by the bot as a suggestion and clarification in the Answer Settings. A new “Include in Clarifications and Suggestions” toggle is added here.
The impact
Early results of the predictive suggestions showed positive metrics. Square was an early adopter and tester of this feature and enabled it in their bot. Since enabling, they saw a 2% increase in the bot’s Recognition rate of typed answers. While 2% sounds small, it is significant considering the tens of thousands of conversations the bot handles. In addition, the fact that this client’s answer flows relied heavily on Quick Replies, shows a big win for this feature. It means Bot builders can spend less effort building tightly structured answer flows and now harness more of the bot’s predictive power.
