How to Double Retention and Dialog Length for your Chatbot

Image courtesy of Geralt on Pixabay

I’m a software engineer currently building chatbots. I’ve published more than twenty custom skills on the Amazon Alexa platform, and last year published my first Facebook Messenger chatbot named Chuck.

I measure success by how much users are educated or entertained by the bot. The best KPI’s for this are retention, and depth of the dialog. Here is my latest insight in how this applies to applications written for Facebook Messenger. The NLU and orchestration layer is written using AWS Lex, as well as AWS Lambda functions written in NodeJS.

What is the Ideal Chatbot?

The ideal chatbot is what we saw with the HAL 9000 in A Space Odyssey. Just a blinking cursor or microphone that we can ask any question. No constraints, just answers. In applying this to software tools around Facebook Messenger, here is what that architecture looks like.

Natural Language Understanding (NLU) models are good at deciphering user requests, and few scale issues. What limits applications is how many features can be written to support the thousands of different requests users make. Once those features proliferate, the challenge becomes discovery of the features — what the bot “knows”. A blinking cursor has limitless options, but a cold conversation starter. Users will have expectations based on the branding of the bot, but little else.

Traditional visual applications handle feature discovery by providing menus and widgets on a device screen that provides options. An effective chatbot needs an approach for this as well. A good approach is to provide context with each response to the user, providing hints at what is possible. Building discovery into the responses encourages dialog, and actively engages the user.

How to Improve a Facebook Messenger Chatbot

Back in December, I began adding buttons to my Facebook Messenger bot in an effort to improve the usability. This was in response to a lack of growth in usage. My bot was growing in features, as it ingested more data and algorithms to manipulate it. Without users knowing of it’s newfound capabilities, it was stuck.

I tried different ways of advertising the features on the Facebook Page, but none were effective. For example, I saw trivial gains through posting video demonstrations of the bot demonstrating new features. Similar results were achieved when writing articles in the news feed. Adding buttons into the bot responses did provide the lift I was looking for.

Designing Buttons into User Interaction

Buttons augment responses by providing clear direction to the user on what’s possible. Let’s start by the initial response the bot has when the user initiates a conversation with “Hello” or an introduction.

Scripting clear language in the response is important, including providing a sample phrase (i.e. “How many calories in a Chicken Sandwich”.) Adding buttons takes this a step further, and proves far more effective. All a user needs to do is click one, and another response is coming their way. It also saves the user time for typing the potential requests. Mobile devices are tricky to navigate spelling and users tend to abbreviate sentances. Buttons simplify the user experience.

Results of Adding Buttons

Now for seeing the progress. I use the analytics tools within Facebook to track detailed usage. It is essential for measuring feedback for changes made within the app. Here’s an example dashboard that is provided if you haven’t used before.

The first success metric is around depth of the dialog. I want users to be educated with my bot. This can be quantified by tracking how many times a given user replies back and forth with the bot before exiting. Here is the chart tracking the amazing growth after I added buttons. More than double the activity per user.

Growth of session length after buttons added in early January.

The next success metric is around retention. If a user believes they received useful information, they will be back. The easiest view on this is around weekly retention, and daily statistics shows similar success.

Retention of users after buttons added in early January.

The first week retention numbers are almost double since I added buttons. Less successful has been the second week retention which is slightly down. Sample size for week 2+ isn’t significant with these cohorts, but will be a trend to monitor in the next month as this grows.

Limitations with Buttons

The biggest limitation is the amount of screen space that buttons take up. The chat dialog becomes overwhelmed with the space being consumed, and with Messenger, there is no font size to shrink this. History of the session gets lost from immediate viewing, but the user can scroll back.

Messenger renders up to three buttons at once, so there is the potential that users limit their thinking to only what they can click. Here is an actual dialog talking about pizza.

These buttons can constrain users thinking, and skew results towards what is presented. For example, the bot knows about Domino’s, but is not picked at the same frequency as those rendered as buttons. I attempt to phrase the label above the buttons to highlight this — “Click button below or type response.”

Conclusion

Buttons are an essential part of designing a chatbot. It introduces some visual constraints, but worth a doubling of retention and transaction depth.

For more information on the underlying technology, here is the source code for the bot, including design documentation.

terrenjpeterson/caloriecounter_caloriecounter - AWS Lex based chatbot that calculates calories based on different fast food restaurants. This was an…_github.com