Conversational Commerce on the Rise: Post-launch and chatbot metrics

So, you’ve just launched your chatbot, your eyes wide with hope and the possibilities of additional engagement and sales. You eagerly await the first user to try your bot, only to discover that they asked a question you didn’t account for in your initial training. Sorry to break it to you; your chatbot journey is far from finished. Welcome to a chatbot’s post-launch period.

How users will turn your chatbot upside down

Let’s get this out of the way right now: It’s almost impossible to launch a perfect chatbot.

You will always have to wait to see how your specific customers actually use it, how they word their questions, what areas they focus on. Of course, ideally, you already did a lot of this work in the initial training, working from actual questions asked by real customers. Nonetheless, preparation will only take you so far. You need actual users’ feedback to smooth down the edges.

This feedback is populated into the bot’s data storage, helping it learn. Data is what sustains the bot and helps it grow, thus making it smarter. Even the most advanced chatbot platforms like IBM Watson are nothing without relevant and meaningful data to point them in the right direction for your use case.

Expectations can change

Actually launching your chatbot may be an eye-opening event, because you may realize that your customers are not at all interested in using it the way you intended for them to use it. Let’s take an actual example of a chatbot project Niteco worked on for a multinational company.

The client had intended the bot to be used to assist customers through the buying process – offering a load of information about products and their usage. What customers did, however, was use this newfound ‘direct line’ in order to get quicker access to customer support and repair services. As the bot was in no way prepared for this focus, satisfaction rates dropped initially, until more data could be collected and entered to meet customer expectations. In this process, not only did Niteco’s client learn more about the actual demands of its customers, they also learned about some shortcomings of their established customer service infrastructure.

Measuring success

Determining customer satisfaction rates by posing a yes-or-no question to the user is one way to measure the success of your chatbot endeavor. But be warned, this may not be the most suitable approach. As anyone who has ever worked with feedback from online users (or people in general) can attest, negative sentiments are much more likely to be expressed than positive ones. That means users of your chatbot who are dissatisfied with their experience are much more likely to tell you so than those users who are satisfied, and the reality of this will naturally skew your results.

Chatbot metrics

While it’s definitely important for you to know when your users are dissatisfied, there are other metrics which may be more suitable to measure your bot’s success, depending on the use case, of course. Here are some of them:

• User numbers: As for any website feature or page, the actual number of users is important for you to gauge the popularity of your bot. It’s especially useful for you to know how many users are visiting any given page, then opting to use the bot. This can tell you which page profits the most from having the bot prominently featured. Sub-categories of this metric would be active and new users.

• Conversation numbers and duration: How many conversations does your chatbot lead every day and how long do they last? Remember that a very short conversation is only good if your customer’s query was answered in a satisfactory way immediately. Otherwise, longer engagement is usually preferable, as this also shows the bot’s ability to hold a conversation.

• User retention: Do your customers remain engaged with the bot throughout the process it is intended to assist them with? Or do they bounce before the bot was able to do its thing? If the latter is the case, it may indicate that you should get rid of any obstacles that are put in the customer’s way.

• Sessions per user: Do your customers opt to use the bot again after they’ve already used it once? In many cases, that would indicate that they were satisfied or at least not entirely dissatisfied with their first encounter. Either way, it can tell you a lot about your users’ needs as well as the topics they are comfortable to leave to a bot.

• Goal completion rate: How many conversations led the user to complete the goal they had set out to achieve? This is an incredibly valuable metric, as it gives you a qualitative overview of the usability of your bot beyond mere numbers. However, it may require you to imply the user’s intention for using the bot, which may sometimes miss the mark.

• Click-through rate: The importance of this metric depends on your use case and may not have the same importance for every chatbot. However, if you want the bot to guide your users toward certain actions, it can be of vital importance for evaluating its performance.

• Fallback rate: As we said before, your users will inevitably ask your bot questions that you didn’t plan for, leaving the bot no choice but to display some variation of the “I’m sorry, I don’t understand” message. It’s important to measure how often this happens as you extend your data sets and learn more about your customers’ needs. Over time, this rate should go down significantly.

These metrics can give you a good idea of how your bot is performing and whether it is actually meeting the demands of your customers. Based on this information, you can adjust and extend your data sets to better meet your customers’ needs and include data that you hadn’t originally planned for.

All in all, developing and rolling out a chatbot doesn’t end with its launch. In fact, the launch is when the real work starts. And that work then continues. Like a child that heads off on their first day of school, your bot has much to learn and will be surprised by much of the information with which it is presented. Like a parent, you can do your best to prepare it for that first day, but there’s only so much you can do. The real learning happens at school or, in your bot’s case, in actual use.