AI & chatbot analytics tools: Off the shelf or build your own?

The world of chatbots and conversational AI is evolving rapidly, with numerous companies offering an array of solutions or incorporating them into their products and services.

As a business owner looking to drive improvement, however, you may find yourself underwhelmed with the analytics available within these platforms.

Bot platforms typically provide limited access to the data, surface-level reporting with metrics for volume, containment, and intent data. Chatbot teams are forced to supplement their efforts with fragmented tools and manual reporting using tools like SQL queries, manual search, and more.

These native reporting options often leave you feeling unfulfilled, as you struggle to answer basic questions:

• What are my customers talking to the bot about?
• How effective is the bot at answering their queries?
• Do my customers like the bot experience?
• Is the bot helping the business save (or make) money; how can I prove it?
• What improvements do I prioritize?
• What metrics should I be sharing with my key stakeholders?

And suddenly, you find yourself thinking about building chatbot analytics software.

But, hold on: before you pull the trigger on the time and resources needed to build in-house bot analytics tools, consider these six important points, including must-have aspects of any chatbot analytics software.

To determine which is the right solution for your needs, it’s important to begin by defining a list of key features that should be built into your chatbot analytics software, no matter where it comes from.

While doing so it’s important to think in the short-term, identifying features to get off the ground as a minimum viable product (MVP). However, it’s more important for the team to have a clear vision of how the product is going to look after a few iterations. A team without an extremely strong background in chatbots and conversational AI will always struggle to understand the features that might be needed in the future. Building a comprehensive vision for a product is the key to the success of the product. A shortlist of essential chatbot data analytics features must include:

Conversation topics: Effective bot analytics must be able to extract “primary contact reasons” from customer interactions with both your chatbot and live agents. By utilizing AI techniques like topic modeling, you should be able to identify the real reasons your customers are engaging with you. However, achieving this level of insight requires a solution backed by AI expertise.

Automation:  Accurately measuring how well the bot is automating conversations goes beyond looking at simple bot containment metrics. Bot platforms often miss this broader measurement, taking into account only whether a conversation was contained. However, a basic containment metric overlooks whether or not a conversation met the customer’s needs and satisfaction. A more advanced analytics platform will take into account additional signals and provide an accurate view of bot automation performance, as in Calabrio Bot Analytics’ Bot Automation Score (BAS).

Customer Satisfaction: To measure customer satisfaction, many platforms rely on customer surveys. However, often the participation rate is less than 5% and significantly skewed towards negative feedback. Clearly, such an approach is far from ideal. Instead, creating an effective customer satisfaction measurement encompassing 100% of conversations demands a combination of innovative signals and advanced AI models to generate the necessary inputs. Achieving this level of intelligence requires strong AI and NLU expertise that would need to be hired or developed internally. Of course, again, an advanced solution like Calabrio Bot Analytics is capable of analyzing CSAT without the use of surveys, with its Bot Experience Score (BES).

New Intent Discovery: Conversational bots are built using natural language understanding and intents as the core mechanism to match customer needs with bot responses. However, there will always be certain things that the bot doesn’t understand, and it’s crucial to keep track of these.

Using an external topic modeler, you can identify topics in conversation data that are missing from the existing intents. Even better, it can be configured to run on a 24-hour basis, uncovering new intents, or topics, as they emerge. This feature is highly valuable as it will enable you to expand their intent repository and capture more of the conversations their customers want to have, thus improving the performance of their bot.

False positive identification: NLU-based bots may occasionally make mistakes, incorrectly understanding customer needs and following the wrong path. Bot platforms typically don’t provide false positive identification. To address this issue, you need a custom-built AI technique separate from NLU/classifier design.

Financial impact: Understanding the financial impact of your chatbot program is essential to justify past and future investments. Early-stage financial metrics may not be clear, and bot platforms might not readily disclose this information. Being able to connect the dots between contact center savings, cost per bot conversation, and the impact of bot improvements on both of these measures will inform sound business decisions around the cost of Conversational AI.

Building a robust bot analytics engine requires connectivity to multiple data sources: think chatbot data from multiple platforms or services, voice data, live agent data, etc.

Understanding and deciding how all that data will be stored, accessed, and reported is not an easy task.

Each chatbot platform utilizes different log structures and reports on varying events, requiring you to access and normalize the data to ensure consistency. As bot platforms typically don’t look beyond the bot, you’ll need to build a system that follows conversations from start to finish. Since bot platforms primarily concentrate on the chatbot’s performance, they often neglect the whole customer journey, thereby missing crucial insights. To gain a comprehensive understanding of customer interactions and satisfaction, it is necessary to build a system that tracks the entire conversation, including any escalations to live agents or subsequent follow-ups.

When deciding on an approach to data integration, keep the following questions in mind:

• Where is your conversation data currently stored, and where will it be in the future?
• Do you plan to expand to multiple chatbots, including voicebots?
• Will you store conversation data in a data cloud like Snowflake, BigQuery, or Amazon Redshift, or leave it in log files within the bot and customer care platforms?

Maintaining connectors: Once you’ve established integrations with conversation data sources, it’s crucial to consider the ongoing maintenance of connectors, as source platforms will continuously change.

Choosing between internal teams and vendors: When deciding whether to rely on an internal team or a vendor with pre-built connectors, consider the following:

• Scalability: Vendors with existing partnerships and technology expertise can better integrate and maintain conversation streams with analytics and reporting suites.
• Adaptability: Off-the-shelf chatbot analytics tools are designed to handle various interaction scenarios, enabling smoother integrations as your chatbot needs evolve.

Early-stage decisions will determine future scalability, including connectivity, storage, deployment, and scaling. Poor initial choices can lead to system bottlenecks, data fragmentation, and other challenges that ultimately hamper your ability to scale effectively over the long term.

Data integrations are far from the only infrastructure-related concern. Running analytics on thousands of multilingual conversations can be a resource-intensive task, demanding powerful AI models and extensive cloud services.

To achieve effective chatbot analysis, it is vital to strike the right balance between cost and insight. Managing analytics for multilingual chatbot interactions requires resources including sophisticated AI models and costly cloud services. Balancing model usage is a critical aspect of avoiding exorbitant expenses while simultaneously capturing essential insights from conversation data.

Equally important, you need to evaluate factors like the depth of analysis, the frequency at which the model is run, and the need for real-time analysis. While basic analytics may seem adequate in the early days, over time, the need will expand and the questions about bot performance will become more sophisticated, resulting in more intensive analyses. Will an internal team be as well-equipped as an experienced vendor to deliver the desired performance?

It is well known that IT projects often exceed their estimated timelines and costs. In fact, an older study by McKinsey and Oxford that still rings true today shares that half of all large IT budgets significantly overrun their initial budgets.

In the world of chatbots, it is essential for businesses to grasp the consequences of postponing analytics.

Failing to get a handle on bot performance, and quickly, will adversely affect businesses that depend on these bots for a multitude of customer interactions.

If your chatbot has been in-market for several months already, you probably needed performance insights yesterday. Building your analytics solutions from scratch will certainly lead to delay, leaving revenue, savings, and customer loyalty on the table.

The effectiveness of analytics and reporting solutions is reliant on a team’s ability to learn the tools and extract desired results. Achieving this is easier said than done. These solutions can be complex, requiring comprehensive training programs to maximize insights and chatbot improvement.

Beyond initial training, effective and comprehensive documentation, as well as ongoing training and product support are crucial for continued success, especially in the face of team turnover. In the long run, external vendors dedicated to analytics platforms may be able to offer more extensive and up-to-date documentation and support for your solution.

In determining whether to build or buy analytics and reporting software, it’s crucial to consider development, maintenance, and upgrade costs. Creating a simple spreadsheet to compare costs can aid in understanding which option is more cost-effective. Generally, external vendors offering or building solutions for multiple customers offer more affordable options due to cost amortization, as opposed to internal teams that work at cost for a single customer’s project.

As businesses consider implementing chatbots into their products or services, determining the return on investment (ROI) becomes essential to making the right decision on securing effective analytics. 

When it comes to managing a chatbot program, gaining access to effective, insightful analytics is undeniably crucial to short- and long-term success. The idea of building your own bot analytics system might seem appealing, but it’s important to consider the significant investment of time and resources it entails.

For those with deep domain expertise in Natural Language Processing (NLP), experience working on multi-lingual large language models like GPT and BERT, and AI model training and productionizing in general, building a chatbot analytics platform in-house might be the way to go—so long as it comes with a long runway and hefty resources. However, for the majority of chatbot teams and contact centers, options like Calabrio Bot Analytics offer more immediate returns, sustainable value, and reliable innovation.

Ready to boost your bot performance? Book a demo of Calabrio Bot Analytics today.