Conversation Mining for Customer Service Chatbots

More and more companies are using chatbots in customer service. The large number of chatbots and their interactions with customers produce a huge amount of data, which is useful to track the usage and performance of the chatbot. However, many established performance metrics (e.g., intent scores, conversa-tions per day) could be considered too intuitive to be helpful and are either at a very high level or at the level of single question-answer pairs. Our research aims to address this challenge by presenting a novel approach and system for conver-sation analysis of customer service chatbots. More specifically, we extend estab-lished metrics and concepts with ideas from process mining since every conver-sation with customer service chatbots can be interpreted as a sequence of discrete steps. This paper presents the methodological foundations for our approach, which we call conversation mining, and demonstrates its potential with first in-sights into our prototype. Ultimately, we aim to draw the attention of chatbot researchers and practitioners to the value of conversation data by describing a novel approach for automatically processing and analyzing at a process level

Investigating the user experience of customer service chatbot interaction: a framework for qualitative analysis of chatbot dialogues

The uptake of chatbots for customer service depends on the user experience. For such chatbots, user experience in particular concerns whether the user is provided relevant answers to their queries and the chatbot interaction brings them closer to resolving their problem. Dialogue data from interactions between users and chatbots represents a potentially valuable source of insight into user experience. However, there is a need for knowledge of how to make use of these data. Motivated by this, we present a framework for qualitative analysis of chatbot dialogues in the customer service domain. The framework has been developed across several studies involving two chatbots for customer service, in collaboration with the chatbot hosts. We present the framework and illustrate its application with insights from three case examples. Through the case findings, we show how the framework may provide insight into key drivers of user experience, including response relevance and dialogue helpfulness (Case 1), insight to drive chatbot improvement in practice (Case 2), and insight of theoretical and practical relevance for understanding chatbot user types and interaction patterns (Case 3). On the basis of the findings, we discuss the strengths and limitations of the framework, its theoretical and practical implications, and directions for future work.publishedVersio

Towards Designing a Conversation Mining System for Customer Service Chatbots

Chatbots are increasingly used to provide customer service. However, despite technological advances, customer service chatbots frequently reach their limits in customer interactions. This is not immediately apparent to both chatbot operators (e.g., customer service managers) and chatbot developers because analyzing conversational data is difficult and labor-intensive. To address this problem, our ongoing design science research project aims to develop a conversation mining system for the automated analysis of customer-chatbot conversations. Based on the exploration of large dataset (N= 91,678 conversations) and six interviews with industry experts, we developed the backend of the system. Specifically, we identified and operationalized important criteria for evalu-ating conversations. Our next step will be the evaluation with industry experts. Ultimately, we aim to contribute to research and practice by providing design knowledge for conversation mining systems that leverage the treasure trove of data from customer-chatbot conversations to generate valuable insights for managers and developers

Understanding the user experience of customer service chatbots: An experimental study of chatbot interaction design

Understanding the user experience of chatbots for customer service is essential to realize the potential of this technology. Such chatbots are typically designed for efficient and effective interactions, accentuating pragmatic quality, and there is a need to understand how to make these more pleasant and engaging, strengthening hedonic quality. One promising approach is to design for more humanlike chatbot interactions, that is, interactions resembling those of skilled customer service personnel. In a randomized experiment (n = 35) we investigated two chatbot interaction design features that may strengthen the impression of a humanlike character: (a) topic-led conversations, encouraging customer reflection, in contrast to task-led conversations, aiming for efficient goal completion, and (b) free text interaction, where users interact mainly using their own words, rather than button interaction, where users mainly interact through predefined answer alternatives. dependent variables were participant perceptions of anthropomorphism and social presence, two key concepts related to chatbot human likeness, in addition to pragmatic quality and hedonic quality. To further explore user perceptions of the interaction designs, the study also included semi-structured interviews. Topic-led conversations were found to strengthen anthropomorphism and hedonic quality. A similar effect was not found for free text interaction, reportedly due to lack in chatbot flexibility and adaptivity. Implications for theory and practice are suggested.publishedVersio

How to Convey Resilience: Towards A Taxonomy for Conversational Agent Breakdown Recovery Strategies

Conversational agents (CAs) have permeated our everyday lives in the past decade. Yet, the CAs we encounter today are far from perfect as they are still prone to breakdowns. Studies have shown that breakdowns have an immense impact on the user-CA relationship, user satisfaction, and retention. Therefore, it is important to investigate how to react and recover from breakdowns appropriately so that failures do not impair the CA experience lastingly. Examples for recovery strategies are the assumption of the most likely user intent (CA self-repair) or to ask for clarification (user-repair). In this paper, we iteratively develop a taxonomy to classify breakdown recovery strategies based on studies from scholarly literature and experiements with productive CA instances, and identify the current best practices described using our taxonomy. We aim to synthesize, structure and further the knowledge on breakdown handling and to provide a common language to describe recovery strategies

Leveraging the Potential of Conversational Agents: Quality Criteria for the Continuous Evaluation and Improvement

Contemporary organizations are increasingly adopting conversational agents (CAs) as intelligent and natural language-based solutions for providing services and information. CAs promote new forms of personalization, speed, cost-effectiveness, and automation. However, despite their hype in research and practice, organizations fail to sustain CAs in operations. They struggle to leverage CAs’ potential because they lack knowledge on how to evaluate and improve the quality of CAs throughout their lifecycle. We build on this research gap by conducting a design science research (DSR) project, aggregating insights from the literature and practice to derive a validated set of quality criteria for CAs. Our study contributes to CA research and guides practitioners by providing a blueprint to structure the evaluation of CAs to discover areas for systematic improvement

Investigating the factors of customer experiences using real-life text-based banking chatbot: a qualitative study in Norway

In recent times, banks have increasingly started using chatbots to offer round-the-clock customer service. However, customers experience with this type of technology is not well understood. The aim of this study was to get an in-depth understanding of factors affecting customer experience with a banking chatbot. Eight participants interacted with a real-life banking chatbot to complete a simple task (order a credit/debit card) and a complex task (apply for a housing loan). Semi-structured interviews were then conducted to examine chatbot-related factors (ease of use, miscommunication errors and human-likeness) and user-related factors (perceptions, future behaviors). The findings indicate that the human-like factors like a human personality, use of emojis, willingness to help, and polite communication style, have a positive impact of customer experience with banking chatbots. The chatbot's ability to understand questions was a critical factor. Miscommunication errors have negative impact, especially when the task is a simple one. Takeaway from this study is that banks should inform customers about the limits of the chatbot's abilities. In addition, they should communicate that the chatbot is safe to use for complex tasks. Successful development and implementation of chatbots for customer service require a customer centric approach from banks.publishedVersio

Implementing Choices in Chatbot-initiated Service Interactions: Helpful or Harmful?

Chatbots are increasingly equipped to provide choices for customers to click and choose from when communicating with the chatbots. This research investigates when and why implementing choices enhances or impairs customers’ service experience. Based on the concept of fluency, we posit that the implementation of choices is beneficial only after a conversational breakdown occurs because the value of choice provision for facilitating fluency may not be recognizable or realized in the absence of service breakdowns. We further propose that the implementation of choices is counterproductive when the choice set is perceived as incomprehensive because it decreases the perception of fluency. We conducted several experiments to test these hypotheses. By illuminating when and why choice implementation may help or harm customers during a chatbot-initiated service interaction, we augment the current understanding of a chatbot’s role in customers’ service experience and provide insights for the deployment of choice-equipped chatbots in customer service

Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data

Large language models (LLMs) provide a new way to build chatbots by accepting natural language prompts. Yet, it is unclear how to design prompts to power chatbots to carry on naturalistic conversations while pursuing a given goal, such as collecting self-report data from users. We explore what design factors of prompts can help steer chatbots to talk naturally and collect data reliably. To this aim, we formulated four prompt designs with different structures and personas. Through an online study (N = 48) where participants conversed with chatbots driven by different designs of prompts, we assessed how prompt designs and conversation topics affected the conversation flows and users' perceptions of chatbots. Our chatbots covered 79% of the desired information slots during conversations, and the designs of prompts and topics significantly influenced the conversation flows and the data collection performance. We discuss the opportunities and challenges of building chatbots with LLMs.Comment: 22 pages including Appendix, 7 figures, 7 tables. Accepted to PACM HCI (CSCW 2024

The Impact of Chatbots on the Relationship between Integrated Marketing Communication and Online Purchasing Behavior in The Frontier Market

Artificial Intelligence (AI), applied in many fields, is the core of the fourth technological revolution. In business, AI is used for customer relationship management as applied in the autoresponder systems, i.e., chatbot. Chatbots were an essential tool in the marketing relationship as many companies applied this function to their website; hence, this study analyzed the influence of chatbots on the enterprise's integrated marketing communication (IMC) activities, resulting in impulse purchase behavior and repurchase intention behavior. The mixed research method was used, particularly the in-depth interview and the survey with 886 online consumers, who shop from the online websites with chatbots system in Vietnam as Tiki, Lazada, Sendo, excetera. The research results showed that the perceived usefulness and ease of use of chatbots have positively affected the attitude of online consumers to the IMC activities of businesses. Simultaneously, IMC leads to impulse buying as well as the repurchase intention behavior of customers. The study proposed some managerial implications for an online business to enhance the chatbot functions to consumer behaviors in the website.