Conversation Mining for Customer Service Chatbots

More and more companies are using chatbots in customer service. The large number of chatbots and their interactions with customers produce a huge amount of data, which is useful to track the usage and performance of the chatbot. However, many established performance metrics (e.g., intent scores, conversa-tions per day) could be considered too intuitive to be helpful and are either at a very high level or at the level of single question-answer pairs. Our research aims to address this challenge by presenting a novel approach and system for conver-sation analysis of customer service chatbots. More specifically, we extend estab-lished metrics and concepts with ideas from process mining since every conver-sation with customer service chatbots can be interpreted as a sequence of discrete steps. This paper presents the methodological foundations for our approach, which we call conversation mining, and demonstrates its potential with first in-sights into our prototype. Ultimately, we aim to draw the attention of chatbot researchers and practitioners to the value of conversation data by describing a novel approach for automatically processing and analyzing at a process level

Match or Mismatch? How Matching Personality and Gender between Voice Assistants and Users Affects Trust in Voice Commerce

Despite the ubiquity of voice assistants (VAs), they see limited adoption in the form of voice commerce, an online sales channel using natural language. A key barrier to the widespread use of voice commerce is the lack of user trust. To address this problem, we draw on similarity-attraction theory to investigate how trust is affected when VAs match the user’s personality and gender. We conducted a scenario-based experiment (N = 380) with four VAs designed to have different personalities and genders by customizing only the auditory cues in their voices. The results indicate that a personality match increases trust, while the effect of a gender match on trust is non-significant. Our findings contribute to research by demonstrating that some types of matches between VAs and users are more effective than others. Moreover, we reveal that it is important for practitioners to consider auditory cues when designing VAs for voice commerce

Designing Effective Conversational Repair Strategies for Chatbots

Conversational breakdowns often force users to go through frustrating loops of trial and error when trying to get answers from chatbots. Although research has emphasized the potential of conversational repair strategies in helping users resolve breakdowns, design knowledge for implementing such strategies is scarce. To address this challenge, we are conducting a design science research (DSR) project to design effective repair strategies that help users recover from conversational breakdowns with chatbots. This paper presents the first design cycle, proposing, instantiating, and evaluating our first design principle on identifying the cause of conversational breakdowns. Using 21,736 real-world user messages from a large insurance company, we conducted a cluster analysis of 5,668 messages leading to breakdowns, identified four distinct breakdown types, and built a classifier that can be used to automatically identify breakdown causes in real time. Our research contributes with prescriptive knowledge for designing repair strategies in conversational breakdown situations

The Impact of Conversational Assistance on the Effective Use of Forecasting Support Systems: A Framed Field Experiment

Forecasting support systems (FSSs) support demand planners in important forecasting decisions by offering statistical forecasts. However, planners often rely on their judgment more than on system-based advice which can be detrimental to forecast accuracy. This is caused by a lack of understanding and subsequent lack of trust in the FSS and its advice. To address this problem, we explore the potential of extending the traditional static assistance (e.g., manuals, tooltips) with conversational assistance provided by a conversational assistant that answers planners’ questions. Drawing on the theory of effective use, we aim to conduct a framed field experiment to investigate whether conversational (vs. static) assistance better supports planners in learning the FSS, increases their trust, and ultimately helps them make more accurate forecasting decisions. With our findings, we aim to contribute to research on FSS design and the body of knowledge on the theory of effective use

Measuring Service Encounter Satisfaction with Customer Service Chatbots using Sentiment Analysis

Chatbots are software-based systems designed to interact with humans using text-based natural language and have attracted considerable interest in online service encounters. In this context, service providers face the challenge of measuring chatbot service encounter satisfaction (CSES), as most approaches are limited to post-interaction surveys that are rarely answered and often biased. Asa result, service providers cannot react quickly to service failures and dissatisfied customers. To address this challenge, we investigate the application of automated sentiment analysis methods as a proxy to measure CSES. Therefore, we first compare different sentiment analysis methods. Second, we investigate the relationship between objectively computed sentiment scores of dialogs and subjectively measured CSES values. Third, we evaluate whether this relationship also exists for utterance sequences throughout the dialog. The paper contributes by proposing and applying an automatic and objective approach to use sentiment scores as a proxy to measure CSES

Can Humanizing Voice Assistants Unleash the Potential of Voice Commerce?

Voice commerce allows customers to carry out sales dialogues with voice assistants (VAs) through natural spoken language. However, its adoption remains limited. To help determine how to overcome existing barriers to adoption, we conducted a series of three empirical pre-studies and a laboratory experiment (N = 323) investigating the role of VAs’ humanness in interactions with customers; research has reached no consensus on this matter. Our results reveal that humanizing VAs increases customers’ perceptions of social presence and parasocial interaction, thereby enhancing perceived relationship quality and ultimately leading to increased intentions to shop using the VA. Although, we also find a negative direct effect of humanization on parasocial interaction, it is offset by the larger positive indirect effect via social presence. This may provide one explanation for the inconsistencies in the literature. For practitioners, our findings highlight the importance of careful design in humanizing VAs to increase voice commerce adoption

Towards Designing a NLU Model Improvement System for Customer Service Chatbots

Current customer service chatbots often struggle to meet customer expectations. One reason is that despite advances in artificial intelligence (AI), the natural language understanding (NLU) capabilities of chatbots are often far from perfect. In order to improve them, chatbot managers need to make informed decisions and continuously adapt the chatbot’s NLU model to the specific topics and expressions used by customers. Customer-chatbot interaction data is an excellent source of information for these adjustments because customer messages contain specific topics and linguistic expressions representing the domain of the customer service chatbot. However, extracting insights from such data to improve the chatbot’s NLU, its architecture, and ultimately the conversational experience requires appropriate systems and methods, which are currently lacking. Therefore, we conduct a design science research project to develop a novel artifact based on chatbot interaction data that supports NLU improvement

The New Dream Team? A Review of Human-AI Collaboration Research From a Human Teamwork Perspective

A new generation of information systems based on artificial intelligence (AI) transforms the way we work. However, existing research on human-AI collaboration is scattered across disciplines, highlighting the need for more transparency about the design of human-AI collaboration in organizational contexts. This paper addresses this gap by reviewing the literature on human-AI collaboration through the lens of human teamwork. Our results provide insights into how emerging topics of human-AI collaboration are connected and influence each other. In particular, the review indicates that, with the increasing complexity of organizational settings, human-AI collaboration needs to be designed differently, and team maintenance activities become more important due to increased communicational requirements of humans. Our main contribution is a novel framework of temporal phases in human-AI collaboration, identifying the mechanisms that need to be considered when designing them for organizational contexts. Additionally, we use our framework to derive a future research agenda

Towards Designing a Conversation Mining System for Customer Service Chatbots

Chatbots are increasingly used to provide customer service. However, despite technological advances, customer service chatbots frequently reach their limits in customer interactions. This is not immediately apparent to both chatbot operators (e.g., customer service managers) and chatbot developers because analyzing conversational data is difficult and labor-intensive. To address this problem, our ongoing design science research project aims to develop a conversation mining system for the automated analysis of customer-chatbot conversations. Based on the exploration of large dataset (N= 91,678 conversations) and six interviews with industry experts, we developed the backend of the system. Specifically, we identified and operationalized important criteria for evalu-ating conversations. Our next step will be the evaluation with industry experts. Ultimately, we aim to contribute to research and practice by providing design knowledge for conversation mining systems that leverage the treasure trove of data from customer-chatbot conversations to generate valuable insights for managers and developers

Designing Multimodal BI&A Systems for Face-to-Face Team Interactions

Organizations increasingly assign complex decision-making tasks to teams. However, current business intelligence & analytics (BI&A) systems are primarily designed to support individual decision-makers and, therefore, cannot be used effectively in face-to-face team interactions. To address this challenge, we conduct a design science research (DSR) project to design a multimodal BI&A system that can be used effectively using a combination of touch and speech interaction. Drawing on the theory of effective use and existing guidelines for multimodal user interfaces, we formulated and instantiated three design principles in an artifact. The results of a focus group evaluation indicate that enhancing the BI&A system with a speech facilitates transparent interaction and increases effective use of the system in team interactions. Our DSR project contributes to the body of design knowledge for multimodal BI&A systems by demonstrating how the combination of touch and speech facilitates its effective use in face-to-face team interactions