6 research outputs found
Definition, conceptualisation and measurement of trust
This report documents the program and the outcomes of Dagstuhl Seminar 21381 "Conversational Agent as Trustworthy Autonomous System (Trust-CA)". First, we present the abstracts of the talks delivered by the Seminar’s attendees. Then we report on the origin and process of our six breakout (working) groups. For each group, we describe its contributors, goals and key questions, key insights, and future research. The themes of the groups were derived from a pre-Seminar survey, which also led to a list of suggested readings for the topic of trust in conversational agents. The list is included in this report for references
Training a Chatbot with Microsoft LUIS: Effect of Intent Imbalance on Prediction Accuracy
The 25th International Conference on Intelligent User Interfaces Companion (IUI'20), Cagliari, Italy, 17-20 March 2020Microsoft LUIS is a natural language understanding service used to train Chatbots. Imbalance in the utterance training set may cause the LUIS model to predict the wrong intent for a user's query. We discuss this problem and the training recommendations from Microsoft to improve prediction accuracy with LUIS. We perform batch testing on three training sets created from two existing datasets to explore the effectiveness of these recommendations.Science Foundation IrelandMicrosoft Corporatio
BoTest: a Framework to Test the Quality of Conversational Agents Using Divergent Input Examples
ACM IUI (Intelligent User Interfaces), Tokyo, Japan, 07-11 March 2018Quality of conversational agents is important as users have high expectations. Consequently, poor interactions may lead to the user abandoning the system. In this paper, we propose a framework to test the quality of conversational agents. Our solution transforms working input that the conversational agent accurately recognises to generate divergent input examples that introduce complexity and stress the agent. As the divergent inputs are based on known utterances for which we have the 'normal' outputs, we can assess how robust the conversational agent is to variations in the input. To demonstrate our framework we built ChitChatBot, a simple conversational agent capable of making casual conversation.Science Foundation IrelandLer
Assessing the robustness of conversational agens using paraphrases
Assessing a conversational agent’s understanding
capabilities is critical, as poor user interactions could seal
the agent’s fate at the very beginning of its lifecycle with
users abandoning the system. In this paper we explore the
use of paraphrases as a testing tool for conversational agents.
Paraphrases, which are different ways of expressing the same
intent, are generated based on known working input by performing
lexical substitutions. As the expected outcome for this
newly generated data is known, we can use it to assess the
agent’s robustness to language variation and detect potential
understanding weaknesses. As demonstrated by a case study,
we obtain encouraging results as it appears that this approach
can help anticipate potential understanding shortcomings and
that these shortcomings can be addressed by the generated
paraphrases
Definition, conceptualisation and measurement of trust
This report documents the program and the outcomes of Dagstuhl Seminar 21381 "Conversational Agent as Trustworthy Autonomous System (Trust-CA)". First, we present the abstracts of the talks delivered by the Seminar’s attendees. Then we report on the origin and process of our six breakout (working) groups. For each group, we describe its contributors, goals and key questions, key insights, and future research. The themes of the groups were derived from a pre-Seminar survey, which also led to a list of suggested readings for the topic of trust in conversational agents. The list is included in this report for references