Usability guidelines and evaluation criteria for conversational user interfaces: a heuristic and linguistic approach

Even though conversational user interfaces (CUI) have been studied since the 1950s, it is not yet fully understood what makes them feel natural, intuitive and usable. As a result, their design and evaluation poses major challenges. In this paper, we discuss how CUIs are different from other forms of human computer interaction, and what challenges and opportunities arise from these differences. We provide an overview of relevant linguistic principles for a natural language conversation and look at established high-level usability heuristics to derive a set of 53 technology-agnostic checkpoints specifically for text-based CUIs (a.k.a chatbots). These checkpoints have been evaluated with 15 professionals and academics from the fields of User Experience, Natural Language Processing, Conversation Analysis and linguistics to examine content validity. The resulting list of checkpoints provides both guidelines for the design and criteria for the evaluation of chatbots