Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study

Neural generative models have been become increasingly popular when building conversational agents. They offer flexibility, can be easily adapted to new domains, and require minimal domain engineering. A common criticism of these systems is that they seldom understand or use the available dialog history effectively. In this paper, we take an empirical approach to understanding how these models use the available dialog history by studying the sensitivity of the models to artificially introduced unnatural changes or perturbations to their context at test time. We experiment with 10 different types of perturbations on 4 multi-turn dialog datasets and find that commonly used neural dialog architectures like recurrent and transformer-based seq2seq models are rarely sensitive to most perturbations such as missing or reordering utterances, shuffling words, etc. Also, by open-sourcing our code, we believe that it will serve as a useful diagnostic tool for evaluating dialog systems in the future.Comment: To appear at ACL 2019(oral; nominated for best paper

Dialogue history integration into end-to-end signal-to-concept spoken language understanding systems

This work investigates the embeddings for representing dialog history in spoken language understanding (SLU) systems. We focus on the scenario when the semantic information is extracted directly from the speech signal by means of a single end-to-end neural network model. We proposed to integrate dialogue history into an end-to-end signal-to-concept SLU system. The dialog history is represented in the form of dialog history embedding vectors (so-called h-vectors) and is provided as an additional information to end-to-end SLU models in order to improve the system performance. Three following types of h-vectors are proposed and experimentally evaluated in this paper: (1) supervised-all embeddings predicting bag-of-concepts expected in the answer of the user from the last dialog system response; (2) supervised-freq embeddings focusing on predicting only a selected set of semantic concept (corresponding to the most frequent errors in our experiments); and (3) unsupervised embeddings. Experiments on the MEDIA corpus for the semantic slot filling task demonstrate that the proposed h-vectors improve the model performance.Comment: Accepted for ICASSP 2020 (Submitted: October 21, 2019

Contextual Understanding in Neural Dialog Systems: the Integration of External Knowledge Graphs for Generating Coherent and Knowledge-rich Conversations

The integration of external knowledge graphs has emerged as a powerful approach to enrich conversational AI systems with coherent and knowledge-rich conversations. This paper provides an overview of the integration process and highlights its benefits. Knowledge graphs serve as structured representations of information, capturing the relationships between entities through nodes and edges. They offer an organized and efficient means of representing factual knowledge. External knowledge graphs, such as DBpedia, Wikidata, Freebase, and Google's Knowledge Graph, are pre-existing repositories that encompass a wide range of information across various domains. These knowledge graphs are compiled by aggregating data from diverse sources, including online encyclopedias, databases, and structured repositories. To integrate an external knowledge graph into a conversational AI system, a connection needs to be established between the system and the knowledge graph. This can be achieved through APIs or by importing a copy of the knowledge graph into the AI system's internal storage. Once integrated, the conversational AI system can query the knowledge graph to retrieve relevant information when a user poses a question or makes a statement. When analyzing user inputs, the conversational AI system identifies entities or concepts that require additional knowledge. It then formulates queries to retrieve relevant information from the integrated knowledge graph. These queries may involve searching for specific entities, retrieving related entities, or accessing properties and attributes associated with the entities. The obtained information is used to generate coherent and knowledge-rich responses. By integrating external knowledge graphs, conversational AI systems can augment their internal knowledge base and provide more accurate and up-to-date responses. The retrieved information allows the system to extract relevant facts, provide detailed explanations, or offer additional context. This integration empowers AI systems to deliver comprehensive and insightful responses that enhance user experience. As external knowledge graphs are regularly updated with new information and improvements, conversational AI systems should ensure their integrated knowledge graphs remain current. This can be achieved through periodic updates, either by synchronizing the system's internal representation with the external knowledge graph or by querying the external knowledge graph in real-time

Corpus-Level End-to-End Exploration for Interactive Systems

A core interest in building Artificial Intelligence (AI) agents is to let them interact with and assist humans. One example is Dynamic Search (DS), which models the process that a human works with a search engine agent to accomplish a complex and goal-oriented task. Early DS agents using Reinforcement Learning (RL) have only achieved limited success for (1) their lack of direct control over which documents to return and (2) the difficulty to recover from wrong search trajectories. In this paper, we present a novel corpus-level end-to-end exploration (CE3) method to address these issues. In our method, an entire text corpus is compressed into a global low-dimensional representation, which enables the agent to gain access to the full state and action spaces, including the under-explored areas. We also propose a new form of retrieval function, whose linear approximation allows end-to-end manipulation of documents. Experiments on the Text REtrieval Conference (TREC) Dynamic Domain (DD) Track show that CE3 outperforms the state-of-the-art DS systems.Comment: Accepted into AAAI 202

Ranking Enhanced Dialogue Generation

How to effectively utilize the dialogue history is a crucial problem in multi-turn dialogue generation. Previous works usually employ various neural network architectures (e.g., recurrent neural networks, attention mechanisms, and hierarchical structures) to model the history. However, a recent empirical study by Sankar et al. has shown that these architectures lack the ability of understanding and modeling the dynamics of the dialogue history. For example, the widely used architectures are insensitive to perturbations of the dialogue history, such as words shuffling, utterances missing, and utterances reordering. To tackle this problem, we propose a Ranking Enhanced Dialogue generation framework in this paper. Despite the traditional representation encoder and response generation modules, an additional ranking module is introduced to model the ranking relation between the former utterance and consecutive utterances. Specifically, the former utterance and consecutive utterances are treated as query and corresponding documents, and both local and global ranking losses are designed in the learning process. In this way, the dynamics in the dialogue history can be explicitly captured. To evaluate our proposed models, we conduct extensive experiments on three public datasets, i.e., bAbI, PersonaChat, and JDC. Experimental results show that our models produce better responses in terms of both quantitative measures and human judgments, as compared with the state-of-the-art dialogue generation models. Furthermore, we give some detailed experimental analysis to show where and how the improvements come from.Comment: Accepted at CIKM 202