Generating Question-Answer Hierarchies

The process of knowledge acquisition can be viewed as a question-answer game between a student and a teacher in which the student typically starts by asking broad, open-ended questions before drilling down into specifics (Hintikka, 1981; Hakkarainen and Sintonen, 2002). This pedagogical perspective motivates a new way of representing documents. In this paper, we present SQUASH (Specificity-controlled Question-Answer Hierarchies), a novel and challenging text generation task that converts an input document into a hierarchy of question-answer pairs. Users can click on high-level questions (e.g., "Why did Frodo leave the Fellowship?") to reveal related but more specific questions (e.g., "Who did Frodo leave with?"). Using a question taxonomy loosely based on Lehnert (1978), we classify questions in existing reading comprehension datasets as either "general" or "specific". We then use these labels as input to a pipelined system centered around a conditional neural language model. We extensively evaluate the quality of the generated QA hierarchies through crowdsourced experiments and report strong empirical results.Comment: ACL camera ready + technical note on pipeline modifications for demo (15 pages

combining multimodal external resources for event-based news video retrieval and question answering

Ph.DDOCTOR OF PHILOSOPH

ANSWER SIMILARITY GROUPING AND DIVERSIFICATION IN QUESTION ANSWERING SYSTEMS

The rise in popularity of mobile and voice search has led to a shift in IR from document to passage retrieval for non-factoid questions. Various datasets such as MSMarco, as well as efficient retrieval models have been developed to identify single best answer passages for this task. However, such models do not specifically address questions which could have multiple or alternative answers. In this dissertation, we focus on this new research area that involves studying answer passage relationships and how this could be applied to passage retrieval tasks. We first create a high quality dataset for the answer passage similarity task in the context of question answering. Manual annotation of passage pairs is performed to set the similarity labels, from which answer group information is automatically generated. We next investigate different types of representations, which could be used to create effective clusters. We experiment with various unsupervised representations and show that distributional representations outperform term based representations for this task. Next, weak supervision is leveraged to further improve the cluster modeling performance. We use BERT as the underlying model for training and show the relative performance of various weak signals such as GloVe and term-based Language Modeling for this task. In order to apply these clusters to the answer passage retrieval task for multi-answer questions, we use a modified version of the Maximal Marginal Relevance (MMR) diversification model. We demonstrate that answers retrieved using this model are more diverse i.e, cover more answer types with low redundancy as well as maximize relevance, with respect to the baselines. So far, we used passage clustering as a means to identify answer groups corresponding to a question and apply them in a question answering task. We extend this a step further by looking at related questions within a conversation. For this purpose, we expand the definition of Reciprocal Rank Fusion (RRF) and use this to identify pertinent history passages for such questions. Updated question rewrites generated using these passages are then used to improve the conversational search task. In addition to being the first work that looks at answer relationships, our specific contributions can be summarized as follows: (1) Creation of new datasets with passage similarity and answer type information; (2) Effective passage similarity clustering models using unsupervised representations and weak supervision methods; (3) Applying the passage similarity/clustering information to diversification framework; (4) Identifying good response history candidates using answer passage clustering for the conversational search task

Social Search: retrieving information in Online Social Platforms -- A Survey

Social Search research deals with studying methodologies exploiting social information to better satisfy user information needs in Online Social Media while simplifying the search effort and consequently reducing the time spent and the computational resources utilized. Starting from previous studies, in this work, we analyze the current state of the art of the Social Search area, proposing a new taxonomy and highlighting current limitations and open research directions. We divide the Social Search area into three subcategories, where the social aspect plays a pivotal role: Social Question&Answering, Social Content Search, and Social Collaborative Search. For each subcategory, we present the key concepts and selected representative approaches in the literature in greater detail. We found that, up to now, a large body of studies model users' preferences and their relations by simply combining social features made available by social platforms. It paves the way for significant research to exploit more structured information about users' social profiles and behaviors (as they can be inferred from data available on social platforms) to optimize their information needs further

Contrastive Video Question Answering via Video Graph Transformer

We propose to perform video question answering (VideoQA) in a Contrastive manner via a Video Graph Transformer model (CoVGT). CoVGT's uniqueness and superiority are three-fold: 1) It proposes a dynamic graph transformer module which encodes video by explicitly capturing the visual objects, their relations and dynamics, for complex spatio-temporal reasoning. 2) It designs separate video and text transformers for contrastive learning between the video and text to perform QA, instead of multi-modal transformer for answer classification. Fine-grained video-text communication is done by additional cross-modal interaction modules. 3) It is optimized by the joint fully- and self-supervised contrastive objectives between the correct and incorrect answers, as well as the relevant and irrelevant questions respectively. With superior video encoding and QA solution, we show that CoVGT can achieve much better performances than previous arts on video reasoning tasks. Its performances even surpass those models that are pretrained with millions of external data. We further show that CoVGT can also benefit from cross-modal pretraining, yet with orders of magnitude smaller data. The results demonstrate the effectiveness and superiority of CoVGT, and additionally reveal its potential for more data-efficient pretraining. We hope our success can advance VideoQA beyond coarse recognition/description towards fine-grained relation reasoning of video contents. Our code is available at https://github.com/doc-doc/CoVGT.Comment: Accepted by IEEE T-PAMI'2

Neural models for information retrieval: towards asymmetry sensitive approaches based on attention models

Ce travail se situe dans le contexte de la recherche d'information (RI) utilisant des techniques d'intelligence artificielle (IA) telles que l'apprentissage profond (DL). Il s'intéresse à des tâches nécessitant l'appariement de textes, telles que la recherche ad-hoc, le domaine du questions-réponses et l'identification des paraphrases. L'objectif de cette thèse est de proposer de nouveaux modèles, utilisant les méthodes de DL, pour construire des modèles d'appariement basés sur la sémantique de textes, et permettant de pallier les problèmes de l'inadéquation du vocabulaire relatifs aux représentations par sac de mots, ou bag of words (BoW), utilisées dans les modèles classiques de RI. En effet, les méthodes classiques de comparaison de textes sont basées sur la représentation BoW qui considère un texte donné comme un ensemble de mots indépendants. Le processus d'appariement de deux séquences de texte repose sur l'appariement exact entre les mots. La principale limite de cette approche est l'inadéquation du vocabulaire. Ce problème apparaît lorsque les séquences de texte à apparier n'utilisent pas le même vocabulaire, même si leurs sujets sont liés. Par exemple, la requête peut contenir plusieurs mots qui ne sont pas nécessairement utilisés dans les documents de la collection, notamment dans les documents pertinents. Les représentations BoW ignorent plusieurs aspects, tels que la structure du texte et le contexte des mots. Ces caractéristiques sont très importantes et permettent de différencier deux textes utilisant les mêmes mots et dont les informations exprimées sont différentes. Un autre problème dans l'appariement de texte est lié à la longueur des documents. Les parties pertinentes peuvent être réparties de manières différentes dans les documents d'une collection. Ceci est d'autant vrai dans les documents volumineux qui ont tendance à couvrir un grand nombre de sujets et à inclure un vocabulaire variable. Un document long pourrait ainsi comporter plusieurs passages pertinents qu'un modèle d'appariement doit capturer. Contrairement aux documents longs, les documents courts sont susceptibles de concerner un sujet spécifique et ont tendance à contenir un vocabulaire plus restreint. L'évaluation de leur pertinence est en principe plus simple que celle des documents plus longs. Dans cette thèse, nous avons proposé différentes contributions répondant chacune à l'un des problèmes susmentionnés. Tout d'abord, afin de résoudre le problème d'inadéquation du vocabulaire, nous avons utilisé des représentations distribuées des mots (plongement lexical) pour permettre un appariement basé sur la sémantique entre les différents mots. Ces représentations ont été utilisées dans des applications de RI où la similarité document-requête est calculée en comparant tous les vecteurs de termes de la requête avec tous les vecteurs de termes du document, indifféremment. Contrairement aux modèles proposés dans l'état-de-l'art, nous avons étudié l'impact des termes de la requête concernant leur présence/absence dans un document. Nous avons adopté différentes stratégies d'appariement document/requête. L'intuition est que l'absence des termes de la requête dans les documents pertinents est en soi un aspect utile à prendre en compte dans le processus de comparaison. En effet, ces termes n'apparaissent pas dans les documents de la collection pour deux raisons possibles : soit leurs synonymes ont été utilisés ; soit ils ne font pas partie du contexte des documents en questions. Les méthodes que nous avons proposé permettent, d'une part d'effectuer un appariement inexact entre le document et la requête, et d'une autre part évaluer l'impact des différents termes d'une requête dans le processus d'appariement. Bien que l'utilisation du plongement lexical des mots permet d'effectuer un appariement basé sur la sémantique entre différentes séquences de texte, ces représentations combinées avec les modèles classiques considèrent toujours le texte comme une liste d'éléments indépendants (sac de vecteurs au lieux de sac de mots). Or, la structure du texte aussi bien que l'ordre des mots est très importante. Tout changement dans la structure du texte et/ou l'ordre des mots altère l'information exprimée. Afin de résoudre ce problème, les modèles neuronaux ont été utilisés dans l'appariement de texte. Dans notre cas, nous avons d'abord étudié différents modèles neuronaux de l'état-de-l'art pour la comparaison de textes, ensuite nous avons proposé deux approches principales. Dans un premier temps, nous avons construit un modèle qui tient compte de la structure d'un texte et de l'importance de ses mots. Plus précisément, nous avons combiné un modèle basé sur la position avec un modèle basé sur l'attention pour construire une approche d'appariement de texte exploitant des représentations basées sur la position en combinaison avec une pondération basée sur l'attention des mots. Nous croyons que lorsque le modèle est conscient de la position et de l'importance des mots, les représentations apprises fourniront des caractéristiques plus pertinentes pour le processus de comparaison. Nous avons conclu que la position combinée, dans une configuration asymétrique, à l'attention portée à un mot d'une séquence, permet d'améliorer de façon significative les résultats. Dans un deuxième temps, nous avons analysé différentes applications d'appariement neuronal de texte et les avons regroupé en deux grandes catégories. (1) les problèmes d'appariement symétrique qui consiste à identifier si deux textes, de même nature, sont sémantiquement similaires ; (2) les problèmes d'appariement asymétrique qui consiste à évaluer si un texte d'entrée fournit les informations recherchées dans un autre texte de nature différente. En étudiant les différents modèles neuronaux existants, nous avons constaté que tous les modèles proposés se basent sur une architecture Siamoise globale où les différentes entrées du modèle subissent le même traitement quelque soit la nature de la tâche, (1) ou (2). Afin de prendre en considération la nature de la tâche, nous avons proposé une architecture sensible à l'asymétrie pour l'appariement neuronal de textes. Particulièrement, nous avons utilisé un modèle d'attention pour construire une architecture générale qui étend différents modèles neuronaux de l'état de l'art. Enfin, pour faire face aux problèmes liés à la taille des documents dans la recherche ad-hoc en utilisant les réseaux de neurones, nous avons proposé une approche pour extraire des signaux de pertinence à différents niveaux dans un document long. Notamment, au niveau des mots, des passages et du document complet. Plus précisément, nous avons proposé une architecture globale multi-couche permettant de mesurer la pertinence à différent niveaux, en utilisant les modèles d'attention. Cette architecture est ensuite utilisée pour étendre plusieurs modèles de l'état de l'art et d'examiner l'apport de la pertinence mesurée à différents niveaux. Par ailleurs, nous avons proposé un modèle basé sur l'architecture générale proposée. Il utilise un réseau récurrent afin d'effectuer une sorte d'interaction compétitive entre les passages susceptible d'être pertinent dans un document, et qui sont préalablement sélectionnés.This work is situated in the context of information retrieval (IR) using machine learning (ML) and deep learning (DL) techniques. It concerns different tasks requiring text matching, such as ad-hoc research, question answering and paraphrase identification. The objective of this thesis is to propose new approaches, using DL methods, to construct semantic-based models for text matching, and to overcome the problems of vocabulary mismatch related to the classical bag of word (BoW) representations used in traditional IR models. Indeed, traditional text matching methods are based on the BoW representation, which considers a given text as a set of independent words. The process of matching two sequences of text is based on the exact matching between words. The main limitation of this approach is related to the vocabulary mismatch. This problem occurs when the text sequences to be matched do not use the same vocabulary, even if their subjects are related. For example, the query may contain several words that are not necessarily used in the documents of the collection, including relevant documents. BoW representations ignore several aspects about a text sequence, such as the structure the context of words. These characteristics are important and make it possible to differentiate between two texts that use the same words but expressing different information. Another problem in text matching is related to the length of documents. The relevant parts can be distributed in different ways in the documents of a collection. This is especially true in large documents that tend to cover a large number of topics and include variable vocabulary. A long document could thus contain several relevant passages that a matching model must capture. Unlike long documents, short documents are likely to be relevant to a specific subject and tend to contain a more restricted vocabulary. Assessing their relevance is in principle simpler than assessing the one of longer documents. In this thesis, we have proposed different contributions, each addressing one of the above-mentioned issues. First, in order to solve the problem of vocabulary mismatch, we used distributed representations of words (word embedding) to allow a semantic matching between the different words. These representations have been used in IR applications where document/query similarity is computed by comparing all the term vectors of the query with all the term vectors of the document, regardless. Unlike the models proposed in the state-of-the-art, we studied the impact of query terms regarding their presence/absence in a document. We have adopted different document/query matching strategies. The intuition is that the absence of the query terms in the relevant documents is in itself a useful aspect to be taken into account in the matching process. Indeed, these terms do not appear in documents of the collection for two possible reasons: either their synonyms have been used or they are not part of the context of the considered documents. The methods we have proposed make it possible, on the one hand, to perform an inaccurate matching between the document and the query, and on the other hand, to evaluate the impact of the different terms of a query in the matching process. Although the use of word embedding allows semantic-based matching between different text sequences, these representations combined with classical matching models still consider the text as a list of independent elements (bag of vectors instead of bag of words). However, the structure of the text as well as the order of the words is important. Any change in the structure of the text and/or the order of words alters the information expressed. In order to solve this problem, neural models were used in text matching. In our case, we first studied different neural models from the state-of-the-art of text matching, then we proposed two main approaches. First, we built a model that takes into account the structure of a text and the importance of its words. Specifically, we combined a position-based model with an attention-based model to build a text matching approach using position-based representations combined with attention-based weights of words. We believe that when the model is aware of the position and importance of words, the representations learned will provide more relevant characteristics for the comparison process. We concluded that the combined position, in an asymmetric configuration, with the attention given to a word in a sequence, significantly improves the results. In a second step, we analyzed different neural text matching applications and grouped them into two main categories. (1) symmetric matching problems which consists in identifying if two texts, of the same nature, are semantically similar; (2) asymmetric matching problems which consists in evaluating if an input text provides the information sought in another text of a different nature. By studying the various existing neural models, we have found that all the models proposed are based on a global Siamese architecture where the different inputs of the model undergo the same processing, whatever the nature of the task (1) or (2). In order to take into consideration the nature of the matching task, we proposed an asymmetry sensitive architecture for neural text matching. In particular, we used an attention model to build a general architecture that extends different neural models of the state-of-the-art. Finally, to address problems related to document size in ad-hoc search using neural networks, we proposed an approach to extract relevance signals at different levels in a long document. In particular, at the level of words, passages and the complete document. More precisely, we proposed a global multi-layer architecture to measure relevance at different levels, using attention models. This architecture is then used to extend several state of the art models and to examine the contribution of relevance measured at different levels. Based on this general architecture, we proposed a model that uses a recurrent layer to perform a kind of competitive interactions between the passages that are likely to be relevant in a document, and which are previously selected