A Comparative Study and Analysis of Conversational Search Algorithms to Improve their Reproducibility

openConversational Search is a field of Information Retrieval that is steadily gaining popularity in recent years. A conversational retrieval system aims to engage with the users in conversations using natural language. In this work, we studied, implemented and compared a total of eleven state-of-the-art algorithms and strategies for conversational search. Additionally, we developed a conversational retrieval framework focused on modularity, extensibility and reproducibility, that we used to test said algorithms. The top-performant method we tested, Context Query, obtained an nDCG@3 of 0.43, beating more complex methods, like the ones based on coreference resolution or Large Language models (i.e., BERT), by at least 10%. Concerning the reproducibility aspect, we’ve been able to reach comparable results on several methods for which we had a suitable comparing value.Conversational Search is a field of Information Retrieval that is steadily gaining popularity in recent years. A conversational retrieval system aims to engage with the users in conversations using natural language. In this work, we studied, implemented and compared a total of eleven state-of-the-art algorithms and strategies for conversational search. Additionally, we developed a conversational retrieval framework focused on modularity, extensibility and reproducibility, that we used to test said algorithms. The top-performant method we tested, Context Query, obtained an nDCG@3 of 0.43, beating more complex methods, like the ones based on coreference resolution or Large Language models (i.e., BERT), by at least 10%. Concerning the reproducibility aspect, we’ve been able to reach comparable results on several methods for which we had a suitable comparing value

Tracking Context in Conversational Search: From Utterances to Neural Embeddings

The use of conversational assistants is becoming increasingly more popular among the general public, pushing the research towards more advanced and sophisticated techniques. Hence, there are currently a number of research opportunities to extend the comprehension and applicability of these tasks in everyday systems. These conversational assistants are capable of performing various tasks, such as chitchatting, internal device functions (e.g., setting up an alarm), and searching for information. In the last few years, the interest in conversational search is increasing, not only because of the generalization of conversational assistants but also because conversational search is a step forward in allowing a more natural interaction with the system. To build a system such as this, many components need to work together, since in a conversation, the importance of context is paramount to retrieve the best answers to the user’s questions. In this thesis, the focus was on developing a conversational search system that aims to help people search for information in a natural way. In particular, this system must be able to understand the context where the question is posed, tracking the current state of the conversation and detecting mentions to previous questions and answers. We achieve this by using a context-tracking component based on neural query-rewriting models. Another crucial aspect of the system is to provide the most relevant answers given the question and the conversational history. To achieve this objective, we used state-of-the-art retrieval and re-ranking methods and expanded their architecture to use the conversational context. The results obtained with the system developed achieved state-of-the-art when compared to the baselines present in TREC Conversational Assistance Track (CAsT) 2019.O uso de assistentes conversacionais está a tornar-se cada vez mais popular entre o público em geral, levando à investigação de técnicas mais avançadas e sofisticadas. Consequentemente, existem atualmente várias oportunidades de investigação para estender a compreensão e aplicabilidade destas tarefas em sistemas do quotidiano. Estes assistentes são capazes de efetuar várias tarefas como, por exemplo: ter uma conversa informal, efetuar funções internas ao dispositivo (e.g. colocar um alarme), e pesquisar por informação. Nos últimos anos, o interesse em pesquisa conversacional tem estado a aumentar, não só pela generalização dos assistentes conversacionais, mas também devido a ser um passo em frente para permitir uma interação mais natural com o sistema. Para construir um sistema deste tipo, vários componentes têm de trabalhar em conjunto, uma vez que numa conversa o contexto é da maior importância para recuperar as melhores respostas para as perguntas do utilizador. Nesta tese, o foco foi desenvolver um sistema de pesquisa conversacional para ajudar as pessoas a pesquisar por informação de uma forma natural. Em particular, este sistema tem de ser capaz de compreender o contexto onde a questão é colocada, fazendo tracking do estado atual da conversa e detetando menções a perguntas e respostas anteriores. Com esse objetivo, desenvolvemos um componente de tracking de contexto baseado em modelos neuronais de reescrita de perguntas. Outro aspeto crucial deste sistema é fornecer as respostas mais relevantes dada uma pergunta e o histórico da conversa. Para alcançar este objetivo, utilizámos modelos do estado-da-arte em recuperação de informação e re-ranking e expandimos estas arquiteturas de modo a utilizarem o contexto da conversa. Os resultados obtidos com o sistema desenvolvido atingiram resultados do estado.da-arte quando comparados às baselines submetidas no TREC Conversational Assistance Track (CAsT) 2019

Geographic information extraction from texts

A large volume of unstructured texts, containing valuable geographic information, is available online. This information – provided implicitly or explicitly – is useful not only for scientific studies (e.g., spatial humanities) but also for many practical applications (e.g., geographic information retrieval). Although large progress has been achieved in geographic information extraction from texts, there are still unsolved challenges and issues, ranging from methods, systems, and data, to applications and privacy. Therefore, this workshop will provide a timely opportunity to discuss the recent advances, new ideas, and concepts but also identify research gaps in geographic information extraction

Multi-task learning for effective Open-Retrieval Conversational Question Answering

Conversational Question Answering (ConvQA) is a rapidly growing area of research that aims to improve the search experience for users by allowing for more natural interactions between users and search systems. ConvQA systems are designed to gauge and answer questions in the context of a conversation, taking into account the previous questions and answers in the dialogue. One of the challenges of ConvQA is resolving ambiguities in the user’s questions based on the conversation history. This requires the system to not only consider the question being asked but to also take into account the conversation context to provide relevant and accurate answers. Open-Retrieval Conversational Question Answering (ORConvQA) is a more challenging variant of ConvQA, as it requires the system to retrieve relevant passages from a large collection of documents before extracting the required answers. This task requires the system to effectively search and retrieve the most relevant information, adding further complexity. In order to build an ORConvQA system, to address the ambiguities in conversational questions, a number of approaches have been proposed, such as follow-up question identification, conversational question rewriting, and asking clarifying questions. These approaches can help the system better gauge the user’s intent and context, thereby allowing it to generate more precise and relevant responses. Another challenge in ORConvQA is retrieving relevant passages from a large collection of documents and identifying the most relevant ones based on the conversation context. This is important because the extracted answers need to be based on the relevant passages, in order to ensure accuracy. On the other hand, Multi-Task Learning (MTL) has emerged as a promising approach to facilitate the learning of multiple related tasks by sharing the learner structure in a single model. MTL has gained considerable attention in recent years due to its effectiveness in addressing a diverse range of complex problems within a unified model. Therefore, we argue that learning ORConvQA approaches simultaneously can help to improve the system’s performance. In this thesis, we propose a novel ORConvQA framework leveraging Multi-Task Learning (MTL) to improve the performance of multiple related tasks by sharing their learned structure. By applying MTL to ORConvQA, we aim to leverage the benefits of addressing several related tasks to build a more effective and efficient model that addresses two main challenges: (i) ambiguities in conversational questions; and (ii) retrieving relevant passages from a large collection of documents before extracting the answers. To address ORConvQA effectively, we first propose an ORConvQA framework, which leverages a novel hybrid dynamic MTL method combining Abridged Linear for the main answer extraction task with a Loss-Balanced Task Weighting (LBTW) for the auxiliary related tasks, such as follow-up question identification, yes/no prediction, and unanswerable prediction, so as to automatically fine-tune task weighting during learning, ensuring that each of the tasks’ weights is adjusted by the relative importance of the different tasks. We conduct experiments using QuAC, a large-scale ConvQA dataset. Our results demonstrate the effectiveness of our proposed method, which significantly outperforms both the single-task learning and existing static task weighting methods with improvements ranging from +2.72% to +3.20% in F1 scores. Our findings also show that the performance of using MTL in developing the ORConvQA model is sensitive to the correct selection of the auxiliary tasks as well as to an adequate balancing of the loss rates of these tasks during training by using LBTW. To address the ambiguities in conversational questions, we propose the use of a text generation model with Multi-Task Learning for follow-up question identification and conversational question rewriting. Our derived models are based on text generation models –BART and T5–, and are trained to rewrite the conversational question and identify follow-up questions simultaneously. We evaluate our method using three test sets from the recent LIF (Learning to Identify Follow-up questions) dataset and a test set from the OR-QuAC dataset. Our results show that our proposed method significantly outperforms the single-task learning baselines on the LIF dataset, with statistically significant improvements ranging from +3.5% to +10.5% across all test sets, and also significantly outperforms the single-task learning of question rewriting models for passage retrieval on the OR-QuAC test set. Next, we employ an approach for asking clarifying questions to further address the ambiguities in conversational questions by proposing a novel hybrid method combining the generation and selection processes. Our method leverages Multi-Task Learning, combining the tasks of clarification need classification and the generation of the clarifying question to simultaneously determine when the initial user’s query necessitates a clarifying question and to generate a set of clarifying questions based on the user’s initial query and conversation history. A selection model is used to select the relevant questions from a question pool. To rank the candidate clarifying questions obtained from both the selection and generation approaches, the questions are scored using a text generation model for question classification. By using both the generation and selection approaches, our proposed method is able to generate a comprehensive set of questions while still ensuring that the selected question is relevant to the user’s queries. Our results on the TREC CAsT 2022 datasets demonstrate the effectiveness of our proposed method, which significantly outperforms existing strong baselines with improvements at P@1 by up to 20% on the relevance criteria and 30% on the novelty criteria. Finally, to effectively address our second challenge of retrieving relevant passages from a large collection of documents and extracting the answers, we propose monoQA, which uses a text generation model with Multi-Task Learning for both the reranker and reader. Our model, which is based on the T5 text generation model, is fine-tuned simultaneously for both reranking (in order to improve the precision of the top retrieved passages) and extracting the answer. Our results on the OR-QuAC and OR-CoQA datasets demonstrate the effectiveness of our proposed model, which significantly outperforms existing strong baselines with improvements ranging from +12.31% to +19.51% in MAP and from +5.70% to +23.34% in F1 on all used test sets. Overall, this thesis contributes an effective ORConvQA framework leveraging Multi-Task Learning to address the challenges of resolving ambiguities in conversational questions and retrieving relevant passages from a large collection of documents. Our proposed framework significantly outperforms existing strong baselines on a variety of benchmark datasets, demonstrating the effectiveness of MTL in improving the performance of ORConvQA models