Balancing Reinforcement Learning Training Experiences in Interactive Information Retrieval

Interactive Information Retrieval (IIR) and Reinforcement Learning (RL) share many commonalities, including an agent who learns while interacts, a long-term and complex goal, and an algorithm that explores and adapts. To successfully apply RL methods to IIR, one challenge is to obtain sufficient relevance labels to train the RL agents, which are infamously known as sample inefficient. However, in a text corpus annotated for a given query, it is not the relevant documents but the irrelevant documents that predominate. This would cause very unbalanced training experiences for the agent and prevent it from learning any policy that is effective. Our paper addresses this issue by using domain randomization to synthesize more relevant documents for the training. Our experimental results on the Text REtrieval Conference (TREC) Dynamic Domain (DD) 2017 Track show that the proposed method is able to boost an RL agent's learning effectiveness by 22\% in dealing with unseen situations.Comment: Accepted by SIGIR 202

Corpus-Level End-to-End Exploration for Interactive Systems

A core interest in building Artificial Intelligence (AI) agents is to let them interact with and assist humans. One example is Dynamic Search (DS), which models the process that a human works with a search engine agent to accomplish a complex and goal-oriented task. Early DS agents using Reinforcement Learning (RL) have only achieved limited success for (1) their lack of direct control over which documents to return and (2) the difficulty to recover from wrong search trajectories. In this paper, we present a novel corpus-level end-to-end exploration (CE3) method to address these issues. In our method, an entire text corpus is compressed into a global low-dimensional representation, which enables the agent to gain access to the full state and action spaces, including the under-explored areas. We also propose a new form of retrieval function, whose linear approximation allows end-to-end manipulation of documents. Experiments on the Text REtrieval Conference (TREC) Dynamic Domain (DD) Track show that CE3 outperforms the state-of-the-art DS systems.Comment: Accepted into AAAI 202

Towards Humane Feedback Mechanisms in Exploratory Search

Machine learning (ML) plays a central role in modern information retrieval (IR) systems. We argue that, in IR systems for multi-session exploratory search, there are unexploited opportunities for IR document ranking models to leverage users’ knowledge about the search task to better support users’ search needs. Specifically, we propose a method to enable users to adapt an IR document ranking model according to their information needs, using an interface that supports search strategies and methods for engaging with documents known to be useful when people explore new or complex domains of knowledge. We also discuss the major challenges in creating human-centered machine learning models and interfaces for exploratory search