Efficient query processing for scalable web search

Search engines are exceptionally important tools for accessing information in today’s world. In satisfying the information needs of millions of users, the effectiveness (the quality of the search results) and the efficiency (the speed at which the results are returned to the users) of a search engine are two goals that form a natural trade-off, as techniques that improve the effectiveness of the search engine can also make it less efficient. Meanwhile, search engines continue to rapidly evolve, with larger indexes, more complex retrieval strategies and growing query volumes. Hence, there is a need for the development of efficient query processing infrastructures that make appropriate sacrifices in effectiveness in order to make gains in efficiency. This survey comprehensively reviews the foundations of search engines, from index layouts to basic term-at-a-time (TAAT) and document-at-a-time (DAAT) query processing strategies, while also providing the latest trends in the literature in efficient query processing, including the coherent and systematic reviews of techniques such as dynamic pruning and impact-sorted posting lists as well as their variants and optimisations. Our explanations of query processing strategies, for instance the WAND and BMW dynamic pruning algorithms, are presented with illustrative figures showing how the processing state changes as the algorithms progress. Moreover, acknowledging the recent trends in applying a cascading infrastructure within search systems, this survey describes techniques for efficiently integrating effective learned models, such as those obtained from learning-to-rank techniques. The survey also covers the selective application of query processing techniques, often achieved by predicting the response times of the search engine (known as query efficiency prediction), and making per-query tradeoffs between efficiency and effectiveness to ensure that the required retrieval speed targets can be met. Finally, the survey concludes with a summary of open directions in efficient search infrastructures, namely the use of signatures, real-time, energy-efficient and modern hardware and software architectures

Managing tail latency in large scale information retrieval systems

As both the availability of internet access and the prominence of smart devices continue to increase, data is being generated at a rate faster than ever before. This massive increase in data production comes with many challenges, including efficiency concerns for the storage and retrieval of such large-scale data. However, users have grown to expect the sub-second response times that are common in most modern search engines, creating a problem - how can such large amounts of data continue to be served efficiently enough to satisfy end users? This dissertation investigates several issues regarding tail latency in large-scale information retrieval systems. Tail latency corresponds to the high percentile latency that is observed from a system - in the case of search, this latency typically corresponds to how long it takes for a query to be processed. In particular, keeping tail latency as low as possible translates to a good experience for all users, as tail latency is directly related to the worst-case latency and hence, the worst possible user experience. The key idea in targeting tail latency is to move from questions such as "what is the median latency of our search engine?" to questions which more accurately capture user experience such as "how many queries take more than 200ms to return answers?" or "what is the worst case latency that a user may be subject to, and how often might it occur?" While various strategies exist for efficiently processing queries over large textual corpora, prior research has focused almost entirely on improvements to the average processing time or cost of search systems. As a first contribution, we examine some state-of-the-art retrieval algorithms for two popular index organizations, and discuss the trade-offs between them, paying special attention to the notion of tail latency. This research uncovers a number of observations that are subsequently leveraged for improved search efficiency and effectiveness. We then propose and solve a new problem, which involves processing a number of related queries together, known as multi-queries, to yield higher quality search results. We experiment with a number of algorithmic approaches to efficiently process these multi-queries, and report on the cost, efficiency, and effectiveness trade-offs present with each. Ultimately, we find that some solutions yield a low tail latency, and are hence suitable for use in real-time search environments. Finally, we examine how predictive models can be used to improve the tail latency and end-to-end cost of a commonly used multi-stage retrieval architecture without impacting result effectiveness. By combining ideas from numerous areas of information retrieval, we propose a prediction framework which can be used for training and evaluating several efficiency/effectiveness trade-off parameters, resulting in improved trade-offs between cost, result quality, and tail latency

Learning Gene Interactions and Networks from Perturbation Screens and Expression Data

We investigate a variety of methods to first discover and then understand genetic interactions. Beginning with pairwise interactions, we propose a method for inferring pairwise gene interactions en masse from short- interfering RNA screens. We use the siRNA off-target effects to form a matrix of knocked-down genes, and consider the observed fitness to be a linear combination of individual and pairwise effects in this matrix. These effects can then be inferred using a variety of statistical learning methods. We evaluate two such methods for this task, xyz and glinternet. Using either method, we are able to find interactions in small simulated data sets. Neither method scales to genome-scale data sets, however. In our larger simulations both methods suffer from scalability problems, either with their accuracy or running time. We overcome these limitations by developing our own lasso-based regression method, which takes into account the binary nature of our perturbation screens. Using a compressed sparse representation of the pairwise interaction matrix, and parallelising updates, we are able to run this method on exome-scale data. Generalising from pairwise interactions we then consider network models, in which pairwise gene interactions form edges of a graph. Such networks are often understood in terms of functional modules, groups of genes that act together to perform a task. We develop a method that combines pairwise interaction and gene expression data to effectively find functional modules in simulated data

Research and Technology Objectives and Plans Summary (RTOPS)

This publication represents the NASA research and technology program for FY89. It is a compilation of the Summary portions of each of the RTOPs (Research and Technology Objectives and Plans) used for management review and control of research currently in progress throughout NASA. The RTOP Summary is designed to facilitate communication and coordination among concerned technical personnel in government, in industry, and in universities. The first section containing citations and abstracts of the RTOPs is followed by four indexes: Subject, Technical Monitor, Responsible NASA Organization, and RTOP Number

Research and Technology Objectives and Plans Summary (RTOPS)

This publication represents the NASA research and technology program for FY88. It is a compilation of the Summary portions of each of the RTOPs (Research and Technology Objectives and Plans) used for management review and control of research currently in progress throughout NASA. The RTOP Summary is designed to facilitate communication and coordination among concerned technical personnel in government, in industry, and in universities. The first section containing citations and abstracts of the RTOPs is followed by four indexes: Subject, Technical Monitor, Responsible NASA Organization, and RTOP Number

Research and Technology Objectives and Plans Summary (RTOPS)

The NASA research and technology program for FY 1990 is presented. The summary portions is compiled of each of the RTOPs (Research and Technology Objectives and Plans) used for management review and control of research currently in progress throughout NASA. The RTOP summary is designed to facilitate communication and coordination among concerned technical personnel in government, industry, and universities. The first section containing citations and abstracts of the RTOPs is followed by four indices: Subject; Technical Monitor; Responsible NASA Organization; and RTOP number