Counterfactual Estimation and Optimization of Click Metrics for Search Engines

Optimizing an interactive system against a predefined online metric is particularly challenging, when the metric is computed from user feedback such as clicks and payments. The key challenge is the counterfactual nature: in the case of Web search, any change to a component of the search engine may result in a different search result page for the same query, but we normally cannot infer reliably from search log how users would react to the new result page. Consequently, it appears impossible to accurately estimate online metrics that depend on user feedback, unless the new engine is run to serve users and compared with a baseline in an A/B test. This approach, while valid and successful, is unfortunately expensive and time-consuming. In this paper, we propose to address this problem using causal inference techniques, under the contextual-bandit framework. This approach effectively allows one to run (potentially infinitely) many A/B tests offline from search log, making it possible to estimate and optimize online metrics quickly and inexpensively. Focusing on an important component in a commercial search engine, we show how these ideas can be instantiated and applied, and obtain very promising results that suggest the wide applicability of these techniques

Using contextual information to understand searching and browsing behavior

There is great imbalance in the richness of information on the web and the succinctness and poverty of search requests of web users, making their queries only a partial description of the underlying complex information needs. Finding ways to better leverage contextual information and make search context-aware holds the promise to dramatically improve the search experience of users. We conducted a series of studies to discover, model and utilize contextual information in order to understand and improve users' searching and browsing behavior on the web. Our results capture important aspects of context under the realistic conditions of different online search services, aiming to ensure that our scientific insights and solutions transfer to the operational settings of real world applications

Temporal Information Models for Real-Time Microblog Search

Real-time search in Twitter and other social media services is often biased towards the most recent results due to the “in the moment” nature of topic trends and their ephemeral relevance to users and media in general. However, “in the moment”, it is often difficult to look at all emerging topics and single-out the important ones from the rest of the social media chatter. This thesis proposes to leverage on external sources to estimate the duration and burstiness of live Twitter topics. It extends preliminary research where itwas shown that temporal re-ranking using external sources could indeed improve the accuracy of results. To further explore this topic we pursued three significant novel approaches: (1) multi-source information analysis that explores behavioral dynamics of users, such as Wikipedia live edits and page view streams, to detect topic trends and estimate the topic interest over time; (2) efficient methods for federated query expansion towards the improvement of query meaning; and (3) exploiting multiple sources towards the detection of temporal query intent. It differs from past approaches in the sense that it will work over real-time queries, leveraging on live user-generated content. This approach contrasts with previous methods that require an offline preprocessing step

Users' trust in information resources in the Web environment: a status report

This study has three aims; to provide an overview of the ways in which trust is either assessed or asserted in relation to the use and provision of resources in the Web environment for research and learning; to assess what solutions might be worth further investigation and whether establishing ways to assert trust in academic information resources could assist the development of information literacy; to help increase understanding of how perceptions of trust influence the behaviour of information users

Social Feedback: Social Learning from Interaction History to Support Information Seeking on the Web

Information seeking on the Web has become a central part of many daily activities. Even though information seeking is extremely common, there are many times when these tasks are unsuccessful, because the information found is less than ideal or the task could have been completed more efficiently. In unsuccessful information-seeking tasks, there are often other people who have knowledge or experience that could help improve task success. However, information seekers do not typically look for help from others, because tasks can often be completed alone (even if inefficiently). One of the problems is that web tools provide people with few opportunities to learn from one another’s experiences in ways that would allow them to improve their success. This dissertation presents the idea of social feedback. Social feedback is based on the theory of social learning, which describes how people learn from observing others. In social feedback, observational learning is enabled through the mechanism of interaction history – the traces of activity people create as they interact with the Web. Social feedback systems collect and display interaction history to allow information seekers to learn how to complete their tasks more successfully by observing how other people have behaved in similar situations. The dissertation outlines the design of two social-feedback systems, and describes two studies that demonstrate the real world applicability and feasibility of the idea. The first system supports global learning, by allowing people to learn new search skills and techniques that improve information seeking success in many different tasks. The second system supports local learning, in which people learn how to accomplish specific tasks more effectively and more efficiently. Two further studies are conducted to explore potential real-world challenges to the successful deployment of social feedback systems, such as the privacy concerns associated with the collection and sharing of interaction history. These studies show that social feedback systems can be deployed successfully for supporting real world information seeking tasks. Overall, this research shows that social feedback is a valuable new idea for the social use of information systems, an idea that allows people to learn from one another’s experiences and improve their success in many common real-world tasks

Whittle Index Policy for Crawling Ephemeral Content

We consider a task of scheduling a crawler to retrieve content from several sites with ephemeral content. A user typically loses interest in ephemeral content, like news or posts at social network groups, after several days or hours. Thus, development of timely crawling policy for such ephemeral information sources is very important. We first formulate this problem as an optimal control problem with average reward. The reward can be measured in the number of clicks or relevant search requests. The problem in its initial formulation suffers from the curse of dimensionality and quickly becomes intractable even with moderate number of information sources. Fortunately, this problem admits a Whittle index, which leads to problem decomposition and to a very simple and efficient crawling policy. We derive the Whittle index and provide its theoretical justification