5,837 research outputs found
Counterfactual Estimation and Optimization of Click Metrics for Search Engines
Optimizing an interactive system against a predefined online metric is
particularly challenging, when the metric is computed from user feedback such
as clicks and payments. The key challenge is the counterfactual nature: in the
case of Web search, any change to a component of the search engine may result
in a different search result page for the same query, but we normally cannot
infer reliably from search log how users would react to the new result page.
Consequently, it appears impossible to accurately estimate online metrics that
depend on user feedback, unless the new engine is run to serve users and
compared with a baseline in an A/B test. This approach, while valid and
successful, is unfortunately expensive and time-consuming. In this paper, we
propose to address this problem using causal inference techniques, under the
contextual-bandit framework. This approach effectively allows one to run
(potentially infinitely) many A/B tests offline from search log, making it
possible to estimate and optimize online metrics quickly and inexpensively.
Focusing on an important component in a commercial search engine, we show how
these ideas can be instantiated and applied, and obtain very promising results
that suggest the wide applicability of these techniques
What Users Ask a Search Engine: Analyzing One Billion Russian Question Queries
We analyze the question queries submitted to a large commercial web search engine to get insights about what people ask, and to better tailor the search results to the users’ needs. Based on a dataset of about one billion question queries submitted during the year 2012, we investigate askers’ querying behavior with the support of automatic query categorization. While the importance of question queries is likely to increase, at present they only make up 3–4% of the total search traffic. Since questions are such a small part of the query stream and are more likely to be unique than shorter queries, clickthrough information is typically rather sparse. Thus, query categorization methods based on the categories of clicked web documents do not work well for questions. As an alternative, we propose a robust question query classification method that uses the labeled questions from a large community question answering platform (CQA) as a training set. The resulting classifier is then transferred to the web search questions. Even though questions on CQA platforms tend to be different to web search questions, our categorization method proves competitive with strong baselines with respect to classification accuracy. To show the scalability of our proposed method we apply the classifiers to about one billion question queries and discuss the trade-offs between performance and accuracy that different classification models offer. Our findings reveal what people ask a search engine and also how this contrasts behavior on a CQA platform
Auditing Search Engines for Differential Satisfaction Across Demographics
Many online services, such as search engines, social media platforms, and
digital marketplaces, are advertised as being available to any user, regardless
of their age, gender, or other demographic factors. However, there are growing
concerns that these services may systematically underserve some groups of
users. In this paper, we present a framework for internally auditing such
services for differences in user satisfaction across demographic groups, using
search engines as a case study. We first explain the pitfalls of na\"ively
comparing the behavioral metrics that are commonly used to evaluate search
engines. We then propose three methods for measuring latent differences in user
satisfaction from observed differences in evaluation metrics. To develop these
methods, we drew on ideas from the causal inference literature and the
multilevel modeling literature. Our framework is broadly applicable to other
online services, and provides general insight into interpreting their
evaluation metrics.Comment: 8 pages Accepted at WWW 201
Enhancing Workflow with a Semantic Description of Scientific Intent
Peer reviewedPreprin
Adapting to the Shifting Intent of Search Queries
Search engines today present results that are often oblivious to abrupt
shifts in intent. For example, the query `independence day' usually refers to a
US holiday, but the intent of this query abruptly changed during the release of
a major film by that name. While no studies exactly quantify the magnitude of
intent-shifting traffic, studies suggest that news events, seasonal topics, pop
culture, etc account for 50% of all search queries. This paper shows that the
signals a search engine receives can be used to both determine that a shift in
intent has happened, as well as find a result that is now more relevant. We
present a meta-algorithm that marries a classifier with a bandit algorithm to
achieve regret that depends logarithmically on the number of query impressions,
under certain assumptions. We provide strong evidence that this regret is close
to the best achievable. Finally, via a series of experiments, we demonstrate
that our algorithm outperforms prior approaches, particularly as the amount of
intent-shifting traffic increases.Comment: This is the full version of the paper in NIPS'0
Studying Ransomware Attacks Using Web Search Logs
Cyber attacks are increasingly becoming prevalent and causing significant
damage to individuals, businesses and even countries. In particular, ransomware
attacks have grown significantly over the last decade. We do the first study on
mining insights about ransomware attacks by analyzing query logs from Bing web
search engine. We first extract ransomware related queries and then build a
machine learning model to identify queries where users are seeking support for
ransomware attacks. We show that user search behavior and characteristics are
correlated with ransomware attacks. We also analyse trends in the temporal and
geographical space and validate our findings against publicly available
information. Lastly, we do a case study on 'Nemty', a popular ransomware, to
show that it is possible to derive accurate insights about cyber attacks by
query log analysis.Comment: To appear in the proceedings of SIGIR 202
- …