221 research outputs found
Assessing Human Error Against a Benchmark of Perfection
An increasing number of domains are providing us with detailed trace data on
human decisions in settings where we can evaluate the quality of these
decisions via an algorithm. Motivated by this development, an emerging line of
work has begun to consider whether we can characterize and predict the kinds of
decisions where people are likely to make errors.
To investigate what a general framework for human error prediction might look
like, we focus on a model system with a rich history in the behavioral
sciences: the decisions made by chess players as they select moves in a game.
We carry out our analysis at a large scale, employing datasets with several
million recorded games, and using chess tablebases to acquire a form of ground
truth for a subset of chess positions that have been completely solved by
computers but remain challenging even for the best players in the world.
We organize our analysis around three categories of features that we argue
are present in most settings where the analysis of human error is applicable:
the skill of the decision-maker, the time available to make the decision, and
the inherent difficulty of the decision. We identify rich structure in all
three of these categories of features, and find strong evidence that in our
domain, features describing the inherent difficulty of an instance are
significantly more powerful than features based on skill or time.Comment: KDD 2016; 10 page
Learning to rank from medical imaging data
Medical images can be used to predict a clinical score coding for the
severity of a disease, a pain level or the complexity of a cognitive task. In
all these cases, the predicted variable has a natural order. While a standard
classifier discards this information, we would like to take it into account in
order to improve prediction performance. A standard linear regression does
model such information, however the linearity assumption is likely not be
satisfied when predicting from pixel intensities in an image. In this paper we
address these modeling challenges with a supervised learning procedure where
the model aims to order or rank images. We use a linear model for its
robustness in high dimension and its possible interpretation. We show on
simulations and two fMRI datasets that this approach is able to predict the
correct ordering on pairs of images, yielding higher prediction accuracy than
standard regression and multiclass classification techniques
Web-Scale Bayesian click-through rate prediction for sponsored search advertising in Microsoft's Bing search engine
We describe a new Bayesian click-through rate
(CTR) prediction algorithm used for Sponsored
Search in Microsoft's Bing search engine. The
algorithm is based on a probit regression model
that maps discrete or real-valued input features to
probabilities. It maintains Gaussian beliefs over
weights of the model and performs Gaussian
online updates derived from approximate
message passing. Scalability of the algorithm is
ensured through a principled weight pruning
procedure and an approximate parallel
implementation. We discuss the challenges
arising from evaluating and tuning the predictor
as part of the complex system of sponsored
search where the predictions made by the
algorithm decide about future training sample
composition. Finally, we show experimental
results from the production system and compare
to a calibrated NaĂŻve Bayes algorithm
A network-based dynamical ranking system for competitive sports
From the viewpoint of networks, a ranking system for players or teams in
sports is equivalent to a centrality measure for sports networks, whereby a
directed link represents the result of a single game. Previously proposed
network-based ranking systems are derived from static networks, i.e.,
aggregation of the results of games over time. However, the score of a player
(or team) fluctuates over time. Defeating a renowned player in the peak
performance is intuitively more rewarding than defeating the same player in
other periods. To account for this factor, we propose a dynamic variant of such
a network-based ranking system and apply it to professional men's tennis data.
We derive a set of linear online update equations for the score of each player.
The proposed ranking system predicts the outcome of the future games with a
higher accuracy than the static counterparts.Comment: 6 figure
A Uniform Lower Error Bound for Half-Space Learning
We give a lower bound for the error of any unitarily invariant algorithm learning half-spaces against the uniform or related distributions on the unit sphere. The bound is uniform in the choice of the target half-space and has an exponentially decaying deviation probability in the sample. The technique of proof is related to a proof of the Johnson Lindenstrauss Lemma. We argue that, unlike previous lower bounds, our result is well suited to evaluate the benefits of multi-task or transfer learning, or other cases where an expense in the acquisition of domain knowledge has to be justified
APRIL: Active Preference-learning based Reinforcement Learning
This paper focuses on reinforcement learning (RL) with limited prior
knowledge. In the domain of swarm robotics for instance, the expert can hardly
design a reward function or demonstrate the target behavior, forbidding the use
of both standard RL and inverse reinforcement learning. Although with a limited
expertise, the human expert is still often able to emit preferences and rank
the agent demonstrations. Earlier work has presented an iterative
preference-based RL framework: expert preferences are exploited to learn an
approximate policy return, thus enabling the agent to achieve direct policy
search. Iteratively, the agent selects a new candidate policy and demonstrates
it; the expert ranks the new demonstration comparatively to the previous best
one; the expert's ranking feedback enables the agent to refine the approximate
policy return, and the process is iterated. In this paper, preference-based
reinforcement learning is combined with active ranking in order to decrease the
number of ranking queries to the expert needed to yield a satisfactory policy.
Experiments on the mountain car and the cancer treatment testbeds witness that
a couple of dozen rankings enable to learn a competent policy
Detecting Remote Evolutionary Relationships among Proteins by Large-Scale Semantic Embedding
Virtually every molecular biologist has searched a protein or DNA sequence database to find sequences that are evolutionarily related to a given query. Pairwise sequence comparison methods—i.e., measures of similarity between query and target sequences—provide the engine for sequence database search and have been the subject of 30 years of computational research. For the difficult problem of detecting remote evolutionary relationships between protein sequences, the most successful pairwise comparison methods involve building local models (e.g., profile hidden Markov models) of protein sequences. However, recent work in massive data domains like web search and natural language processing demonstrate the advantage of exploiting the global structure of the data space. Motivated by this work, we present a large-scale algorithm called ProtEmbed, which learns an embedding of protein sequences into a low-dimensional “semantic space.” Evolutionarily related proteins are embedded in close proximity, and additional pieces of evidence, such as 3D structural similarity or class labels, can be incorporated into the learning process. We find that ProtEmbed achieves superior accuracy to widely used pairwise sequence methods like PSI-BLAST and HHSearch for remote homology detection; it also outperforms our previous RankProp algorithm, which incorporates global structure in the form of a protein similarity network. Finally, the ProtEmbed embedding space can be visualized, both at the global level and local to a given query, yielding intuition about the structure of protein sequence space
- …