46 research outputs found
Discovering Valuable Items from Massive Data
Suppose there is a large collection of items, each with an associated cost
and an inherent utility that is revealed only once we commit to selecting it.
Given a budget on the cumulative cost of the selected items, how can we pick a
subset of maximal value? This task generalizes several important problems such
as multi-arm bandits, active search and the knapsack problem. We present an
algorithm, GP-Select, which utilizes prior knowledge about similarity be- tween
items, expressed as a kernel function. GP-Select uses Gaussian process
prediction to balance exploration (estimating the unknown value of items) and
exploitation (selecting items of high value). We extend GP-Select to be able to
discover sets that simultaneously have high utility and are diverse. Our
preference for diversity can be specified as an arbitrary monotone submodular
function that quantifies the diminishing returns obtained when selecting
similar items. Furthermore, we exploit the structure of the model updates to
achieve an order of magnitude (up to 40X) speedup in our experiments without
resorting to approximations. We provide strong guarantees on the performance of
GP-Select and apply it to three real-world case studies of industrial
relevance: (1) Refreshing a repository of prices in a Global Distribution
System for the travel industry, (2) Identifying diverse, binding-affine
peptides in a vaccine de- sign task and (3) Maximizing clicks in a web-scale
recommender system by recommending items to users
Deep Exploration for Recommendation Systems
Modern recommendation systems ought to benefit by probing for and learning
from delayed feedback. Research has tended to focus on learning from a user's
response to a single recommendation. Such work, which leverages methods of
supervised and bandit learning, forgoes learning from the user's subsequent
behavior. Where past work has aimed to learn from subsequent behavior, there
has been a lack of effective methods for probing to elicit informative delayed
feedback. Effective exploration through probing for delayed feedback becomes
particularly challenging when rewards are sparse. To address this, we develop
deep exploration methods for recommendation systems. In particular, we
formulate recommendation as a sequential decision problem and demonstrate
benefits of deep exploration over single-step exploration. Our experiments are
carried out with high-fidelity industrial-grade simulators and establish large
improvements over existing algorithms
Multi-Task Off-Policy Learning from Bandit Feedback
Many practical applications, such as recommender systems and learning to
rank, involve solving multiple similar tasks. One example is learning of
recommendation policies for users with similar movie preferences, where the
users may still rank the individual movies slightly differently. Such tasks can
be organized in a hierarchy, where similar tasks are related through a shared
structure. In this work, we formulate this problem as a contextual off-policy
optimization in a hierarchical graphical model from logged bandit feedback. To
solve the problem, we propose a hierarchical off-policy optimization algorithm
(HierOPO), which estimates the parameters of the hierarchical model and then
acts pessimistically with respect to them. We instantiate HierOPO in linear
Gaussian models, for which we also provide an efficient implementation and
analysis. We prove per-task bounds on the suboptimality of the learned
policies, which show a clear improvement over not using the hierarchical model.
We also evaluate the policies empirically. Our theoretical and empirical
results show a clear advantage of using the hierarchy over solving each task
independently.Comment: 14 pages, 3 figure
Dynamic Slate Recommendation with Gated Recurrent Units and Thompson Sampling
We consider the problem of recommending relevant content to users of an internet platform in the form of lists of items, called slates. We introduce a variational Bayesian Recurrent Neural Net recommender system that acts on time series of interactions between the internet platform and the user, and which scales to real world industrial situations. The recommender system is tested both online on real users, and on an offline dataset collected from a Norwegian web-based marketplace, FINN.no, that is made public for research. This is one of the first publicly available datasets which includes all the slates that are presented to users as well as which items (if any) in the slates were clicked on. Such a data set allows us to move beyond the common assumption that implicitly assumes that users are considering all possible items at each interaction. Instead we build our likelihood using the items that are actually in the slate, and evaluate the strengths and weaknesses of both approaches theoretically and in experiments. We also introduce a hierarchical prior for the item parameters based on group memberships. Both item parameters and user preferences are learned probabilistically. Furthermore, we combine our model with bandit strategies to ensure learning, and introduce `in-slate Thompson Sampling' which makes use of the slates to maximise explorative opportunities. We show experimentally that explorative recommender strategies perform on par or above their greedy counterparts. Even without making use of exploration to learn more effectively, click rates increase simply because of improved diversity in the recommended slates
Improved sequential decision-making with structural priors: Enhanced treatment personalization with historical data
Personalizing treatments for patients involves a period where different treatments out of a set of available treatments are tried until an optimal treatment is found, for particular patient characteristics. To minimize suffering and other costs, it is critical to minimize this search. When treatments have primarily short-term effects, the search can be performed with multi-armed bandit algorithms (MABs). However, these typically require long exploration periods to guarantee optimality. With historical data, it is possible to recover a structure incorporating the prior knowledge of the types of patients that can be encountered, and the conditional reward models for those patient types. Such structural priors can be used to reduce the treatment exploration period for enhanced applicability in the real world. This thesis presents work on designing MAB algorithms that find optimal treatments quickly, by incorporating a structural prior for patient types in the form of a latent variable model. Theoretical guarantees for the algorithms, including a lower and a matching upper bound, and an empirical study is provided, showing that incorporating latent structural priors is beneficial. Another line of work in this thesis is the design of simulators for evaluating treatment policies and comparing algorithms. A new simulator for benchmarking estimators of causal effects, the Alzheimer’s Disease Causal estimation Benchmark (ADCB) is presented. ADCB combines data-driven simulation with subject-matter knowledge for high realism and causal verifiability. The design of the simulator is discussed, and to demonstrate its utility, the results of a usage scenario for evaluating estimators of causal effects are outlined