5,557 research outputs found
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Machine translation is a natural candidate problem for reinforcement learning
from human feedback: users provide quick, dirty ratings on candidate
translations to guide a system to improve. Yet, current neural machine
translation training focuses on expensive human-generated reference
translations. We describe a reinforcement learning algorithm that improves
neural machine translation systems from simulated human feedback. Our algorithm
combines the advantage actor-critic algorithm (Mnih et al., 2016) with the
attention-based neural encoder-decoder architecture (Luong et al., 2015). This
algorithm (a) is well-designed for problems with a large action space and
delayed rewards, (b) effectively optimizes traditional corpus-level machine
translation metrics, and (c) is robust to skewed, high-variance, granular
feedback modeled after actual human behaviors.Comment: 11 pages, 5 figures, In Proceedings of Empirical Methods in Natural
Language Processing (EMNLP) 201
A Shared Task on Bandit Learning for Machine Translation
We introduce and describe the results of a novel shared task on bandit
learning for machine translation. The task was organized jointly by Amazon and
Heidelberg University for the first time at the Second Conference on Machine
Translation (WMT 2017). The goal of the task is to encourage research on
learning machine translation from weak user feedback instead of human
references or post-edits. On each of a sequence of rounds, a machine
translation system is required to propose a translation for an input, and
receives a real-valued estimate of the quality of the proposed translation for
learning. This paper describes the shared task's learning and evaluation setup,
using services hosted on Amazon Web Services (AWS), the data and evaluation
metrics, and the results of various machine translation architectures and
learning protocols.Comment: Conference on Machine Translation (WMT) 201
Seamlessly Unifying Attributes and Items: Conversational Recommendation for Cold-Start Users
Static recommendation methods like collaborative filtering suffer from the
inherent limitation of performing real-time personalization for cold-start
users. Online recommendation, e.g., multi-armed bandit approach, addresses this
limitation by interactively exploring user preference online and pursuing the
exploration-exploitation (EE) trade-off. However, existing bandit-based methods
model recommendation actions homogeneously. Specifically, they only consider
the items as the arms, being incapable of handling the item attributes, which
naturally provide interpretable information of user's current demands and can
effectively filter out undesired items. In this work, we consider the
conversational recommendation for cold-start users, where a system can both ask
the attributes from and recommend items to a user interactively. This important
scenario was studied in a recent work. However, it employs a hand-crafted
function to decide when to ask attributes or make recommendations. Such
separate modeling of attributes and items makes the effectiveness of the system
highly rely on the choice of the hand-crafted function, thus introducing
fragility to the system. To address this limitation, we seamlessly unify
attributes and items in the same arm space and achieve their EE trade-offs
automatically using the framework of Thompson Sampling. Our Conversational
Thompson Sampling (ConTS) model holistically solves all questions in
conversational recommendation by choosing the arm with the maximal reward to
play. Extensive experiments on three benchmark datasets show that ConTS
outperforms the state-of-the-art methods Conversational UCB (ConUCB) and
Estimation-Action-Reflection model in both metrics of success rate and average
number of conversation turns.Comment: TOIS 202
Effective Evaluation using Logged Bandit Feedback from Multiple Loggers
Accurately evaluating new policies (e.g. ad-placement models, ranking
functions, recommendation functions) is one of the key prerequisites for
improving interactive systems. While the conventional approach to evaluation
relies on online A/B tests, recent work has shown that counterfactual
estimators can provide an inexpensive and fast alternative, since they can be
applied offline using log data that was collected from a different policy
fielded in the past. In this paper, we address the question of how to estimate
the performance of a new target policy when we have log data from multiple
historic policies. This question is of great relevance in practice, since
policies get updated frequently in most online systems. We show that naively
combining data from multiple logging policies can be highly suboptimal. In
particular, we find that the standard Inverse Propensity Score (IPS) estimator
suffers especially when logging and target policies diverge -- to a point where
throwing away data improves the variance of the estimator. We therefore propose
two alternative estimators which we characterize theoretically and compare
experimentally. We find that the new estimators can provide substantially
improved estimation accuracy.Comment: KDD 201
- …