110,743 research outputs found
Statistical Rejection Sampling Improves Preference Optimization
Improving the alignment of language models with human preferences remains an
active research challenge. Previous approaches have primarily utilized
Reinforcement Learning from Human Feedback (RLHF) via online RL methods such as
Proximal Policy Optimization (PPO). Recently, offline methods such as Sequence
Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have
emerged as attractive alternatives, offering improvements in stability and
scalability while maintaining competitive performance. SLiC refines its loss
function using sequence pairs sampled from a supervised fine-tuned (SFT)
policy, while DPO directly optimizes language models based on preference data,
foregoing the need for a separate reward model. However, the maximum likelihood
estimator (MLE) of the target optimal policy requires labeled preference pairs
sampled from that policy. DPO's lack of a reward model constrains its ability
to sample preference pairs from the optimal policy, and SLiC is restricted to
sampling preference pairs only from the SFT policy. To address these
limitations, we introduce a novel approach called Statistical Rejection
Sampling Optimization (RSO) that aims to source preference data from the target
optimal policy using rejection sampling, enabling a more accurate estimation of
the optimal policy. We also propose a unified framework that enhances the loss
functions used in both SLiC and DPO from a preference modeling standpoint.
Through extensive experiments across three diverse tasks, we demonstrate that
RSO consistently outperforms both SLiC and DPO on evaluations from both Large
Language Model (LLM) and human raters
Active Inverse Reward Design
Designers of AI agents often iterate on the reward function in a
trial-and-error process until they get the desired behavior, but this only
guarantees good behavior in the training environment. We propose structuring
this process as a series of queries asking the user to compare between
different reward functions. Thus we can actively select queries for maximum
informativeness about the true reward. In contrast to approaches asking the
designer for optimal behavior, this allows us to gather additional information
by eliciting preferences between suboptimal behaviors. After each query, we
need to update the posterior over the true reward function from observing the
proxy reward function chosen by the designer. The recently proposed Inverse
Reward Design (IRD) enables this. Our approach substantially outperforms IRD in
test environments. In particular, it can query the designer about
interpretable, linear reward functions and still infer non-linear ones
A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
We present a tutorial on Bayesian optimization, a method of finding the
maximum of expensive cost functions. Bayesian optimization employs the Bayesian
technique of setting a prior over the objective function and combining it with
evidence to get a posterior function. This permits a utility-based selection of
the next observation to make on the objective function, which must take into
account both exploration (sampling from areas of high uncertainty) and
exploitation (sampling areas likely to offer improvement over the current best
observation). We also present two detailed extensions of Bayesian optimization,
with experiments---active user modelling with preferences, and hierarchical
reinforcement learning---and a discussion of the pros and cons of Bayesian
optimization based on our experiences
The Green Choice: Learning and Influencing Human Decisions on Shared Roads
Autonomous vehicles have the potential to increase the capacity of roads via
platooning, even when human drivers and autonomous vehicles share roads.
However, when users of a road network choose their routes selfishly, the
resulting traffic configuration may be very inefficient. Because of this, we
consider how to influence human decisions so as to decrease congestion on these
roads. We consider a network of parallel roads with two modes of
transportation: (i) human drivers who will choose the quickest route available
to them, and (ii) ride hailing service which provides an array of autonomous
vehicle ride options, each with different prices, to users. In this work, we
seek to design these prices so that when autonomous service users choose from
these options and human drivers selfishly choose their resulting routes, road
usage is maximized and transit delay is minimized. To do so, we formalize a
model of how autonomous service users make choices between routes with
different price/delay values. Developing a preference-based algorithm to learn
the preferences of the users, and using a vehicle flow model related to the
Fundamental Diagram of Traffic, we formulate a planning optimization to
maximize a social objective and demonstrate the benefit of the proposed routing
and learning scheme.Comment: Submitted to CDC 201
Deep reinforcement learning from human preferences
For sophisticated reinforcement learning (RL) systems to interact usefully
with real-world environments, we need to communicate complex goals to these
systems. In this work, we explore goals defined in terms of (non-expert) human
preferences between pairs of trajectory segments. We show that this approach
can effectively solve complex RL tasks without access to the reward function,
including Atari games and simulated robot locomotion, while providing feedback
on less than one percent of our agent's interactions with the environment. This
reduces the cost of human oversight far enough that it can be practically
applied to state-of-the-art RL systems. To demonstrate the flexibility of our
approach, we show that we can successfully train complex novel behaviors with
about an hour of human time. These behaviors and environments are considerably
more complex than any that have been previously learned from human feedback
- …