83 research outputs found
Inverse Preference Learning: Preference-based RL without a Reward Function
Reward functions are difficult to design and often hard to align with human
intent. Preference-based Reinforcement Learning (RL) algorithms address these
problems by learning reward functions from human feedback. However, the
majority of preference-based RL methods na\"ively combine supervised reward
models with off-the-shelf RL algorithms. Contemporary approaches have sought to
improve performance and query complexity by using larger and more complex
reward architectures such as transformers. Instead of using highly complex
architectures, we develop a new and parameter-efficient algorithm, Inverse
Preference Learning (IPL), specifically designed for learning from offline
preference data. Our key insight is that for a fixed policy, the -function
encodes all information about the reward function, effectively making them
interchangeable. Using this insight, we completely eliminate the need for a
learned reward function. Our resulting algorithm is simpler and more
parameter-efficient. Across a suite of continuous control and robotics
benchmarks, IPL attains competitive performance compared to more complex
approaches that leverage transformer-based and non-Markovian reward functions
while having fewer algorithmic hyperparameters and learned network parameters.
Our code is publicly released
Altruistic Autonomy: Beating Congestion on Shared Roads
Traffic congestion has large economic and social costs. The introduction of
autonomous vehicles can potentially reduce this congestion, both by increasing
network throughput and by enabling a social planner to incentivize users of
autonomous vehicles to take longer routes that can alleviate congestion on more
direct roads. We formalize the effects of altruistic autonomy on roads shared
between human drivers and autonomous vehicles. In this work, we develop a
formal model of road congestion on shared roads based on the fundamental
diagram of traffic. We consider a network of parallel roads and provide
algorithms that compute optimal equilibria that are robust to additional
unforeseen demand. We further plan for optimal routings when users have varying
degrees of altruism. We find that even with arbitrarily small altruism, total
latency can be unboundedly better than without altruism, and that the best
selfish equilibrium can be unboundedly better than the worst selfish
equilibrium. We validate our theoretical results through microscopic traffic
simulations and show average latency decrease of a factor of 4 from worst-case
selfish equilibrium to the optimal equilibrium when autonomous vehicles are
altruistic.Comment: Accepted to Workshop on the Algorithmic Foundations of Robotics
(WAFR) 201
- …