4 research outputs found
Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
Reinforcement learning (RL) requires either manually specifying a reward
function, which is often infeasible, or learning a reward model from a large
amount of human feedback, which is often very expensive. We study a more
sample-efficient alternative: using pretrained vision-language models (VLMs) as
zero-shot reward models (RMs) to specify tasks via natural language. We propose
a natural and general approach to using VLMs as reward models, which we call
VLM-RMs. We use VLM-RMs based on CLIP to train a MuJoCo humanoid to learn
complex tasks without a manually specified reward function, such as kneeling,
doing the splits, and sitting in a lotus position. For each of these tasks, we
only provide a single sentence text prompt describing the desired task with
minimal prompt engineering. We provide videos of the trained agents at:
https://sites.google.com/view/vlm-rm. We can improve performance by providing a
second ``baseline'' prompt and projecting out parts of the CLIP embedding space
irrelevant to distinguish between goal and baseline. Further, we find a strong
scaling effect for VLM-RMs: larger VLMs trained with more compute and data are
better reward models. The failure modes of VLM-RMs we encountered are all
related to known capability limitations of current VLMs, such as limited
spatial reasoning ability or visually unrealistic environments that are far
off-distribution for the VLM. We find that VLM-RMs are remarkably robust as
long as the VLM is large enough. This suggests that future VLMs will become
more and more useful reward models for a wide range of RL applications
Picking the low-hanging fruit: testing new physics at scale with active learning
Since the discovery of the Higgs boson, testing the many possible extensions to the Standard Model has become a key challenge in particle physics. This paper discusses a new method for predicting the compatibility of new physics theories with existing experimental data from particle colliders. Using machine learning, the technique obtained comparable results to previous methods (>90% precision and recall) with only a fraction of their computing resources (<10%). This makes it possible to test models that were impossible to probe before, and allows for large-scale testing of new physics theories