7 research outputs found
Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning
Training generally capable agents that perform well in unseen dynamic
environments is a long-term goal of robot learning. Quality Diversity
Reinforcement Learning (QD-RL) is an emerging class of reinforcement learning
(RL) algorithms that blend insights from Quality Diversity (QD) and RL to
produce a collection of high performing and behaviorally diverse policies with
respect to a behavioral embedding. Existing QD-RL approaches have thus far
taken advantage of sample-efficient off-policy RL algorithms. However, recent
advances in high-throughput, massively parallelized robotic simulators have
opened the door for algorithms that can take advantage of such parallelism, and
it is unclear how to scale existing off-policy QD-RL methods to these new
data-rich regimes. In this work, we take the first steps to combine on-policy
RL methods, specifically Proximal Policy Optimization (PPO), that can leverage
massive parallelism, with QD, and propose a new QD-RL method with these
high-throughput simulators and on-policy training in mind. Our proposed
Proximal Policy Gradient Arborescence (PPGA) algorithm yields a 4x improvement
over baselines on the challenging humanoid domain.Comment: Submitted to Neurips 202
Deep Surrogate Assisted Generation of Environments
Recent progress in reinforcement learning (RL) has started producing
generally capable agents that can solve a distribution of complex environments.
These agents are typically tested on fixed, human-authored environments. On the
other hand, quality diversity (QD) optimization has been proven to be an
effective component of environment generation algorithms, which can generate
collections of high-quality environments that are diverse in the resulting
agent behaviors. However, these algorithms require potentially expensive
simulations of agents on newly generated environments. We propose Deep
Surrogate Assisted Generation of Environments (DSAGE), a sample-efficient QD
environment generation algorithm that maintains a deep surrogate model for
predicting agent behaviors in new environments. Results in two benchmark
domains show that DSAGE significantly outperforms existing QD environment
generation algorithms in discovering collections of environments that elicit
diverse behaviors of a state-of-the-art RL agent and a planning agent.Comment: 24 pages, 15 figures, supplemental website at
https://dsagepaper.github.io
Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealing
Pre-training a diverse set of neural network controllers in simulation has
enabled robots to adapt online to damage in robot locomotion tasks. However,
finding diverse, high-performing controllers requires expensive network
training and extensive tuning of a large number of hyperparameters. On the
other hand, Covariance Matrix Adaptation MAP-Annealing (CMA-MAE), an evolution
strategies (ES)-based quality diversity algorithm, does not have these
limitations and has achieved state-of-the-art performance on standard QD
benchmarks. However, CMA-MAE cannot scale to modern neural network controllers
due to its quadratic complexity. We leverage efficient approximation methods in
ES to propose three new CMA-MAE variants that scale to high dimensions. Our
experiments show that the variants outperform ES-based baselines in benchmark
robotic locomotion tasks, while being comparable with or exceeding
state-of-the-art deep reinforcement learning-based quality diversity
algorithms.Comment: Source code and videos available at https://scalingcmamae.github.i
Surrogate Assisted Generation of Human-Robot Interaction Scenarios
As human-robot interaction (HRI) systems advance, so does the difficulty of
evaluating and understanding the strengths and limitations of these systems in
different environments and with different users. To this end, previous methods
have algorithmically generated diverse scenarios that reveal system failures in
a shared control teleoperation task. However, these methods require directly
evaluating generated scenarios by simulating robot policies and human actions.
The computational cost of these evaluations limits their applicability in more
complex domains. Thus, we propose augmenting scenario generation systems with
surrogate models that predict both human and robot behaviors. In the shared
control teleoperation domain and a more complex shared workspace collaboration
task, we show that surrogate assisted scenario generation efficiently
synthesizes diverse datasets of challenging scenarios. We demonstrate that
these failures are reproducible in real-world interactions.Comment: 18 pages; 9 figures; 2 table
Development and Benchmarking of Open Force Field v1.0.0, the Parsley Small Molecule Force Field
We present a methodology for defining and optimizing a general force field for classical molecular simulations, and we describe its use to derive the Open Force Field 1.0.0 small molecule force field, code-named Parsley. Rather than traditional atom-typing, our approach builds on the SMIRKS-native Open Force Field (SMIRNOFF) parameter assignment formalism, which handles increases in the diversity and specificity of the force field definition without needlessly increasing the complexity of the specification. Parameters are optimized with the ForceBalance tool, based on reference quantum chemical data that include torsion potential energy profiles, optimized gas-phase structures, and vibrational frequencies. These quantum reference data are computed and are maintained with QCArchive, an open-source and freely available distributed computing and database software ecosystem. In this initial application of the method, we present essentially a full optimization of all valence parameters and report tests of the resulting force field against compounds and data types outside the training set. These tests show improvements in optimized geometries and conformational energetics and demonstrate that Parsley\u27s accuracy for liquid properties is similar to that of other general force fields, as is accuracy on binding free energies. We find that this initial Parsley force field affords accuracy similar to that of other general force fields when used to calculate relative binding free energies spanning 199 protein-ligand systems. Additionally, the resulting infrastructure allows us to rapidly optimize an entire new force field with minimal human intervention
Development and Benchmarking of Open Force Field v1.0.0the Parsley Small-Molecule Force Field
We present a methodology for defining and optimizing a general force field for classical molecular simulations, and we describe its use to derive the Open Force Field 1.0.0 small-molecule force field, codenamed Parsley. Rather than using traditional atom typing, our approach is built on the SMIRKS-native Open Force Field (SMIRNOFF) parameter assignment formalism, which handles increases in the diversity and specificity of the force field definition without needlessly increasing the complexity of the specification. Parameters are optimized with the ForceBalance tool, based on reference quantum chemical data that include torsion potential energy profiles, optimized gas-phase structures, and vibrational frequencies. These quantum reference data are computed and are maintained with QCArchive, an open-source and freely available distributed computing and database software ecosystem. In this initial application of the method, we present essentially a full optimization of all valence parameters and report tests of the resulting force field against compounds and data types outside the training set. These tests show improvements in optimized geometries and conformational energetics and demonstrate that Parsley's accuracy for liquid properties is similar to that of other general force fields, as is accuracy on binding free energies. We find that this initial Parsley force field affords accuracy similar to that of other general force fields when used to calculate relative binding free energies spanning 199 protein-ligand systems. Additionally, the resulting infrastructure allows us to rapidly optimize an entirely new force field with minimal human intervention