1,633 research outputs found
Machine Learning for Fluid Mechanics
The field of fluid mechanics is rapidly advancing, driven by unprecedented
volumes of data from field measurements, experiments and large-scale
simulations at multiple spatiotemporal scales. Machine learning offers a wealth
of techniques to extract information from data that could be translated into
knowledge about the underlying fluid mechanics. Moreover, machine learning
algorithms can augment domain knowledge and automate tasks related to flow
control and optimization. This article presents an overview of past history,
current developments, and emerging opportunities of machine learning for fluid
mechanics. It outlines fundamental machine learning methodologies and discusses
their uses for understanding, modeling, optimizing, and controlling fluid
flows. The strengths and limitations of these methods are addressed from the
perspective of scientific inquiry that considers data as an inherent part of
modeling, experimentation, and simulation. Machine learning provides a powerful
information processing framework that can enrich, and possibly even transform,
current lines of fluid mechanics research and industrial applications.Comment: To appear in the Annual Reviews of Fluid Mechanics, 202
Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
Evolution strategies (ES) are a family of black-box optimization algorithms
able to train deep neural networks roughly as well as Q-learning and policy
gradient methods on challenging deep reinforcement learning (RL) problems, but
are much faster (e.g. hours vs. days) because they parallelize better. However,
many RL problems require directed exploration because they have reward
functions that are sparse or deceptive (i.e. contain local optima), and it is
unknown how to encourage such exploration with ES. Here we show that algorithms
that have been invented to promote directed exploration in small-scale evolved
neural networks via populations of exploring agents, specifically novelty
search (NS) and quality diversity (QD) algorithms, can be hybridized with ES to
improve its performance on sparse or deceptive deep RL tasks, while retaining
scalability. Our experiments confirm that the resultant new algorithms, NS-ES
and two QD algorithms, NSR-ES and NSRA-ES, avoid local optima encountered by ES
to achieve higher performance on Atari and simulated robots learning to walk
around a deceptive trap. This paper thus introduces a family of fast, scalable
algorithms for reinforcement learning that are capable of directed exploration.
It also adds this new family of exploration algorithms to the RL toolbox and
raises the interesting possibility that analogous algorithms with multiple
simultaneous paths of exploration might also combine well with existing RL
algorithms outside ES
Evolutionary Reinforcement Learning: A Survey
Reinforcement learning (RL) is a machine learning approach that trains agents
to maximize cumulative rewards through interactions with environments. The
integration of RL with deep learning has recently resulted in impressive
achievements in a wide range of challenging tasks, including board games,
arcade games, and robot control. Despite these successes, there remain several
crucial challenges, including brittle convergence properties caused by
sensitive hyperparameters, difficulties in temporal credit assignment with long
time horizons and sparse rewards, a lack of diverse exploration, especially in
continuous search space scenarios, difficulties in credit assignment in
multi-agent reinforcement learning, and conflicting objectives for rewards.
Evolutionary computation (EC), which maintains a population of learning agents,
has demonstrated promising performance in addressing these limitations. This
article presents a comprehensive survey of state-of-the-art methods for
integrating EC into RL, referred to as evolutionary reinforcement learning
(EvoRL). We categorize EvoRL methods according to key research fields in RL,
including hyperparameter optimization, policy search, exploration, reward
shaping, meta-RL, and multi-objective RL. We then discuss future research
directions in terms of efficient methods, benchmarks, and scalable platforms.
This survey serves as a resource for researchers and practitioners interested
in the field of EvoRL, highlighting the important challenges and opportunities
for future research. With the help of this survey, researchers and
practitioners can develop more efficient methods and tailored benchmarks for
EvoRL, further advancing this promising cross-disciplinary research field
ES-ENAS: Blackbox Optimization over Hybrid Spaces via Combinatorial and Continuous Evolution
We consider the problem of efficient blackbox optimization over a large
hybrid search space, consisting of a mixture of a high dimensional continuous
space and a complex combinatorial space. Such examples arise commonly in
evolutionary computation, but also more recently, neuroevolution and
architecture search for Reinforcement Learning (RL) policies. Unfortunately
however, previous mutation-based approaches suffer in high dimensional
continuous spaces both theoretically and practically. We thus instead propose
ES-ENAS, a simple joint optimization procedure by combining Evolutionary
Strategies (ES) and combinatorial optimization techniques in a highly scalable
and intuitive way, inspired by the one-shot or supernet paradigm introduced in
Efficient Neural Architecture Search (ENAS). Through this relatively simple
marriage between two different lines of research, we are able to gain the best
of both worlds, and empirically demonstrate our approach by optimizing BBOB
functions over hybrid spaces as well as combinatorial neural network
architectures via edge pruning and quantization on popular RL benchmarks. Due
to the modularity of the algorithm, we also are able incorporate a wide variety
of popular techniques ranging from use of different continuous and
combinatorial optimizers, as well as constrained optimization.Comment: 22 pages. See
https://github.com/google-research/google-research/tree/master/es_enas for
associated cod
Bridging adaptive management and reinforcement learning for more robust decisions
From out-competing grandmasters in chess to informing high-stakes healthcare
decisions, emerging methods from artificial intelligence are increasingly
capable of making complex and strategic decisions in diverse, high-dimensional,
and uncertain situations. But can these methods help us devise robust
strategies for managing environmental systems under great uncertainty? Here we
explore how reinforcement learning, a subfield of artificial intelligence,
approaches decision problems through a lens similar to adaptive environmental
management: learning through experience to gradually improve decisions with
updated knowledge. We review where reinforcement learning (RL) holds promise
for improving evidence-informed adaptive management decisions even when
classical optimization methods are intractable. For example, model-free deep RL
might help identify quantitative decision strategies even when models are
nonidentifiable. Finally, we discuss technical and social issues that arise
when applying reinforcement learning to adaptive management problems in the
environmental domain. Our synthesis suggests that environmental management and
computer science can learn from one another about the practices, promises, and
perils of experience-based decision-making.Comment: In press at Philosophical Transactions of the Royal Society
Text Generation with Efficient (Soft) Q-Learning
Maximum likelihood estimation (MLE) is the predominant algorithm for training
text generation models. This paradigm relies on direct supervision examples,
which is not applicable to many applications, such as generating adversarial
attacks or generating prompts to control language models. Reinforcement
learning (RL) on the other hand offers a more flexible solution by allowing
users to plug in arbitrary task metrics as reward. Yet previous RL algorithms
for text generation, such as policy gradient (on-policy RL) and Q-learning
(off-policy RL), are often notoriously inefficient or unstable to train due to
the large sequence space and the sparse reward received only at the end of
sequences. In this paper, we introduce a new RL formulation for text generation
from the soft Q-learning perspective. It further enables us to draw from the
latest RL advances, such as path consistency learning, to combine the best of
on-/off-policy updates, and learn effectively from sparse reward. We apply the
approach to a wide range of tasks, including learning from noisy/negative
examples, adversarial attacks, and prompt generation. Experiments show our
approach consistently outperforms both task-specialized algorithms and the
previous RL methods. On standard supervised tasks where MLE prevails, our
approach also achieves competitive performance and stability by training text
generation from scratch.Comment: Code available at
https://github.com/HanGuo97/soft-Q-learning-for-text-generatio
Meta-Learning in Neural Networks: A Survey
The field of meta-learning, or learning-to-learn, has seen a dramatic rise in
interest in recent years. Contrary to conventional approaches to AI where tasks
are solved from scratch using a fixed learning algorithm, meta-learning aims to
improve the learning algorithm itself, given the experience of multiple
learning episodes. This paradigm provides an opportunity to tackle many
conventional challenges of deep learning, including data and computation
bottlenecks, as well as generalization. This survey describes the contemporary
meta-learning landscape. We first discuss definitions of meta-learning and
position it with respect to related fields, such as transfer learning and
hyperparameter optimization. We then propose a new taxonomy that provides a
more comprehensive breakdown of the space of meta-learning methods today. We
survey promising applications and successes of meta-learning such as few-shot
learning and reinforcement learning. Finally, we discuss outstanding challenges
and promising areas for future research
- …