93 research outputs found
Continual Reinforcement Learning in 3D Non-stationary Environments
High-dimensional always-changing environments constitute a hard challenge for
current reinforcement learning techniques. Artificial agents, nowadays, are
often trained off-line in very static and controlled conditions in simulation
such that training observations can be thought as sampled i.i.d. from the
entire observations space. However, in real world settings, the environment is
often non-stationary and subject to unpredictable, frequent changes. In this
paper we propose and openly release CRLMaze, a new benchmark for learning
continually through reinforcement in a complex 3D non-stationary task based on
ViZDoom and subject to several environmental changes. Then, we introduce an
end-to-end model-free continual reinforcement learning strategy showing
competitive results with respect to four different baselines and not requiring
any access to additional supervised signals, previously encountered
environmental conditions or observations.Comment: Accepted in the CLVision Workshop at CVPR2020: 13 pages, 4 figures, 5
table
Efficient Parallel Reinforcement Learning Framework using the Reactor Model
Parallel Reinforcement Learning (RL) frameworks are essential for mapping RL
workloads to multiple computational resources, allowing for faster generation
of samples, estimation of values, and policy improvement. These computational
paradigms require a seamless integration of training, serving, and simulation
workloads. Existing frameworks, such as Ray, are not managing this
orchestration efficiently, especially in RL tasks that demand intensive
input/output and synchronization between actors on a single node. In this
study, we have proposed a solution implementing the reactor model, which
enforces a set of actors to have a fixed communication pattern. This allows the
scheduler to eliminate work needed for synchronization, such as acquiring and
releasing locks for each actor or sending and processing coordination-related
messages. Our framework, Lingua Franca (LF), a coordination language based on
the reactor model, also supports true parallelism in Python and provides a
unified interface that allows users to automatically generate dataflow graphs
for RL tasks. In comparison to Ray on a single-node multi-core compute
platform, LF achieves 1.21x and 11.62x higher simulation throughput in OpenAI
Gym and Atari environments, reduces the average training time of synchronized
parallel Q-learning by 31.2%, and accelerates multi-agent RL inference by
5.12x.Comment: 10 pages, 11 figure
RLgraph: Modular Computation Graphs for Deep Reinforcement Learning
Reinforcement learning (RL) tasks are challenging to implement, execute and
test due to algorithmic instability, hyper-parameter sensitivity, and
heterogeneous distributed communication patterns. We argue for the separation
of logical component composition, backend graph definition, and distributed
execution. To this end, we introduce RLgraph, a library for designing and
executing reinforcement learning tasks in both static graph and define-by-run
paradigms. The resulting implementations are robust, incrementally testable,
and yield high performance across different deep learning frameworks and
distributed backends
- …