15,903 research outputs found
A neurally plausible model learns successor representations in partially observable environments
Animals need to devise strategies to maximize returns while interacting with
their environment based on incoming noisy sensory observations. Task-relevant
states, such as the agent's location within an environment or the presence of a
predator, are often not directly observable but must be inferred using
available sensory information. Successor representations (SR) have been
proposed as a middle-ground between model-based and model-free reinforcement
learning strategies, allowing for fast value computation and rapid adaptation
to changes in the reward function or goal locations. Indeed, recent studies
suggest that features of neural responses are consistent with the SR framework.
However, it is not clear how such representations might be learned and computed
in partially observed, noisy environments. Here, we introduce a neurally
plausible model using distributional successor features, which builds on the
distributed distributional code for the representation and computation of
uncertainty, and which allows for efficient value function computation in
partially observed environments via the successor representation. We show that
distributional successor features can support reinforcement learning in noisy
environments in which direct learning of successful policies is infeasible
Count-Based Exploration with the Successor Representation
In this paper we introduce a simple approach for exploration in reinforcement
learning (RL) that allows us to develop theoretically justified algorithms in
the tabular case but that is also extendable to settings where function
approximation is required. Our approach is based on the successor
representation (SR), which was originally introduced as a representation
defining state generalization by the similarity of successor states. Here we
show that the norm of the SR, while it is being learned, can be used as a
reward bonus to incentivize exploration. In order to better understand this
transient behavior of the norm of the SR we introduce the substochastic
successor representation (SSR) and we show that it implicitly counts the number
of times each state (or feature) has been observed. We use this result to
introduce an algorithm that performs as well as some theoretically
sample-efficient approaches. Finally, we extend these ideas to a deep RL
algorithm and show that it achieves state-of-the-art performance in Atari 2600
games when in a low sample-complexity regime.Comment: This paper appears in the Proceedings of the 34th AAAI Conference on
Artificial Intelligence (AAAI 2020
Searching for rewards in graph-structured spaces
How do people generalize and explore structured spaces? We study human behavior on a multi-armed bandit task, where rewards are influenced by the connectivity structure of a graph. A detailed predictive model comparison shows that a Gaussian Process regression model using a diffusion kernel is able to best describe participant choices, and also predict judgments about expected reward and confidence. This model unifies psychological models of function learning with the Successor Representation used in reinforcement learning, thereby building a bridge between different models of generalization
Relational Representations in Reinforcement Learning: Review and Open Problems
This paper is about representation in RL.We discuss some of the concepts in representation and generalization in reinforcement learning and argue for higher-order representations, instead of the commonly used propositional representations. The paper contains a small review of current reinforcement learning systems using higher-order representations, followed by a brief discussion. The paper ends with research directions and open problems.\u
- ā¦