42 research outputs found
A Novel Confidence-Based Algorithm for Structured Bandits
We study finite-armed stochastic bandits where the rewards of each arm might
be correlated to those of other arms. We introduce a novel phased algorithm
that exploits the given structure to build confidence sets over the parameters
of the true bandit problem and rapidly discard all sub-optimal arms. In
particular, unlike standard bandit algorithms with no structure, we show that
the number of times a suboptimal arm is selected may actually be reduced thanks
to the information collected by pulling other arms. Furthermore, we show that,
in some structures, the regret of an anytime extension of our algorithm is
uniformly bounded over time. For these constant-regret structures, we also
derive a matching lower bound. Finally, we demonstrate numerically that our
approach better exploits certain structures than existing methods.Comment: AISTATS 202
Sequential Transfer in Reinforcement Learning with a Generative Model
We are interested in how to design reinforcement learning agents that
provably reduce the sample complexity for learning new tasks by transferring
knowledge from previously-solved ones. The availability of solutions to related
problems poses a fundamental trade-off: whether to seek policies that are
expected to achieve high (yet sub-optimal) performance in the new task
immediately or whether to seek information to quickly identify an optimal
solution, potentially at the cost of poor initial behavior. In this work, we
focus on the second objective when the agent has access to a generative model
of state-action pairs. First, given a set of solved tasks containing an
approximation of the target one, we design an algorithm that quickly identifies
an accurate solution by seeking the state-action pairs that are most
informative for this purpose. We derive PAC bounds on its sample complexity
which clearly demonstrate the benefits of using this kind of prior knowledge.
Then, we show how to learn these approximate tasks sequentially by reducing our
transfer setting to a hidden Markov model and employing spectral methods to
recover its parameters. Finally, we empirically verify our theoretical findings
in simple simulated domains.Comment: ICML 202
Active Coverage for PAC Reinforcement Learning
Collecting and leveraging data with good coverage properties plays a crucial
role in different aspects of reinforcement learning (RL), including reward-free
exploration and offline learning. However, the notion of "good coverage" really
depends on the application at hand, as data suitable for one context may not be
so for another. In this paper, we formalize the problem of active coverage in
episodic Markov decision processes (MDPs), where the goal is to interact with
the environment so as to fulfill given sampling requirements. This framework is
sufficiently flexible to specify any desired coverage property, making it
applicable to any problem that involves online exploration. Our main
contribution is an instance-dependent lower bound on the sample complexity of
active coverage and a simple game-theoretic algorithm, CovGame, that nearly
matches it. We then show that CovGame can be used as a building block to solve
different PAC RL tasks. In particular, we obtain a simple algorithm for PAC
reward-free exploration with an instance-dependent sample complexity that, in
certain MDPs which are "easy to explore", is lower than the minimax one. By
further coupling this exploration algorithm with a new technique to do implicit
eliminations in policy space, we obtain a computationally-efficient algorithm
for best-policy identification whose instance-dependent sample complexity
scales with gaps between policy values.Comment: Accepted at COLT 202
Towards Instance-Optimality in Online PAC Reinforcement Learning
Several recent works have proposed instance-dependent upper bounds on the
number of episodes needed to identify, with probability , an
-optimal policy in finite-horizon tabular Markov Decision
Processes (MDPs). These upper bounds feature various complexity measures for
the MDP, which are defined based on different notions of sub-optimality gaps.
However, as of now, no lower bound has been established to assess the
optimality of any of these complexity measures, except for the special case of
MDPs with deterministic transitions. In this paper, we propose the first
instance-dependent lower bound on the sample complexity required for the PAC
identification of a near-optimal policy in any tabular episodic MDP.
Additionally, we demonstrate that the sample complexity of the PEDEL algorithm
of \cite{Wagenmaker22linearMDP} closely approaches this lower bound.
Considering the intractability of PEDEL, we formulate an open question
regarding the possibility of achieving our lower bound using a
computationally-efficient algorithm
Gradient-Aware Model-based Policy Search
Traditional model-based reinforcement learning approaches learn a model of
the environment dynamics without explicitly considering how it will be used by
the agent. In the presence of misspecified model classes, this can lead to poor
estimates, as some relevant available information is ignored. In this paper, we
introduce a novel model-based policy search approach that exploits the
knowledge of the current agent policy to learn an approximate transition model,
focusing on the portions of the environment that are most relevant for policy
improvement. We leverage a weighting scheme, derived from the minimization of
the error on the model-based policy gradient estimator, in order to define a
suitable objective function that is optimized for learning the approximate
transition model. Then, we integrate this procedure into a batch policy
improvement algorithm, named Gradient-Aware Model-based Policy Search (GAMPS),
which iteratively learns a transition model and uses it, together with the
collected trajectories, to compute the new policy parameters. Finally, we
empirically validate GAMPS on benchmark domains analyzing and discussing its
properties
Reinforcement Learning in Linear MDPs: Constant Regret and Representation Selection
International audienceWe study the role of the representation of state-action value functions in regret minimization in finite-horizon Markov Decision Processes (MDPs) with linear structure. We first derive a necessary condition on the representation, called universally spanning optimal features (UNISOFT), to achieve constant regret in any MDP with linear reward function. This result encompasses the well-known settings of low-rank MDPs and, more generally, zero inherent Bellman error (also known as the Bellman closure assumption). We then demonstrate that this condition is also sufficient for these classes of problems by deriving a constant regret bound for two optimistic algorithms (LSVI-UCB and ELEANOR). Finally, we propose an algorithm for representation selection and we prove that it achieves constant regret when one of the given representations, or a suitable combination of them, satisfies the UNISOFT condition
Sequential Transfer in Reinforcement Learning with a Generative Model
We are interested in how to design reinforcement learning agents that provably reduce the sample complexity for learning new tasks by transferring knowledge from previously-solved ones. The availability of solutions to related problems poses a fundamental trade-off: whether to seek policies that are expected to immediately achieve high (yet sub-optimal) performance in the new task or whether to seek information to quickly identify an optimal solution, potentially at the cost of poor initial behaviour. In this work, we focus on the second objective when the agent has access to a generative model of state-action pairs. First, given a set of solved tasks containing an approximation of the target one, we design an algorithm that quickly identifies an accurate solution by seeking the state-action pairs that are most informative for this purpose. We derive PAC bounds on its sample complexity which clearly demonstrate the benefits of using this kind of prior knowledge. Then, we show how to learn these approximate tasks sequentially by reducing our transfer setting to a hidden Markov model and employing spectral methods to recover its parameters. Finally, we empirically verify our theoretical findings in simple simulated domains
An Asymptotically Optimal Primal-Dual Incremental Algorithm for Contextual Linear Bandits
In the contextual linear bandit setting, algorithms built on the optimism principle fail to exploit the structure of the problem and have been shown to be asymptotically suboptimal. In this paper, we follow recent approaches of deriving asymptotically optimal algorithms from problem-dependent regret lower bounds and we introduce a novel algorithm improving over the state-of-the-art along multiple dimensions. We build on a reformulation of the lower bound, where context distribution and exploration policy are decoupled, and we obtain an algorithm robust to unbalanced context distributions. Then, using an incremental primal-dual approach to solve the Lagrangian relaxation of the lower bound, we obtain a scalable and computationally efficient algorithm. Finally, we remove forced exploration and build on confidence intervals of the optimization problem to encourage a minimum level of exploration that is better adapted to the problem structure. We demonstrate the asymptotic optimality of our algorithm, while providing both problem-dependent and worst-case finite-time regret guarantees. Our bounds scale with the logarithm of the number of arms, thus avoiding the linear dependence common in all related prior works. Notably, we establish minimax optimality for any learning horizon in the special case of non-contextual linear bandits. Finally, we verify that our algorithm obtains better empirical performance than state-of-the-art baselines
Adversarial Inverse Reinforcement Learning with Changing Dynamics
Most work on inverse reinforcement learning, the problem of recovering the unknown reward function being optimized by a decision-making agent, has focused on cases where optimal
demonstrations are provided under single dynamics. We analyze the more general settings where the learner has access to sub-optimal demonstrations under several different dynamics.
We argue that several problems, such as learning under covariate shift or risk aversion, can be modeled in this way.
We propose an adversarial formulation where the learner tries to imitate a constrained, worst-case estimate of the demonstrator’s control policy. We adopt the method of Lagrange multipliers to remove the constraints and produce a convex optimization problem.
We prove that the constraints imposed by the multiple dynamics lead to an NP-Hard optimization subproblem, the computation of a deterministic policy maximizing the total expected reward from several different Markov decision processes. We propose a tractable approximation by reducing the latter to the optimal control of partially observable Markov decision processes.
We show the performance of our algorithm on two synthetic data problems. In the first one, we try to recover the reward function of a randomly generated Markov decision process, while in the second we try to rationalize a robot navigating through a grid and demonstrating goal-directed behavior