12,139 research outputs found
On the Sample Complexity of the Linear Quadratic Regulator
This paper addresses the optimal control problem known as the linear quadratic regulator in the case when the dynamics are unknown. We propose a multistage procedure, called Coarse-ID control, that estimates a model from a few experimental trials, estimates the error in that model with respect to the truth, and then designs a controller using both the model and uncertainty estimate. Our technique uses contemporary tools from random matrix theory to bound the error in the estimation procedure. We also employ a recently developed approach to control synthesis called System Level Synthesis that enables robust control design by solving a quasi-convex optimization problem. We provide end-to-end bounds on the relative error in control cost that are optimal in the number of parameters and that highlight salient properties of the system to be controlled such as closed-loop sensitivity and optimal control magnitude. We show experimentally that the Coarse-ID approach enables efficient computation of a stabilizing controller in regimes where simple control schemes that do not take the model uncertainty into account fail to stabilize the true system
On the Sample Complexity of the Linear Quadratic Gaussian Regulator
In this paper we provide direct data-driven expressions for the Linear
Quadratic Regulator (LQR), the Kalman filter, and the Linear Quadratic Gaussian
(LQG) controller using a finite dataset of noisy input, state, and output
trajectories. We show that our data-driven expressions are consistent, since
they converge as the number of experimental trajectories increases, we
characterize their convergence rate, and quantify their error as a function of
the system and data properties. These results complement the body of literature
on data-driven control and finite-sample analysis, and provide new ways to
solve canonical control and estimation problems that do not assume, nor require
the estimation of, a model of the system and noise and do not rely on solving
implicit equations.Comment: Accepted to CDC 202
On the Sample Complexity of the Linear Quadratic Regulator
This paper addresses the optimal control problem known as the linear quadratic regulator in the case when the dynamics are unknown. We propose a multistage procedure, called Coarse-ID control, that estimates a model from a few experimental trials, estimates the error in that model with respect to the truth, and then designs a controller using both the model and uncertainty estimate. Our technique uses contemporary tools from random matrix theory to bound the error in the estimation procedure. We also employ a recently developed approach to control synthesis called System Level Synthesis that enables robust control design by solving a quasi-convex optimization problem. We provide end-to-end bounds on the relative error in control cost that are optimal in the number of parameters and that highlight salient properties of the system to be controlled such as closed-loop sensitivity and optimal control magnitude. We show experimentally that the Coarse-ID approach enables efficient computation of a stabilizing controller in regimes where simple control schemes that do not take the model uncertainty into account fail to stabilize the true system
Learning Zero-Sum Linear Quadratic Games with Improved Sample Complexity
Zero-sum Linear Quadratic (LQ) games are fundamental in optimal control and
can be used (i) as a dynamic game formulation for risk-sensitive or robust
control, or (ii) as a benchmark setting for multi-agent reinforcement learning
with two competing agents in continuous state-control spaces. In contrast to
the well-studied single-agent linear quadratic regulator problem, zero-sum LQ
games entail solving a challenging nonconvex-nonconcave min-max problem with an
objective function that lacks coercivity. Recently, Zhang et al. discovered an
implicit regularization property of natural policy gradient methods which is
crucial for safety-critical control systems since it preserves the robustness
of the controller during learning. Moreover, in the model-free setting where
the knowledge of model parameters is not available, Zhang et al. proposed the
first polynomial sample complexity algorithm to reach an
-neighborhood of the Nash equilibrium while maintaining the desirable
implicit regularization property. In this work, we propose a simpler nested
Zeroth-Order (ZO) algorithm improving sample complexity by several orders of
magnitude. Our main result guarantees a
sample complexity under the same
assumptions using a single-point ZO estimator. Furthermore, when the estimator
is replaced by a two-point estimator, our method enjoys a better
sample complexity. Our key
improvements rely on a more sample-efficient nested algorithm design and finer
control of the ZO natural gradient estimation error
Sample Complexity of Data-Driven Stochastic LQR with Multiplicative Uncertainty
This paper studies the sample complexity of the stochastic Linear Quadratic
Regulator when applied to systems with multiplicative noise. We assume that the
covariance of the noise is unknown and estimate it using the sample covariance,
which results in suboptimal behaviour. The main contribution of this paper is
then to bound the suboptimality of the methodology and prove that it decreases
with 1/N, where N denotes the amount of samples. The methodology easily
generalizes to the case where the mean is unknown and to the distributionally
robust case studied in a previous work of the authors. The analysis is mostly
based on results from matrix function perturbation analysis
- …