ADAPS: Autonomous Driving Via Principled Simulations

Autonomous driving has gained significant advancements in recent years. However, obtaining a robust control policy for driving remains challenging as it requires training data from a variety of scenarios, including rare situations (e.g., accidents), an effective policy architecture, and an efficient learning mechanism. We propose ADAPS for producing robust control policies for autonomous vehicles. ADAPS consists of two simulation platforms in generating and analyzing accidents to automatically produce labeled training data, and a memory-enabled hierarchical control policy. Additionally, ADAPS offers a more efficient online learning mechanism that reduces the number of iterations required in learning compared to existing methods such as DAGGER. We present both theoretical and experimental results. The latter are produced in simulated environments, where qualitative and quantitative results are generated to demonstrate the benefits of ADAPS.Comment: Accepted to ICRA201

Imperfect Knowledge and the Pitfalls of Optimal Control Monetary Policy

This paper examines the robustness characteristics of optimal control policies derived under the assumption of rational expectations to alternative models of expectations formation and uncertainty about the natural rates of interest and unemployment. We assume that agents have imperfect knowledge about the precise structure of the economy and form expectations using a forecasting model that they continuously update based on incoming data. We also allow for central bank uncertainty regarding the natural rates of interest and unemployment. We find that the optimal control policy derived under the assumption of perfect knowledge about the structure of the economy can perform poorly when knowledge is imperfect. These problems are exacerbated by natural rate uncertainty, even when the central bank's estimates of natural rates are efficient. We show that the optimal control approach can be made more robust to the presence of imperfect knowledge by deemphasizing the stabilization of real economic activity and interest rates relative to inflation in the central bank loss function. That is, robustness to the presence of imperfect knowledge about the economy provides an incentive to employ a "conservative" central banker. We then examine two types of simple monetary policy rules from the literature that have been found to be robust to model misspecification in other contexts. We find that these policies are robust to the alternative models of learning that we study and natural rate uncertainty and outperform the optimal control policy and generally perform as well as the robust optimal control policy that places less weight on stabilizing economic activity and interest rates.Rational expectations, robust Control, model uncertainty, natural rate of unemployment,natural rate of interest

Imperfect knowledge and the pitfalls of optimal control monetary policy

This paper examines the robustness characteristics of optimal control policies derived under the assumption of rational expectations to alternative models of expectations formation and uncertainty about the natural rates of interest and unemployment. We assume that agents have imperfect knowledge about the precise structure of the economy and form expectations using a forecasting model that they continuously update based on incoming data. We also allow for central bank uncertainty regarding the natural rates of interest and unemployment. We find that the optimal control policy derived under the assumption of perfect knowledge about the structure of the economy can perform poorly when knowledge is imperfect. These problems are exacerbated by natural rate uncertainty, even when the central bank's estimates of natural rates are efficient. We show that the optimal control approach can be made more robust to the presence of imperfect knowledge by deemphasizing the stabilization of real economic activity and interest rates relative to inflation in the central bank loss function. That is, robustness to the presence of imperfect knowledge about the economy provides an incentive to employ a "conservative" central banker. We then examine two types of simple monetary policy rules from the literature that have been found to be robust to model misspecification in other contexts. We find that these policies are robust to the alternative models of learning that we study and natural rate uncertainty and outperform the optimal control policy and generally perform as well as the robust optimal control policy that places less weight on stabilizing economic activity and interest rates.Rational expectations (Economic theory) ; Monetary policy ; Econometric models

Efficient Deep Learning of Robust Policies from MPC using Imitation and Tube-Guided Data Augmentation

Imitation Learning (IL) has been increasingly employed to generate computationally efficient policies from task-relevant demonstrations provided by Model Predictive Control (MPC). However, commonly employed IL methods are often data- and computationally-inefficient, as they require a large number of MPC demonstrations, resulting in long training times, and they produce policies with limited robustness to disturbances not experienced during training. In this work, we propose an IL strategy to efficiently compress a computationally expensive MPC into a Deep Neural Network (DNN) policy that is robust to previously unseen disturbances. By using a robust variant of the MPC, called Robust Tube MPC (RTMPC), and leveraging properties from the controller, we introduce a computationally-efficient Data Aggregation (DA) method that enables a significant reduction of the number of MPC demonstrations and training time required to generate a robust policy. Our approach opens the possibility of zero-shot transfer of a policy trained from a single MPC demonstration collected in a nominal domain, such as a simulation or a robot in a lab/controlled environment, to a new domain with previously-unseen bounded model errors/perturbations. Numerical and experimental evaluations performed using linear and nonlinear MPC for agile flight on a multirotor show that our method outperforms strategies commonly employed in IL (such as DAgger and DR) in terms of demonstration-efficiency, training time, and robustness to perturbations unseen during training.Comment: Under review. arXiv admin note: text overlap with arXiv:2109.0991

Episodic Bayesian Optimal Control with Unknown Randomness Distributions

Stochastic optimal control with unknown randomness distributions has been studied for a long time, encompassing robust control, distributionally robust control, and adaptive control. We propose a new episodic Bayesian approach that incorporates Bayesian learning with optimal control. In each episode, the approach learns the randomness distribution with a Bayesian posterior and subsequently solves the corresponding Bayesian average estimate of the true problem. The resulting policy is exercised during the episode, while additional data/observations of the randomness are collected to update the Bayesian posterior for the next episode. We show that the resulting episodic value functions and policies converge almost surely to their optimal counterparts of the true problem if the parametrized model of the randomness distribution is correctly specified. We further show that the asymptotic convergence rate of the episodic value functions is of the order $O(N^{-1/2})$. We develop an efficient computational method based on stochastic dual dynamic programming for a class of problems that have convex value functions. Our numerical results on a classical inventory control problem verify the theoretical convergence results and demonstrate the effectiveness of the proposed computational method