Probabilistic models for data efficient reinforcement learning

Trial-and-error based reinforcement learning (RL) has seen rapid advancements in recent times, especially with the advent of deep neural networks. However, the standard deep learning methods often overlook the progress made in control theory by treating systems as black-box. We propose a model-based RL framework based on probabilistic Model Predictive Control (MPC). In particular, we propose to learn a probabilistic transition model using Gaussian Processes (GPs) to incorporate model uncertainty into long-term predictions, thereby, reducing the impact of model errors. We provide theoretical guarantees for first-order optimality in the GP-based transition models with deterministic approximate inference for long-term planning. We demonstrate that our approach not only achieves the state-of-the-art data efficiency, but also is a principled way for RL in constrained environments. When the true state of the dynamical system cannot be fully observed the standard model based methods cannot be directly applied. For these systems an additional step of state estimation is needed. We propose distributed message passing for state estimation in non-linear dynamical systems. In particular, we propose to use expectation propagation (EP) to iteratively refine the state estimate, i.e., the Gaussian posterior distribution on the latent state. We show two things: (a) Classical Rauch-Tung-Striebel (RTS) smoothers, such as the extended Kalman smoother (EKS) or the unscented Kalman smoother (UKS), are special cases of our message passing scheme; (b) running the message passing scheme more than once can lead to significant improvements over the classical RTS smoothers. We show the explicit connection between message passing with EP and well-known RTS smoothers and provide a practical implementation of the suggested algorithm. Furthermore, we address convergence issues of EP by generalising this framework to damped updates and the consideration of general -divergences. Probabilistic models can also be used to generate synthetic data. In model based RL we use ’synthetic’ data as a proxy to real environments and in order to achieve high data efficiency. The ability to generate high-fidelity synthetic data is crucial when available (real) data is limited as in RL or where privacy and data protection standards allow only for limited use of the given data, e.g., in medical and financial data-sets. Current state-of-the-art methods for synthetic data generation are based on generative models, such as Generative Adversarial Networks (GANs). Even though GANs have achieved remarkable results in synthetic data generation, they are often challenging to interpret. Furthermore, GAN-based methods can suffer when used with mixed real and categorical variables. Moreover, the loss function (discriminator loss) design itself is problem specific, i.e., the generative model may not be useful for tasks it was not explicitly trained for. In this paper, we propose to use a probabilistic model as a synthetic data generator. Learning the probabilistic model for the data is equivalent to estimating the density of the data. Based on the copula theory, we divide the density estimation task into two parts, i.e., estimating univariate marginals and estimating the multivariate copula density over the univariate marginals. We use normalising flows to learn both the copula density and univariate marginals. We benchmark our method on both simulated and real data-sets in terms of density estimation as well as the ability to generate high-fidelity synthetic data.Open Acces

Discrete Time Systems

Discrete-Time Systems comprehend an important and broad research field. The consolidation of digital-based computational means in the present, pushes a technological tool into the field with a tremendous impact in areas like Control, Signal Processing, Communications, System Modelling and related Applications. This book attempts to give a scope in the wide area of Discrete-Time Systems. Their contents are grouped conveniently in sections according to significant areas, namely Filtering, Fixed and Adaptive Control Systems, Stability Problems and Miscellaneous Applications. We think that the contribution of the book enlarges the field of the Discrete-Time Systems with signification in the present state-of-the-art. Despite the vertiginous advance in the field, we also believe that the topics described here allow us also to look through some main tendencies in the next years in the research area

Um estudo sobre métodos de determinação de estados e parâmetros de máquinas síncronas de polos salientes

Orientador: Mateus GiesbrechtDissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Elétrica e de ComputaçãoResumo: As máquinas síncronas de polos salientes desempenham um papel fundamental na análise de estabilidade de sistemas elétricos de potência, especialmente em países cuja maior parte da energia gerada provém de fontes hidráulicas. Os modelos elétricos equivalentes que descrevem o comportamento dessas máquinas são compostos por diversos parâmetros, os quais são utilizados em uma ampla gama de estudos. No presente trabalho, estudam-se e propõem-se técnicas de estimação de estados e parâmetros de máquinas síncronas de polos salientes. A princípio, as equações de tensão, de fluxos concatenados, de potência e de movimento são desenvolvidas com as devidas unidades de medida, tanto em variáveis de máquina quanto em variáveis projetadas sobre um plano ortogonal que gira na velocidade elétrica do rotor. Na maior parte da literatura, essas unidades não são explicitadas no equacionamento. Dentre os parâmetros elétricos dos modelos das máquinas síncronas de polos salientes, as reatâncias de magnetização são os que mais influenciam o comportamento da máquina em condições de regime permanente senoidal. Desta forma, apresenta-se uma nova abordagem à estimação do ângulo de carga dessas máquinas e o subsequente cálculo das reatâncias de magnetização a partir de condições de carga específicas -- o desempenho do método proposto é avaliado em dados de simulação e em dados reais de operação de um gerador síncrono de grande porte. Algumas abordagens à determinação de parâmetros requerem que a máquina seja posta fora de operação para que ensaios específicos possam ser realizados. Dentre eles, um dos mais empregados na determinação de parâmetros transitórios e de regime permanente é o ensaio de rejeição de carga; assim, este ensaio também é analisado e aperfeiçoado por um método automatizado de separação de soma de exponenciais baseado em projeção de variáveis. Por tratar-se de um sistema multivariável e altamente não linear, diferentes observadores de estado também são utilizados para se determinarem estados e parâmetros de máquinas síncronas em tempo hábil e com precisão satisfatória. Este trabalho apresenta uma abordagem não linear recursivamente aplicável à estimação de fluxos concatenados, correntes de enrolamentos amortecedores, ângulo de carga e reatâncias de magnetização de máquinas síncronas de polos salientes por meio da filtragem de partículas. Um modelo não linear de oitava ordem é considerado e apenas as medições realizadas nos terminais da armadura e do campo durante regime permanente se fazem necessárias para estimar as referidas grandezasAbstract: Salient-pole synchronous machines play a fundamental role in the stability analysis of electrical power systems, especially in countries where most of the generated energy comes from hydraulic sources. The electrical equivalent models that describe the behavior of these machines are composed of several electrical parameters, which are used in a wide range of studies. In the present work, techniques for estimating states and parameters of salient-pole synchronous machines are studied and proposed. A priori, the voltage, flux linkage, power, and motion equations are developed with the appropriate units included, both in machine variables and in variables projected on an orthogonal plane rotating in the rotor's electrical speed. In most of the literature, these units are not explained in the equation process. Among the electrical parameters, the magnetizing reactances are the ones that most influence the machine behavior under transient and steady-state conditions. In this way, a new approach to estimate the load angle of these machines and the subsequent calculation of the magnetizing reactances from specific load conditions are presented -- the performance of the proposed method is evaluated by means of simulation data and by operating data of a large synchronous generator. Some approaches to determine parameters require the machine to be taken out of operation, so that specific tests may be performed. Among them, one of the most used to determine transient and steady-state parameters is the load rejection test; thus, this test is also analyzed and refined by an automated method based on variable projection for separating the resulting sum-of-exponentials. Since the machines are highly nonlinear, multivariate, dynamic systems, different state observers seek to solve the state estimation problem in a timely manner and with satisfactory accuracy. This work presents a nonlinear and recursive approach for the estimation of flux linkages per second, amortisseur winding currents, load angle, and magnetizing reactances of salient-pole synchronous machines by means of the particle filtering. An eighth-order nonlinear model is considered, and only measurements taken at the machine terminals are necessary to estimate these quantitiesMestradoAutomaçãoMestre em Engenharia Elétrica162015/2018-6CNPq

Feedback Systems: An Introduction for Scientists and Engineers

This book provides an introduction to the basic principles and tools for the design and analysis of feedback systems. It is intended to serve a diverse audience of scientists and engineers who are interested in understanding and utilizing feedback in physical, biological, information and social systems.We have attempted to keep the mathematical prerequisites to a minimum while being careful not to sacrifice rigor in the process. We have also attempted to make use of examples from a variety of disciplines, illustrating the generality of many of the tools while at the same time showing how they can be applied in specific application domains. A major goal of this book is to present a concise and insightful view of the current knowledge in feedback and control systems. The field of control started by teaching everything that was known at the time and, as new knowledge was acquired, additional courses were developed to cover new techniques. A consequence of this evolution is that introductory courses have remained the same for many years, and it is often necessary to take many individual courses in order to obtain a good perspective on the field. In developing this book, we have attempted to condense the current knowledge by emphasizing fundamental concepts. We believe that it is important to understand why feedback is useful, to know the language and basic mathematics of control and to grasp the key paradigms that have been developed over the past half century. It is also important to be able to solve simple feedback problems using back-of-the-envelope techniques, to recognize fundamental limitations and difficult control problems and to have a feel for available design methods. This book was originally developed for use in an experimental course at Caltech involving students from a wide set of backgrounds. The course was offered to undergraduates at the junior and senior levels in traditional engineering disciplines, as well as first- and second-year graduate students in engineering and science. This latter group included graduate students in biology, computer science and physics. Over the course of several years, the text has been classroom tested at Caltech and at Lund University, and the feedback from many students and colleagues has been incorporated to help improve the readability and accessibility of the material. Because of its intended audience, this book is organized in a slightly unusual fashion compared to many other books on feedback and control. In particular, we introduce a number of concepts in the text that are normally reserved for second-year courses on control and hence often not available to students who are not control systems majors. This has been done at the expense of certain traditional topics, which we felt that the astute student could learn independently and are often explored through the exercises. Examples of topics that we have included are nonlinear dynamics, Lyapunov stability analysis, the matrix exponential, reachability and observability, and fundamental limits of performance and robustness. Topics that we have deemphasized include root locus techniques, lead/lag compensation and detailed rules for generating Bode and Nyquist plots by hand. Several features of the book are designed to facilitate its dual function as a basic engineering text and as an introduction for researchers in natural, information and social sciences. The bulk of the material is intended to be used regardless of the audience and covers the core principles and tools in the analysis and design of feedback systems. Advanced sections, marked by the “dangerous bend” symbol shown here, contain material that requires a slightly more technical background, of the sort that would be expected of senior undergraduates in engineering. A few sections are marked by two dangerous bend symbols and are intended for readers with more specialized backgrounds, identified at the beginning of the section. To limit the length of the text, several standard results and extensions are given in the exercises, with appropriate hints toward their solutions. To further augment the printed material contained here, a companion web site has been developed and is available from the publisher’s web page: http://press.princeton.edu/titles/8701.html The web site contains a database of frequently asked questions, supplemental examples and exercises, and lecture material for courses based on this text. The material is organized by chapter and includes a summary of the major points in the text as well as links to external resources. The web site also contains the source code for many examples in the book, as well as utilities to implement the techniques described in the text. Most of the code was originally written using MATLAB M-files but was also tested with LabView MathScript to ensure compatibility with both packages. Many files can also be run using other scripting languages such as Octave, SciLab, SysQuake and Xmath. The first half of the book focuses almost exclusively on state space control systems. We begin in Chapter 2 with a description of modeling of physical, biological and information systems using ordinary differential equations and difference equations. Chapter 3 presents a number of examples in some detail, primarily as a reference for problems that will be used throughout the text. Following this, Chapter 4 looks at the dynamic behavior of models, including definitions of stability and more complicated nonlinear behavior. We provide advanced sections in this chapter on Lyapunov stability analysis because we find that it is useful in a broad array of applications and is frequently a topic that is not introduced until later in one’s studies. The remaining three chapters of the first half of the book focus on linear systems, beginning with a description of input/output behavior in Chapter 5. In Chapter 6, we formally introduce feedback systems by demonstrating how state space control laws can be designed. This is followed in Chapter 7 by material on output feedback and estimators. Chapters 6 and 7 introduce the key concepts of reachability and observability, which give tremendous insight into the choice of actuators and sensors, whether for engineered or natural systems. The second half of the book presents material that is often considered to be from the field of “classical control.” This includes the transfer function, introduced in Chapter 8, which is a fundamental tool for understanding feedback systems. Using transfer functions, one can begin to analyze the stability of feedback systems using frequency domain analysis, including the ability to reason about the closed loop behavior of a system from its open loop characteristics. This is the subject of Chapter 9, which revolves around the Nyquist stability criterion. In Chapters 10 and 11, we again look at the design problem, focusing first on proportional-integral-derivative (PID) controllers and then on the more general process of loop shaping. PID control is by far the most common design technique in control systems and a useful tool for any student. The chapter on frequency domain design introduces many of the ideas of modern control theory, including the sensitivity function. In Chapter 12, we combine the results from the second half of the book to analyze some of the fundamental trade-offs between robustness and performance. This is also a key chapter illustrating the power of the techniques that have been developed and serving as an introduction for more advanced studies. The book is designed for use in a 10- to 15-week course in feedback systems that provides many of the key concepts needed in a variety of disciplines. For a 10-week course, Chapters 1–2, 4–6 and 8–11 can each be covered in a week’s time, with the omission of some topics from the final chapters. A more leisurely course, spread out over 14–15 weeks, could cover the entire book, with 2 weeks on modeling (Chapters 2 and 3) — particularly for students without much background in ordinary differential equations — and 2 weeks on robust performance (Chapter 12). The mathematical prerequisites for the book are modest and in keeping with our goal of providing an introduction that serves a broad audience. We assume familiarity with the basic tools of linear algebra, including matrices, vectors and eigenvalues. These are typically covered in a sophomore-level course on the subject, and the textbooks by Apostol [10], Arnold [13] and Strang [187] can serve as good references. Similarly, we assume basic knowledge of differential equations, including the concepts of homogeneous and particular solutions for linear ordinary differential equations in one variable. Apostol [10] and Boyce and DiPrima [42] cover this material well. Finally, we also make use of complex numbers and functions and, in some of the advanced sections, more detailed concepts in complex variables that are typically covered in a junior-level engineering or physics course in mathematical methods. Apostol [9] or Stewart [186] can be used for the basic material, with Ahlfors [6], Marsden and Hoffman [146] or Saff and Snider [172] being good references for the more advanced material. We have chosen not to include appendices summarizing these various topics since there are a number of good books available. One additional choice that we felt was important was the decision not to rely on a knowledge of Laplace transforms in the book. While their use is by far the most common approach to teaching feedback systems in engineering, many students in the natural and information sciences may lack the necessary mathematical background. Since Laplace transforms are not required in any essential way, we have included them only in an advanced section intended to tie things together for students with that background. Of course, we make tremendous use of transfer functions, which we introduce through the notion of response to exponential inputs, an approach we feel is more accessible to a broad array of scientists and engineers. For classes in which students have already had Laplace transforms, it should be quite natural to build on this background in the appropriate sections of the text. Acknowledgments: The authors would like to thank the many people who helped during the preparation of this book. The idea for writing this book came in part from a report on future directions in control [155] to which Stephen Boyd, Roger Brockett, John Doyle and Gunter Stein were major contributors. Kristi Morgansen and Hideo Mabuchi helped teach early versions of the course at Caltech on which much of the text is based, and Steve Waydo served as the head TA for the course taught at Caltech in 2003–2004 and provided numerous comments and corrections. Charlotta Johnsson and Anton Cervin taught from early versions of the manuscript in Lund in 2003–2007 and gave very useful feedback. Other colleagues and students who provided feedback and advice include Leif Andersson, John Carson, K. Mani Chandy, Michel Charpentier, Domitilla Del Vecchio, Kate Galloway, Per Hagander, Toivo Henningsson Perby, Joseph Hellerstein, George Hines, Tore Hägglund, Cole Lepine, Anders Rantzer, Anders Robertsson, Dawn Tilbury and Francisco Zabala. The reviewers for Princeton University Press and Tom Robbins at NI Press also provided valuable comments that significantly improved the organization, layout and focus of the book. Our editor, Vickie Kearn, was a great source of encouragement and help throughout the publishing process. Finally, we would like to thank Caltech, Lund University and the University of California at Santa Barbara for providing many resources, stimulating colleagues and students, and pleasant working environments that greatly aided in the writing of this book

Estimation and control of non-linear and hybrid systems with applications to air-to-air guidance

Issued as Progress report, and Final report, Project no. E-21-67

The implementation of a generalised robust adaptive controller

An adaptive controller is developed, comprising a robust parameter estimator and an explicit pole assignment controller design. The controller is reformulated to have a standard PID structure. A practical implementation is facilitated on a digital microcomputer, connected to a physical process. Test results are presented for this real process subject to variable dead-time and an external disturbance. Simulation results are also presented, for a nominally non minimum-phase process subject to variable dead-time and large open-loop gain changes. Robust performance is demonstrated under all of these circumstances. Recommendations are given for the choices and considerations required in a robust practical implementation. Much research has been done in the field of adaptive control over the past few decades. However, a let needs to be learned about the robustness of adaptive control algorithms. This research investigates the implementation of a practical adaptive control algorithm, with numerous features incorporated to improve the robust performance of such a controller. Parameter estimation is performed using Recursive Least Squares (RLS), with various signal conditioning filters to reduce estimator sensitivity to noise and modelling errors. The control design is based on closed-loop pole assignment, with adaptive feed forward compensation included. Further, provision is made in both the estimation model and the feedback control structure to eliminate deterministic immeasurable disturbances, and to track deterministic set point variations. This is based on the Internal Model Principle. Measured random disturbance signals are included in the estimation model, for which "transfer function" polynomial coefficients are estimated and then used in the feed forward control d e sign. A new shift- operator, namely the 6-operator, is used in all controller and estimator formulations. This has been shown to have better numerical properties and to correspond more closely to continuous-time control, than the traditional q operator of z-domain discrete control. A practical implementation on a digital computer is investigated, applied to a real plant typical of an industrial application. Simulation results are also obtained for plant with non minimum-phase zeros and variable dead-time

Self-tuning controllers via the state space

Imperial Users onl

Electromagnetic transitions as a probe for Superdeformation in 28Si

There is great interest in superdeformation in light nuclei, A < 40 region, in particular alpha-conjugate nuclei in the sd-shell. Enhanced collectivity for such light systems opens new opportunities to test nuclear-structure theories. Antisymmetrized molecular dynamics, large-scale shell model and beyond mean-field calculations which predict superdeformed structures in these regions can be validated and tuned with the aid of experimental evidence of superdeformed bands in light nuclei. A J(pi) = 6+ state at 12.865 MeV in 28Si with a measured B(E2) value of more than 25 W.u for the transition to the 10.946-MeV J(pi) = 4+ state is indicative of a highly collective transition and has been thought to form part of a candidate SD band. Measurements of in-band electromagnetic transitions are required to fully describe this proposed SD band. The CAGRA campaign is a combination of small-angle inelastic scattering with high resolution gamma-ray spectroscopy. This method preferentially populates low-spin and isoscalar natural parity states. A 12 Clover + 4 LaBr3 array was used in coincidence with the high resolution Grand Raiden spectrometer to momentum analyse inelastically scattered alpha-particles. The experiment was performed at the Research Center for Nuclear Physics (RNCP) of Osaka University, Japan. This thesis will focus on the experimental challenges, analysis and results of the 28Si(alpha,alpha') reaction at 9.1 degrees with E_beam = 130 MeV impinged on a natural Si target. The first upper limits of the in-band gamma-ray transition strength of 6.08 W.u from the J(pi) = 4+ to J(pi) = 2+ of the proposed superdeformed band in 28Si has been measured. This has the potential to constrain future theoretical predictions of superdeformation in 28Si

Orbit Estimation of Non-Cooperative Maneuvering Spacecraft

Due to the ever increasing congestion of the space environment, there is an increased demand for real-time situation awareness of all objects in space. An unknown spacecraft maneuver changes the predicted orbit, complicates tracking, and degrades estimate accuracies. Traditional orbit estimation routines are implemented, tested, and compared to a multiple model format that adaptively handles unknown maneuvers. Multiple Model Adaptive Estimation is implemented in an original way to track a non-cooperative satellite by covariance inflation and filtering-through a maneuver. Parameters for successful instantaneous maneuver reconstruction are analyzed. Variable State Dimension estimation of a continuously maneuvering spacecraft is investigated. A requirements based analysis is performed on short arc orbital solutions. Large covariance propagation of potential maneuvers is explored. Using ground-based radars, several thousand simulations are run to develop new techniques to estimate orbits during and after both instantaneous and continuous maneuvers. The new methods discovered are more accurate by a factor of 700 after only a single pass when compared to non-adaptive methods. The algorithms, tactics, and analysis complement on-going efforts to improve Space Situational Awareness and dynamic modeling

Kinematic State Estimation using Multiple DGPS/MEMS-IMU Sensors

Animals have evolved over billions of years and understanding these complex and intertwined systems have potential to advance the technology in the field of sports science, robotics and more. As such, a gait analysis using Motion Capture (MOCAP) technology is the subject of a number of research and development projects aimed at obtaining quantitative measurements. Existing MOCAP technology has limited the majority of studies to the analysis of the steady-state locomotion in a controlled (indoor) laboratory environment. MOCAP systems such as the optical, non-optical acoustic and non-optical magnetic MOCAP systems require predefined capture volume and controlled environmental conditions whilst the non-optical mechanical MOCAP system impedes the motion of the subject. Although the non-optical inertial MOCAP system allows MOCAP in an outdoor environment, it suffers from measurement noise and drift and lacks global trajectory information. The accuracy of these MOCAP systems are known to decrease during the tracking of the transient locomotion. Quantifying the manoeuvrability of animals in their natural habitat to answer the question “Why are animals so manoeuvrable?” remains a challenge. This research aims to develop an outdoor MOCAP system that will allow tracking of the steady-state as well as the transient locomotion of an animal in its natural habitat outside a controlled laboratory condition. A number of researchers have developed novel MOCAP systems with the same aim of creating an outdoor MOCAP system that is aimed at tracking the motion outside a controlled laboratory (indoor) environment with unlimited capture volume. These novel MOCAP systems are either not validated against the commercial MOCAP systems or do not have comparable sub-millimetre accuracy as the commercial MOCAP systems. The developed DGPS/MEMS-IMU multi-receiver fusion MOCAP system was assessed to have global trajectory accuracy of _0:0394m, relative limb position accuracy of _0:006497m. To conclude the research, several recommendations are made to improve the developed MOCAP system and to prepare for a field-testing with a wild animal from a family of a terrestrial megafauna