An Efficient Off-Policy Reinforcement Learning Algorithm for the Continuous-Time LQR Problem

In this paper, an off-policy reinforcement learning algorithm is designed to solve the continuous-time LQR problem using only input-state data measured from the system. Different from other algorithms in the literature, we propose the use of a specific persistently exciting input as the exploration signal during the data collection step. We then show that, using this persistently excited data, the solution of the matrix equation in our algorithm is guaranteed to exist and to be unique at every iteration. Convergence of the algorithm to the optimal control input is also proven. Moreover, we formulate the policy evaluation step as the solution of a Sylvester-transpose equation, which increases the efficiency of its solution. Finally, a method to determine a stabilizing policy to initialize the algorithm using only measured data is proposed.Comment: 7 page

An iterative Newton\u27s method for output-feedback LQR design for large-scale systems with guaranteed convergence

The paper proposes a novel iterative output-feedback control design procedure, with necessary and sufficient stability conditions, for linear time-invariant systems within the linear quadratic regulator (LQR) framework. The proposed iterative method has a guaranteed convergence from an initial Lyapunov matrix, obtained for any stabilizing state-feedback gain, to a stabilizing output-feedback solution. Another contribution of the proposed method is that it is computationally much more tractable then algorithms in the literature, since it solves only a Lyapunov equation at each iteration step. Therefore, the proposed algorithm succeed in high dimensional problems where other, state-of-the-art methods fails. Finally, numerical examples illustrate the effectiveness of the proposed method

Novel Results on Output-Feedback LQR Design

This paper provides novel developments in output-feedback stabilization for linear time-invariant systems within the linear quadratic regulator (LQR) framework. First, we derive the necessary and sufficient conditions for output-feedback stabilizability in connection with the LQR framework. Then, we propose a novel iterative Newton\u27s method for output-feedback LQR design and a computationally efficient modified approach that requires solving only a Lyapunov equation\ua0at each iteration step. We show that the proposed modified approach guarantees convergence from a stabilizing state-feedback to a stabilizing output-feedback solution and succeeds in solving high dimensional problems where other, state-of-the-art methods, fail. Finally, numerical examples illustrate the effectiveness of the proposed methods

Acoustic source localization : exploring theory and practice

Over the past few decades, noise pollution became an important issue in modern society. This has led to an increased effort in the industry to reduce noise. Acoustic source localization methods determine the location and strength of the vibrations which are the cause of sound based onmeasurements of the sound field. This thesis describes a theoretical study of many facets of the acoustic source localization problem as well as the development, implementation and validation of new source localization methods. The main objective is to increase the range of applications of inverse acoustics and to develop accurate and computationally efficient methods for each of these applications. Four applications are considered. Firstly, the inverse acoustic problem is considered where the source and the measurement points are located on two parallel planes. A new fast method to solve this problem is developed and it is compared to the existing method planar nearfield acoustic holography (PNAH) from a theoretical point of view, as well as by means of simulations and experiments. Both methods are fast but the newmethod yields more robust and accurate results. Secondly, measurements in inverse acoustics are often point-by-point or full array measurements. However a straightforward and cost-effective alternative to these approaches is a sensor or array which moves through the sound field during the measurement to gather sound field information. The same numerical techniques make it possible to apply inverse acoustics to the case where the source moves and the sensors are fixed in space. It is shown that the inverse methods such as the inverse boundary element method (IBEM) can be applied to this problem. To arrive at an accurate representation of the sound field, an optimized signal processing method is applied and it is shown experimentally that this method leads to accurate results. Thirdly, a theoretical framework is established for the inverse acoustical problem where the sound field and the source are represented by a cross-spectral matrix. This problem is important in inverse acoustics because it occurs in the inverse calculation of sound intensity. The existing methods for this problem are analyzed from a theoretical point of view using this framework and a new method is derived from it. A simulation study indicates that the new method improves the results by 30% in some cases and the results are similar otherwise. Finally, the localization of point sources in the acoustic near field is considered. MUltiple SIgnal Classification (MUSIC) is newly applied to the Boundary element method (BEM) for this purpose. It is shown that this approach makes it possible to localize point sources accurately even if the noise level is extremely high or if the number of sensors is low