Finite-horizon optimal control of linear and a class of nonlinear systems

Traditionally, optimal control of dynamical systems with known system dynamics is obtained in a backward-in-time and offline manner either by using Riccati or Hamilton-Jacobi-Bellman (HJB) equation. In contrast, in this dissertation, finite-horizon optimal regulation has been investigated for both linear and nonlinear systems in a forward-in-time manner when system dynamics are uncertain. Value and policy iterations are not used while the value function (or Q-function for linear systems) and control input are updated once a sampling interval consistent with standard adaptive control. First, the optimal adaptive control of linear discrete-time systems with unknown system dynamics is presented in Paper I by using Q-learning and Bellman equation while satisfying the terminal constraint. A novel update law that uses history information of the cost to go is derived. Paper II considers the design of the linear quadratic regulator in the presence of state and input quantization. Quantization errors are eliminated via a dynamic quantizer design and the parameter update law is redesigned from Paper I. Furthermore, an optimal adaptive state feedback controller is developed in Paper III for the general nonlinear discrete-time systems in affine form without the knowledge of system dynamics. In Paper IV, a NN-based observer is proposed to reconstruct the state vector and identify the dynamics so that the control scheme from Paper III is extended to output feedback. Finally, the optimal regulation of quantized nonlinear systems with input constraint is considered in Paper V by introducing a non-quadratic cost functional. Closed-loop stability is demonstrated for all the controller designs developed in this dissertation by using Lyapunov analysis while all the proposed schemes function in an online and forward-in-time manner so that they are practically viable --Abstract, page iv

Neural network solution for suboptimal control of non-holonomic chained form system

In this paper, we develop fixed-final time nearly optimal control laws for a class of non-holonomic chained form systems by using neural networks to approximately solve a Hamilton-Jacobi-Bellman equation. A certain time-folding method is applied to recover uniform complete controllability for the chained form system. This method requires an innovative design of a certain dynamic control component. Using this time-folding method, the chained form system is mapped into a controllable linear system for which controllers can systematically be designed to ensure exponential or asymptotic stability as well as nearly optimal performance. The result is a neural network feedback controller that has time-varying coefficients found by a priori offline tuning. The results of this paper are demonstrated in an example

Decentralized Optimal Control With Application In Power System

An output-feedback decentralized optimal controller is proposed for power systems with renewable energy penetration. Renewable energy source is modeled similar to the classical generator model and is equipped with the unified power flow controller (UPFC). The transient performance of power system is considered and stability of the dynamical states are investigated. An offline decentralized optimal controller is designed that utilizes only the local states. The network comprises conventional synchronous generators as well as renewable sources with inverter equipped with UPFC. Subsequently, the optimal decentralized controller is compared to the initial stabilizing controller used to obtain the optimal controller. An online decentralized optimal controller is designed for discrete-time system. Two neuro networks are utilized to estimate value function and optimal control strategy. Furthermore, a novel observer-based decentralized optimal controller is developed on small scale discrete-time power system. The system is trained followed by least square rules and successive approximation. Simulation results on IEEE 14-, 30-, and 118-bus power system benchmarks shows satisfactory performance of the online decentralized controller. And also, simulation results demonstrate great performance of the observer and the optimal controller compare to the centralized optimal controller

Higher-Order Methods for Determining Optimal Controls and Their Sensitivities

The solution of optimal control problems through the Hamilton-Jacobi-Bellman (HJB) equation offers guaranteed satisfaction of both the necessary and sufficient conditions for optimality. However, finding an exact solution to the HJB equation is a near impossible task for many optimal control problems. This thesis presents an approximation method for solving finite-horizon optimal control problems involving nonlinear dynamical systems. The method uses finite-order approximations of the partial derivatives of the cost-to-go function, and successive higher-order differentiations of the HJB equation. Natural byproducts of the proposed method provide sensitivities of the controls to changes in the initial states, which can be used to approximate the solution to neighboring optimal control problems. For highly nonlinear problems, the method is modified to calculate control sensitivities about a nominal trajectory. In this framework, the method is shown to provide accurate control sensitivities at much lower orders of approximation. Several numerical examples are presented to illustrate both applications of the approximation method

Approximate dynamic programming based solutions for fixed-final-time optimal control and optimal switching

Optimal solutions with neural networks (NN) based on an approximate dynamic programming (ADP) framework for new classes of engineering and non-engineering problems and associated difficulties and challenges are investigated in this dissertation. In the enclosed eight papers, the ADP framework is utilized for solving fixed-final-time problems (also called terminal control problems) and problems with switching nature. An ADP based algorithm is proposed in Paper 1 for solving fixed-final-time problems with soft terminal constraint, in which, a single neural network with a single set of weights is utilized. Paper 2 investigates fixed-final-time problems with hard terminal constraints. The optimality analysis of the ADP based algorithm for fixed-final-time problems is the subject of Paper 3, in which, it is shown that the proposed algorithm leads to the global optimal solution providing certain conditions hold. Afterwards, the developments in Papers 1 to 3 are used to tackle a more challenging class of problems, namely, optimal control of switching systems. This class of problems is divided into problems with fixed mode sequence (Papers 4 and 5) and problems with free mode sequence (Papers 6 and 7). Each of these two classes is further divided into problems with autonomous subsystems (Papers 4 and 6) and problems with controlled subsystems (Papers 5 and 7). Different ADP-based algorithms are developed and proofs of convergence of the proposed iterative algorithms are presented. Moreover, an extension to the developments is provided for online learning of the optimal switching solution for problems with modeling uncertainty in Paper 8. Each of the theoretical developments is numerically analyzed using different real-world or benchmark problems --Abstract, page v

A Series Solution Framework for Finite-time Optimal Feedback Control, H-infinity Control and Games

The Bolza-form of the finite-time constrained optimal control problem leads to the Hamilton-Jacobi-Bellman (HJB) equation with terminal boundary conditions and tobe- determined parameters. In general, it is a formidable task to obtain analytical and/or numerical solutions to the HJB equation. This dissertation presents two novel polynomial expansion methodologies for solving optimal feedback control problems for a class of polynomial nonlinear dynamical systems with terminal constraints. The first approach uses the concept of higher-order series expansion methods. Specifically, the Series Solution Method (SSM) utilizes a polynomial series expansion of the cost-to-go function with time-dependent coefficient gains that operate on the state variables and constraint Lagrange multipliers. A significant accomplishment of the dissertation is that the new approach allows for a systematic procedure to generate optimal feedback control laws that exactly satisfy various types of nonlinear terminal constraints. The second approach, based on modified Galerkin techniques for the solution of terminally constrained optimal control problems, is also developed in this dissertation. Depending on the time-interval, nonlinearity of the system, and the terminal constraints, the accuracy and the domain of convergence of the algorithm can be related to the order of truncation of the functional form of the optimal cost function. In order to limit the order of the expansion and still retain improved midcourse performance, a waypoint scheme is developed. The waypoint scheme has the dual advantages of reducing computational efforts and gain-storage requirements. This is especially true for autonomous systems. To illustrate the theoretical developments, several aerospace application-oriented examples are presented, including a minimum-fuel orbit transfer problem. Finally, the series solution method is applied to the solution of a class of partial differential equations that arise in robust control and differential games. Generally, these problems lead to the Hamilton-Jacobi-Isaacs (HJI) equation. A method is presented that allows this partial differential equation to be solved using the structured series solution approach. A detailed investigation, with several numerical examples, is presented on the Nash and Pareto-optimal nonlinear feedback solutions with a general terminal payoff. Other significant applications are also discussed for one-dimensional problems with control inequality constraints and parametric optimization