Near Optimal Neural Network-Based Output Feedback Control of Affine Nonlinear Discrete-Time Systems

In this paper, a novel online reinforcement learning neural network (NN)-based optimal output feedback controller, referred to as adaptive critic controller, is proposed for affine nonlinear discrete-time systems, to deliver a desired tracking performance. The adaptive critic design consist of three entities, an observer to estimate the system states, an action network that produces optimal control input and a critic that evaluates the performance of the action network. The critic is termed adaptive as it adapts itself to output the optimal cost-to-go function which is based on the standard Bellman equation. By using the Lyapunov approach, the uniformly ultimate boundedness (UUB) of the estimation and tracking errors and weight estimates is demonstrated. The effectiveness of the controller is evaluated for the task of nanomanipulation in a simulation environment

Formation control of mobile robots and unmanned aerial vehicles

In this dissertation, the nonlinear control of nonholonomic mobile robot formations and unmanned aerial vehicle (UAV) formations is undertaken and presented in six papers. In the first paper, an asymptotically stable combined kinematic/torque control law is developed for leader-follower based formation control of mobile robots using backstepping. A neural network (NN) is introduced along with robust integral of the sign of the error (RISE) feedback to approximate the dynamics of the follower as well as its leader using online weight tuning. Subsequently, in the second paper, a novel NN observer is designed to estimate the linear and angular velocities of both the follower and its leader robot and a NN output feedback control law is developed. On the other hand, in the third paper, a NN-based output feedback control law is presented for the control of an underactuated quad rotor UAV, and a NN virtual control input scheme is proposed which allows all six degrees of freedom to be controlled using only four control inputs. The results of this paper are extended to include the control of quadrotor UAV formations, and a novel three-dimensional leader-follower framework is proposed in the fourth paper. Next, in the fifth paper, the discrete-time nonlinear optimal control is undertaken using two online approximators (OLA\u27s) to solve the infinite horizon Hamilton-Jacobi-Bellman (HJB) equation forward-in-time to achieve nearly optimal regulation and tracking control. In contrast, paper six utilizes a single OLA to solve the infinite horizon HJB and Hamilton-Jacobi-Isaacs (HJI) equations forward-intime for the near optimal regulation and tracking control of continuous affine nonlinear systems. The effectiveness of the optimal tracking controllers proposed in the fifth and sixth papers are then demonstrated using nonholonomic mobile robot formation control --Abstract, page iv

Finite-horizon optimal control of linear and a class of nonlinear systems

Traditionally, optimal control of dynamical systems with known system dynamics is obtained in a backward-in-time and offline manner either by using Riccati or Hamilton-Jacobi-Bellman (HJB) equation. In contrast, in this dissertation, finite-horizon optimal regulation has been investigated for both linear and nonlinear systems in a forward-in-time manner when system dynamics are uncertain. Value and policy iterations are not used while the value function (or Q-function for linear systems) and control input are updated once a sampling interval consistent with standard adaptive control. First, the optimal adaptive control of linear discrete-time systems with unknown system dynamics is presented in Paper I by using Q-learning and Bellman equation while satisfying the terminal constraint. A novel update law that uses history information of the cost to go is derived. Paper II considers the design of the linear quadratic regulator in the presence of state and input quantization. Quantization errors are eliminated via a dynamic quantizer design and the parameter update law is redesigned from Paper I. Furthermore, an optimal adaptive state feedback controller is developed in Paper III for the general nonlinear discrete-time systems in affine form without the knowledge of system dynamics. In Paper IV, a NN-based observer is proposed to reconstruct the state vector and identify the dynamics so that the control scheme from Paper III is extended to output feedback. Finally, the optimal regulation of quantized nonlinear systems with input constraint is considered in Paper V by introducing a non-quadratic cost functional. Closed-loop stability is demonstrated for all the controller designs developed in this dissertation by using Lyapunov analysis while all the proposed schemes function in an online and forward-in-time manner so that they are practically viable --Abstract, page iv

Reinforcement Learning Based Dual-Control Methodology for Complex Nonlinear Discrete-Time Systems with Application to Spark Engine EGR Operation

A novel reinforcement-learning-based dual-control methodology adaptive neural network (NN) controller is developed to deliver a desired tracking performance for a class of complex feedback nonlinear discrete-time systems, which consists of a second-order nonlinear discrete-time system in nonstrict feedback form and an affine nonlinear discrete-time system, in the presence of bounded and unknown disturbances. For example, the exhaust gas recirculation (EGR) operation of a spark ignition (SI) engine is modeled by using such a complex nonlinear discrete-time system. A dual-controller approach is undertaken where primary adaptive critic NN controller is designed for the nonstrict feedback nonlinear discrete-time system whereas the secondary one for the affine nonlinear discrete-time system but the controllers together offer the desired performance. The primary adaptive critic NN controller includes an NN observer for estimating the states and output, an NN critic, and two action NNs for generating virtual control and actual control inputs for the nonstrict feedback nonlinear discrete-time system, whereas an additional critic NN and an action NN are included for the affine nonlinear discrete-time system by assuming the state availability. All NN weights adapt online towards minimization of a certain performance index, utilizing gradient-descent-based rule. Using Lyapunov theory, the uniformly ultimate boundedness (UUB) of the closed-loop tracking error, weight estimates, and observer estimates are shown. The adaptive critic NN controller performance is evaluated on an SI engine operating with high EGR levels where the controller objective is to reduce cyclic dispersion in heat release while minimizing fuel intake. Simulation and experimental results indicate that engine out emissions drop significantly at 20% EGR due to reduction in dispersion in heat release thus verifying the dual-control approach

Episodic Learning with Control Lyapunov Functions for Uncertain Robotic Systems

Many modern nonlinear control methods aim to endow systems with guaranteed properties, such as stability or safety, and have been successfully applied to the domain of robotics. However, model uncertainty remains a persistent challenge, weakening theoretical guarantees and causing implementation failures on physical systems. This paper develops a machine learning framework centered around Control Lyapunov Functions (CLFs) to adapt to parametric uncertainty and unmodeled dynamics in general robotic systems. Our proposed method proceeds by iteratively updating estimates of Lyapunov function derivatives and improving controllers, ultimately yielding a stabilizing quadratic program model-based controller. We validate our approach on a planar Segway simulation, demonstrating substantial performance improvements by iteratively refining on a base model-free controller

Decentralized adaptive neural network control of interconnected nonlinear dynamical systems with application to power system

Traditional nonlinear techniques cannot be directly applicable to control large scale interconnected nonlinear dynamic systems due their sheer size and unavailability of system dynamics. Therefore, in this dissertation, the decentralized adaptive neural network (NN) control of a class of nonlinear interconnected dynamic systems is introduced and its application to power systems is presented in the form of six papers. In the first paper, a new nonlinear dynamical representation in the form of a large scale interconnected system for a power network free of algebraic equations with multiple UPFCs as nonlinear controllers is presented. Then, oscillation damping for UPFCs using adaptive NN control is discussed by assuming that the system dynamics are known. Subsequently, the dynamic surface control (DSC) framework is proposed in continuous-time not only to overcome the need for the subsystem dynamics and interconnection terms, but also to relax the explosion of complexity problem normally observed in traditional backstepping. The application of DSC-based decentralized control of power system with excitation control is shown in the third paper. On the other hand, a novel adaptive NN-based decentralized controller for a class of interconnected discrete-time systems with unknown subsystem and interconnection dynamics is introduced since discrete-time is preferred for implementation. The application of the decentralized controller is shown on a power network. Next, a near optimal decentralized discrete-time controller is introduced in the fifth paper for such systems in affine form whereas the sixth paper proposes a method for obtaining the L2-gain near optimal control while keeping a tradeoff between accuracy and computational complexity. Lyapunov theory is employed to assess the stability of the controllers --Abstract, page iv