Reinforcement Learning Based Dual-Control Methodology for Complex Nonlinear Discrete-Time Systems with Application to Spark Engine EGR Operation

A novel reinforcement-learning-based dual-control methodology adaptive neural network (NN) controller is developed to deliver a desired tracking performance for a class of complex feedback nonlinear discrete-time systems, which consists of a second-order nonlinear discrete-time system in nonstrict feedback form and an affine nonlinear discrete-time system, in the presence of bounded and unknown disturbances. For example, the exhaust gas recirculation (EGR) operation of a spark ignition (SI) engine is modeled by using such a complex nonlinear discrete-time system. A dual-controller approach is undertaken where primary adaptive critic NN controller is designed for the nonstrict feedback nonlinear discrete-time system whereas the secondary one for the affine nonlinear discrete-time system but the controllers together offer the desired performance. The primary adaptive critic NN controller includes an NN observer for estimating the states and output, an NN critic, and two action NNs for generating virtual control and actual control inputs for the nonstrict feedback nonlinear discrete-time system, whereas an additional critic NN and an action NN are included for the affine nonlinear discrete-time system by assuming the state availability. All NN weights adapt online towards minimization of a certain performance index, utilizing gradient-descent-based rule. Using Lyapunov theory, the uniformly ultimate boundedness (UUB) of the closed-loop tracking error, weight estimates, and observer estimates are shown. The adaptive critic NN controller performance is evaluated on an SI engine operating with high EGR levels where the controller objective is to reduce cyclic dispersion in heat release while minimizing fuel intake. Simulation and experimental results indicate that engine out emissions drop significantly at 20% EGR due to reduction in dispersion in heat release thus verifying the dual-control approach

Decentralized Optimal Control With Application In Power System

An output-feedback decentralized optimal controller is proposed for power systems with renewable energy penetration. Renewable energy source is modeled similar to the classical generator model and is equipped with the unified power flow controller (UPFC). The transient performance of power system is considered and stability of the dynamical states are investigated. An offline decentralized optimal controller is designed that utilizes only the local states. The network comprises conventional synchronous generators as well as renewable sources with inverter equipped with UPFC. Subsequently, the optimal decentralized controller is compared to the initial stabilizing controller used to obtain the optimal controller. An online decentralized optimal controller is designed for discrete-time system. Two neuro networks are utilized to estimate value function and optimal control strategy. Furthermore, a novel observer-based decentralized optimal controller is developed on small scale discrete-time power system. The system is trained followed by least square rules and successive approximation. Simulation results on IEEE 14-, 30-, and 118-bus power system benchmarks shows satisfactory performance of the online decentralized controller. And also, simulation results demonstrate great performance of the observer and the optimal controller compare to the centralized optimal controller

Disturbance observer-based neural network control of cooperative multiple manipulators with input saturation

In this paper, the complex problems of internal forces and position control are studied simultaneously and a disturbance observer-based radial basis function neural network (RBFNN) control scheme is proposed to: 1) estimate the unknown parameters accurately; 2) approximate the disturbance experienced by the system due to input saturation; and 3) simultaneously improve the robustness of the system. More specifically, the proposed scheme utilizes disturbance observers, neural network (NN) collaborative control with an adaptive law, and full state feedback. Utilizing Lyapunov stability principles, it is shown that semiglobally uniformly bounded stability is guaranteed for all controlled signals of the closed-loop system. The effectiveness of the proposed controller as predicted by the theoretical analysis is verified by comparative experimental studies

Adaptive dynamic programming with eligibility traces and complexity reduction of high-dimensional systems

This dissertation investigates the application of a variety of computational intelligence techniques, particularly clustering and adaptive dynamic programming (ADP) designs especially heuristic dynamic programming (HDP) and dual heuristic programming (DHP). Moreover, a one-step temporal-difference (TD(0)) and n-step TD (TD(λ)) with their gradients are utilized as learning algorithms to train and online-adapt the families of ADP. The dissertation is organized into seven papers. The first paper demonstrates the robustness of model order reduction (MOR) for simulating complex dynamical systems. Agglomerative hierarchical clustering based on performance evaluation is introduced for MOR. This method computes the reduced order denominator of the transfer function by clustering system poles in a hierarchical dendrogram. Several numerical examples of reducing techniques are taken from the literature to compare with our work. In the second paper, a HDP is combined with the Dyna algorithm for path planning. The third paper uses DHP with an eligibility trace parameter (λ) to track a reference trajectory under uncertainties for a nonholonomic mobile robot by using a first-order Sugeno fuzzy neural network structure for the critic and actor networks. In the fourth and fifth papers, a stability analysis for a model-free action-dependent HDP(λ) is demonstrated with batch- and online-implementation learning, respectively. The sixth work combines two different gradient prediction levels of critic networks. In this work, we provide a convergence proofs. The seventh paper develops a two-hybrid recurrent fuzzy neural network structures for both critic and actor networks. They use a novel n-step gradient temporal-difference (gradient of TD(λ)) of an advanced ADP algorithm called value-gradient learning (VGL(λ)), and convergence proofs are given. Furthermore, the seventh paper is the first to combine the single network adaptive critic with VGL(λ). --Abstract, page iv