Reinforcement Learning Based Output-feedback Controller for Complex Nonlinear Discrete-Time Systems

A novel reinforcement-learning based output-adaptive neural network (NN) controller, also referred as the adaptive-critic NN controller, is developed to track a desired trajectory for a class of complex feedback nonlinear discrete-time systems in the presence of bounded and unknown disturbances. This nonlinear discrete-time system consists of a second order system in nonstrict form and an affine nonlinear discrete-time system tightly coupled together. Two adaptive critic NN controllers are designed - primary one for the nonstrict system and the secondary one for the affine system. A Lyapunov function shows the uniformly ultimate boundedness (UUB) of the closed-loop tracking error, weight estimates and observer estimates. Separation principle and certainty equivalence principles are relaxed, persistency of excitation condition is not required and linear in the unknown parameter assumption is not needed. The performance of this controller is evaluated on a spark ignition (SI) engine operating with high exhaust gas recirculation (EGR) levels where the objective is to reduce cyclic dispersion in heat release

Reinforcement Learning Based Dual-Control Methodology for Complex Nonlinear Discrete-Time Systems with Application to Spark Engine EGR Operation

A novel reinforcement-learning-based dual-control methodology adaptive neural network (NN) controller is developed to deliver a desired tracking performance for a class of complex feedback nonlinear discrete-time systems, which consists of a second-order nonlinear discrete-time system in nonstrict feedback form and an affine nonlinear discrete-time system, in the presence of bounded and unknown disturbances. For example, the exhaust gas recirculation (EGR) operation of a spark ignition (SI) engine is modeled by using such a complex nonlinear discrete-time system. A dual-controller approach is undertaken where primary adaptive critic NN controller is designed for the nonstrict feedback nonlinear discrete-time system whereas the secondary one for the affine nonlinear discrete-time system but the controllers together offer the desired performance. The primary adaptive critic NN controller includes an NN observer for estimating the states and output, an NN critic, and two action NNs for generating virtual control and actual control inputs for the nonstrict feedback nonlinear discrete-time system, whereas an additional critic NN and an action NN are included for the affine nonlinear discrete-time system by assuming the state availability. All NN weights adapt online towards minimization of a certain performance index, utilizing gradient-descent-based rule. Using Lyapunov theory, the uniformly ultimate boundedness (UUB) of the closed-loop tracking error, weight estimates, and observer estimates are shown. The adaptive critic NN controller performance is evaluated on an SI engine operating with high EGR levels where the controller objective is to reduce cyclic dispersion in heat release while minimizing fuel intake. Simulation and experimental results indicate that engine out emissions drop significantly at 20% EGR due to reduction in dispersion in heat release thus verifying the dual-control approach

Near Optimal Output-Feedback Control of Nonlinear Discrete-Time Systems in Nonstrict Feedback Form with Application to Engines

A novel reinforcement-learning based output-adaptive neural network (NN) controller, also referred as the adaptive-critic NN controller, is developed to track a desired trajectory for a class of complex nonlinear discrete-time systems in the presence of bounded and unknown disturbances. The controller includes an observer for estimating states and the outputs, critic, and two action NNs for generating virtual, and actual control inputs. The critic approximates certain strategic utility function and the action NNs are used to minimize both the strategic utility function and their outputs. All NN weights adapt online towards minimization of a performance index, utilizing gradient-descent based rule. A Lyapunov function proves the uniformly ultimate boundedness (UUB) of the closed-loop tracking error, weight, and observer estimation. Separation principle and certainty equivalence principles are relaxed; persistency of excitation condition and linear in the unknown parameter assumption is not needed. The performance of this controller is evaluated on a spark ignition (SI) engine operating with high exhaust gas recirculation (EGR) levels and experimental results are demonstrated

Reinforcement Learning-Based Output Feedback Control of Nonlinear Systems with Input Constraints

A novel neural network (NN) -based output feedback controller with magnitude constraints is designed to deliver a desired tracking performance for a class of multi-input-multi-output (MIMO) discrete-time strict feedback nonlinear systems. Reinforcement learning in discrete time is proposed for the output feedback controller, which uses three NN: 1) a NN observer to estimate the system states with the input-output data; 2) a critic NN to approximate certain strategic utility function; and 3) an action NN to minimize both the strategic utility function and the unknown dynamics estimation errors. The magnitude constraints are manifested as saturation nonlinearities in the output feedback controller design. Using the Lyapunov approach, the uniformly ultimate boundedness (UUB) of the state estimation errors, the tracking errors and weight estimates is shown

Reinforcement-Learning-Based Output-Feedback Control of Nonstrict Nonlinear Discrete-Time Systems with Application to Engine Emission Control

A novel reinforcement-learning-based output adaptive neural network (NN) controller, which is also referred to as the adaptive-critic NN controller, is developed to deliver the desired tracking performance for a class of nonlinear discrete-time systems expressed in nonstrict feedback form in the presence of bounded and unknown disturbances. The adaptive-critic NN controller consists of an observer, a critic, and two action NNs. The observer estimates the states and output, and the two action NNs provide virtual and actual control inputs to the nonlinear discrete-time system. The critic approximates a certain strategic utility function, and the action NNs minimize the strategic utility function and control inputs. All NN weights adapt online toward minimization of a performance index, utilizing the gradient-descent-based rule, in contrast with iteration-based adaptive-critic schemes. Lyapunov functions are used to show the stability of the closed-loop tracking error, weights, and observer estimates. Separation and certainty equivalence principles, persistency of excitation condition, and linearity in the unknown parameter assumption are not needed. Experimental results on a spark ignition (SI) engine operating lean at an equivalence ratio of 0.75 show a significant (25%) reduction in cyclic dispersion in heat release with control, while the average fuel input changes by less than 1% compared with the uncontrolled case. Consequently, oxides of nitrogen (NOx) drop by 30%, and unburned hydrocarbons drop by 16% with control. Overall, NOx\u27s are reduced by over 80% compared with stoichiometric levels

Neural Network Control of Spark Ignition Engines with High EGR Levels

Research has shown substantial reductions in the oxides of nitrogen (NOx) concentrations by using 10% to 25% exhaust gas recirculation (EGR) in spark ignition (SI) engines [1]. However under high EGR levels the engine exhibits strong cyclic dispersion in heat release which may lead to instability and unsatisfactory performance. A suite of neural network (NN)-based output feedback controllers with and without reinforcement learning is developed to control the SI engine at high levels of EGR even when the engine dynamics are unknown by using fuel as the control input. A separate control loop was designed for controlling EGR levels. The neural network controllers consists of three NN: a) A NN observer to estimate the states of the engine such as total fuel and air; b) a second NN for generating virtual input; and c) a third NN for generating actual control input. For reinforcement learning, an additional NN is used as the critic. The stability analysis of the closed loop system is given and the boundedness of all signals is ensured without separation principle. Online training is used for the adaptive NN and no offline training phase is needed. Experimental results obtained by testing the controller on a research engine indicate an 80% drop of NOx from stoichiometric levels using 10% EGR. Moreover, unburned hydrocarbons drop by 25% due to NN control as compared to the uncontrolled scenario

Neural network control of nonstrict feedback and nonaffine nonlinear discrete-time systems with application to engine control

In this dissertation, neural networks (NN) approximate unknown nonlinear functions in the system equations, unknown control inputs, and cost functions for two different classes of nonlinear discrete-time systems. Employing NN in closed-loop feedback systems requires that weight update algorithms be stable...Controllers are developed and applied to a nonlinear, discrete-time system of equations for a spark ignition engine model to reduce the cyclic dispersion of heat release --Abstract, page iv

Output Feedback Controller for Operation of Spark Ignition Engines at Lean Conditions Using Neural Networks

Spark ignition (SI) engines operating at very lean conditions demonstrate significant nonlinear behavior by exhibiting cycle-to-cycle bifurcation of heat release. Past literature suggests that operating an engine under such lean conditions can significantly reduce NO emissions by as much as 30% and improve fuel efficiency by as much as 5%-10%. At lean conditions, the heat release per engine cycle is not close to constant, as it is when these engines operate under stoichiometric conditions where the equivalence ratio is 1.0. A neural network controller employing output feedback has shown ability in simulation to reduce the nonlinear cyclic dispersion observed under lean operating conditions. This neural network (NN) output controller consists of three NNs: a) an NN observer to estimate the states of the engine such as total fuel and air; b) a second NN for generating virtual input; and c) a third NN for generating actual control input. The uniform ultimate boundedness of all closed-loop signals is demonstrated by using the Lyapunov analysis without using the separation principle. Persistency of the excitation condition, the certainty equivalence principle, and the linearity in the unknown parameter assumptions are also relaxed. The controller is implemented for a research engine as a program running on an embeddable PC that communicates with the engine through a custom hardware interface, and the results are similar to those observed in simulation. Experimental results at an equivalence ratio of 0.77 show a drop in NO emissions by around 98% from stoichiometric levels with an improvement of fuel efficiency by 5%. A 30% drop in unburned hydrocarbons from uncontrolled case is observed at this equivalence ratio of 0.77. Similar performance was observed with the controller on a different engine

Reinforcement-learning based output-feedback controller for nonlinear discrete-time system with application to spark ignition engines operating lean and EGR

A spark ignition (SI) engine can be described by non-strict feedback nonlinear discrete-time system with the output dependent upon on the states in a nonlinear manner. The controller developed in this thesis utilizes the inherent universal approximation property of neural networks (NN) to simplify the design process and solve the non-causality problem inherent with traditional designs. It also exploits a long-term performance index called the strategic utility function to minimize and assist in updating of the NN weights; therefore, an optimal controller can be realized. Finally, through Lyapunov equations, the controller guarantees stability --Abstract, page iv

An Iterative Learning Control Design Method for Nonlinear Discrete-Time Systems with Unknown Iteration-Varying Parameters and Control Direction

An iterative learning control (ILC) scheme is designed for a class of nonlinear discrete-time dynamical systems with unknown iteration-varying parameters and control direction. The iteration-varying parameters are described by a high-order internal model (HOIM) such that the unknown parameters in the current iteration are a linear combination of the counterparts in the previous certain iterations. Under the framework of ILC, the learning convergence condition is derived through rigorous analysis. It is shown that the adaptive ILC law can achieve perfect tracking of system state in presence of iteration-varying parameters and unknown control direction. The effectiveness of the proposed control scheme is verified by simulations