Near Optimal Output-Feedback Control of Nonlinear Discrete-Time Systems in Nonstrict Feedback Form with Application to Engines

A novel reinforcement-learning based output-adaptive neural network (NN) controller, also referred as the adaptive-critic NN controller, is developed to track a desired trajectory for a class of complex nonlinear discrete-time systems in the presence of bounded and unknown disturbances. The controller includes an observer for estimating states and the outputs, critic, and two action NNs for generating virtual, and actual control inputs. The critic approximates certain strategic utility function and the action NNs are used to minimize both the strategic utility function and their outputs. All NN weights adapt online towards minimization of a performance index, utilizing gradient-descent based rule. A Lyapunov function proves the uniformly ultimate boundedness (UUB) of the closed-loop tracking error, weight, and observer estimation. Separation principle and certainty equivalence principles are relaxed; persistency of excitation condition and linear in the unknown parameter assumption is not needed. The performance of this controller is evaluated on a spark ignition (SI) engine operating with high exhaust gas recirculation (EGR) levels and experimental results are demonstrated

Reinforcement Learning Based Dual-Control Methodology for Complex Nonlinear Discrete-Time Systems with Application to Spark Engine EGR Operation

A novel reinforcement-learning-based dual-control methodology adaptive neural network (NN) controller is developed to deliver a desired tracking performance for a class of complex feedback nonlinear discrete-time systems, which consists of a second-order nonlinear discrete-time system in nonstrict feedback form and an affine nonlinear discrete-time system, in the presence of bounded and unknown disturbances. For example, the exhaust gas recirculation (EGR) operation of a spark ignition (SI) engine is modeled by using such a complex nonlinear discrete-time system. A dual-controller approach is undertaken where primary adaptive critic NN controller is designed for the nonstrict feedback nonlinear discrete-time system whereas the secondary one for the affine nonlinear discrete-time system but the controllers together offer the desired performance. The primary adaptive critic NN controller includes an NN observer for estimating the states and output, an NN critic, and two action NNs for generating virtual control and actual control inputs for the nonstrict feedback nonlinear discrete-time system, whereas an additional critic NN and an action NN are included for the affine nonlinear discrete-time system by assuming the state availability. All NN weights adapt online towards minimization of a certain performance index, utilizing gradient-descent-based rule. Using Lyapunov theory, the uniformly ultimate boundedness (UUB) of the closed-loop tracking error, weight estimates, and observer estimates are shown. The adaptive critic NN controller performance is evaluated on an SI engine operating with high EGR levels where the controller objective is to reduce cyclic dispersion in heat release while minimizing fuel intake. Simulation and experimental results indicate that engine out emissions drop significantly at 20% EGR due to reduction in dispersion in heat release thus verifying the dual-control approach

Neural-Network-Based State Feedback Control of a Nonlinear Discrete-Time System in Nonstrict Feedback Form

In this paper, a suite of adaptive neural network (NN) controllers is designed to deliver a desired tracking performance for the control of an unknown, second-order, nonlinear discrete-time system expressed in nonstrict feedback form. In the first approach, two feedforward NNs are employed in the controller with tracking error as the feedback variable whereas in the adaptive critic NN architecture, three feedforward NNs are used. In the adaptive critic architecture, two action NNs produce virtual and actual control inputs, respectively, whereas the third critic NN approximates certain strategic utility function and its output is employed for tuning action NN weights in order to attain the near-optimal control action. Both the NN control methods present a well-defined controller design and the noncausal problem in discrete-time backstepping design is avoided via NN approximation. A comparison between the controller methodologies is highlighted. The stability analysis of the closed-loop control schemes is demonstrated. The NN controller schemes do not require an offline learning phase and the NN weights can be initialized at zero or random. Results show that the performance of the proposed controller schemes is highly satisfactory while meeting the closed-loop stability

Neural network control of nonstrict feedback and nonaffine nonlinear discrete-time systems with application to engine control

In this dissertation, neural networks (NN) approximate unknown nonlinear functions in the system equations, unknown control inputs, and cost functions for two different classes of nonlinear discrete-time systems. Employing NN in closed-loop feedback systems requires that weight update algorithms be stable...Controllers are developed and applied to a nonlinear, discrete-time system of equations for a spark ignition engine model to reduce the cyclic dispersion of heat release --Abstract, page iv

Reinforcement-Learning-Based Output-Feedback Control of Nonstrict Nonlinear Discrete-Time Systems with Application to Engine Emission Control

A novel reinforcement-learning-based output adaptive neural network (NN) controller, which is also referred to as the adaptive-critic NN controller, is developed to deliver the desired tracking performance for a class of nonlinear discrete-time systems expressed in nonstrict feedback form in the presence of bounded and unknown disturbances. The adaptive-critic NN controller consists of an observer, a critic, and two action NNs. The observer estimates the states and output, and the two action NNs provide virtual and actual control inputs to the nonlinear discrete-time system. The critic approximates a certain strategic utility function, and the action NNs minimize the strategic utility function and control inputs. All NN weights adapt online toward minimization of a performance index, utilizing the gradient-descent-based rule, in contrast with iteration-based adaptive-critic schemes. Lyapunov functions are used to show the stability of the closed-loop tracking error, weights, and observer estimates. Separation and certainty equivalence principles, persistency of excitation condition, and linearity in the unknown parameter assumption are not needed. Experimental results on a spark ignition (SI) engine operating lean at an equivalence ratio of 0.75 show a significant (25%) reduction in cyclic dispersion in heat release with control, while the average fuel input changes by less than 1% compared with the uncontrolled case. Consequently, oxides of nitrogen (NOx) drop by 30%, and unburned hydrocarbons drop by 16% with control. Overall, NOx\u27s are reduced by over 80% compared with stoichiometric levels

Neural Network Controller Development and Implementation for Spark Ignition Engines with High EGR Levels

Past research has shown substantial reductions in the oxides of nitrogen (NOx) concentrations by using 10% -25% exhaust gas recirculation (EGR) in spark ignition (SI) engines (see Dudek and Sain, 1989). However, under high EGR levels, the engine exhibits strong cyclic dispersion in heat release which may lead to instability and unsatisfactory performance preventing commercial engines to operate with high EGR levels. A neural network (NN)-based output feedback controller is developed to reduce cyclic variation in the heat release under high levels of EGR even when the engine dynamics are unknown by using fuel as the control input. A separate control loop was designed for controlling EGR levels. The stability analysis of the closed-loop system is given and the boundedness of the control input is demonstrated by relaxing separation principle, persistency of excitation condition, certainty equivalence principle, and linear in the unknown parameter assumptions. Online training is used for the adaptive NN and no offline training phase is needed. This online learning feature and model-free approach is used to demonstrate the applicability of the controller on a different engine with minimal effort. Simulation results demonstrate that the cyclic dispersion is reduced significantly using the proposed controller when implemented on an engine model that has been validated experimentally. For a single cylinder research engine fitted with a modern four-valve head (Ricardo engine), experimental results at 15% EGR indicate that cyclic dispersion was reduced 33% by the controller, an improvement of fuel efficiency by 2%, and a 90% drop in NOx from stoichiometric operation without EGR was observed. Moreover, unburned hydrocarbons (uHC) drop by 6% due to NN control as compared to the uncontrolled scenario due to the drop in cyclic dispersion. Similar performance was observed with the controller on a different engine

Reinforcement Learning Based Output-Feedback Control of Nonlinear Nonstrict Feedback Discrete-Time Systems with Application to Engines

A novel reinforcement-learning based output-adaptive neural network (NN) controller, also referred as the adaptive-critic NN controller, is developed to track a desired trajectory for a class of complex nonlinear discrete-time systems in the presence of bounded and unknown disturbances. The controller includes an observer for estimating states and the outputs, critic, and two action NNs for generating virtual, and actual control inputs. The critic approximates certain strategic utility function and the action NNs are used to minimize both the strategic utility function and their outputs. All NN weights adapt online towards minimization of a performance index, utilizing gradient-descent based rule. A Lyapunov function proves the uniformly ultimate boundedness (UUB) of the closed-loop tracking error, weight, and observer estimation. Separation principle and certainty equivalence principles are relaxed; persistency of excitation condition and linear in the unknown parameter assumption is not needed. The performance of this adaptive critic NN controller is evaluated through simulation with the Daw engine model in lean mode. The objective is to reduce the cyclic dispersion in heat release by using the controller

Issues on Stability of ADP Feedback Controllers for Dynamical Systems

This paper traces the development of neural-network (NN)-based feedback controllers that are derived from the principle of adaptive/approximate dynamic programming (ADP) and discusses their closed-loop stability. Different versions of NN structures in the literature, which embed mathematical mappings related to solutions of the ADP-formulated problems called “adaptive critics” or “action-critic” networks, are discussed. Distinction between the two classes of ADP applications is pointed out. Furthermore, papers in “model-free” development and model-based neurocontrollers are reviewed in terms of their contributions to stability issues. Recent literature suggests that work in ADP-based feedback controllers with assured stability is growing in diverse forms

Reinforcement-learning based output-feedback controller for nonlinear discrete-time system with application to spark ignition engines operating lean and EGR

A spark ignition (SI) engine can be described by non-strict feedback nonlinear discrete-time system with the output dependent upon on the states in a nonlinear manner. The controller developed in this thesis utilizes the inherent universal approximation property of neural networks (NN) to simplify the design process and solve the non-causality problem inherent with traditional designs. It also exploits a long-term performance index called the strategic utility function to minimize and assist in updating of the NN weights; therefore, an optimal controller can be realized. Finally, through Lyapunov equations, the controller guarantees stability --Abstract, page iv

Technological Solutions for Energy Security and Sustainability

This paper addresses the question: how can we minimize the expected time between now and the time when we achieve three measures of sustainability and security together -- independence from oil in cars and trucks, very deep reductions in greenhouse gas emissions and deep reductions in natural gas for electricity? Specific new technologies and metrics for progress are discussed, in context, linked to new information from IEEE, NSF, the State of the Future project and other sources