Reinforcement Learning Based Dual-Control Methodology for Complex Nonlinear Discrete-Time Systems with Application to Spark Engine EGR Operation

A novel reinforcement-learning-based dual-control methodology adaptive neural network (NN) controller is developed to deliver a desired tracking performance for a class of complex feedback nonlinear discrete-time systems, which consists of a second-order nonlinear discrete-time system in nonstrict feedback form and an affine nonlinear discrete-time system, in the presence of bounded and unknown disturbances. For example, the exhaust gas recirculation (EGR) operation of a spark ignition (SI) engine is modeled by using such a complex nonlinear discrete-time system. A dual-controller approach is undertaken where primary adaptive critic NN controller is designed for the nonstrict feedback nonlinear discrete-time system whereas the secondary one for the affine nonlinear discrete-time system but the controllers together offer the desired performance. The primary adaptive critic NN controller includes an NN observer for estimating the states and output, an NN critic, and two action NNs for generating virtual control and actual control inputs for the nonstrict feedback nonlinear discrete-time system, whereas an additional critic NN and an action NN are included for the affine nonlinear discrete-time system by assuming the state availability. All NN weights adapt online towards minimization of a certain performance index, utilizing gradient-descent-based rule. Using Lyapunov theory, the uniformly ultimate boundedness (UUB) of the closed-loop tracking error, weight estimates, and observer estimates are shown. The adaptive critic NN controller performance is evaluated on an SI engine operating with high EGR levels where the controller objective is to reduce cyclic dispersion in heat release while minimizing fuel intake. Simulation and experimental results indicate that engine out emissions drop significantly at 20% EGR due to reduction in dispersion in heat release thus verifying the dual-control approach

Composite learning backstepping control with guaranteed exponential stability and robustness

Adaptive backstepping control provides a feasible solution to achieve asymptotic tracking for mismatched uncertain nonlinear systems. However, input-to-state stability depends on high-gain feedback generated by nonlinear damping terms, and closed-loop exponential stability with parameter convergence involves a stringent condition named persistent excitation (PE). This paper proposes a composite learning backstepping control (CLBC) strategy based on modular backstepping and high-order tuners to compensate for the transient process of parameter estimation and achieve closed-loop exponential stability without the nonlinear damping terms and the PE condition. A novel composite learning mechanism that maximizes the staged exciting strength is designed for parameter estimation, such that parameter convergence can be achieved under a condition of interval excitation (IE) or even partial IE that is strictly weaker than PE. An extra prediction error is employed in the adaptive law to ensure the transient performance without nonlinear damping terms. The exponential stability of the closed-loop system is proved rigorously under the partial IE or IE condition. Simulations have demonstrated the effectiveness and superiority of the proposed method in both parameter estimation and control compared to state-of-the-art methods

Adaptive Controller Algorithm for 2-DOF Humanoid Robot Arm

AbstractA computational model of human motor control for a nonlinear 2 degrees-of-freedom (DOF) robot arm to mimic humanlike behavior is developed and presented in this paper. The model is based on a simple mathematical model of a 2-segment compound pendulum which mimics the human upper arm and forearm. Using the Lagrangian and Euler-Lagrange equations, the 2-DOF dynamic equations were successfully derived and solved using Euler's method. Two types of controllers; a feedback Proportional-Derivative (PD) controller and a feedforward controller, were combined into the model. The algorithm exhibited learning of the necessary torque required in performing the desired Position Control via Specific Trajectory (PCST) rehabilitative task via feedback control and using it as the feedforward torque in subsequent trial motions. After 30 trials, the mean absolute error with respect to the desired motion of the upper arm, showed a decrease from 0.09533 to 0.005859, and the forearm motion from 0.3526 to 0.006138. This decrement trend in mean absolute errorwith increase in number of trials is consistent with the adaptive control strategy of the human arm known as the Feedback Error Learning (FEL) strategy

Optimal Adaptive Output Regulation of Uncertain Nonlinear Discrete-Time Systems using Lifelong Concurrent Learning

This Paper Addresses Neural Network (NN) based Optimal Adaptive Regulation of Uncertain Nonlinear Discrete-Time Systems in Affine Form using Output Feedback Via Lifelong Concurrent Learning. First, an Adaptive NN Observer is Introduced to Estimate Both the State Vector and Control Coefficient Matrix, and its NN Weights Are Adjusted using Both Output Error and Concurrent Learning Term to Relax the Persistency Excitation (PE) Condition. Next, by Utilizing an Actor-Critic Framework for Estimating the Value Functional and Control Policy, the Critic Network Weights Are Tuned Via Both Temporal Different Error and Concurrent Learning Schemes through a Replay Buffer. the Actor NN Weights Are Tuned using Control Policy Errors. to Attain Lifelong Learning for Performing Effectively during Multiple Tasks, an Elastic Weight Consolidation Term is Added to the Critic NN Weight Tuning Law. the State Estimation, Regulation, and the Weight Estimation Errors of the Observer, Actor and Critic NNs Are Demonstrated to Be Bounded When Performing Tasks by using Lyapunov Analysis. Simulation Results Are Carried Out to Verify the Effectiveness of the Proposed Approach on a Vander Pol Oscillator. Finally, Extension to Optimal Tracking is Given Briefly

Robust online adaptive neural network control for the regulation of treadmill exercises

The paper proposes a robust online adaptive neural network control scheme for an automated treadmill system. The proposed control scheme is based on Feedback-Error Learning Approach (FELA), by using which the plant Jacobian calculation problem is avoided. Modification of the learning algorithm is proposed to solve the overtraining issue, guaranteeing to system stability and system convergence. As an adaptive neural network controller can adapt itself to deal with system uncertainties and external disturbances, this scheme is very suitable for treadmill exercise regulation when the model of the exerciser is unknown or inaccurate. In this study, exercise intensity (measured by heart rate) is regulated by simultaneously manipulating both treadmill speed and gradient in order to achieve fast tracking for which a single input multi output (SIMO) adaptive neural network controller has been designed. Real-time experiment result confirms that robust performance for nonlinear multivariable system under model uncertainties and unknown external disturbances can indeed be achieved. © 2011 IEEE

Neural-Network-Based State Feedback Control of a Nonlinear Discrete-Time System in Nonstrict Feedback Form

In this paper, a suite of adaptive neural network (NN) controllers is designed to deliver a desired tracking performance for the control of an unknown, second-order, nonlinear discrete-time system expressed in nonstrict feedback form. In the first approach, two feedforward NNs are employed in the controller with tracking error as the feedback variable whereas in the adaptive critic NN architecture, three feedforward NNs are used. In the adaptive critic architecture, two action NNs produce virtual and actual control inputs, respectively, whereas the third critic NN approximates certain strategic utility function and its output is employed for tuning action NN weights in order to attain the near-optimal control action. Both the NN control methods present a well-defined controller design and the noncausal problem in discrete-time backstepping design is avoided via NN approximation. A comparison between the controller methodologies is highlighted. The stability analysis of the closed-loop control schemes is demonstrated. The NN controller schemes do not require an offline learning phase and the NN weights can be initialized at zero or random. Results show that the performance of the proposed controller schemes is highly satisfactory while meeting the closed-loop stability

Output regulation of Euler-Lagrange systems based on error and velocity feedback

Based on a certainty equivalence property, we propose an adaptive internal model control law that solves global robust output regulation of uncertain Euler-Lagrange (EL) systems based only on error (or relative position) and velocity feedback. The proposed controller does not require apriori knowledge of reference signal and its derivatives, which are commonly assumed in literature. It enables a self-learning mechanism of the closed-loop EL systems where the adaptive internal model-based controller is able to learn the desired trajectories and adapt itself to the uncertain plant parameters. Furthermore, the analysis offers insights to the design of internal model-based output regulation for multivariable nonlinear systems with uniform vector relative degree two

Composite adaptive locally weighted learning control for multi-constraint nonlinear systems

A composite adaptive locally weighted learning (LWL) control approach is proposed for a class of uncertain nonlinear systems with system constraints, including state constraints and asymmetric control saturation in this paper. The system constraints are tackled by considering the control input as an extended state variable and introducing barrier Lyapunov functions (BLFs) into the backstepping procedure. The system uncertainty is approximated by a composite adaptive LWL neural networks (NNs), where a prediction error is constructed via a series-parallel identification model, and NN weights are updated by both the tracking error and the prediction error. The update law with composite error feedback improves uncertainty approximation accuracy and trajectory tracking accuracy. The feasibility and effectiveness of the proposed approach have been demonstrated by formal proof and simulation results

Adaptive Deep Learning for High-Dimensional Hamilton-Jacobi-Bellman Equations

Computing optimal feedback controls for nonlinear systems generally requires solving Hamilton-Jacobi-Bellman (HJB) equations, which are notoriously difficult when the state dimension is large. Existing strategies for high-dimensional problems often rely on specific, restrictive problem structures, or are valid only locally around some nominal trajectory. In this paper, we propose a data-driven method to approximate semi-global solutions to HJB equations for general high-dimensional nonlinear systems and compute candidate optimal feedback controls in real-time. To accomplish this, we model solutions to HJB equations with neural networks (NNs) trained on data generated without discretizing the state space. Training is made more effective and data-efficient by leveraging the known physics of the problem and using the partially-trained NN to aid in adaptive data generation. We demonstrate the effectiveness of our method by learning solutions to HJB equations corresponding to the attitude control of a six-dimensional nonlinear rigid body, and nonlinear systems of dimension up to 30 arising from the stabilization of a Burgers'-type partial differential equation. The trained NNs are then used for real-time feedback control of these systems.Comment: Added section on validation error computation. Updated convergence test formula and associated result