Adaptive Critic Designs

We discuss a variety of adaptive critic designs (ACDs) for neurocontrol. These are suitable for learning in noisy, nonlinear, and nonstationary environments. They have common roots as generalizations of dynamic programming for neural reinforcement learning approaches. Our discussion of these origins leads to an explanation of three design families: heuristic dynamic programming, dual heuristic programming, and globalized dual heuristic programming (GDHP). The main emphasis is on DHP and GDHP as advanced ACDs. We suggest two new modifications of the original GDHP design that are currently the only working implementations of GDHP. They promise to be useful for many engineering applications in the areas of optimization and optimal control. Based on one of these modifications, we present a unified approach to all ACDs. This leads to a generalized training procedure for ACD

Stable Adaptive Control Using New Critic Designs

Classical adaptive control proves total-system stability for control of linear plants, but only for plants meeting very restrictive assumptions. Approximate Dynamic Programming (ADP) has the potential, in principle, to ensure stability without such tight restrictions. It also offers nonlinear and neural extensions for optimal control, with empirically supported links to what is seen in the brain. However, the relevant ADP methods in use today -- TD, HDP, DHP, GDHP -- and the Galerkin-based versions of these all have serious limitations when used here as parallel distributed real-time learning systems; either they do not possess quadratic unconditional stability (to be defined) or they lead to incorrect results in the stochastic case. (ADAC or Q-learning designs do not help.) After explaining these conclusions, this paper describes new ADP designs which overcome these limitations. It also addresses the Generalized Moving Target problem, a common family of static optimization problems, and describes a way to stabilize large-scale economic equilibrium models, such as the old long-term energy model of DOE.Comment: Includes general reviews of alternative control technologies and reinforcement learning. 4 figs, >70p., >200 eqs. Implementation details, stability analysis. Included in 9/24/98 patent disclosure. pdf version uploaded 2012, based on direct conversion of the original word/html file, because of issues of format compatabilit

Dynamic Re-Optimization of a Fed-Batch Fermentor using Adaptive Critic Designs

Traditionally, fed-batch biochemical process optimization and control uses complicated off-line optimizers, with no online model adaptation or re-optimization. This study demonstrates the applicability of a class of adaptive critic designs for online re-optimization and control of an aerobic fed-batch fermentor. Specifically, the performance of an entire class of adaptive critic designs, viz., heuristic dynamic programming, dual heuristic programming and generalized dual heuristic programming, was demonstrated to be superior to that of a heuristic random optimizer, on optimization of a fed-batch fermentor operation producing monoclonal antibodie

Choice of Utility Functions for Adaptive Critic Designs

This paper first presents a general overview of Adaptive Critic Designs (ACDs) and their existing control applications. It describes the importance of the right choice of utility functions for the development of critic networks and their convergence to the cost-to-go function J. A closer look into the step by step derivation of a utility function for the design of an ACD nonlinear optimal neurocontroller to replace/augment the conventional controllers, the automatic voltage regulator and governor, in a power system consisting of a generator connected to the power grid is described and some results are presented

Online Reinforcement Learning-Based Neural Network Controller Design for Affine Nonlinear Discrete-Time Systems

In this paper, a novel reinforcement learning neural network (NN)-based controller, referred to adaptive critic controller, is proposed for general multi-input and multi- output affine unknown nonlinear discrete-time systems in the presence of bounded disturbances. Adaptive critic designs consist of two entities, an action network that produces optimal solution and a critic that evaluates the performance of the action network. The critic is termed adaptive as it adapts itself to output the optimal cost-to-go function and the action network is adapted simultaneously based on the information from the critic. In our online learning method, one NN is designated as the critic NN, which approximates the Bellman equation. An action NN is employed to derive the control signal to track a desired system trajectory while minimizing the cost function. Online updating weight tuning schemes for these two NNs are also derived and uniformly ultimate boundedness (UUB) of the tracking error and weight estimates is shown. The effectiveness of the controller is evaluated on a two-link robotic arm system

Adaptive Critic Designs for Optimal Control of Power Systems

The increasing complexity of the modern power grid highlights the need for advanced modeling and control techniques for effective control of excitation, turbine and flexible AC transmission systems (FACTS). The crucial factors affecting the modern power systems today is voltage and load flow control. Simulation studies in the PSCAD/EMTDC environment and realtime laboratory experimental studies carried out are described and the results show the successful control of the power system elements and the entire power system with adaptive and optimal neurocontrol schemes. Performances of the neurocontrollers are compared with the conventional PI controllers for damping under different operating conditions for small and large disturbances

Fully Evolvable Optimal Neurofuzzy Controller Using Adaptive Critic Designs

A near-optimal neurofuzzy external controller is designed in this paper for a static compensator (STATCOM) in a multimachine power system. The controller provides an auxiliary reference signal for the STATCOM in such a way that it improves the damping of the rotor speed deviations of its neighboring generators. A zero-order Takagi-Sugeno fuzzy rule base constitutes the core of the controller. A heuristic dynamic programming (HDP) based approach is used to further train the controller and enable it to provide nonlinear near-optimal control at different operating conditions of the power system. Based on the connectionist systems theory, the parameters of the neurofuzzy controller, including the membership functions, undergo training. Simulation results are provided that compare the performance of the neurofuzzy controller with and without updating the fuzzy set parameters. Simulation results indicate that updating the membership functions can noticeably improve the performance of the controller and reduce the size of the STATCOM, which leads to lower capital investment

Adaptive Critic Designs Based Coupled Neurocontrollers for a Static Compensator

A novel nonlinear optimal neurocontroller for a static compensator (STATCOM) connected to a power system, using artificial neural networks, is presented in this paper. The heuristic dynamic programming (HDP) method, a member of the adaptive critic designs (ACD) family, is used for the design of the STATCOM neurocontroller. The proposed controller is a nonlinear optimal controller that provides coupled control for the line voltage and the dc link voltage regulation loops of the STATCOM. An action dependent approach is used, in which the controller is independent of a model of the network. Moreover, a proportional-integrator approach allows the neurocontroller to deal with the actual signals rather than the deviations. Simulation results are provided to show that the proposed ACD based neurocontroller is more effective in controlling the STATCOM compared to finely tuned conventional PI controllers

Adaptive Critic Designs and Their Implementations on Different Neural Network Architectures

The design of nonlinear optimal neurocontrollers based on the Adaptive Critic Designs (ACDs) family of algorithms has recently attracted interest. This paper presents a summary of these algorithms, and compares their performance when implemented on two different types of artificial neural networks, namely the multilayer perceptron neural network (MLPNN) and the radial basis function neural network (RBFNN). As an example for the application of the ACDs, the control of synchronous generator on an electric power grid is considered and results are presented to compare the different ACD family members and their implementations on different neural network architectures

Optimal Control Synthesis of a Class of Nonlinear Systems Using Single Network Adaptive Critics

Adaptive critic (AC) neural network solutions to optimal control designs using dynamic programming has reduced the need of complex computations and storage requirements that typical dynamic programming requires. In this paper, a single network adaptive critic (SNAC) is presented. This approach is applicable to a class of nonlinear systems where the optimal control (stationary) equation is explicitly solvable for control in terms of state and costate variables. The SNAC architecture offers three potential advantages; a simpler architecture, significant savings of computational load and reduction in approximation errors. In order to demonstrate these benefits, a real-life micro-electro-mechanical-system (MEMS) problem has been solved. This demonstrates that the SNAC technique is applicable for complex engineering systems. Both AC and SNAC approaches are compared in terms of some metrics