Adaptive dynamic programming with eligibility traces and complexity reduction of high-dimensional systems

This dissertation investigates the application of a variety of computational intelligence techniques, particularly clustering and adaptive dynamic programming (ADP) designs especially heuristic dynamic programming (HDP) and dual heuristic programming (DHP). Moreover, a one-step temporal-difference (TD(0)) and n-step TD (TD(λ)) with their gradients are utilized as learning algorithms to train and online-adapt the families of ADP. The dissertation is organized into seven papers. The first paper demonstrates the robustness of model order reduction (MOR) for simulating complex dynamical systems. Agglomerative hierarchical clustering based on performance evaluation is introduced for MOR. This method computes the reduced order denominator of the transfer function by clustering system poles in a hierarchical dendrogram. Several numerical examples of reducing techniques are taken from the literature to compare with our work. In the second paper, a HDP is combined with the Dyna algorithm for path planning. The third paper uses DHP with an eligibility trace parameter (λ) to track a reference trajectory under uncertainties for a nonholonomic mobile robot by using a first-order Sugeno fuzzy neural network structure for the critic and actor networks. In the fourth and fifth papers, a stability analysis for a model-free action-dependent HDP(λ) is demonstrated with batch- and online-implementation learning, respectively. The sixth work combines two different gradient prediction levels of critic networks. In this work, we provide a convergence proofs. The seventh paper develops a two-hybrid recurrent fuzzy neural network structures for both critic and actor networks. They use a novel n-step gradient temporal-difference (gradient of TD(λ)) of an advanced ADP algorithm called value-gradient learning (VGL(λ)), and convergence proofs are given. Furthermore, the seventh paper is the first to combine the single network adaptive critic with VGL(λ). --Abstract, page iv

Model Order Reduction based on Agglomerative Hierarchical Clustering

This paper presents an improved method for reducing high-order dynamical system models via clustering. Agglomerative hierarchical clustering based on performance evaluation (HC-PE) is introduced for model order reduction. This method computes the reduced order denominator of the transfer function model by clustering system poles in a hierarchical dendrogram. The base layer represents an nth order system, which is used to calculate each successive layer to reduce the model order until finally reaching a second-order system. HC-PE uses a mean-squared error (MSE) in every reduced order, which modifies the pole placement process. The coefficients for the numerator of the reduced model are calculated by using the Padé approximation (PA) or alternatively a genetic algorithm (GA). Several numerical examples of reducing techniques are taken from the literature to compare with HC-PE. Two classes of results are shown in this paper. The first sets are single-input single-output models that range from simple models to 48th order systems. The second sets of experiments are with a multi-input multioutput model. We demonstrate the best performance for HC-PE through minimum MSEs compared with other methods. Furthermore, the robustness of HC-PE combined with PA or GA is confirmed by evaluating the third-order reduced model for the triple-link inverted pendulum model by adding a disturbance impulse signal and by changing model parameters. The relevant stability proofs are provided in Appendixes A and B in the supplementary material. HC-PE with PA slightly outperforms its performance with GA, but both approaches are attractive alternatives to other published methods