Global convergence of a new hybrid Gauss-Newton structured BFGS method for nonlinear least squares problems

2009-2010 > Academic research: refereed > Publication in refereed journalVersion of RecordPublishe

Development of Adaptive and Factorized Neural Models for MPC of Industrial Systems

Many industrial processes have non-linear and time-varying dynamics, for which the control and optimization require further investigations. Adaptive modelling techniques using radial basis function (RBF) networks often provide competitive modelling performances but encounter slow recovery speed when processes operating regions are shifted largely. In addition, RBF networks based model predictive control results as a non-linear programming problem, which restricts the application to fast dynamic systems. To these targets, the thesis presents the development of adaptive and factorized RBF network models. Model predictive control (MPC) based on the factorized RBF model is applied to a non-linear proton exchange membrane fuel cell (PEMFC) stack system. The main contents include three parts: RBF model adaptation; model factorization and fast long-range prediction; and MPC for the PEMFC stack system. The adaptive RBF model employs the recursive orthogonal least squares (ROLS) algorithm for both structure and parameter adaptation. In decomposing the regression matrix of the RBF model, the R matrix is obtained. Principles for adding centres and pruning centres are developed based on the manipulation of the R matrix. While the modelling accuracy is remained, the developed structure adaptation algorithm ensures the model size to be kept to the minimum. At the same time, the RBF model parameters are optimized in terms of minimum Frobenius norm of the model prediction error. A simulation example is used to evaluate the developed adaptive RBF model, and the model performance in output prediction is superior over the existing methods. Considering that a model with fast long-range prediction is needed for the MPC of fast dynamic systems, a f-step factorization algorithm is developed for the RBF model. The model structure is re-arranged so that the unknown future process outputs are not required for output prediction. Therefore, the accumulative error caused by recursive calculation in normal neural network model is avoided. Furthermore, as the information for output prediction is explicitly divided into the past information and the future information, the optimization of the control variable in the MPC based on this developed factorized model can be solved much faster than the normal NARX-RBF model. The developed model adaptation algorithm can be applied to this f-step factorized model to achieve fast and adaptive model prediction. Finally, the developed factorized RBF model is applied to the MPC of a PEMFC stack system with a popular industrial benchmark model in Simulink developed at Michigan University. The optimization algorithms for quadratic and non-linear system without and with constraints are presented and discussed for application purpose in the NMPC. Simulation results confirm the effectiveness of the developed model in both smooth tracking performance and less optimization time used. Conclusions and further work are given at the end of the thesis. Major contributions of the research have been outlined and achievements are checked against the objectives assigned. Further work is also suggested to extend the developed work to industrial applications in real-time simulation. This is to further examine the effectiveness of developed models. Extensive investigations are also recommended on the optimization problems to improve the existing algorithms

A stochastic quasi Newton method for molecular simulations

In this thesis the Langevin equation with a space-dependent alternative mobility matrix has been considered. Simulations of a complex molecular system with many different length and time scales based on the fundamental equations of motion take a very long simulation time before capturing the functional and relevant motions. This problem is called critical slowing down. To avoid this problem multi-scale simulation methods are applied, which permits the use of different size time steps and thus enables acceleration of the relevant (slow) movements. The aim of this thesis is to develop a stochastic quasi Newton method, such that by incorporating multi-scaling the relevant motions are effectively taken with larger time step. Due to the integration of the slow motions with a larger time step, critical slowing down can be avoided. The proposed stochastic quasi Newton method enables automatic multi-scaling in the Langevin dynamics and contributes to efficient calculation of the noise term. The construction of the proposed method also enables the construction of a limited memory version for the mobility. This results in a method where less storage is needed. Together with the reduction of the computation time and the multi-scaling property, a powerful method for molecular simulations has been provided.UBL - phd migration 201

Adam through a Second-Order Lens

Research into optimisation for deep learning is characterised by a tension between the computational efficiency of first-order, gradient-based methods (such as SGD and Adam) and the theoretical efficiency of second-order, curvature-based methods (such as quasi-Newton methods and K-FAC). We seek to combine the benefits of both approaches into a single computationally-efficient algorithm. Noting that second-order methods often depend on stabilising heuristics (such as Levenberg-Marquardt damping), we propose AdamQLR: an optimiser combining damping and learning rate selection techniques from K-FAC (Martens and Grosse, 2015) with the update directions proposed by Adam, inspired by considering Adam through a second-order lens. We evaluate AdamQLR on a range of regression and classification tasks at various scales, achieving competitive generalisation performance vs runtime.Comment: 28 pages, 15 figures, 4 tables. Submitted to ICLR 202