ARCHITECTURE OPTIMIZATION, TRAINING CONVERGENCE AND NETWORK ESTIMATION ROBUSTNESS OF A FULLY CONNECTED RECURRENT NEURAL NETWORK

Recurrent neural networks (RNN) have been rapidly developed in recent years. Applications of RNN can be found in system identification, optimization, image processing, pattern reorganization, classification, clustering, memory association, etc. In this study, an optimized RNN is proposed to model nonlinear dynamical systems. A fully connected RNN is developed first which is modified from a fully forward connected neural network (FFCNN) by accommodating recurrent connections among its hidden neurons. In addition, a destructive structure optimization algorithm is applied and the extended Kalman filter (EKF) is adopted as a network\u27s training algorithm. These two algorithms can seamlessly work together to generate the optimized RNN. The enhancement of the modeling performance of the optimized network comes from three parts: 1) its prototype - the FFCNN has advantages over multilayer perceptron network (MLP), the most widely used network, in terms of modeling accuracy and generalization ability; 2) the recurrency in RNN network make it more capable of modeling non-linear dynamical systems; and 3) the structure optimization algorithm further improves RNN\u27s modeling performance in generalization ability and robustness. Performance studies of the proposed network are highlighted in training convergence and robustness. For the training convergence study, the Lyapunov method is used to adapt some training parameters to guarantee the training convergence, while the maximum likelihood method is used to estimate some other parameters to accelerate the training process. In addition, robustness analysis is conducted to develop a robustness measure considering uncertainties propagation through RNN via unscented transform. Two case studies, the modeling of a benchmark non-linear dynamical system and a tool wear progression in hard turning, are carried out to testify the development in this dissertation. The work detailed in this dissertation focuses on the creation of: (1) a new method to prove/guarantee the training convergence of RNN, and (2) a new method to quantify the robustness of RNN using uncertainty propagation analysis. With the proposed study, RNN and related algorithms are developed to model nonlinear dynamical system which can benefit modeling applications such as the condition monitoring studies in terms of robustness and accuracy in the future

A Recurrent Neural Network Enhanced Unscented Kalman Filter for Human Motion Prediction

This paper presents a deep learning enhanced adaptive unscented Kalman filter (UKF) for predicting human arm motion in the context of manufacturing. Unlike previous network-based methods that solely rely on captured human motion data, which is represented as bone vectors in this paper, we incorporate a human arm dynamic model into the motion prediction algorithm and use the UKF to iteratively forecast human arm motions. Specifically, a Lagrangian-mechanics-based physical model is employed to correlate arm motions with associated muscle forces. Then a Recurrent Neural Network (RNN) is integrated into the framework to predict future muscle forces, which are transferred back to future arm motions based on the dynamic model. Given the absence of measurement data for future human motions that can be input into the UKF to update the state, we integrate another RNN to directly predict human future motions and treat the prediction as surrogate measurement data fed into the UKF. A noteworthy aspect of this study involves the quantification of uncertainties associated with both the data-driven and physical models in one unified framework. These quantified uncertainties are used to dynamically adapt the measurement and process noises of the UKF over time. This adaption, driven by the uncertainties of the RNN models, addresses inaccuracies stemming from the data-driven model and mitigates discrepancies between the assumed and true physical models, ultimately enhancing the accuracy and robustness of our predictions. Compared to the traditional RNN-based prediction, our method demonstrates improved accuracy and robustness in extensive experimental validations of various types of human motions

DiffHybrid-UQ: Uncertainty Quantification for Differentiable Hybrid Neural Modeling

The hybrid neural differentiable models mark a significant advancement in the field of scientific machine learning. These models, integrating numerical representations of known physics into deep neural networks, offer enhanced predictive capabilities and show great potential for data-driven modeling of complex physical systems. However, a critical and yet unaddressed challenge lies in the quantification of inherent uncertainties stemming from multiple sources. Addressing this gap, we introduce a novel method, DiffHybrid-UQ, for effective and efficient uncertainty propagation and estimation in hybrid neural differentiable models, leveraging the strengths of deep ensemble Bayesian learning and nonlinear transformations. Specifically, our approach effectively discerns and quantifies both aleatoric uncertainties, arising from data noise, and epistemic uncertainties, resulting from model-form discrepancies and data sparsity. This is achieved within a Bayesian model averaging framework, where aleatoric uncertainties are modeled through hybrid neural models. The unscented transformation plays a pivotal role in enabling the flow of these uncertainties through the nonlinear functions within the hybrid model. In contrast, epistemic uncertainties are estimated using an ensemble of stochastic gradient descent (SGD) trajectories. This approach offers a practical approximation to the posterior distribution of both the network parameters and the physical parameters. Notably, the DiffHybrid-UQ framework is designed for simplicity in implementation and high scalability, making it suitable for parallel computing environments. The merits of the proposed method have been demonstrated through problems governed by both ordinary and partial differentiable equations

A critical review of improved deep learning methods for the remaining useful life prediction of lithium-ion batteries.

As widely used for secondary energy storage, lithium-ion batteries have become the core component of the power supply system and accurate remaining useful life prediction is the key to ensure its reliability. Because of the complex working characteristics of lithium-ion batteries as well as the model parameter changing along with the aging process, the accuracy of the online remaining useful life prediction is difficult but urgent to be improved for the reliable power supply application. The deep learning algorithm improves the accuracy of the remaining useful life prediction, which also reduces the characteristic testing time requirement, providing the possibility to improve the power profitability of predictive energy management. This article analyzes, reviews, classifies, and compares different adaptive mathematical models on deep learning algorithms for the remaining useful life prediction. The features are identified for the modeling ability, according to which the adaptive prediction methods are classified. The specific criteria are defined to evaluate different modeling accuracy in the deep learning calculation procedure. The key features of effective life prediction are used to draw relevant conclusions and suggestions are provided, in which the high-accuracy deep convolutional neural network — extreme learning machine algorithm is chosen to be utilized for the stable remaining useful life prediction of lithium-ion batteries

Novel Computational Methods for State Space Filtering

The state-space formulation for time-dependent models has been long used invarious applications in science and engineering. While the classical Kalman filter(KF) provides optimal posterior estimation under linear Gaussian models, filteringin nonlinear and non-Gaussian environments remains challenging.Based on the Monte Carlo approximation, the classical particle filter (PF) can providemore precise estimation under nonlinear non-Gaussian models. However, it suffers fromparticle degeneracy. Drawing from optimal transport theory, the stochastic map filter(SMF) accommodates a solution to this problem, but its performance is influenced bythe limited flexibility of nonlinear map parameterisation. To account for these issues,a hybrid particle-stochastic map filter (PSMF) is first proposed in this thesis, wherethe two parts of the split likelihood are assimilated by the PF and SMF, respectively.Systematic resampling and smoothing are employed to alleviate the particle degeneracycaused by the PF. Furthermore, two PSMF variants based on the linear and nonlinearmaps (PSMF-L and PSMF-NL) are proposed, and their filtering performance is comparedwith various benchmark filters under different nonlinear non-Gaussian models.Although achieving accurate filtering results, the particle-based filters require expensive computations because of the large number of samples involved. Instead, robustKalman filters (RKFs) provide efficient solutions for the linear models with heavy-tailednoise, by adopting the recursive estimation framework of the KF. To exploit the stochasticcharacteristics of the noise, the use of heavy-tailed distributions which can fit variouspractical noises constitutes a viable solution. Hence, this thesis also introduces a novelRKF framework, RKF-SGαS, where the signal noise is assumed to be Gaussian and theheavy-tailed measurement noise is modelled by the sub-Gaussian α-stable (SGαS) distribution. The corresponding joint posterior distribution of the state vector and auxiliaryrandom variables is estimated by the variational Bayesian (VB) approach. Four differentminimum mean square error (MMSE) estimators of the scale function are presented.Besides, the RKF-SGαS is compared with the state-of-the-art RKFs under three kinds ofheavy-tailed measurement noises, and the simulation results demonstrate its estimationaccuracy and efficiency.One notable limitation of the proposed RKF-SGαS is its reliance on precise modelparameters, and substantial model errors can potentially impede its filtering performance. Therefore, this thesis also introduces a data-driven RKF method, referred to asRKFnet, which combines the conventional RKF framework with a deep learning technique. An unsupervised scheduled sampling technique (USS) is proposed to improve theistability of the training process. Furthermore, the advantages of the proposed RKFnetare quantified with respect to various traditional RKFs

Overview of Machine Learning Methods for Lithium-Ion Battery Remaining Useful Lifetime Prediction

Lithium-ion batteries play an indispensable role, from portable electronic devices to electric vehicles and home storage systems. Even though they are characterized by superior performance than most other storage technologies, their lifetime is not unlimited and has to be predicted to ensure the economic viability of the battery application. Furthermore, to ensure the optimal battery system operation, the remaining useful lifetime (RUL) prediction has become an essential feature of modern battery management systems (BMSs). Thus, the prediction of RUL of Lithium-ion batteries has become a hot topic for both industry and academia. The purpose of this work is to review, classify, and compare different machine learning (ML)-based methods for the prediction of the RUL of Lithium-ion batteries. First, this article summarizes and classifies various Lithium-ion battery RUL estimation methods that have been proposed in recent years. Secondly, an innovative method was selected for evaluation and compared in terms of accuracy and complexity. DNN is more suitable for RUL prediction due to its strong independent learning ability and generalization ability. In addition, the challenges and prospects of BMS and RUL prediction research are also put forward. Finally, the development of various methods is summarized

Deep Convolutional Architectures for Extrapolative Forecast in Time-dependent Flow Problems

Physical systems whose dynamics are governed by partial differential equations (PDEs) find applications in numerous fields, from engineering design to weather forecasting. The process of obtaining the solution from such PDEs may be computationally expensive for large-scale and parameterized problems. In this work, deep learning techniques developed especially for time-series forecasts, such as LSTM and TCN, or for spatial-feature extraction such as CNN, are employed to model the system dynamics for advection dominated problems. These models take as input a sequence of high-fidelity vector solutions for consecutive time-steps obtained from the PDEs and forecast the solutions for the subsequent time-steps using auto-regression; thereby reducing the computation time and power needed to obtain such high-fidelity solutions. The models are tested on numerical benchmarks (1D Burgers' equation and Stoker's dam break problem) to assess the long-term prediction accuracy, even outside the training domain (extrapolation). Non-intrusive reduced-order modelling techniques such as deep auto-encoder networks are utilized to compress the high-fidelity snapshots before feeding them as input to the forecasting models in order to reduce the complexity and the required computations in the online and offline stages. Deep ensembles are employed to perform uncertainty quantification of the forecasting models, which provides information about the variance of the predictions as a result of the epistemic uncertainties

Modelling and estimation in lithium-ion batteries: a literature review

Lithium-ion batteries are widely recognised as the leading technology for electrochemical energy storage. Their applications in the automotive industry and integration with renewable energy grids highlight their current significance and anticipate their substantial future impact. However, battery management systems, which are in charge of the monitoring and control of batteries, need to consider several states, like the state of charge and the state of health, which cannot be directly measured. To estimate these indicators, algorithms utilising mathematical models of the battery and basic measurements like voltage, current or temperature are employed. This review focuses on a comprehensive examination of various models, from complex but close to the physicochemical phenomena to computationally simpler but ignorant of the physics; the estimation problem and a formal basis for the development of algorithms; and algorithms used in Li-ion battery monitoring. The objective is to provide a practical guide that elucidates the different models and helps to navigate the different existing estimation techniques, simplifying the process for the development of new Li-ion battery applications.This research received support from the Spanish Ministry of Science and Innovation under projects MAFALDA (PID2021-126001OB-C31 funded by MCIN/AEI/10.13039/501100011033/ ERDF,EU) and MASHED (TED2021-129927B-I00), and by FI Joan Oró grant (code 2023 FI-1 00827), cofinanced by the European Union.Peer ReviewedPostprint (published version

Degradation Vector Fields with Uncertainty Considerations

The focus of this work is on capturing uncertainty in remaining useful life (RUL) estimates for machinery and constructing some latent dynamics that aid in interpreting those results. This is primarily achieved through sequential deep generative models known as Dynamical Variational Autoencoders (DVAEs). These allow for the construction of latent dynamics related to the RUL estimates while being a probabilistic model that can quantify the uncertainties of the estimates