Robust standard gradient descent algorithm for ARX models using Aitken acceleration technique

A robust standard gradient descent (SGD) algorithm for ARX models using the Aitken acceleration method is developed. Considering that the SGD algorithm has slow convergence rates and is sensitive to the step size, a robust and accelerative SGD (RA-SGD) algorithm is derived. This algorithm is based on the Aitken acceleration method, and its convergence rate is improved from linear convergence to at least quadratic convergence in general. Furthermore, the RA-SGD algorithm is always convergent with no limitation of the step size. Both the convergence analysis and the simulation examples demonstrate that the presented algorithm is effective

Dynamic non-linear system modelling using wavelet-based soft computing techniques

The enormous number of complex systems results in the necessity of high-level and cost-efficient modelling structures for the operators and system designers. Model-based approaches offer a very challenging way to integrate a priori knowledge into the procedure. Soft computing based models in particular, can successfully be applied in cases of highly nonlinear problems. A further reason for dealing with so called soft computational model based techniques is that in real-world cases, many times only partial, uncertain and/or inaccurate data is available. Wavelet-Based soft computing techniques are considered, as one of the latest trends in system identification/modelling. This thesis provides a comprehensive synopsis of the main wavelet-based approaches to model the non-linear dynamical systems in real world problems in conjunction with possible twists and novelties aiming for more accurate and less complex modelling structure. Initially, an on-line structure and parameter design has been considered in an adaptive Neuro- Fuzzy (NF) scheme. The problem of redundant membership functions and consequently fuzzy rules is circumvented by applying an adaptive structure. The growth of a special type of Fungus (Monascus ruber van Tieghem) is examined against several other approaches for further justification of the proposed methodology. By extending the line of research, two Morlet Wavelet Neural Network (WNN) structures have been introduced. Increasing the accuracy and decreasing the computational cost are both the primary targets of proposed novelties. Modifying the synoptic weights by replacing them with Linear Combination Weights (LCW) and also imposing a Hybrid Learning Algorithm (HLA) comprising of Gradient Descent (GD) and Recursive Least Square (RLS), are the tools utilised for the above challenges. These two models differ from the point of view of structure while they share the same HLA scheme. The second approach contains an additional Multiplication layer, plus its hidden layer contains several sub-WNNs for each input dimension. The practical superiority of these extensions is demonstrated by simulation and experimental results on real non-linear dynamic system; Listeria Monocytogenes survival curves in Ultra-High Temperature (UHT) whole milk, and consolidated with comprehensive comparison with other suggested schemes. At the next stage, the extended clustering-based fuzzy version of the proposed WNN schemes, is presented as the ultimate structure in this thesis. The proposed Fuzzy Wavelet Neural network (FWNN) benefitted from Gaussian Mixture Models (GMMs) clustering feature, updated by a modified Expectation-Maximization (EM) algorithm. One of the main aims of this thesis is to illustrate how the GMM-EM scheme could be used not only for detecting useful knowledge from the data by building accurate regression, but also for the identification of complex systems. The structure of FWNN is based on the basis of fuzzy rules including wavelet functions in the consequent parts of rules. In order to improve the function approximation accuracy and general capability of the FWNN system, an efficient hybrid learning approach is used to adjust the parameters of dilation, translation, weights, and membership. Extended Kalman Filter (EKF) is employed for wavelet parameters adjustment together with Weighted Least Square (WLS) which is dedicated for the Linear Combination Weights fine-tuning. The results of a real-world application of Short Time Load Forecasting (STLF) further re-enforced the plausibility of the above technique

Developing dynamic machine learning surrogate models of physics-based industrial process simulation models

Abstract. Dynamic physics-based models of industrial processes can be computationally heavy which prevents using them in some applications, e.g. in process operator training. Suitability of machine learning in creating surrogate models of a physics-based unit operation models was studied in this research. The main motivation for this was to find out if machine learning model can be accurate enough to replace the corresponding physics-based components in dynamic modelling and simulation software Apros® which is developed by VTT Technical Research Centre of Finland Ltd and Fortum. This study is part of COCOP project, which receive funding from EU, and INTENS project that is Business Finland funded. The research work was divided into a literature study and an experimental part. In the literature study, the steps of modelling with data-driven methods were studied and artificial neural network architectures suitable for dynamic modelling were investigated. Based on that, four neural network architectures were chosen for the case studies. In the first case study, linear and nonlinear autoregressive models with exogenous inputs (ARX and NARX respectively) were used in modelling dynamic behaviour of a water tank process build in Apros®. In the second case study, also Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) were considered and compared with the previously mentioned ARX and NARX models. The workflow from selecting the input and output variables for the machine learning model and generating the datasets in Apros® to implement the machine learning models back to Apros® was defined. Keras is an open source neural network library running on Python that was utilised in the model generation framework which was developed as a part of this study. Keras library is a very popular library that allow fast experimenting. The framework make use of random hyperparameter search and each model is tested on a validation dataset in dynamic manner, i.e. in multi-step-ahead configuration, during the optimisation. The best models based in terms of average normalised root mean squared error (NRMSE) is selected for further testing. The results of the case studies show that accurate multi-step-ahead models can be built using recurrent artificial neural networks. In the first case study, the linear ARX model achieved slightly better NRMSE value than the nonlinear one, but the accuracy of both models was on a very good level with the average NRMSE being lower than 0.1 %. The generalisation ability of the models was tested using multiple datasets and the models proved to generalise well. In the second case study, there were more difference between the models’ accuracies. This was an expected result as the studied process contains nonlinearities and thus the linear ARX model performed worse in predicting some output variables than the nonlinear ones. On the other hand, ARX model performed better with some other output variables. However, also in the second case study the model NRMSE values were on good level, being 1.94–3.60 % on testing dataset. Although the workflow to implement machine learning models in Apros® using its Python binding was defined, the actual implementation need more work. Experimenting with Keras neural network models in Apros® was noticed to slow down the simulation even though the model was fast when testing it outside of Apros®. The Python binding in Apros® do not seem to cause overhead to the calculation process which is why further investigating is needed. It is obvious that the machine learning model must be very accurate if it is to be implemented in Apros® because it needs to be able interact with the physics-based model. The actual accuracy requirement that Apros® sets should be also studied to know if and in which direction the framework made for this study needs to be developed.Dynaamisten surrogaattimallien kehittäminen koneoppimismenetelmillä teollisuusprosessien fysiikkapohjaisista simulaatiomalleista. Tiivistelmä. Teollisuusprosessien toimintaa jäljittelevät dynaamiset fysiikkapohjaiset simulaatiomallit voivat laajuudesta tai yksityiskohtien määrästä johtuen olla laskennallisesti raskaita. Tämä voi rajoittaa simulaatiomallin käyttöä esimerkiksi prosessioperaattorien koulutuksessa ja hidastaa simulaattorin avulla tehtävää prosessien optimointia. Tässä tutkimuksessa selvitettiin koneoppimismenetelmillä luotujen mallien soveltuvuutta fysiikkapohjaisten yksikköoperaatiomallien surrogaattimallinnukseen. Fysiikkapohjaiset mallit on luotu teollisuusprosessien dynaamiseen mallinnukseen ja simulointiin kehitetyllä Apros®-ohjelmistolla, jota kehittää Teknologian tutkimuskeskus VTT Oy ja Fortum. Työ on osa COCOP-projektia, joka saa rahoitusta EU:lta, ja INTENS-projektia, jota rahoittaa Business Finland. Työ on jaettu kirjallisuusselvitykseen ja kahteen kokeelliseen case-tutkimukseen. Kirjallisuusosiossa selvitettiin datapohjaisen mallinnuksen eri vaiheet ja tutkittiin dynaamiseen mallinnukseen soveltuvia neuroverkkorakenteita. Tämän perusteella valittiin neljä neuroverkkoarkkitehtuuria case-tutkimuksiin. Ensimmäisessä case-tutkimuksessa selvitettiin lineaarisen ja epälineaarisen autoregressive model with exogenous inputs (ARX ja NARX) -mallin soveltuvuutta pinnankorkeuden säädöllä varustetun vesisäiliömallin dynaamisen käyttäytymisen mallintamiseen. Toisessa case-tutkimuksessa tarkasteltiin edellä mainittujen mallityyppien lisäksi Long Short-Term Memory (LSTM) ja Gated Recurrent Unit (GRU) -verkkojen soveltuvuutta power-to-gas prosessin metanointireaktorin dynaamiseen mallinnukseen. Työssä selvitettiin surrogaattimallinnuksen vaiheet korvattavien yksikköoperaatiomallien ja siihen liittyvien muuttujien valinnasta datan generointiin ja koneoppimismallien implementointiin Aprosiin. Koneoppimismallien rakentamiseen tehtiin osana työtä Python-sovellus, joka hyödyntää Keras Python-kirjastoa neuroverkkomallien rakennuksessa. Keras on suosittu kirjasto, joka mahdollistaa nopean neuroverkkomallien kehitysprosessin. Työssä tehty sovellus hyödyntää neuroverkkomallien hyperparametrien optimoinnissa satunnaista hakua. Jokaisen optimoinnin aikana luodun mallin tarkkuutta dynaamisessa simuloinnissa mitataan erillistä aineistoa käyttäen. Jokaisen mallityypin paras malli valitaan NRMSE-arvon perusteella seuraaviin testeihin. Case-tutkimuksen tuloksien perusteella neuroverkoilla voidaan saavuttaa korkea tarkkuus dynaamisessa simuloinnissa. Ensimmäisessä case-tutkimuksessa lineaarinen ARX-malli oli hieman epälineaarista tarkempi, mutta molempien mallityyppien tarkkuus oli hyvä (NRMSE alle 0.1 %). Mallien yleistyskykyä mitattiin simuloimalla usealla aineistolla, joiden perusteella yleistyskyky oli hyvällä tasolla. Toisessa case-tutkimuksessa vastemuuttujien tarkkuuden välillä oli eroja lineaarisen ja epälineaaristen mallityyppien välillä. Tämä oli odotettu tulos, sillä joidenkin mallinnettujen vastemuuttujien käyttäytyminen on epälineaarista ja näin ollen lineaarinen ARX-malli suoriutui niiden mallintamisesta epälineaarisia malleja huonommin. Toisaalta lineaarinen ARX-malli oli tarkempi joidenkin vastemuuttujien mallinnuksessa. Kaiken kaikkiaan mallinnus onnistui hyvin myös toisessa case-tutkimuksessa, koska käytetyillä mallityypeillä saavutettiin 1.94–3.60 % NRMSE-arvo testidatalla simuloitaessa. Koneoppimismallit saatiin sisällytettyä Apros-malliin käyttäen Python-ominaisuutta, mutta prosessi vaatii lisäselvitystä, jotta mallit saadaan toimimaan yhdessä. Testien perusteella Keras-neuroverkkojen käyttäminen näytti hidastavan simulaatiota, vaikka neuroverkkomalli oli nopea Aprosin ulkopuolella. Aprosin Python-ominaisuus ei myöskään näytä itsessään aiheuttavan hitautta, jonka takia asiaa tulisi selvittää mallien implementoinnin mahdollistamiseksi. Koneoppimismallin tulee olla hyvin tarkka toimiakseen vuorovaikutuksessa fysiikkapohjaisen mallin kanssa. Jatkotutkimuksen ja Python-sovelluksen kehittämisen kannalta on tärkeää selvittää mikä on Aprosin koneoppimismalleille asettama tarkkuusvaatimus

Partial Least Squares and Neural Network Based Identification of Process Dynamics & Design of Neural Controllers

Present study implemented the Neural network (NN) and Partial least squares (PLS) based identification of process dynamics for single-input single output (SISO) as well as multi-input multi-output (MIMO)systems. In the present study, the Neural network (NN) based controller design has been implemented for a non-linear continuous bioreactor process. Multilayer feed forward networks (FFNN) were used as direct inverse neural network (DINN) controllers as well as IMC based NN controllers. The training as well as testing database was created by perturbing the open loop process with pseudo random signals (PRS).DINN controllers performed effectively for set-point tracking. To address the disturbance rejection problems, which are very likely to be faced by the bioreactors, the IMC based neural control architecture was proposed with suitable choice of filter and disturbance transfer function. To assess the controllability of the various configurations, like conventional turbidostat and nutristat& concentration turbidostat and nutristat, the offset or degree of disturbance rejection by the proposed IMC based NN controllers were utilized. The ‘concentration turbidostat’using the feed substrate concentration as the manipulated variable was found to be the best control configuration among the continuous bioreactor configurations.A (2×2) distillation column was simulated to generate the time series data consisting various inputs and outputs of the process. Multivariate statistical technique PLS was used to relate the scores of input and output matrices. ARX as well as linear least squares techniques were used for inner relation development between the input-output scores. The PLS model of the distillation column dynamics could simulate the process with reasonable accuracy

Multivariable System Identification of a Continuous Binary Distillation Column

Distillation is a process that is commonly used in industries for separation purpose. A distillation column is a multivariable system which shows nonlinear dynamic behavior due to its nonlinear vapor-liquid equilibrium. In order to gain better product quality and lower energy consumption of the distillation column, an effective model based control system is needed to allow the process to be operated over a certain operating range. In control engineering, System Identification is considered as a well suited approach for developing an approximate model for the nonlinear system. In this study, System Identification technique is applied to predict the top and bottom product composition by focusing the temperature of the distillation column. The process in the column is based on the distillation of a binary mixture of Isopropyl Alcohol and Acetone. The experimental data obtained from the distillation column was used for estimation and validation of simulated models. During analysis, different types of linear and nonlinear models were developed and are compared to predict the best model which can be effectively used for designing the control system of the distillation column. Among the linear models such as; Autoregressive with Exogenous Input (ARX), Autoregressive Moving Average with Exogenous inputs (ARMAX), Linear State Space (LSS) model and Continuous Process Model were developed and compared with each other. The results of this comparison reveals that the perf01mance of LSS model is efficient and hence it was further used to improve the modeling approach and compared with other nonlinear models. A Nonlinear State Space (NSS) model was developed by the combination of LSS and Neural Network (NN) and is compared solely with NN and ANFIS identification model. The simulation results show that the developed NSS model is well capable of defining the dynan1ics of the plant based on the best fit criteria and residual performance. In addition to this, NSS model predicted the best statistical measurement of the nonlinear system. This approach is helpful for designing the efficient control system for online separation process of the plant

Particle size estimation of hydrocyclone overflow

Bibliography : leaves 83-86.This dissertation describes the development of a robust hydrocyclone particle size estimation model that will form the basis of an industrial soft-sensor. Aside from increased throughput, efficient product regulation constitutes the primary function of a milling circuit control system. Before the milling circuit product size can be regulated, it should be measured or estimated. The particle size estimation algorithm developed provides a reliable estimate of the product size and will compliment or replace conventional size measurement devices

Linear and nonlinear parametric hydrodynamic models for wave energy converters identified from recorded data

Ocean waves represent an important resource of renewable energy, which can provide a significant support to the development of more sustainable energy solutions and to the reduction ofCO2 emissions. The amount of extracted energy from the ocean waves can be increased by optimizing the geometry and the control strategy of the wave energy converter (WEC), which both require mathematical hydrodynamic models, able to correctly describe the WEC-fluid interaction. In general, the construction of a model is based on physical laws describing the system under investigation. The hydrodynamic laws are the foundation for a complete description of the WEC-fluid interaction, but their solution represents a very complex and challenging problem. Different approaches to hydrodynamic WEC-fluid interaction modelling, such as computational fluid dynamics (CFD) and linear potential theory (LPT), lead to different mathematical models, each one characterised by different accuracy and computational speed. Fully nonlinear CFD models are able to describe the full range of hydrodynamic effects, but are very computationally expensive. On the other hand, LPT is based on the strong assumptions of inviscid fluid, irrotational flow, small waves and small body motion, which completely remove the hydrodynamic nonlinearity of the WEC-fluid interaction. Linear models have good computational speed, but are not able to properly describe nonlinear hydrodynamic effects, which are relevant in some WEC power production conditions, since WECs are designed to operate over a wide range of wave amplitudes, experience large motions, and generate viscous drag and vortex shedding. The main objective of this thesis is to propose and investigate an alternative pragmatic framework, for hydrodynamic model construction, based on system identification methodologies. The goal is to obtain models which are between the CFD and LPT extremes, a good compromise able to describe the most important nonlinearities of the physical system, without requiring excessively computational time. The identified models remain sufficiently fast and simple to run in real-time. System identification techniques can ‘inject’ into the model only the information contained in the identification data; therefore, the models obtained from LPT data are not able to describe nonlinear hydrodynamic effects. In this thesis, instead of traditional LPT data, experimental wave tank data (both numerical wave tank (NWT), implemented with a CFD software package, and real wave tank (RWT)) are proposed for hydrodynamic model identification, since CFD-NWT and RWT data can contain the full range of nonlinear hydrodynamic effects. In this thesis, different typologies of wave tank experiments and excitation signals are investigated in order to generate informative data and reduce the experiment duration. Indeed, the reduction of the experiment duration represents an important advantage since, in the case of a CFD-NWT, the amount of computation time can become unsustainable whereas, in the case of a RWT, a set of long tank experiments corresponds to an increase of the facility renting costs

Modeling and Estimation of Cardiorespiratory Function, with Application to Mechanical Ventilation

Evidence-based medicine is at the heart of current medical practice where clinical decisions are driven by research data. However, most current therapy recommendations follow generalized protocols and guidelines that are based on epidemiological (population) studies and thus not suited for the individual patient's demands. Patient-tailored therapies are considered, hence, an unmet clinical need. We believe that mathematical models of the physiology can attend to such a clinical need, because they can be tuned to the individual patient. Such models provide a sound mathematical framework for personalized clinical decisions. In particular, physiological models in medicine can serve the following two purposes: 1) They can be an efficient tool to quantify cardiopulmonary dynamics, conduct virtual clinical/physiological experiments, and investigate the effects of specific treatments. 2) Model-based estimation techniques can assess physiological parameters or variables, which are otherwise impractical or dangerous to measure; they can effectively tune a generic model to become patient-specific, able to mimic the behavior of a particular patient. In this thesis, we propose a series of modifications to a previously developed cardiopulmonary model (CP Model) in order to better replicate heart-lung interaction phenomena that are typically observed under mechanical ventilation, hence allowing for a more accurate analysis of ventilation-induced changes in cardiac function. The response of this modified model is validated with experimental data collected during mechanical ventilation conditions. Further, as an industrial application of mathematical models, we present a patient emulator system that comprises the modified CP Model, a physical ventilator, and a piston-cylinder arrangement that serves as an electrical-to-hydraulic transducer. The modified CP Model then serves as the virtual patient that is being ventilated, where disease conditions can be instilled. Such a system is designed to offer a well-controlled experimental environment for ventilator manufacturers to efficaciously test and compare ventilation modalities and therapies, thereby enhancing their verification and validation manufacturing processes. Finally, we develop a model-based approach to estimate (noninvasively) the function of the cardiovascular system, in terms of cardiac performance (i.e., cardiac output) and the dynamics of the systemic arterial tree (i.e., time constant). With this technique, we envision to provide continuous and real-time bedside monitoring of changes in cardiovascular function, such as those induced by changes in ventilator settings