Defining and applying prediction performance metrics on a recurrent NARX time series model.

International audienceNonlinear autoregressive moving average with exogenous inputs (NARMAX) models have been successfully demonstrated for modeling the input-output behavior of many complex systems. This paper deals with the proposition of a scheme to provide time series prediction. The approach is based on a recurrent NARX model obtained by linear combination of a recurrent neural network (RNN) output and the real data output. Some prediction metrics are also proposed to assess the quality of predictions. This metrics enable to compare different prediction schemes and provide an objective way to measure how changes in training or prediction model (Neural network architecture) affect the quality of predictions. Results show that the proposed NARX approach consistently outperforms the prediction obtained by the RNN neural network

Intelligent Calibration and Virtual Sensing for Integrated Low-Cost Air Quality Sensors

This paper presents the development of air quality low-cost sensors (LCS) with improved accuracy features. The LCS features integrate machine learning based calibration models and virtual sensors. LCS performances are analyzed and some LCS variables with low performance are improved through intelligent field-calibrations. Meteorological variables are calibrated using linear dynamic models. While, due to the non-linear relationship to reference instruments, fine particulate matter (PM2.5) are calibrated using non-linear machine learning models. However, due to sensor drifts or faults, carbon dioxide (CO2) does not present correlation to reference instrument. As a result, the LCS for CO2 is not feasible to be calibrated. Hence, to estimate the CO2 concentration, mathematical models are developed to be integrated in the calibrated LCS, known as a virtual sensor. In addition, another virtual sensor is developed to demonstrate the capability of estimating air pollutant concentrations, e.g. black carbon, when the physical sensor devices are not available. In our paper, calibration models and virtual sensors are established using corresponding reference instruments that are installed on two reference stations. This strategy generalizes the models of calibration and virtual sensing which then allows LCS to be deployed in field independently with a high accuracy. Our proposed methodology enables scaling-up accurate air pollution mapping appropriate for smart cities.Peer reviewe

Solar Radiation Prediction Using NARX Model

The human brain, like every vital organ, is constituted of neurons. It is through this organ that we can learn and reason, reflect and memorize. The geniality of human brain and more particularly of its neurons motivates several researchers to interest to this research and to benefit from its biological aspect. The idea was to reproduce, in an artificial way, the behaviors observed in man. It was in 1943 that the first artificial neural network (ANN) was created by Warren McCulloch and Walter Pitts. It is a simple elementary processor imitating the structure and the functioning from the biological neuron. Artificial neural network is characterized by its capacity to learning and generalizing. It represents a very powerful tool. It provided multiple solutions to different complex problems. In these recent years, its effectiveness is proved in various researches fields. ANN is subdivided on two main groups, the static and dynamic neural network. The choice of the one or the other neural network type depends to the application to be processed and the complexity of model. For static neural network, information propagates in a single direction, layer by layer, and from the inlet to the outlet. They are generally used in various applications such as classifications, pattern recognition, and functions approximation. For the dynamic neural network dynamic neural network is not limited. Each neuron can send and receive information from all other neurons. The dynamic neural network architecture includes frequently one or more cycles which necessarily contain at least one delay connection. This gives rise to the dynamism notion. This neural network type is more complex than the static one, but it is more efficient for some particular applications such as dynamic modeling, monitoring, and process control. In this chapter, nonlinear autoregressive models with exogenous input (NARX) model, as type of dynamic neural network, will be used to the solar radiation prediction. Simulation results will be presented to prove the effectiveness of this model compared to those obtained using the static one

Developing dynamic machine learning surrogate models of physics-based industrial process simulation models

Abstract. Dynamic physics-based models of industrial processes can be computationally heavy which prevents using them in some applications, e.g. in process operator training. Suitability of machine learning in creating surrogate models of a physics-based unit operation models was studied in this research. The main motivation for this was to find out if machine learning model can be accurate enough to replace the corresponding physics-based components in dynamic modelling and simulation software Apros® which is developed by VTT Technical Research Centre of Finland Ltd and Fortum. This study is part of COCOP project, which receive funding from EU, and INTENS project that is Business Finland funded. The research work was divided into a literature study and an experimental part. In the literature study, the steps of modelling with data-driven methods were studied and artificial neural network architectures suitable for dynamic modelling were investigated. Based on that, four neural network architectures were chosen for the case studies. In the first case study, linear and nonlinear autoregressive models with exogenous inputs (ARX and NARX respectively) were used in modelling dynamic behaviour of a water tank process build in Apros®. In the second case study, also Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) were considered and compared with the previously mentioned ARX and NARX models. The workflow from selecting the input and output variables for the machine learning model and generating the datasets in Apros® to implement the machine learning models back to Apros® was defined. Keras is an open source neural network library running on Python that was utilised in the model generation framework which was developed as a part of this study. Keras library is a very popular library that allow fast experimenting. The framework make use of random hyperparameter search and each model is tested on a validation dataset in dynamic manner, i.e. in multi-step-ahead configuration, during the optimisation. The best models based in terms of average normalised root mean squared error (NRMSE) is selected for further testing. The results of the case studies show that accurate multi-step-ahead models can be built using recurrent artificial neural networks. In the first case study, the linear ARX model achieved slightly better NRMSE value than the nonlinear one, but the accuracy of both models was on a very good level with the average NRMSE being lower than 0.1 %. The generalisation ability of the models was tested using multiple datasets and the models proved to generalise well. In the second case study, there were more difference between the models’ accuracies. This was an expected result as the studied process contains nonlinearities and thus the linear ARX model performed worse in predicting some output variables than the nonlinear ones. On the other hand, ARX model performed better with some other output variables. However, also in the second case study the model NRMSE values were on good level, being 1.94–3.60 % on testing dataset. Although the workflow to implement machine learning models in Apros® using its Python binding was defined, the actual implementation need more work. Experimenting with Keras neural network models in Apros® was noticed to slow down the simulation even though the model was fast when testing it outside of Apros®. The Python binding in Apros® do not seem to cause overhead to the calculation process which is why further investigating is needed. It is obvious that the machine learning model must be very accurate if it is to be implemented in Apros® because it needs to be able interact with the physics-based model. The actual accuracy requirement that Apros® sets should be also studied to know if and in which direction the framework made for this study needs to be developed.Dynaamisten surrogaattimallien kehittäminen koneoppimismenetelmillä teollisuusprosessien fysiikkapohjaisista simulaatiomalleista. Tiivistelmä. Teollisuusprosessien toimintaa jäljittelevät dynaamiset fysiikkapohjaiset simulaatiomallit voivat laajuudesta tai yksityiskohtien määrästä johtuen olla laskennallisesti raskaita. Tämä voi rajoittaa simulaatiomallin käyttöä esimerkiksi prosessioperaattorien koulutuksessa ja hidastaa simulaattorin avulla tehtävää prosessien optimointia. Tässä tutkimuksessa selvitettiin koneoppimismenetelmillä luotujen mallien soveltuvuutta fysiikkapohjaisten yksikköoperaatiomallien surrogaattimallinnukseen. Fysiikkapohjaiset mallit on luotu teollisuusprosessien dynaamiseen mallinnukseen ja simulointiin kehitetyllä Apros®-ohjelmistolla, jota kehittää Teknologian tutkimuskeskus VTT Oy ja Fortum. Työ on osa COCOP-projektia, joka saa rahoitusta EU:lta, ja INTENS-projektia, jota rahoittaa Business Finland. Työ on jaettu kirjallisuusselvitykseen ja kahteen kokeelliseen case-tutkimukseen. Kirjallisuusosiossa selvitettiin datapohjaisen mallinnuksen eri vaiheet ja tutkittiin dynaamiseen mallinnukseen soveltuvia neuroverkkorakenteita. Tämän perusteella valittiin neljä neuroverkkoarkkitehtuuria case-tutkimuksiin. Ensimmäisessä case-tutkimuksessa selvitettiin lineaarisen ja epälineaarisen autoregressive model with exogenous inputs (ARX ja NARX) -mallin soveltuvuutta pinnankorkeuden säädöllä varustetun vesisäiliömallin dynaamisen käyttäytymisen mallintamiseen. Toisessa case-tutkimuksessa tarkasteltiin edellä mainittujen mallityyppien lisäksi Long Short-Term Memory (LSTM) ja Gated Recurrent Unit (GRU) -verkkojen soveltuvuutta power-to-gas prosessin metanointireaktorin dynaamiseen mallinnukseen. Työssä selvitettiin surrogaattimallinnuksen vaiheet korvattavien yksikköoperaatiomallien ja siihen liittyvien muuttujien valinnasta datan generointiin ja koneoppimismallien implementointiin Aprosiin. Koneoppimismallien rakentamiseen tehtiin osana työtä Python-sovellus, joka hyödyntää Keras Python-kirjastoa neuroverkkomallien rakennuksessa. Keras on suosittu kirjasto, joka mahdollistaa nopean neuroverkkomallien kehitysprosessin. Työssä tehty sovellus hyödyntää neuroverkkomallien hyperparametrien optimoinnissa satunnaista hakua. Jokaisen optimoinnin aikana luodun mallin tarkkuutta dynaamisessa simuloinnissa mitataan erillistä aineistoa käyttäen. Jokaisen mallityypin paras malli valitaan NRMSE-arvon perusteella seuraaviin testeihin. Case-tutkimuksen tuloksien perusteella neuroverkoilla voidaan saavuttaa korkea tarkkuus dynaamisessa simuloinnissa. Ensimmäisessä case-tutkimuksessa lineaarinen ARX-malli oli hieman epälineaarista tarkempi, mutta molempien mallityyppien tarkkuus oli hyvä (NRMSE alle 0.1 %). Mallien yleistyskykyä mitattiin simuloimalla usealla aineistolla, joiden perusteella yleistyskyky oli hyvällä tasolla. Toisessa case-tutkimuksessa vastemuuttujien tarkkuuden välillä oli eroja lineaarisen ja epälineaaristen mallityyppien välillä. Tämä oli odotettu tulos, sillä joidenkin mallinnettujen vastemuuttujien käyttäytyminen on epälineaarista ja näin ollen lineaarinen ARX-malli suoriutui niiden mallintamisesta epälineaarisia malleja huonommin. Toisaalta lineaarinen ARX-malli oli tarkempi joidenkin vastemuuttujien mallinnuksessa. Kaiken kaikkiaan mallinnus onnistui hyvin myös toisessa case-tutkimuksessa, koska käytetyillä mallityypeillä saavutettiin 1.94–3.60 % NRMSE-arvo testidatalla simuloitaessa. Koneoppimismallit saatiin sisällytettyä Apros-malliin käyttäen Python-ominaisuutta, mutta prosessi vaatii lisäselvitystä, jotta mallit saadaan toimimaan yhdessä. Testien perusteella Keras-neuroverkkojen käyttäminen näytti hidastavan simulaatiota, vaikka neuroverkkomalli oli nopea Aprosin ulkopuolella. Aprosin Python-ominaisuus ei myöskään näytä itsessään aiheuttavan hitautta, jonka takia asiaa tulisi selvittää mallien implementoinnin mahdollistamiseksi. Koneoppimismallin tulee olla hyvin tarkka toimiakseen vuorovaikutuksessa fysiikkapohjaisen mallin kanssa. Jatkotutkimuksen ja Python-sovelluksen kehittämisen kannalta on tärkeää selvittää mikä on Aprosin koneoppimismalleille asettama tarkkuusvaatimus

Online voltage prediction using gaussian process regression for fault-tolerant photovoltaic standalone applications

This paper presents a fault detection system for photovoltaic standalone applications based on Gaussian Process Regression (GPR). The installation is a communication repeater from the Confederacion Hidrografica del Ebro (CHE), public institution which manages the hydrographic system of Aragon, Spain. Therefore, fault-tolerance is a mandatory requirement, complex to fulfill since it depends on the meteorology, the state of the batteries and the power demand. To solve it, we propose an online voltage prediction solution where GPR is applied in a real and large dataset of two years to predict the behavior of the installation up to 48 hour. The dataset captures electrical and thermal measures of the lead-acid batteries which sustain the installation. In particular, the crucial aspect to avoid failures is to determine the voltage at the end of the night, so different GPR methods are studied. Firstly, the photovoltaic standalone installation is described, along with the dataset. Then, there is an overview of GPR, emphasizing in the key aspects to deal with real and large datasets. Besides, three online recursive multistep GPR model alternatives are tailored, justifying the selection of the hyperparameters: Regular GPR, Sparse GPR and Multiple Experts (ME) GPR. An exhaustive assessment is performed, validating the results with those obtained by Long Short-Term Memory (LSTM) and Nonlinear Autoregressive Exogenous Model (NARX) networks. A maximum error of 127 mV and 308 mV at the end of the night with Sparse and ME, respectively, corroborates GPR as a promising tool

The Challenge of Machine Learning in Space Weather Nowcasting and Forecasting

The numerous recent breakthroughs in machine learning (ML) make imperative to carefully ponder how the scientific community can benefit from a technology that, although not necessarily new, is today living its golden age. This Grand Challenge review paper is focused on the present and future role of machine learning in space weather. The purpose is twofold. On one hand, we will discuss previous works that use ML for space weather forecasting, focusing in particular on the few areas that have seen most activity: the forecasting of geomagnetic indices, of relativistic electrons at geosynchronous orbits, of solar flares occurrence, of coronal mass ejection propagation time, and of solar wind speed. On the other hand, this paper serves as a gentle introduction to the field of machine learning tailored to the space weather community and as a pointer to a number of open challenges that we believe the community should undertake in the next decade. The recurring themes throughout the review are the need to shift our forecasting paradigm to a probabilistic approach focused on the reliable assessment of uncertainties, and the combination of physics-based and machine learning approaches, known as gray-box.Comment: under revie

Reconstruction of gasoline engine in-cylinder pressures using recurrent neural networks

Knowledge of the pressure inside the combustion chamber of a gasoline engine would provide very useful information regarding the quality and consistency of combustion and allow significant improvements in its control, leading to improved efficiency and refinement. While measurement using incylinder pressure transducers is common in laboratory tests, their use in production engines is very limited due to cost and durability constraints. This thesis seeks to exploit the time series prediction capabilities of recurrent neural networks in order to build an inverse model accepting crankshaft kinematics or cylinder block vibrations as inputs for the reconstruction of in-cylinder pressures. Success in this endeavour would provide information to drive a real time combustion control strategy using only sensors already commonly installed on production engines. A reference data set was acquired from a prototype Ford in-line 3 cylinder direct injected, spark ignited gasoline engine of 1.125 litre swept volume. Data acquired concentrated on low speed (1000-2000 rev/min), low load (10-30 Nm brake torque) test conditions. The experimental work undertaken is described in detail, along with the signal processing requirements to treat the data prior to presentation to a neural network. The primary problem then addressed is the reliable, efficient training of a recurrent neural network to result in an inverse model capable of predicting cylinder pressures from data not seen during the training phase, this unseen data includes examples from speed and load ranges other than those in the training case. The specific recurrent network architecture investigated is the non-linear autoregressive with exogenous inputs (NARX) structure. Teacher forced training is investigated using the reference engine data set before a state of the art recurrent training method (Robust Adaptive Gradient Descent – RAGD) is implemented and the influence of the various parameters surrounding input vectors, network structure and training algorithm are investigated. Optimum parameters for data, structure and training algorithm are identified

Machine Learning Modeling for Energy Consumption of Residential and Commercial Sectors

Energy has a strategic role in the economic and social development of countries. In the last few decades, energy demand has been increasing exponentially across the world, and predicting energy demand has become one of the main concerns in many countries. The residential and commercial sectors constitute about 34.7% of global energy consumption. Anticipating energy demand in these sectors will help governments to supply energy sources and to develop their sustainable energy plans such as using renewable and non-renewable energy potentials for the development of a secure and environmentally friendly energy system. Modeling energy consumption in the residential and commercial sectors enables identification of the influential economic, social, and technological factors, resulting in a secure level of energy supply. In this paper, we forecast residential and commercial energy demands in Iran using three different machine learning methods, including multiple linear regression, logarithmic multiple linear regression methods, and nonlinear autoregressive with exogenous input artificial neural networks. These models are developed based on several factors, including the share of renewable energy sources in final energy consumption, gross domestic production, population, natural gas price, and the electricity price. According to the results of the three machine learning methods applied in our study, by 2040, Iranian residential and commercial energy consumption will be 76.97, 96.42 and 128.09 Mtoe, respectively. Results show that Iran must develop and implement new policies to increase the share of renewable energy supply in final energy consumption.Peer reviewe

Machine learning supported forecasting of baseline energy consumption for industrial processes

Abstract. The purpose of the thesis was to study and evaluate machine learning supported methods in order to forecast a baseline energy consumption from time-series data of energy-intensive industry. In addition, time-series anomaly detection methods were studied and the anomaly detection accuracy of them was evaluated with hourly and daily average energy consumption data. In the experimental part of the thesis a simulation scenario was established for hourly average data of two factories. The energy baseline was identified dynamically with week-ahead time-series forecasting by utilizing previous 52 weeks of data in the model training. In addition, model adaptation was considered in the simulation scenario. Predictor variables of the models were designed to imitate natural calendar effect. The energy baseline data of factory A was used to evaluate five linear and non-linear model structures. An average ensemble model structure appeared to outperform other model structures resulting in mean absolute percentage error of 9.3% for validation data of Factory A. The generalization ability of the model structure was evaluated with the data of factory B. For factory B the average ensemble model resulted in mean absolute percentage error of 9.9% for validation data. Overall, the results seemed promising especially as the set of input variables remained relatively simple as more precise subject matter expertise was not available during variable design and selection phase.Koneoppimiseen perustuva ominaisenergian kulutuksen ennustaminen teollisissa prosesseissa . Tiivistelmä. Diplomityön tavoitteena oli tutkia ja evaluoida koneoppimiseen pohjautuvia menetelmiä energiaintensiivisen teollisuuden aikasarjamuotoisen energiankulutusdatan käsittelyssä energiankulutuksen perusuran ennustamiseksi. Lisäksi työssä tutkittiin aikasarjadatan anomaliantunnistusmenetelmiä ja evaluoitiin niiden kykyä tunnistaa poikkeamia tuntija päiväkeskiarvoresoluutioisessa energiankulutusdatassa. Työn kokeellisessa osiossa muodostettiin simulaatioskenaario kahden eri tehtaan vuosien 2020 sekä 2021 tuntikeskiarvoisten energiankulutusaineistojen mallinnukselle. Perusura muodostettiin dynaamisesti kerrallaan viikoksi eteenpäin aikasarjaennusteena edellisen 52 viikon aineistoa mallin opetuksessa hyödyntäen. Mallinnusskenaariossa huomioitiin lisäksi mallin suorituskyvylle olennainen adaptaatioproseduuri. Mallien selittävinä muuttujina käytettiin eksploratiivisen data-analyysin pohjalta luotuja luonnollista kalenterivaikutusta imitoivia muuttujia. Tehtaan A aineistolla evaluoitiin viittä eri lineaarista ja epälineaarista mallirakennetta. Parhaimmaksi mallirakenteeksi osoittautui keskiarvoyhdistelmämalli, jolle ennusteen keskimääräinen suhteellinen virhe oli 9,3 % validointiaineistolla. Mallirakenteen yleistyvyyttä testattiin toisen tehtaan (B) vastaavan ajanjakson aineistolla. Tehtaan B aineistolle keskiarvoyhdistelmämallin ennusteen keskimääräinen suhteellinen virhe oli 9,9 % validointiaineistolla. Tuloksia voidaan yleisesti ottaen pitää lupaavina etenkin, kun mallien tulomuuttujajoukko jäi verrattain yksinkertaiseksi, sillä tarkempaa aiheasiantuntemusta ei ollut saatavilla

Conjoint approach of the "residual" prediction and the nonlinear autoregressive neural network increases the forecast precision of the base model

An algorithm based on predicting the residuals of the nonlinear autoregressive neural network model with external input (NARX), which can improve the prediction accuracy, was proposed. Data of the concentration of one of the main greenhouse gases methane (CH4) on the Arctic Island of Belyy, Russia, were used for prediction. A time interval, which was characterized by high daily fluctuations in the CH4 concentration was selected. The forecast accuracy was determined by the mean absolute error (MAE), root mean squared error (RMSE) and root mean squared relative error (RMSRE) errors. The use of the algorithm allowed to increase the forecast accuracy from 11% for RMSE to 20% for RMSRE. © 2020 American Institute of Physics Inc.. All rights reserved