Probabilistic surrogate modeling of offshore wind-turbine loads with chained Gaussian processes

Heteroscedastic Gaussian process regression, based on the concept of chained Gaussian processes, is used to build surrogates to predict site-specific loads on an offshore wind turbine. Stochasticity in the inflow turbulence and irregular waves results in load responses that are best represented as random variables rather than deterministic values. Moreover, the effect of these stochastic sources on the loads depends strongly on the mean environmental conditions -- for instance, at low mean wind speeds, inflow turbulence produces much less variability in loads than at high wind speeds. Statistically, this is known as heteroscedasticity. Deterministic and most stochastic surrogates do not account for the heteroscedastic noise, giving an incomplete and potentially misleading picture of the structural response. In this paper, we draw on the recent advancements in statistical inference to train a heteroscedastic surrogate model on a noisy database to predict the conditional pdf of the response. The model is informed via 10-minute load statistics of the IEA-10MW-RWT subject to both aero- and hydrodynamic loads, simulated with OpenFAST. Its performance is assessed against the standard Gaussian process regression. The predicted mean is similar in both models, but the heteroscedastic surrogate approximates the large-scale variance of the responses significantly better.Comment: 10 pages. To be published in the IOP Journal of Physics: Conference Series. To be presented at TORQUE 202

A supervised learning framework in the context of multiple annotators

The increasing popularity of crowdsourcing platforms, i.e., Amazon Mechanical Turk, is changing how datasets for supervised learning are built. In these cases, instead of having datasets labeled by one source (which is supposed to be an expert who provided the absolute gold standard), we have datasets labeled by multiple annotators with different and unknown expertise. Hence, we face a multi-labeler scenario, which typical supervised learning models cannot tackle. For such a reason, much attention has recently been given to the approaches that capture multiple annotators’ wisdom. However, such methods residing on two key assumptions: the labeler’s performance does not depend on the input space and independence among the annotators, which are hardly feasible in real-world settings..

A supervised learning framework in the context of multiple annotators

The increasing popularity of crowdsourcing platforms, i.e., Amazon Mechanical Turk, is changing how datasets for supervised learning are built. In these cases, instead of having datasets labeled by one source (which is supposed to be an expert who provided the absolute gold standard), we have datasets labeled by multiple annotators with different and unknown expertise. Hence, we face a multi-labeler scenario, which typical supervised learning models cannot tackle. For such a reason, much attention has recently been given to the approaches that capture multiple annotators’ wisdom. However, such methods residing on two key assumptions: the labeler’s performance does not depend on the input space and independence among the annotators, which are hardly feasible in real-world settings..

Development of a methodology for the diagnosis of internal combustion engines using non-invasive measurements based on the use of interpretable neural networks applicable to databases with multiple annotators

Pressure is one of the essential variables that give information for engine condition and monitoring. Direct recording of this signal is complex and invasive, while the angular velocity can be measured easily. Nonetheless, the challenge is to predict the cylinder pressure using the shaft kinematics accurately. On the other hand, the increasing popularity of crowdsourcing platforms, i.e., Amazon Mechanical Turk, changes how datasets for supervised learning are built. In these cases, instead of having datasets labeled by one source (which is supposed to be an expert who provided the absolute gold standard), databases holding multiple annotators are provided. However, most state-of-the-art methods devoted to learning from multiple experts assume that the labeler's behavior is homogeneous across the input feature space. Besides, independence constraints are imposed on annotators' outputs. This document presents a Regularized Chained Deep Neural Network to deal with classification tasks from multiple annotators. In this thesis, we develop 2 strategies aiming to avoid intrusive techniques that are commonly used to diagnose Internal Combustion Engines (ICE). The first consist of a time-delay neural network (TDNN), interpreted as a finite pulse response (FIR) filter to estimate the in-cylinder pressure of a single-cylinder ICE from fluctuations in shaft angular velocity. The experiments are conducted over data obtained from an ICE operating in 12 different states by changing the angular velocity and load. The TDNN's delay is adjusted to get the highest possible correlation-based score. Our methodology can predict pressure with an R2>0.9, avoiding complicated pre-processing steps. The second technique, termed RCDNN, jointly predicts the ground truth label and the annotators' performance from input space samples. In turn, RCDNN codes interdependencies among the experts by analyzing the layers' weights and includes l1, l2, and Monte-Carlo Dropout-based regularizers to deal with the overfitting issue in deep learning models. Obtained results (using both simulated and real-world annotators) demonstrate that RCDNN can deal with multi-labelers scenarios for classification tasks, defeating state-of-the-art techniques.La presión es una de las variables esenciales que dan información para el estado del motor y su monitorización. El registro directo de esta señal es complejo e invasivo, mientras que la velocidad angular puede medirse fácilmente. No obstante, el reto consiste en predecir la presión del cilindro utilizando la cinemática del eje con precisión. Por otro lado, la creciente popularidad de las plataformas de crowdsourcing, por ejemplo, Amazon Mechanical Turk, cambia la forma de construir conjuntos de datos para el aprendizaje supervisado. En estos casos, en lugar de tener conjuntos de datos etiquetados por una sola fuente (que se supone que es un experto que proporcionó el estándar de oro absoluto), se proporcionan bases de datos con múltiples anotadores. Sin embargo, la mayoría de los métodos de vanguardia dedicados al aprendizaje a partir de múltiples expertos suponen que el comportamiento del etiquetador es homogéneo en todo el espacio de características de entrada. Además, se imponen restricciones de independencia a los resultados de los anotadores. Este documento presenta una Red Neuronal Profunda Encadenada Regularizada para abordar tareas de clasificación a partir de múltiples anotadores. En esta tesis, desarrollamos dos estrategias con el objetivo de evitar las técnicas intrusivas que se utilizan habitualmente para diagnosticar motores de combustión interna (ICE). La primera consiste en una red neuronal de retardo temporal (TDNN), interpretada como un filtro de respuesta de pulso finito (FIR) para estimar la presión en el cilindro de un ICE de un solo cilindro a partir de las fluctuaciones de la velocidad angular del eje. Los experimentos se realizan sobre datos obtenidos de un ICE que opera en 12 estados diferentes cambiando la velocidad angular y la carga. El retardo de la TDNN se ajusta para obtener la mayor puntuación posible basada en la correlación. Nuestra metodología puede predecir la presión con un R2>0,9, evitando complicados pasos de preprocesamiento.MaestríaMagíster en Ingeniería EléctricaContent 1 Introduction 10 1.1 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.2 Justification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.3 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.3.1 General objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.3.2 Specific objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2 TDNN-based Engine In-cylinder Pressure Estimation from Shaft Velocity Spectral Representation 18 2.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2.1 Time Delay Neural Network fundamentals . . . . . . . . . . . . . . . 19 2.2.2 Harmonic prediction performance based on Magnitude-Squared Coherence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.3 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.3.1 Engine Measurements, Data Acquisition, and Preprocessing . . . . . 22 2.3.2 Pressure signal estimation . . . . . . . . . . . . . . . . . . . . . . . . 26 2.4 Results and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.5 Conclusions and future work . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3 Master Thesis: Content 3 Regularized Chained Deep Neural Network Classifier for Multiple Annotators 37 3.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.2 Materials and Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.2.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.2.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.3 Experimental set-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.3.1 Tested datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.3.2 RCDNN detailed architecture and training . . . . . . . . . . . . . . . 46 3.3.3 Results and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.3.4 Introducing spammers and malicious annotators . . . . . . . . . . . . 55 3.3.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4 Final Remarks 58 4.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.1.1 TDNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.1.2 RCDNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.2.1 TDNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.2.2 RCDNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.3 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Variational Optimisation for Non-conjugate Likelihood Gaussian Process Models

In this thesis we address the problems associated to non-conjugate likelihood Gaussian process models, i.e., probabilistic models where the likelihood function and the Gaussian process priors are non-conjugate. Such problems include intractability, scalability, and poor local optima solutions for the parameters and hyper-parameters of the models. Particularly, in this thesis we address the aforementioned issues in the context of probabilistic models, where the likelihood’s parameters are modelled as latent parameter functions drawn from correlated Gaussian processes. We study three ways to generate such latent parameter functions: 1. from a linear model of coregionalisation; 2. from convolution processes, i.e., a convolution integral between smoothing kernels and Gaussian process priors; and 3. using variational inducing kernels, an alternative form to generate the latent parameter functions through the convolution processes formalism, by using a double convolution integral. We borrow ideas from different variational optimisation mechanisms, that consist on introducing a variational (or exploratory) distribution over the model so as to build objective functions that: allow us to deal with intractability as well as enabling scalability when needing to hand massive amounts of data observations. Also, such variational optimisations mechanisms grant us to perform inference of the model hyper-parameters together with the posterior’s parameters through a fully natural gradient optimisation scheme; a useful scheme for tackling the problem of poor local optima solutions. Such variational optimisation mechanisms have been broadly studied in the context of reinforcement and Bayesian deep learning showing to be successful exploratory-learning tools; nonetheless, they have not been much studied in the context of Gaussian process models, so we provide a study of their performance in said context

Stacked Modelling Framework

The thesis develops a predictive modeling framework based on stacked Gaussian processes and applies it to two main applications in environmental and chemical en- gineering. First, a network of independently trained Gaussian processes (StackedGP) is introduced to obtain analytical predictions of quantities of interest (model out- puts) with quantified uncertainties. StackedGP framework supports component- based modeling in different fields such as environmental and chemical science, en- hances predictions of quantities of interest through a cascade of intermediate predic- tions usually addressed by cokriging, and propagates uncertainties through emulated dynamical systems driven by uncertain forcing variables. By using analytical first and second-order moments of a Gaussian process with uncertain inputs using squared ex- ponential and polynomial kernels, approximated expectations of model outputs that require an arbitrary composition of functions can be obtained. The performance of the proposed nonparametric stacked model in model composition and cascading predictions is measured in different applications and datasets. The framework has been evaluated in a wildfire and mineral resource problem using real data, and its application to time-series prediction is demonstrated in a 2D puff advection problem. In additions, the StackedGP is introduced to one of challenging environmental problems, prediction of mycotoxins. In this part of the work, we develop a stacked Gaussian process using both field and wet-lab measurements to predict fungal toxin (aflatoxin) concentrations in corn in South Carolina. While most of the aflatoxin contamination issues associated with the post-harvest period in the U.S. can be con- trolled with expensive testing, a systematic and economical approach is lacking to determine how the pre-harvest aflatoxin risk adversely affects crop producers as afla- toxin is virtually unobservable on a geographical and temporal scale. This information gap carries significant cost burdens for grain producers and is filled by the proposed stacked Gaussian process. The novelty of this part is two fold. First, the aflatoxin probabilistic maps are obtained using an analytical scheme to propagate the uncer- tainty through the stacked Gaussian process. The model predictions are validated both at the Gaussian process component level and at the system level for the entire stacked Gaussian process using historical field data. Second, a novel derivation is introduced to calculate the analytical covariance of aflatoxin production at two ge- ographical locations. Similar with kriging/Gaussian process, this is used to predict aflatoxin at unobserved locations using measurements at nearby locations but with the prior mean and covariance provided by the stacked Gaussian process. As field measurements arrive, this measurement update scheme may be used in targeted field inspections and warning farmers of emerging aflatoxin contaminations. Lastly, we apply the stackedGP framework in a chemical engineering application. Computational catalyst discovery involves identification of a meaningful model and suitable descriptors that determine the catalyst properties. First, we study the impact of combining various descriptors (e.g. reaction energies, metal descriptors, and bond counts) for modeling transition state energies (TS) based on a database of adsorption and TS energies across transition metal surfaces {Palladium (PD_111), Platinum (PT_111), Nickel (NI_111), Ruthenium (RU_0001), and Rhodium (RH_111)} for the decarboxylation and decarbonylation of propionic acid, a chemistry characteristic for biomass conversion. Results of different machine learning models for more than 1330 of these descriptor combinations suggest that there is no statistically significant difference between linear and non-linear models when using the right combination of reactant energies, metal descriptors, and bond counts. However, linear models are inferior when not including bond count and metal descriptors. Furthermore, when there are missing data for reaction steps on all metals, conventional linear scaling is inferior to linear and nonlinear models with proper choice of descriptors that are surprisingly robust. Finally, the stackedGP framework is evaluated in modeling the adsorption and transition state energies as a function of metal descriptors with data from all metal surfaces. By getting these energies, the Turn-Over-Frequency (TOF) can be estimated using micro-kinetic models