Bayesian Smoothing with Gaussian Processes Using Fourier Basis Functions in the spectralGP Package

The spectral representation of stationary Gaussian processes via the Fourier basis provides a computationally efficient specification of spatial surfaces and nonparametric regression functions for use in various statistical models. I describe the representation in detail and introduce the spectralGP package in R for computations. Because of the large number of basis coefficients, some form of shrinkage is necessary; I focus on a natural Bayesian approach via a particular parameterized prior structure that approximates stationary Gaussian processes on a regular grid. I review several models from the literature for data that do not lie on a grid, suggest a simple model modification, and provide example code demonstrating MCMC sampling using the spectralGP package. I describe reasons that mixing can be slow in certain situations and provide some suggestions for MCMC techniques to improve mixing, also with example code, and some general recommendations grounded in experience.

Multifidelity Information Fusion Algorithms for High-Dimensional Systems and Massive Data sets

We develop a framework for multifidelity information fusion and predictive inference in high-dimensional input spaces and in the presence of massive data sets. Hence, we tackle simultaneously the “big N" problem for big data and the curse of dimensionality in multivariate parametric problems. The proposed methodology establishes a new paradigm for constructing response surfaces of high-dimensional stochastic dynamical systems, simultaneously accounting for multifidelity in physical models as well as multifidelity in probability space. Scaling to high dimensions is achieved by data-driven dimensionality reduction techniques based on hierarchical functional decompositions and a graph-theoretic approach for encoding custom autocorrelation structure in Gaussian process priors. Multifidelity information fusion is facilitated through stochastic autoregressive schemes and frequency-domain machine learning algorithms that scale linearly with the data. Taking together these new developments leads to linear complexity algorithms as demonstrated in benchmark problems involving deterministic and stochastic fields in up to 10⁵ input dimensions and 10⁵ training points on a standard desktop computer

Bayesian Smoothing with Gaussian Processes Using Fourier Basis Functions in the spectralGP Package

The spectral representation of stationary Gaussian processes via the Fourier basis provides a computationally efficient specification of spatial surfaces and nonparametric regression functions for use in various statistical models. I describe the representation in detail and introduce the spectralGP package in R for computations. Because of the large number of basis coefficients, some form of shrinkage is necessary; I focus on a natural Bayesian approach via a particular parameterized prior structure that approximates stationary Gaussian processes on a regular grid. I review several models from the literature for data that do not lie on a grid, suggest a simple model modification, and provide example code demonstrating MCMC sampling using the spectralGP package. I describe reasons that mixing can be slow in certain situations and provide some suggestions for MCMC techniques to improve mixing, also with example code, and some general recommendations grounded in experience

A Framework for Evaluating Approximation Methods for Gaussian Process Regression

Gaussian process (GP) predictors are an important component of many Bayesian approaches to machine learning. However, even a straightforward implementation of Gaussian process regression (GPR) requires O(n^2) space and O(n^3) time for a data set of n examples. Several approximation methods have been proposed, but there is a lack of understanding of the relative merits of the different approximations, and in what situations they are most useful. We recommend assessing the quality of the predictions obtained as a function of the compute time taken, and comparing to standard baselines (e.g., Subset of Data and FITC). We empirically investigate four different approximation algorithms on four different prediction problems, and make our code available to encourage future comparisons

Epistemic Modeling Uncertainty of Rapid Neural Network Ensembles for Adaptive Learning

Emulator embedded neural networks, which are a type of physics informed neural network, leverage multi-fidelity data sources for efficient design exploration of aerospace engineering systems. Multiple realizations of the neural network models are trained with different random initializations. The ensemble of model realizations is used to assess epistemic modeling uncertainty caused due to lack of training samples. This uncertainty estimation is crucial information for successful goal-oriented adaptive learning in an aerospace system design exploration. However, the costs of training the ensemble models often become prohibitive and pose a computational challenge, especially when the models are not trained in parallel during adaptive learning. In this work, a new type of emulator embedded neural network is presented using the rapid neural network paradigm. Unlike the conventional neural network training that optimizes the weights and biases of all the network layers by using gradient-based backpropagation, rapid neural network training adjusts only the last layer connection weights by applying a linear regression technique. It is found that the proposed emulator embedded neural network trains near-instantaneously, typically without loss of prediction accuracy. The proposed method is demonstrated on multiple analytical examples, as well as an aerospace flight parameter study of a generic hypersonic vehicle

A Probabilistic Machine Learning Approach for the Uncertainty Quantification of Electronic Circuits Based on Gaussian Process Regression

This paper introduces a probabilistic machine learning framework for the uncertainty quantification (UQ) of electronic circuits based on Gaussian process regression (GPR). As opposed to classical surrogate modeling techniques, GPR inherently provides information on the model uncertainty. The main contribution of this work is twofold. First, it describes how, in an UQ scenario, the model uncertainty can be combined with the uncertainty of the input design parameters to provide confidence bounds for the statistical estimates of the system outputs, such as moments and probability distributions. These confidence bounds allows assessing the accuracy of the predicted statistics. Second, in order to deal with dynamic multi-output systems, principal component analysis (PCA) is effectively employed to compress the time-dependent output variables into a smaller set of components, for which the training of individual GPR models becomes feasible. The uncertainty on the principal components is then propagated back to the original output variables. Several application examples, ranging from a trivial RLC circuit to real-life designs, are used to illustrate and validate the advocated approach

Radio Map Estimation: A Data-Driven Approach to Spectrum Cartography

Radio maps characterize quantities of interest in radio communication environments, such as the received signal strength and channel attenuation, at every point of a geographical region. Radio map estimation typically entails interpolative inference based on spatially distributed measurements. In this tutorial article, after presenting some representative applications of radio maps, the most prominent radio map estimation methods are discussed. Starting from simple regression, the exposition gradually delves into more sophisticated algorithms, eventually touching upon state-of-the-art techniques. To gain insight into this versatile toolkit, illustrative toy examples will also be presented