Analysis of the Gibbs sampler for hierarchical inverse problems

Many inverse problems arising in applications come from continuum models where the unknown parameter is a field. In practice the unknown field is discretized resulting in a problem in $\mathbb{R}^N$, with an understanding that refining the discretization, that is increasing $N$, will often be desirable. In the context of Bayesian inversion this situation suggests the importance of two issues: (i) defining hyper-parameters in such a way that they are interpretable in the continuum limit $N \to \infty$ and so that their values may be compared between different discretization levels; (ii) understanding the efficiency of algorithms for probing the posterior distribution, as a function of large $N.$ Here we address these two issues in the context of linear inverse problems subject to additive Gaussian noise within a hierarchical modelling framework based on a Gaussian prior for the unknown field and an inverse-gamma prior for a hyper-parameter, namely the amplitude of the prior variance. The structure of the model is such that the Gibbs sampler can be easily implemented for probing the posterior distribution. Subscribing to the dogma that one should think infinite-dimensionally before implementing in finite dimensions, we present function space intuition and provide rigorous theory showing that as $N$ increases, the component of the Gibbs sampler for sampling the amplitude of the prior variance becomes increasingly slower. We discuss a reparametrization of the prior variance that is robust with respect to the increase in dimension; we give numerical experiments which exhibit that our reparametrization prevents the slowing down. Our intuition on the behaviour of the prior hyper-parameter, with and without reparametrization, is sufficiently general to include a broad class of nonlinear inverse problems as well as other families of hyper-priors.Comment: to appear, SIAM/ASA Journal on Uncertainty Quantificatio

Exploring, tailoring, and traversing the solution landscape of a phase-shaped CARS process

Pulse shaping techniques are used to improve the selectivity of broadband CARS experiments, and to reject the overwhelming background. Knowledge about the fitness landscape and the capability of tailoring it is crucial for both fundamental insight and performing an efficient optimization of phase shapes. We use an evolutionary algorithm to find the optimal spectral phase of the broadband pump and probe beams in a background-suppressed shaped CARS process. We then investigate the shapes, symmetries, and topologies of the landscape contour lines around the optimal solution and also around the point corresponding to zero phase. We demonstrate the significance of the employed phase bases in achieving convex contour lines, suppressed local optima, and high optimization fitness with a few (and even a single) optimization parameter

Functional Linear Mixed Models for Irregularly or Sparsely Sampled Data

We propose an estimation approach to analyse correlated functional data which are observed on unequal grids or even sparsely. The model we use is a functional linear mixed model, a functional analogue of the linear mixed model. Estimation is based on dimension reduction via functional principal component analysis and on mixed model methodology. Our procedure allows the decomposition of the variability in the data as well as the estimation of mean effects of interest and borrows strength across curves. Confidence bands for mean effects can be constructed conditional on estimated principal components. We provide R-code implementing our approach. The method is motivated by and applied to data from speech production research

Rethinking LDA: moment matching for discrete ICA

We consider moment matching techniques for estimation in Latent Dirichlet Allocation (LDA). By drawing explicit links between LDA and discrete versions of independent component analysis (ICA), we first derive a new set of cumulant-based tensors, with an improved sample complexity. Moreover, we reuse standard ICA techniques such as joint diagonalization of tensors to improve over existing methods based on the tensor power method. In an extensive set of experiments on both synthetic and real datasets, we show that our new combination of tensors and orthogonal joint diagonalization techniques outperforms existing moment matching methods.Comment: 30 pages; added plate diagrams and clarifications, changed style, corrected typos, updated figures. in Proceedings of the 29-th Conference on Neural Information Processing Systems (NIPS), 201

Fast matrix computations for functional additive models

It is common in functional data analysis to look at a set of related functions: a set of learning curves, a set of brain signals, a set of spatial maps, etc. One way to express relatedness is through an additive model, whereby each individual function $g_{i}\left(x\right)$ is assumed to be a variation around some shared mean $f(x)$. Gaussian processes provide an elegant way of constructing such additive models, but suffer from computational difficulties arising from the matrix operations that need to be performed. Recently Heersink & Furrer have shown that functional additive model give rise to covariance matrices that have a specific form they called quasi-Kronecker (QK), whose inverses are relatively tractable. We show that under additional assumptions the two-level additive model leads to a class of matrices we call restricted quasi-Kronecker, which enjoy many interesting properties. In particular, we formulate matrix factorisations whose complexity scales only linearly in the number of functions in latent field, an enormous improvement over the cubic scaling of na\"ive approaches. We describe how to leverage the properties of rQK matrices for inference in Latent Gaussian Models

Control-Relevant System Identification using Nonlinear Volterra and Volterra-Laguerre Models

One of the key impediments to the wide-spread use of nonlinear control in industry is the availability of suitable nonlinear models. Empirical models, which are obtained from only the process input-output data, present a convenient alternative to the more involved fundamental models. An important advantage of the empirical models is that their structure can be chosen so as to facilitate the controller design problem. Many of the widely used empirical model structures are linear, and in some cases this basic model formulation may not be able to adequately capture the nonlinear process dynamics. One of the commonly used nonlinear dynamic empirical model structures is the Volterra model, and this work develops a systematic approach to the identification of third-order Volterra and Volterra-Laguerre models from process input-output data.First, plant-friendly input sequences are designed that exploit the Volterra model structure and use the prediction error variance (PEV) expression as a metric of model fidelity. Second, explicit estimator equations are derived for the linear, nonlinear diagonal, and higher-order sub-diagonal kernels using the tailored input sequences. Improvements in the sequence design are also presented which lead to a significant reduction in the amount of data required for identification. Finally, the third-order off-diagonal kernels are estimated using a cross-correlation approach. As an application of this technique, an isothermal polymerization reactor case study is considered.In order to overcome the noise sensitivity and highly parameterized nature of Volterra models, they are projected onto an orthonormal Laguerre basis. Two important variables that need to be selected for the projection are the Laguerre pole and the number of Laguerre filters. The Akaike Information Criterion (AIC) is used as a criterion to determine projected model quality. AIC includes contributions from both model size and model quality, with the latter characterized by the sum-squared error between the Volterra and the Volterra-Laguerre model outputs. Reduced Volterra-Laguerre models were also identified, and the control-relevance of identified Volterra-Laguerre models was evaluated in closed-loop using the model predictive control framework. Thus, this work presents a complete treatment of the problem of identifying nonlinear control-relevant Volterra and Volterra-Laguerre models from input-output data