Sequential Monte Carlo EM for multivariate probit models

Multivariate probit models (MPM) have the appealing feature of capturing some of the dependence structure between the components of multidimensional binary responses. The key for the dependence modelling is the covariance matrix of an underlying latent multivariate Gaussian. Most approaches to MLE in multivariate probit regression rely on MCEM algorithms to avoid computationally intensive evaluations of multivariate normal orthant probabilities. As an alternative to the much used Gibbs sampler a new SMC sampler for truncated multivariate normals is proposed. The algorithm proceeds in two stages where samples are first drawn from truncated multivariate Student $t$ distributions and then further evolved towards a Gaussian. The sampler is then embedded in a MCEM algorithm. The sequential nature of SMC methods can be exploited to design a fully sequential version of the EM, where the samples are simply updated from one iteration to the next rather than resampled from scratch. Recycling the samples in this manner significantly reduces the computational cost. An alternative view of the standard conditional maximisation step provides the basis for an iterative procedure to fully perform the maximisation needed in the EM algorithm. The identifiability of MPM is also thoroughly discussed. In particular, the likelihood invariance can be embedded in the EM algorithm to ensure that constrained and unconstrained maximisation are equivalent. A simple iterative procedure is then derived for either maximisation which takes effectively no computational time. The method is validated by applying it to the widely analysed Six Cities dataset and on a higher dimensional simulated example. Previous approaches to the Six Cities overly restrict the parameter space but, by considering the correct invariance, the maximum likelihood is quite naturally improved when treating the full unrestricted model.Comment: 26 pages, 2 figures. In press, Computational Statistics & Data Analysi

Gibbs flow for approximate transport with applications to Bayesian computation

Let $\pi_{0}$ and $\pi_{1}$ be two distributions on the Borel space $(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d}))$. Any measurable function $T:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ such that $Y=T(X)\sim\pi_{1}$ if $X\sim\pi_{0}$ is called a transport map from $\pi_{0}$ to $\pi_{1}$. For any $\pi_{0}$ and $\pi_{1}$, if one could obtain an analytical expression for a transport map from $\pi_{0}$ to $\pi_{1}$, then this could be straightforwardly applied to sample from any distribution. One would map draws from an easy-to-sample distribution $\pi_{0}$ to the target distribution $\pi_{1}$ using this transport map. Although it is usually impossible to obtain an explicit transport map for complex target distributions, we show here how to build a tractable approximation of a novel transport map. This is achieved by moving samples from $\pi_{0}$ using an ordinary differential equation with a velocity field that depends on the full conditional distributions of the target. Even when this ordinary differential equation is time-discretized and the full conditional distributions are numerically approximated, the resulting distribution of mapped samples can be efficiently evaluated and used as a proposal within sequential Monte Carlo samplers. We demonstrate significant gains over state-of-the-art sequential Monte Carlo samplers at a fixed computational complexity on a variety of applications.Comment: Significantly revised with new methodology and numerical example

Learning Bayesian Networks from Ordinal Data

Bayesian networks are a powerful framework for studying the dependency structure of variables in a complex system. The problem of learning Bayesian networks is tightly associated with the given data type. Ordinal data, such as stages of cancer, rating scale survey questions, and letter grades for exams, are ubiquitous in applied research. However, existing solutions are mainly for continuous and nominal data. In this work, we propose an iterative score-and-search method - called the Ordinal Structural EM (OSEM) algorithm - for learning Bayesian networks from ordinal data. Unlike traditional approaches designed for nominal data, we explicitly respect the ordering amongst the categories. More precisely, we assume that the ordinal variables originate from marginally discretizing a set of Gaussian variables, whose structural dependence in the latent space follows a directed acyclic graph. Then, we adopt the Structural EM algorithm and derive closed-form scoring functions for efficient graph searching. Through simulation studies, we illustrate the superior performance of the OSEM algorithm compared to the alternatives and analyze various factors that may influence the learning accuracy. Finally, we demonstrate the practicality of our method with a real-world application on psychological survey data from 408 patients with co-morbid symptoms of obsessive-compulsive disorder and depression

Robust and Efficient Probabilistic Approaches towards Parameter Identification and Model Updating

In engineering, the virtual behaviour of structures under operational and extreme conditions are investigated using mathematical or physics-based models. To obtain numerical responses that best reflect the structure under investigation, the physical input parameters describing the geometric, material, and damping properties of these models need to be identified or inferred. However, the presence of uncertainty poses significant challenges in parameter identification. Often, these uncertainties would stem from the following: 1) the aleatory uncertainty due the variations in the response measurements of nominal identical structures under same loading conditions due to manufacturing and material variability, thus, leading to the parameter not having a single "true" parameter value representation; 2) the epistemic uncertainty associated with the "fuzziness" to the knowledge of the parameter(s) as a result of the experimental data/measurements being usually affected by "noise"; and 3) the model uncertainty due to the modelling errors associated with the failure of the model in capturing the physics of the problem. This presents the need to not only perform an inference on the parameter(s), but also quantify the uncertainty associated with the estimates. An approach towards this would be Bayesian model updating, which serves as the context of this dissertation. The dissertation provides details to the efficient and robust approaches towards probabilistic parameter identification and model updating via the aforementioned approach. To realize this, an extensive literature review on Bayesian inference and the existing sampling tools is provided. This is done to identify the key research gaps, as well as limitations to the current sampling algorithms. From there, the Transitional Ensemble Markov Chain Monte Carlo sampler is proposed to which its strengths include its robustness in sampling from skewed distributions, quicker computational time, and the removal of any need for tuning by the users. To demonstrate this, the algorithm has been implemented on both numerical and real-world examples. The latter involves a structural health monitoring problem and the recent NASA-Langley Uncertainty Quantification challenge. Following which, the analysis is extended towards inferring time-varying parameter(s) via on-line Bayesian inference. This motivated the development of the Sequential Ensemble Monte Carlo sampler to which its strengths include its robustness in identifying the most probable Markov kernel under uncertainty. Such strengths are demonstrated through the experimental example involving a single-storey structure subjected to a time-varying Coulomb friction. Finally, the dissertation presents an approach to merge Artificial Intelligence tools with Bayesian statistics towards the probabilistic prediction of material properties for Nuclear power plant structures. Such development seeks to enable the Artificial Intelligence models to provide a more robust probabilistic prediction on the material properties under very limited data and model uncertainty. For the interest of the relevant practitioners, the algorithms to the proposed methods presented in the dissertation are made accessible on OpenCOSSAN, an open-source software for uncertainty quantification, as well as GitHub