Variational Bayesian Inference: Message Passing Schemes and Streamlined Multilevel Data Analysis

University of Technology Sydney. Faculty of Science.Mean-field variational Bayes (MFVB) is a deterministic technique for approximating intractable integrals arising in Bayesian inference. They are typically used for making approximate inference for parameters in complex statistical models. Most of its foundational literature and applications are in Machine Learning. However, in the age of “Big Data”, and by extension large sample sizes, MFVB has become an important tool in Statistics. The approximating schemes afforded by MFVB rely on heavy algebraic derivations across the model. The emergence of Big Data has resulted in more complex statistical models, making the process of formulating an MFVB algorithm cumbersome. Fortunately, the MFVB updating scheme can be simplified by representing the parameters of the statistical model in a probabilistic graph. The derivations are made more efficient by decomposing the required computations into calculations that are local to each node in the graph. We focus on constructing variational Bayesian inference algorithms based on a modularised format known as variational message passing (VMP), which is founded upon the notion of messages passed between fragments on a factor graph. Primitive functions, which represent the localised messages over factor graph fragments, are derived and can be called upon for direct implementation into arbitrarily large statistical models. The MFVB and VMP approaches result in superficially different algorithms, but converge to identical posterior density function approximations because they are founded upon the same optimisation problem. For complex statistical models, VMP has the advantage that the iterative updates are adjusted into a modularised format by taking advantage of the localised computations afforded by variational Bayesian methods. The resulting algorithm is a sequence of fragment-based functions that represent a compartmentalisation of the required algebra and computer coding. Despite the computational convenience of VMP algorithms over their MFVB counterparts, the speed of both classes is limited for multilevel data models, such as Gaussian response linear mixed models. Statistical inference on such models requires standard matrix operations, such as inversion and matrix-vector multiplication, on sparse matrices, which are difficult to achieve efficiently. Furthermore, computational storage issues restrict the size of such models. Streamlined matrix algebraic results are necessary for implementing fast frequentist and variational Bayesian inference, which is not inhibited by storage-greedy sparse matrix operations, on multilevel data models. This thesis develops factor graph fragment functions that can be used to build complex statistical models and achieves streamlined matrix algebraic derivations for multilevel data analysis