Statistical Inference for Models with Intractable Normalizing Constants

In this dissertation, we have proposed two new algorithms for statistical inference for models with intractable normalizing constants: the Monte Carlo Metropolis-Hastings algorithm and the Bayesian Stochastic Approximation Monte Carlo algorithm. The MCMH algorithm is a Monte Carlo version of the Metropolis-Hastings algorithm. At each iteration, it replaces the unknown normalizing constant ratio by a Monte Carlo estimate. Although the algorithm violates the detailed balance condition, it still converges, as shown in the paper, to the desired target distribution under mild conditions. The BSAMC algorithm works by simulating from a sequence of approximated distributions using the SAMC algorithm. A strong law of large numbers has been established for BSAMC estimators under mild conditions. One significant advantage of our algorithms over the auxiliary variable MCMC methods is that they avoid the requirement for perfect samples, and thus it can be applied to many models for which perfect sampling is not available or very expensive. In addition, although the normalizing constant approximation is also involved in BSAMC, BSAMC can perform very robustly to initial guesses of parameters due to the powerful ability of SAMC in sample space exploration. BSAMC has also provided a general framework for approximated Bayesian inference for the models for which the likelihood function is intractable: sampling from a sequence of approximated distributions with their average converging to the target distribution. With these two illustrated algorithms, we have demonstrated how the SAMCMC method can be applied to estimate the parameters of ERGMs, which is one of the typical examples of statistical models with intractable normalizing constants. We showed that the resulting estimate is consistent, asymptotically normal and asymptotically efficient. Compared to the MCMLE and SSA methods, a significant advantage of SAMCMC is that it overcomes the model degeneracy problem. The strength of SAMCMC comes from its varying truncation mechanism, which enables SAMCMC to avoid the model degeneracy problem through re-initialization. MCMLE and SSA do not possess the re-initialization mechanism, and tend to converge to a solution near the starting point, so they often fail for the models which suffer from the model degeneracy problem

Approximate Methods For Otherwise Intractable Problems

Recent Monte Carlo methods have expanded the scope of the Bayesian statistical approach. In some situations however, computational methods are often impractically burdensome. We present new methods which reduce this burden and aim to extend the Bayesian toolkit further. This thesis is partitioned into three parts. The first part builds on the Approximate Bayesian Computation (ABC) method. Existing ABC methods often suffer from a local trapping problem which causes inefficient sampling. We present a new ABC framework which overcomes this problem and additionally allows for model selection as a by-product. We demonstrate that this framework conducts ABC inference with an adaptive ABC kernel and extend the framework to specify this kernel in a completely automated way. Furthermore, the ABC part of the thesis also presents a novel methodology for multifidelity ABC. This method constructs a computationally efficient sampler that minimises the approximation error induced by performing early acceptance with a low fidelity model. The second part of the thesis extends the Reversible Jump Monte Carlo method. Reversible Jump methods often suffer from poor mixing. It is possible to construct a “bridge” of intermediate models to facilitate the model transition. However, this scales poorly to big datasets because it requires many evaluations of the model likelihoods. Here we present a new method which greatly improves the scalability at the cost of some approximation error. However, we show that under weak conditions this error is well controlled and convergence is still achieved. The third part of the thesis introduces a multifidelity spatially clustered Gaussian process model. This model enables cheap modelling of nonstationary spatial statistical problems. The model outperforms existing methodology which perform poorly when predicting output at new spatial locations

Monte Carlo methods for intractable and doubly intractable density estimation

This thesis is concerned with Monte Carlo methods for intractable and doubly intractable density estimation. The primary focus is on the likelihood free method of Approximate Bayesian inference, where the presence of an intractable likelihood term necessitates the need for various approximation procedures. We propose a novel Sequential Monte Carlo based algorithm and demonstrate the significant efficiency (computational and statistical) improvements compared to the widely used SMC-ABC, in numerical experiments for a simple Gaussian model and a more realistic random network model. Further, we investigate a recently proposed algorithm, called SAMC-ABC, an adaptive MCMC algorithm where we also demonstrate some advantages over ABC-MCMC; primarily in the reduction of variance of the estimated means although at a cost of increased bias for which we propose a potential correction. In addition, we provide theoretical guarantees of ergodicity and convergence of another newly proposed algorithm termed Adaptive Noisy Exchange, that is aimed at problems of intractable normalising constants where regular MCMC cannot be employed. Finally, we propose potential improvements and future research directions for all of the considered algorithms

Trajectory averaging for stochastic approximation MCMC algorithms

The subject of stochastic approximation was founded by Robbins and Monro [Ann. Math. Statist. 22 (1951) 400--407]. After five decades of continual development, it has developed into an important area in systems control and optimization, and it has also served as a prototype for the development of adaptive algorithms for on-line estimation and control of stochastic systems. Recently, it has been used in statistics with Markov chain Monte Carlo for solving maximum likelihood estimation problems and for general simulation and optimizations. In this paper, we first show that the trajectory averaging estimator is asymptotically efficient for the stochastic approximation MCMC (SAMCMC) algorithm under mild conditions, and then apply this result to the stochastic approximation Monte Carlo algorithm [Liang, Liu and Carroll J. Amer. Statist. Assoc. 102 (2007) 305--320]. The application of the trajectory averaging estimator to other stochastic approximation MCMC algorithms, for example, a stochastic approximation MLE algorithm for missing data problems, is also considered in the paper.Comment: Published in at http://dx.doi.org/10.1214/10-AOS807 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

Permutation Testing and Semiparametric Regression: Efficient Computation, Tests of Matrix Structure, and l1 Smoothing Penalties

Part I: Permutation Testing Chapters 1 and 2: Fast Approximation of Small p-values in Permutation Tests by Partitioning the Permutations Researchers in genetics and other life sciences commonly use permutation tests to evaluate differences between groups. Permutation tests have desirable properties, including exactness if data are exchangeable, and are applicable even when the distribution of the test statistic is analytically intractable. However, permutation tests can be computationally intensive. We propose both an asymptotic approximation and a resampling algorithm for quickly estimating small permutation p-values (e.g. <10^-6) for the difference and ratio of means in two-sample tests. Our methods are based on the distribution of test statistics within and across partitions of the permutations, which we define. We present our methods and demonstrate their use through simulations and an application to cancer genomic data. Through simulations, we find that our resampling algorithm is more computationally efficient than another leading alternative, particularly for extremely small p-values (e.g. <10^-30). Through application to cancer genomic data, we find that our methods can successfully identify up- and down-regulated genes. While we focus on the difference and ratio of means, we speculate that our approaches may work in other settings Chapter 3: Tests of Matrix Structure for Construct Validation Psychologists and other behavioral scientists are frequently interested in whether a questionnaire reliably measures a latent construct. Attempts to address this issue are referred to as construct validation. We describe nonparametric hypothesis testing procedures to assess matrix structures, which can be used for construct validation. These methods are based on a quadratic assignment framework, and can be used either by themselves or to check the robustness of other methods. We investigate the performance of these matrix structure tests through simulations, and demonstrate their use by analyzing a big five personality traits questionnaire administered as part of the Health and Retirement Study. We also derive the rate of convergence for our overall test to better understand its behavior. Part II: Semiparametric regression Chapter 4: P-Splines with an l1 Penalty for Repeated Measures P-splines are penalized B-splines, in which finite order differences in coefficients are typically penalized with an l2 norm. P-splines can be used for semiparametric regression and can include random effects to account for within-subject variability. In addition to l2 penalties, l1-type penalties have been used in nonparametric and semiparametric regression to achieve greater flexibility, such as in locally adaptive regression splines, l1 trend filtering, and the fused lasso additive model. However, there has been less focus on using l1 penalties in P-splines, particularly for estimating conditional means. We demonstrate the potential benefits of using an l1 penalty in P-splines, with an emphasis on fitting non-smooth functions. We propose an estimation procedure using the alternating direction method of multipliers and cross validation, and provide degrees of freedom and approximate confidence bands based on a ridge approximation to the l1 penalized fit. We also demonstrate potential uses through simulations and an application to electrodermal activity data collected as part of a stress study.PHDBiostatisticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/138723/1/bdsegal_1.pd

Advances in statistical methodology and analysis in a study of ARC syndrome

This thesis presents statistical analysis and methodology development for a systems analysis of ARC syndrome. ARC is a genetic disease caused by mutations in one of two proteins, VPS33B and VIPAS39, of whose function little is known. Transcriptomic and metabolomic data are analysed to identify differentially expressed genes and pathways, and to highlight processes which are perturbed. Results consistently point to processes involved in cell polarisation and cell-cell adhesion, which is corroborated by experimental work. Beneficial suggestions for future experimental work are included and have already yielded interesting results. Motivated by the desire to incorporate knowledge of genetic dependencies into this analysis, methodology is developed to enable Bayesian inference for ‘doublyintractable distributions’. These models have a likelihood normalising term which is a function of unknown model parameters and which cannot be computed. This means that standard methods for sampling from the posterior, such as Markov chain Monte Carlo (MCMC), cannot be used. In the developed method, the likelihood is expressed as an infinite series which is then stochastically truncated. These unbiased, but possibly negative, estimates can then be used in a Pseudo-marginal MCMC scheme to compute expectations with respect to the posterior. Finally, methodology is developed to enable unbiased estimation for models in which data can be generated but no tractable likelihood is available. The main motivation for this is stochastic kinetic models used to describe complex and heterogeneous biological systems, but models of this type can be found across the sciences. Approximate Bayesian Computation is used to define a sequence of consistent Monte Carlo estimates, and these are then combined to produce an estimator which is unbiased with respect to the true posterior. Both approaches are demonstrated on a range of examples followed by a critical assessment of their strengths and weaknesses

Modeling of Locally Scaled Spatial Point Processes, and Applications in Image Analysis

Spatial point processes provide a statistical framework for modeling random arrangements of objects, which is of relevance in a variety of scientific disciplines, including ecology, spatial epidemiology and material science. Describing systematic spatial variations within this framework and developing methods for estimating parameters from empirical data constitute an active area of research. Image analysis, in particular, provides a range of scenarios to which point process models are applicable. Typical examples are images of trees in remote sensing, cells in biology, or composite structures in material science. Due to its real-world orientation and versatility, the class of the recently developed locally scaled point processes appears particularly suitable for the modeling of spatial object patterns. An unknown normalizing constant in the likelihood, however, makes inference complicated and requires elaborate techniques. This work presents an efficient Bayesian inference concept for locally scaled point processes. The suggested optimization procedure is applied to images of cross-sections through the stems of maize plants, where the goal is to accurately describe and classify different genotypes based on the spatial arrangement of their vascular bundles. A further spatial point process framework is specifically provided for the estimation of shape from texture. Texture learning and the estimation of surface orientation are two important tasks in pattern analysis and computer vision. Given the image of a scene in three-dimensional space, a frequent goal is to derive global geometrical knowledge, e.g. information on camera positioning and angle, from the local textural characteristics in the image. The statistical framework proposed comprises locally scaled point process strategies as well as the draft of a Bayesian marked point process model for inferring shape from texture