Nonparametric estimation of first passage time distributions in flowgraph models

Statistical flowgraphs represent multistate semi-Markov processes using integral transforms of transition time distributions between adjacent states; these are combined algebraically and inverted to derive parametric estimates for first passage time distributions between nonadjacent states. This dissertation extends previous work in the field by developing estimation methods for flowgraphs using empirical transforms based on sample data, with no assumption of specific parametric probability models for transition times. We prove strong convergence of empirical flowgraph results to the exact parametric results; develop alternatives for numerical inversion of empirical transforms and compare them in terms of computational complexity, accuracy, and ability to determine error bounds; discuss (with examples) the difficulties of determining confidence bands for distribution estimates obtained in this way; develop confidence intervals for moment-based quantities such as the mean; and show how methods based on empirical transforms can be modified to accommodate censored data. Several applications of the nonparametric method, based on reliability and survival data, are presented in detail

Probabilistic Programming for Deep Learning

We propose the idea of deep probabilistic programming, a synthesis of advances for systems at the intersection of probabilistic modeling and deep learning. Such systems enable the development of new probabilistic models and inference algorithms that would otherwise be impossible: enabling unprecedented scales to billions of parameters, distributed and mixed precision environments, and AI accelerators; integration with neural architectures for modeling massive and high-dimensional datasets; and the use of computation graphs for automatic differentiation and arbitrary manipulation of probabilistic programs for flexible inference and model criticism. After describing deep probabilistic programming, we discuss applications in novel variational inference algorithms and deep probabilistic models. First, we introduce the variational Gaussian process (VGP), a Bayesian nonparametric variational family, which adapts its shape to match complex posterior distributions. The VGP generates approximate posterior samples by generating latent inputs and warping them through random non-linear mappings; the distribution over random mappings is learned during inference, enabling the transformed outputs to adapt to varying complexity of the true posterior. Second, we introduce hierarchical implicit models (HIMs). HIMs combine the idea of implicit densities with hierarchical Bayesian modeling, thereby defining models via simulators of data with rich hidden structure

Neural Empirical Bayes

Peer reviewe

Graph-Based Change-Point Detection

We consider the testing and estimation of change-points -- locations where the distribution abruptly changes -- in a data sequence. A new approach, based on scan statistics utilizing graphs representing the similarity between observations, is proposed. The graph-based approach is non-parametric, and can be applied to any data set as long as an informative similarity measure on the sample space can be defined. Accurate analytic approximations to the significance of graph-based scan statistics for both the single change-point and the changed interval alternatives are provided. Simulations reveal that the new approach has better power than existing approaches when the dimension of the data is moderate to high. The new approach is illustrated on two applications: The determination of authorship of a classic novel, and the detection of change in a network over time

Polynomially Adjusted Saddlepoint Density Approximations

This thesis aims at obtaining improved bona fide density estimates and approximants by means of adjustments applied to the widely used saddlepoint approximation. Said adjustments are determined by solving systems of equations resulting from a moment-matching argument. A hybrid density approximant that relies on the accuracy of the saddlepoint approximation in the distributional tails is introduced as well. A certain representation of noncentral indefinite quadratic forms leads to an initial approximation whose parameters are evaluated by simultaneously solving four equations involving the cumulants of the target distribution. A saddlepoint approximation to the distribution of quadratic forms is also discussed. By way of application, accurate approximations to the distributions of the Durbin-Watson statistic and a certain portmanteau test statistic are determined. It is explained that the moments of the latter can be evaluated either by defining an expected value operator via the symbolic approach or by resorting to a recursive relationship between the joint moments and cumulants of sums of products of quadratic forms. As well, the bivariate case is addressed by applying a polynomial adjustment to the product of the saddlepoint approximations of the marginal densities of the standardized variables. Furthermore, extensions to the context of density estimation are formulated and applied to several univariate and bivariate data sets. In this instance, sample moments and empirical cumulant-generating functions are utilized in lieu of their theoretical analogues. Interestingly, the methodology herein advocated for approximating bivariate distributions not only yields density estimates whose functional forms readily lend themselves to algebraic manipulations, but also gives rise to copula density functions that prove significantly more flexible than the conventional functional type

Advances in Semi-Nonparametric Density Estimation and Shrinkage Regression

This thesis advocates the use of shrinkage and penalty techniques for estimating the parameters of a regression model that comprises both parametric and nonparametric components and develops semi-nonparametric density estimation methodologies that are applicable in a regression context. First, a moment-based approach whereby a univariate or bivariate density function is approximated by means of a suitable initial density function that is adjusted by a linear combination of orthogonal polynomials is introduced. Such adjustments are shown to be mathematically equivalent to making use of standard polynomials in one or two variables. Once extended to apply to density estimation, in which case the sample moments are being utilized, the proposed technique readily lends itself to the modeling of massive univariate or bivariate data sets. As well, the resulting density functions are shown to be expressible as kernel density estimates via the Christoffel-Darboux formula. Additionally, it is established that a set of n observations is entirely specified by its first n moments. It is also explained that a univariate bona fide density approximation can be obtained by assuming that the derivative of the logarithm of the density function under consideration is expressible as a rational function or a polynomial. An explicit representation of the density function so obtained is derived and jointly sufficient statistics for its parameters are identified. Then, extensions of the proposed methodology to density estimation and multivariate settings are discussed. As a matter of fact, this approach constitutes a generalization of Pearson\u27s system of frequency curves. Several illustrative examples are presented including regression applications. Finally, an iterative algorithm involving shrinkage and pretest techniques is introduced for estimating the parameters of a certain semi-nonparametric model. It is theoretically established and numerically verified that the proposed estimators are more accurate than the unrestricted ones. This methodology is successfully applied to a mass spectrometry data set

Probabilistic Numerics and Uncertainty in Computations

We deliver a call to arms for probabilistic numerical methods: algorithms for numerical tasks, including linear algebra, integration, optimization and solving differential equations, that return uncertainties in their calculations. Such uncertainties, arising from the loss of precision induced by numerical calculation with limited time or hardware, are important for much contemporary science and industry. Within applications such as climate science and astrophysics, the need to make decisions on the basis of computations with large and complex data has led to a renewed focus on the management of numerical uncertainty. We describe how several seminal classic numerical methods can be interpreted naturally as probabilistic inference. We then show that the probabilistic view suggests new algorithms that can flexibly be adapted to suit application specifics, while delivering improved empirical performance. We provide concrete illustrations of the benefits of probabilistic numeric algorithms on real scientific problems from astrometry and astronomical imaging, while highlighting open problems with these new algorithms. Finally, we describe how probabilistic numerical methods provide a coherent framework for identifying the uncertainty in calculations performed with a combination of numerical algorithms (e.g. both numerical optimisers and differential equation solvers), potentially allowing the diagnosis (and control) of error sources in computations.Comment: Author Generated Postprint. 17 pages, 4 Figures, 1 Tabl