191 research outputs found
Approximating Probability Densities by Iterated Laplace Approximations
The Laplace approximation is an old, but frequently used method to
approximate integrals for Bayesian calculations. In this paper we develop an
extension of the Laplace approximation, by applying it iteratively to the
residual, i.e., the difference between the current approximation and the true
function. The final approximation is thus a linear combination of multivariate
normal densities, where the coefficients are chosen to achieve a good fit to
the target distribution. We illustrate on real and artificial examples that the
proposed procedure is a computationally efficient alternative to current
approaches for approximation of multivariate probability densities. The
R-package iterLap implementing the methods described in this article is
available from the CRAN servers.Comment: to appear in Journal of Computational and Graphical Statistics,
http://pubs.amstat.org/loi/jcg
Time-varying Learning and Content Analytics via Sparse Factor Analysis
We propose SPARFA-Trace, a new machine learning-based framework for
time-varying learning and content analytics for education applications. We
develop a novel message passing-based, blind, approximate Kalman filter for
sparse factor analysis (SPARFA), that jointly (i) traces learner concept
knowledge over time, (ii) analyzes learner concept knowledge state transitions
(induced by interacting with learning resources, such as textbook sections,
lecture videos, etc, or the forgetting effect), and (iii) estimates the content
organization and intrinsic difficulty of the assessment questions. These
quantities are estimated solely from binary-valued (correct/incorrect) graded
learner response data and a summary of the specific actions each learner
performs (e.g., answering a question or studying a learning resource) at each
time instance. Experimental results on two online course datasets demonstrate
that SPARFA-Trace is capable of tracing each learner's concept knowledge
evolution over time, as well as analyzing the quality and content organization
of learning resources, the question-concept associations, and the question
intrinsic difficulties. Moreover, we show that SPARFA-Trace achieves comparable
or better performance in predicting unobserved learner responses than existing
collaborative filtering and knowledge tracing approaches for personalized
education
Model-based machine learning
Several decades of research in the field of machine learning have resulted in a multitude of different algorithms for solving a broad range of problems. To tackle a new application, a researcher typically tries to map their problem onto one of these existing methods, often influenced by their familiarity with specific algorithms and by the availability of corresponding software implementations. In this study, we describe an alternative methodology for applying machine learning, in which a bespoke solution is formulated for each new application. The solution is expressed through a compact modelling language, and the corresponding custom machine learning code is then generated automatically. This model-based approach offers several major advantages, including the opportunity to create highly tailored models for specific scenarios, as well as rapid prototyping and comparison of a range of alternative models. Furthermore, newcomers to the field of machine learning do not have to learn about the huge range of traditional methods, but instead can focus their attention on understanding a single modelling environment. In this study, we show how probabilistic graphical models, coupled with efficient inference algorithms, provide a very flexible foundation for model-based machine learning, and we outline a large-scale commercial application of this framework involving tens of millions of users. We also describe the concept of probabilistic programming as a powerful software environment for model-based machine learning, and we discuss a specific probabilistic programming language called Infer.NET, which has been widely used in practical applications
Noise and nonlinearities in high-throughput data
High-throughput data analyses are becoming common in biology, communications,
economics and sociology. The vast amounts of data are usually represented in
the form of matrices and can be considered as knowledge networks. Spectra-based
approaches have proved useful in extracting hidden information within such
networks and for estimating missing data, but these methods are based
essentially on linear assumptions. The physical models of matching, when
applicable, often suggest non-linear mechanisms, that may sometimes be
identified as noise. The use of non-linear models in data analysis, however,
may require the introduction of many parameters, which lowers the statistical
weight of the model. According to the quality of data, a simpler linear
analysis may be more convenient than more complex approaches.
In this paper, we show how a simple non-parametric Bayesian model may be used
to explore the role of non-linearities and noise in synthetic and experimental
data sets.Comment: 12 pages, 3 figure
A Geometric Variational Approach to Bayesian Inference
We propose a novel Riemannian geometric framework for variational inference
in Bayesian models based on the nonparametric Fisher-Rao metric on the manifold
of probability density functions. Under the square-root density representation,
the manifold can be identified with the positive orthant of the unit
hypersphere in L2, and the Fisher-Rao metric reduces to the standard L2 metric.
Exploiting such a Riemannian structure, we formulate the task of approximating
the posterior distribution as a variational problem on the hypersphere based on
the alpha-divergence. This provides a tighter lower bound on the marginal
distribution when compared to, and a corresponding upper bound unavailable
with, approaches based on the Kullback-Leibler divergence. We propose a novel
gradient-based algorithm for the variational problem based on Frechet
derivative operators motivated by the geometry of the Hilbert sphere, and
examine its properties. Through simulations and real-data applications, we
demonstrate the utility of the proposed geometric framework and algorithm on
several Bayesian models
Learning a Factor Model via Regularized PCA
We consider the problem of learning a linear factor model. We propose a
regularized form of principal component analysis (PCA) and demonstrate through
experiments with synthetic and real data the superiority of resulting estimates
to those produced by pre-existing factor analysis approaches. We also establish
theoretical results that explain how our algorithm corrects the biases induced
by conventional approaches. An important feature of our algorithm is that its
computational requirements are similar to those of PCA, which enjoys wide use
in large part due to its efficiency
Fabular: regression formulas as probabilistic programming
Regression formulas are a domain-specific language adopted by several R packages for describing an important and useful class of statistical models: hierarchical linear regressions. Formulas are succinct, expressive, and clearly popular, so are they a useful addition to probabilistic programming languages? And what do they mean? We propose a core calculus of hierarchical linear regression, in which regression coefficients are themselves defined by nested regressions (unlike in R). We explain how our calculus captures the essence of the formula DSL found in R. We describe the design and implementation of Fabular, a version of the Tabular schema-driven probabilistic programming language, enriched with formulas based on our regression calculus. To the best of our knowledge, this is the first formal description of the core ideas of R's formula notation, the first development of a calculus of regression formulas, and the first demonstration of the benefits of composing regression formulas and latent variables in a probabilistic programming language.Adam Ścibior received travel support from the DARPA PPAML programme. Marcin Szymczak was supported by Microsoft Research through its PhD Scholarship Programme.This is the author accepted manuscript. The final version is available from the Association of Computer Machinery via http://dx.doi.org/10.1145/2837614.283765
On the Use of Upper Trust Bounds in Constrained Bayesian Optimization Infill Criterion
In order to handle constrained optimization problems with a large number of design variables, a new approach has been proposed to address constraints in a surrogate-based optimization framework. This approach focuses on sequential enrichment using adaptive surrogate models based on Bayesian optimization approach, and Gaussian process models. A constraints criterion using the uncertainty estimation of the Gaussian process models is introduced. Different evolutions of the algorithm, based on the accuracy of the constraints surrogate models, are used for selecting the infill sample points. The resulting algorithm has been tested on the well known modified Branin optimization problem
- …