12 research outputs found
Particle Gibbs for Bayesian Additive Regression Trees
Additive regression trees are flexible non-parametric models and popular
off-the-shelf tools for real-world non-linear regression. In application
domains, such as bioinformatics, where there is also demand for probabilistic
predictions with measures of uncertainty, the Bayesian additive regression
trees (BART) model, introduced by Chipman et al. (2010), is increasingly
popular. As data sets have grown in size, however, the standard
Metropolis-Hastings algorithms used to perform inference in BART are proving
inadequate. In particular, these Markov chains make local changes to the trees
and suffer from slow mixing when the data are high-dimensional or the best
fitting trees are more than a few layers deep. We present a novel sampler for
BART based on the Particle Gibbs (PG) algorithm (Andrieu et al., 2010) and a
top-down particle filtering algorithm for Bayesian decision trees
(Lakshminarayanan et al., 2013). Rather than making local changes to individual
trees, the PG sampler proposes a complete tree to fit the residual. Experiments
show that the PG sampler outperforms existing samplers in many settings
The Binary Space Partitioning-Tree Process
The Mondrian process represents an elegant and powerful approach for space
partition modelling. However, as it restricts the partitions to be
axis-aligned, its modelling flexibility is limited. In this work, we propose a
self-consistent Binary Space Partitioning (BSP)-Tree process to generalize the
Mondrian process. The BSP-Tree process is an almost surely right continuous
Markov jump process that allows uniformly distributed oblique cuts in a
two-dimensional convex polygon. The BSP-Tree process can also be extended using
a non-uniform probability measure to generate direction differentiated cuts.
The process is also self-consistent, maintaining distributional invariance
under a restricted subdomain. We use Conditional-Sequential Monte Carlo for
inference using the tree structure as the high-dimensional variable. The
BSP-Tree process's performance on synthetic data partitioning and relational
modelling demonstrates clear inferential improvements over the standard
Mondrian process and other related methods
Random Tessellation Forests
Space partitioning methods such as random forests and the Mondrian process
are powerful machine learning methods for multi-dimensional and relational
data, and are based on recursively cutting a domain. The flexibility of these
methods is often limited by the requirement that the cuts be axis aligned. The
Ostomachion process and the self-consistent binary space partitioning-tree
process were recently introduced as generalizations of the Mondrian process for
space partitioning with non-axis aligned cuts in the two dimensional plane.
Motivated by the need for a multi-dimensional partitioning tree with non-axis
aligned cuts, we propose the Random Tessellation Process (RTP), a framework
that includes the Mondrian process and the binary space partitioning-tree
process as special cases. We derive a sequential Monte Carlo algorithm for
inference, and provide random forest methods. Our process is self-consistent
and can relax axis-aligned constraints, allowing complex inter-dimensional
dependence to be captured. We present a simulation study, and analyse gene
expression data of brain tissue, showing improved accuracies over other
methods.Comment: 11 pages, 4 figure
Bayesian additive regression trees for probabilistic programming
Bayesian additive regression trees (BART) is a non-parametric method to approximate functions. It is a black-box method based on the sum of many trees where priors are used to regularize inference, mainly by restricting trees’ learning capacity so that no individual tree is able to explain the data, but rather the sum of trees. We discuss BART in the context of probabilistic programming languages (PPLs), specifically we introduce a BART implementation extending PyMC, a Python library for probabilistic programming. We present a few examples of models that can be built using this probabilistic programming-oriented version of BART, discuss recommendations for sample diagnostics and selection of model hyperparameters, and finally we close with limitations of the current approach and future extensions.Fil: Quiroga Andiñach, Miriana Esther. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - San Luis. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi". Universidad Nacional de San Luis. Facultad de Ciencias Físico, Matemáticas y Naturales. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi"; ArgentinaFil: Garay, Pablo Germán. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - San Luis. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi". Universidad Nacional de San Luis. Facultad de Ciencias Físico, Matemáticas y Naturales. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi"; ArgentinaFil: Alonso, Juan Manuel. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - San Luis. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi". Universidad Nacional de San Luis. Facultad de Ciencias Físico, Matemáticas y Naturales. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi"; ArgentinaFil: Loyola, Juan Martin. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - San Luis. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi". Universidad Nacional de San Luis. Facultad de Ciencias Físico, Matemáticas y Naturales. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi"; ArgentinaFil: Martín, Osvaldo Antonio. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - San Luis. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi". Universidad Nacional de San Luis. Facultad de Ciencias Físico, Matemáticas y Naturales. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi"; Argentin
Bayesian additive regression trees for probabilistic programming
Bayesian additive regression trees (BART) is a non-parametric method to
approximate functions. It is a black-box method based on the sum of many trees
where priors are used to regularize inference, mainly by restricting trees'
learning capacity so that no individual tree is able to explain the data, but
rather the sum of trees. We discuss BART in the context of probabilistic
programming languages (PPL), i.e., we present BART as a primitive that can be
used as a component of a probabilistic model rather than as a standalone model.
Specifically, we introduce the Python library PyMC-BART, which works by
extending PyMC, a library for probabilistic programming. We showcase a few
examples of models that can be built using PyMC-BART, discuss recommendations
for the selection of hyperparameters, and finally, we close with limitations of
our implementation and future directions for improvement.Comment: 22 pages, 17 figure
BART-based inference for Poisson processes
The effectiveness of Bayesian Additive Regression Trees (BART) has been
demonstrated in a variety of contexts including non parametric regression and
classification. Here we introduce a BART scheme for estimating the intensity of
inhomogeneous Poisson Processes. Poisson intensity estimation is a vital task
in various applications including medical imaging, astrophysics and network
traffic analysis. Our approach enables full posterior inference of the
intensity in a nonparametric regression setting. We demonstrate the performance
of our scheme through simulation studies on synthetic and real datasets in one
and two dimensions, and compare our approach to alternative approaches
Classic and Bayesian Tree-Based Methods
Tree-based methods are nonparametric techniques and machine-learning methods for data prediction and exploratory modeling. These models are one of valuable and powerful tools among data mining methods and can be used for predicting different types of outcome (dependent) variable: (e.g., quantitative, qualitative, and time until an event occurs (survival data)). Tree model is called classification tree/regression tree/survival tree based on the type of outcome variable. These methods have some advantages over against traditional statistical methods such as generalized linear models (GLMs), discriminant analysis, and survival analysis. Some of these advantages are: without requiring to determine assumptions about the functional form between outcome variable and predictor (independent) variables, invariant to monotone transformations of predictor variables, useful for dealing with nonlinear relationships and high-order interactions, deal with different types of predictor variable, ease of interpretation and understanding results without requiring to have statistical experience, robust to missing values, outliers, and multicollinearity. Several classic and Bayesian tree algorithms are proposed for classification and regression trees, and in this chapter, we provide a review of these algorithms and appropriate criteria for determining the predictive performance of them