8,141 research outputs found
Minimax Rates for High-Dimensional Random Tessellation Forests
Random forests are a popular class of algorithms used for regression and
classification. The algorithm introduced by Breiman in 2001 and many of its
variants are ensembles of randomized decision trees built from axis-aligned
partitions of the feature space. One such variant, called Mondrian forests, was
proposed to handle the online setting and is the first class of random forests
for which minimax rates were obtained in arbitrary dimension. However, the
restriction to axis-aligned splits fails to capture dependencies between
features, and random forests that use oblique splits have shown improved
empirical performance for many tasks. In this work, we show that a large class
of random forests with general split directions also achieve minimax rates in
arbitrary dimension. This class includes STIT forests, a generalization of
Mondrian forests to arbitrary split directions, as well as random forests
derived from Poisson hyperplane tessellations. These are the first results
showing that random forest variants with oblique splits can obtain minimax
optimality in arbitrary dimension. Our proof technique relies on the novel
application of the theory of stationary random tessellations in stochastic
geometry to statistical learning theory.Comment: 20 page
TreeGrad: Transferring Tree Ensembles to Neural Networks
Gradient Boosting Decision Tree (GBDT) are popular machine learning
algorithms with implementations such as LightGBM and in popular machine
learning toolkits like Scikit-Learn. Many implementations can only produce
trees in an offline manner and in a greedy manner. We explore ways to convert
existing GBDT implementations to known neural network architectures with
minimal performance loss in order to allow decision splits to be updated in an
online manner and provide extensions to allow splits points to be altered as a
neural architecture search problem. We provide learning bounds for our neural
network.Comment: Technical Report on Implementation of Deep Neural Decision Forests
Algorithm. To accompany implementation here:
https://github.com/chappers/TreeGrad. Update: Please cite as: Siu, C. (2019).
"Transferring Tree Ensembles to Neural Networks". International Conference on
Neural Information Processing. Springer, 2019. arXiv admin note: text overlap
with arXiv:1909.1179
- …