69,258 research outputs found
TreeGrad: Transferring Tree Ensembles to Neural Networks
Gradient Boosting Decision Tree (GBDT) are popular machine learning
algorithms with implementations such as LightGBM and in popular machine
learning toolkits like Scikit-Learn. Many implementations can only produce
trees in an offline manner and in a greedy manner. We explore ways to convert
existing GBDT implementations to known neural network architectures with
minimal performance loss in order to allow decision splits to be updated in an
online manner and provide extensions to allow splits points to be altered as a
neural architecture search problem. We provide learning bounds for our neural
network.Comment: Technical Report on Implementation of Deep Neural Decision Forests
Algorithm. To accompany implementation here:
https://github.com/chappers/TreeGrad. Update: Please cite as: Siu, C. (2019).
"Transferring Tree Ensembles to Neural Networks". International Conference on
Neural Information Processing. Springer, 2019. arXiv admin note: text overlap
with arXiv:1909.1179
NCART: Neural Classification and Regression Tree for Tabular Data
Deep learning models have become popular in the analysis of tabular data, as
they address the limitations of decision trees and enable valuable applications
like semi-supervised learning, online learning, and transfer learning. However,
these deep-learning approaches often encounter a trade-off. On one hand, they
can be computationally expensive when dealing with large-scale or
high-dimensional datasets. On the other hand, they may lack interpretability
and may not be suitable for small-scale datasets. In this study, we propose a
novel interpretable neural network called Neural Classification and Regression
Tree (NCART) to overcome these challenges. NCART is a modified version of
Residual Networks that replaces fully-connected layers with multiple
differentiable oblivious decision trees. By integrating decision trees into the
architecture, NCART maintains its interpretability while benefiting from the
end-to-end capabilities of neural networks. The simplicity of the NCART
architecture makes it well-suited for datasets of varying sizes and reduces
computational costs compared to state-of-the-art deep learning models.
Extensive numerical experiments demonstrate the superior performance of NCART
compared to existing deep learning models, establishing it as a strong
competitor to tree-based models
Enhancing Decision Tree based Interpretation of Deep Neural Networks through L1-Orthogonal Regularization
One obstacle that so far prevents the introduction of machine learning models
primarily in critical areas is the lack of explainability. In this work, a
practicable approach of gaining explainability of deep artificial neural
networks (NN) using an interpretable surrogate model based on decision trees is
presented. Simply fitting a decision tree to a trained NN usually leads to
unsatisfactory results in terms of accuracy and fidelity. Using L1-orthogonal
regularization during training, however, preserves the accuracy of the NN,
while it can be closely approximated by small decision trees. Tests with
different data sets confirm that L1-orthogonal regularization yields models of
lower complexity and at the same time higher fidelity compared to other
regularizers.Comment: 8 pages, 18th IEEE International Conference on Machine Learning and
Applications (ICMLA) 201
- …