27 research outputs found
Tree ensemble kernels for Bayesian optimization with known constraints over mixed-feature spaces
Tree ensembles can be well-suited for black-box optimization tasks such as
algorithm tuning and neural architecture search, as they achieve good
predictive performance with little or no manual tuning, naturally handle
discrete feature spaces, and are relatively insensitive to outliers in the
training data. Two well-known challenges in using tree ensembles for black-box
optimization are (i) effectively quantifying model uncertainty for exploration
and (ii) optimizing over the piece-wise constant acquisition function. To
address both points simultaneously, we propose using the kernel interpretation
of tree ensembles as a Gaussian Process prior to obtain model variance
estimates, and we develop a compatible optimization formulation for the
acquisition function. The latter further allows us to seamlessly integrate
known constraints to improve sampling efficiency by considering
domain-knowledge in engineering settings and modeling search space symmetries,
e.g., hierarchical relationships in neural architecture search. Our framework
performs as well as state-of-the-art methods for unconstrained black-box
optimization over continuous/discrete features and outperforms competing
methods for problems combining mixed-variable feature spaces and known input
constraints.Comment: 27 pages, 9 figures, 4 table
Stenkul Fiord Palynostratigraphy
These data include raw counts of pollen and spores at Stenkul Fiord. Attached are also richness calculations and Shannon diversity & evenness estimates
Stenkul Fiord Palynostratigraphy
These data include raw counts of pollen and spores at Stenkul Fiord
ENTMOOT: A framework for optimization over ensemble tree models
Gradient boosted trees and other regression tree models perform well in a wide range of real-world, industrial applications. These tree models (i) offer insight into important prediction features, (ii) effectively manage sparse data, and (iii) have excellent prediction capabilities. Despite their advantages, they are generally unpopular for decision-making tasks and black-box optimization, which is due to their difficult-to-optimize structure and the lack of a reliable uncertainty measure. ENTMOOT is our new framework for integrating (already trained) tree models into larger optimization problems. The contributions of ENTMOOT include: (i) explicitly introducing a reliable uncertainty measure that is compatible with tree models, (ii) solving the larger optimization problems that incorporate these uncertainty aware tree models, (iii) proving that the solutions are globally optimal, i.e. no better solution exists. In particular, we show how the ENTMOOT approach allows a simple integration of tree models into decision-making and black-box optimization, where it proves as a strong competitor to commonly-used frameworks