2 research outputs found
Rethinking Default Values: a Low Cost and Efficient Strategy to Define Hyperparameters
Machine Learning (ML) algorithms have been increasingly applied to problems
from several different areas. Despite their growing popularity, their
predictive performance is usually affected by the values assigned to their
hyperparameters (HPs). As consequence, researchers and practitioners face the
challenge of how to set these values. Many users have limited knowledge about
ML algorithms and the effect of their HP values and, therefore, do not take
advantage of suitable settings. They usually define the HP values by trial and
error, which is very subjective, not guaranteed to find good values and
dependent on the user experience. Tuning techniques search for HP values able
to maximize the predictive performance of induced models for a given dataset,
but have the drawback of a high computational cost. Thus, practitioners use
default values suggested by the algorithm developer or by tools implementing
the algorithm. Although default values usually result in models with acceptable
predictive performance, different implementations of the same algorithm can
suggest distinct default values. To maintain a balance between tuning and using
default values, we propose a strategy to generate new optimized default values.
Our approach is grounded on a small set of optimized values able to obtain
predictive performance values better than default settings provided by popular
tools. After performing a large experiment and a careful analysis of the
results, we concluded that our approach delivers better default values.
Besides, it leads to competitive solutions when compared to tuned values,
making it easier to use and having a lower cost. We also extracted simple rules
to guide practitioners in deciding whether to use our new methodology or a HP
tuning approach.Comment: 44 pages, 13 figure
A meta-learning recommender system for hyperparameter tuning: predicting when tuning improves SVM classifiers
For many machine learning algorithms, predictive performance is critically
affected by the hyperparameter values used to train them. However, tuning these
hyperparameters can come at a high computational cost, especially on larger
datasets, while the tuned settings do not always significantly outperform the
default values. This paper proposes a recommender system based on meta-learning
to identify exactly when it is better to use default values and when to tune
hyperparameters for each new dataset. Besides, an in-depth analysis is
performed to understand what they take into account for their decisions,
providing useful insights. An extensive analysis of different categories of
meta-features, meta-learners, and setups across 156 datasets is performed.
Results show that it is possible to accurately predict when tuning will
significantly improve the performance of the induced models. The proposed
system reduces the time spent on optimization processes, without reducing the
predictive performance of the induced models (when compared with the ones
obtained using tuned hyperparameters). We also explain the decision-making
process of the meta-learners in terms of linear separability-based hypotheses.
Although this analysis is focused on the tuning of Support Vector Machines, it
can also be applied to other algorithms, as shown in experiments performed with
decision trees.Comment: 49 pages, 11 figure