3 research outputs found
Rethinking Default Values: a Low Cost and Efficient Strategy to Define Hyperparameters
Machine Learning (ML) algorithms have been increasingly applied to problems
from several different areas. Despite their growing popularity, their
predictive performance is usually affected by the values assigned to their
hyperparameters (HPs). As consequence, researchers and practitioners face the
challenge of how to set these values. Many users have limited knowledge about
ML algorithms and the effect of their HP values and, therefore, do not take
advantage of suitable settings. They usually define the HP values by trial and
error, which is very subjective, not guaranteed to find good values and
dependent on the user experience. Tuning techniques search for HP values able
to maximize the predictive performance of induced models for a given dataset,
but have the drawback of a high computational cost. Thus, practitioners use
default values suggested by the algorithm developer or by tools implementing
the algorithm. Although default values usually result in models with acceptable
predictive performance, different implementations of the same algorithm can
suggest distinct default values. To maintain a balance between tuning and using
default values, we propose a strategy to generate new optimized default values.
Our approach is grounded on a small set of optimized values able to obtain
predictive performance values better than default settings provided by popular
tools. After performing a large experiment and a careful analysis of the
results, we concluded that our approach delivers better default values.
Besides, it leads to competitive solutions when compared to tuned values,
making it easier to use and having a lower cost. We also extracted simple rules
to guide practitioners in deciding whether to use our new methodology or a HP
tuning approach.Comment: 44 pages, 13 figure
Automatic selection of clustering algorithms using supervised graph embedding
The widespread adoption of machine learning (ML) techniques and the extensive
expertise required to apply them have led to increased interest in automated ML
solutions that reduce the need for human intervention. One of the main
challenges in applying ML to previously unseen problems is algorithm selection
- the identification of high-performing algorithm(s) for a given dataset, task,
and evaluation measure. This study addresses the algorithm selection challenge
for data clustering, a fundamental task in data mining that is aimed at
grouping similar objects. We present MARCO-GE, a novel meta-learning approach
for the automated recommendation of clustering algorithms. MARCO-GE first
transforms datasets into graphs and then utilizes a graph convolutional neural
network technique to extract their latent representation. Using the embedding
representations obtained, MARCO-GE trains a ranking meta-model capable of
accurately recommending top-performing algorithms for a new dataset and
clustering evaluation measure. Extensive evaluation on 210 datasets, 13
clustering algorithms, and 10 clustering measures demonstrates the
effectiveness of our approach and its superiority in terms of predictive and
generalization performance over state-of-the-art clustering meta-learning
approaches
A meta-learning recommender system for hyperparameter tuning: predicting when tuning improves SVM classifiers
For many machine learning algorithms, predictive performance is critically
affected by the hyperparameter values used to train them. However, tuning these
hyperparameters can come at a high computational cost, especially on larger
datasets, while the tuned settings do not always significantly outperform the
default values. This paper proposes a recommender system based on meta-learning
to identify exactly when it is better to use default values and when to tune
hyperparameters for each new dataset. Besides, an in-depth analysis is
performed to understand what they take into account for their decisions,
providing useful insights. An extensive analysis of different categories of
meta-features, meta-learners, and setups across 156 datasets is performed.
Results show that it is possible to accurately predict when tuning will
significantly improve the performance of the induced models. The proposed
system reduces the time spent on optimization processes, without reducing the
predictive performance of the induced models (when compared with the ones
obtained using tuned hyperparameters). We also explain the decision-making
process of the meta-learners in terms of linear separability-based hypotheses.
Although this analysis is focused on the tuning of Support Vector Machines, it
can also be applied to other algorithms, as shown in experiments performed with
decision trees.Comment: 49 pages, 11 figure