23,222 research outputs found
Is One Hyperparameter Optimizer Enough?
Hyperparameter tuning is the black art of automatically finding a good
combination of control parameters for a data miner. While widely applied in
empirical Software Engineering, there has not been much discussion on which
hyperparameter tuner is best for software analytics. To address this gap in the
literature, this paper applied a range of hyperparameter optimizers (grid
search, random search, differential evolution, and Bayesian optimization) to
defect prediction problem. Surprisingly, no hyperparameter optimizer was
observed to be `best' and, for one of the two evaluation measures studied here
(F-measure), hyperparameter optimization, in 50\% cases, was no better than
using default configurations.
We conclude that hyperparameter optimization is more nuanced than previously
believed. While such optimization can certainly lead to large improvements in
the performance of classifiers used in software analytics, it remains to be
seen which specific optimizers should be applied to a new dataset.Comment: 7 pages, 2 columns, accepted for SWAN1
DP-HyPO: An Adaptive Private Hyperparameter Optimization Framework
Hyperparameter optimization, also known as hyperparameter tuning, is a widely
recognized technique for improving model performance. Regrettably, when
training private ML models, many practitioners often overlook the privacy risks
associated with hyperparameter optimization, which could potentially expose
sensitive information about the underlying dataset. Currently, the sole
existing approach to allow privacy-preserving hyperparameter optimization is to
uniformly and randomly select hyperparameters for a number of runs,
subsequently reporting the best-performing hyperparameter. In contrast, in
non-private settings, practitioners commonly utilize "adaptive" hyperparameter
optimization methods such as Gaussian process-based optimization, which select
the next candidate based on information gathered from previous outputs. This
substantial contrast between private and non-private hyperparameter
optimization underscores a critical concern. In our paper, we introduce
DP-HyPO, a pioneering framework for "adaptive" private hyperparameter
optimization, aiming to bridge the gap between private and non-private
hyperparameter optimization. To accomplish this, we provide a comprehensive
differential privacy analysis of our framework. Furthermore, we empirically
demonstrate the effectiveness of DP-HyPO on a diverse set of real-world and
synthetic datasets
Basic Enhancement Strategies When Using Bayesian Optimization for Hyperparameter Tuning of Deep Neural Networks
Compared to the traditional machine learning models, deep neural networks (DNN) are known to be highly sensitive to the choice of hyperparameters. While the required time and effort for manual tuning has been rapidly decreasing for the well developed and commonly used DNN architectures, undoubtedly DNN hyperparameter optimization will continue to be a major burden whenever a new DNN architecture needs to be designed, a new task needs to be solved, a new dataset needs to be addressed, or an existing DNN needs to be improved further. For hyperparameter optimization of general machine learning problems, numerous automated solutions have been developed where some of the most popular solutions are based on Bayesian Optimization (BO). In this work, we analyze four fundamental strategies for enhancing BO when it is used for DNN hyperparameter optimization. Specifically, diversification, early termination, parallelization, and cost function transformation are investigated. Based on the analysis, we provide a simple yet robust algorithm for DNN hyperparameter optimization - DEEP-BO (Diversified, Early-termination-Enabled, and Parallel Bayesian Optimization). When evaluated over six DNN benchmarks, DEEP-BO mostly outperformed well-known solutions including GP-Hedge, BOHB, and the speed-up variants that use Median Stopping Rule or Learning Curve Extrapolation. In fact, DEEP-BO consistently provided the top, or at least close to the top, performance over all the benchmark types that we have tested. This indicates that DEEP-BO is a robust solution compared to the existing solutions. The DEEP-BO code is publicly available at <uri>https://github.com/snu-adsl/DEEP-BO</uri>
- …