A Modified Bayesian Optimization based Hyper-Parameter Tuning Approach for Extreme Gradient Boosting

It is already reported in the literature that the performance of a machine learning algorithm is greatly impacted by performing proper Hyper-Parameter optimization. One of the ways to perform Hyper-Parameter optimization is by manual search but that is time consuming. Some of the common approaches for performing Hyper-Parameter optimization are Grid search Random search and Bayesian optimization using Hyperopt. In this paper, we propose a brand new approach for hyperparameter improvement i.e. Randomized-Hyperopt and then tune the hyperparameters of the XGBoost i.e. the Extreme Gradient Boosting algorithm on ten datasets by applying Random search, Randomized-Hyperopt, Hyperopt and Grid Search. The performances of each of these four techniques were compared by taking both the prediction accuracy and the execution time into consideration. We find that the Randomized-Hyperopt performs better than the other three conventional methods for hyper-paramter optimization of XGBoost.Comment: Pre-review version of the paper submitted to IEEE 2019 Fifteenth International Conference on Information Processing (ICINPRO). The paper is accepted for publicatio

Algorithms for Hyper-Parameter Optimization

International audienceSeveral recent advances to the state of the art in image classification benchmarks have come from better configurations of existing techniques rather than novel ap- proaches to feature learning. Traditionally, hyper-parameter optimization has been the job of humans because they can be very efficient in regimes where only a few trials are possible. Presently, computer clusters and GPU processors make it pos- sible to run more trials and we show that algorithmic approaches can find better results. We present hyper-parameter optimization results on tasks of training neu- ral networks and deep belief networks (DBNs). We optimize hyper-parameters using random search and two new greedy sequential methods based on the ex- pected improvement criterion. Random search has been shown to be sufficiently efficient for learning neural networks for several datasets, but we show it is unreli- able for training DBNs. The sequential algorithms are applied to the most difficult DBN learning problems from [1] and find significantly better results than the best previously reported. This work contributes novel techniques for making response surface models P(y|x) in which many elements of hyper-parameter assignment (x) are known to be irrelevant given particular values of other elements

Hyper-parameter Optimization of a Convolutional Neural Network

In the world of machine learning, neural networks have become a powerful pattern recognition technique that gives a user the ability to interpret high-dimensional data whereas conventional methods, such as logistic regression, would fail. There exists many different types of neural networks, each containing its own set of hyper-parameters that are dependent on the type of analysis required, but the focus of this paper will be on the hyper-parameters of convolutional neural networks. Convolutional neural networks are commonly used for classifications of visual imagery. For example, if you were to build a network for the purpose of predicting a specific animal, it would hopefully output, with high fidelity, the correct classification of a new animal introduced to the model. Traditionally, hyper-parameters were rarely optimized because it required a lot of computational power and time. If hyper-parameters were adjusted, analysts would manually change a few hyper-parameters, re-run the model, and hopefully get a better classification accuracy. However, because of the advancements in technology, hyper-parameter tuning can now be done through complex and powerful optimization algorithms to improve the model. This paper implements and compares three different optimization techniques: random search, Bayesian Optimization with Gaussian Process, and tree of parzen estimator approach. The best performing technique is then improved through the Kiefer-Wolfowitz approximation

Comparative Study of Surrogate Techniques for CNN Hyperparameter Optimization

Optimizing hyper parameters in Convolutional Neural networks is a tedious process for many researchers and practitioners. It requires a high degree of expertise or experience to optimise the hyper parameters, and manual optimisation is likely to be biased. To date, methods or approaches to automate hyper parameter optimization include grid search, random search, and Genetic Algorithms (GAs). However, evaluating large number of sample points in the hyperparameter configuration space, as is typically required by these methods, is computationally expensive process. Hence, the objective of this paper is to explore regression as a surrogate technique in CNN hyperparameter optimisation. Performance in terms of accuracy, error rate, training time and coefficient of determination (R2) are evaluated and recorded. Although there is no significant performance difference between the resulting optimized Deep Learning and state-of-the-art on CIFAR-10 datasets, using regression as a surrogate technique for CNN hyperparameter optimization contributes to minimising the time taken for the optimization process, a benefit which has not been fully explored in the literature to the best of the author’s knowledge