A Modified Bayesian Optimization based Hyper-Parameter Tuning Approach for Extreme Gradient Boosting

It is already reported in the literature that the performance of a machine learning algorithm is greatly impacted by performing proper Hyper-Parameter optimization. One of the ways to perform Hyper-Parameter optimization is by manual search but that is time consuming. Some of the common approaches for performing Hyper-Parameter optimization are Grid search Random search and Bayesian optimization using Hyperopt. In this paper, we propose a brand new approach for hyperparameter improvement i.e. Randomized-Hyperopt and then tune the hyperparameters of the XGBoost i.e. the Extreme Gradient Boosting algorithm on ten datasets by applying Random search, Randomized-Hyperopt, Hyperopt and Grid Search. The performances of each of these four techniques were compared by taking both the prediction accuracy and the execution time into consideration. We find that the Randomized-Hyperopt performs better than the other three conventional methods for hyper-paramter optimization of XGBoost.Comment: Pre-review version of the paper submitted to IEEE 2019 Fifteenth International Conference on Information Processing (ICINPRO). The paper is accepted for publicatio

Above ground biomass and carbon sequestration estimation -Implementation of a sentinel-2 based exploratory workflow

Dissertation submitted in partial fulfilment of the requirements for the Degree of Master of Science in Geospatial TechnologiesThis work presents a Sentinel-2 based exploratory work ow for the estimation of Above Ground Biomass (AGB) and Carbon Sequestration (CS) in a subtropical forest. In the last decades, remote sensing-based studies on AGB have been widely investigated alongside with a variety of sensors, features and Machine Learning (ML) algorithms. Up-to-date and reliable mapping of such measures have been increasingly required by international commitments under the climate convention as well as by sustainable forest management practices. The proposed approach consists of 5 major steps: 1) generation of several Vegetation Indices (VI), biophysical parameters and texture measures; 2) feature selection with Mean Decrease in Impurity (MDI), Mean Decrease in Accuracy (MDA), L1 Regularization (LASSO), and Principal Component Analysis (PCA); 3) feature selection testing with k-Nearest Neighbour (kNN), Random Forest (RF), Extreme Gradient Boosting (XGB), and Arti cial Neural Network (ANN); 4) hyper-parameters ne-tuning with Grid Search, Random Search and Bayesian Optimization; and nally, 5) model explanation with the SHapley Additive exPlanations (SHAP) package, which to this day has not been investigated in the context of AGB mapping. The following results were obtained: 1) MDI was chosen as the best performing feature selection method by the XGB and the Deep Neural Network (DNN), MDA was chosen by the RF and the kNN, while LASSO was chosen by the Shallow Neural Network (SNN) and the Linear Neural Network (LNN); 2) before hyper-parameters optimization, the Deep Neural Network (DNN) yielded the best performance with a Root Mean Squared Error (RMSE) of 42.30 t=ha; 3) after hyper-parameters ne-tuning with Bayesian Optimization, the XGB model yielded the best performance with a RMSE of 37.79 t=ha; 4) model explanation with SHAP allowed for a deeper understanding of the features impact on the model predictions. Finally, the predicted AGB throughout the study area showed an average value of 83 t=ha, ranging from 0 t=ha to 346.56 t=ha. The related CS was estimated by using a conversion factor of 0.47

An Evolutionary Optimization Algorithm for Automated Classical Machine Learning

Machine learning is an evolving branch of computational algorithms that allow computers to learn from experiences, make predictions, and solve different problems without being explicitly programmed. However, building a useful machine learning model is a challenging process, requiring human expertise to perform various proper tasks and ensure that the machine learning\u27s primary objective --determining the best and most predictive model-- is achieved. These tasks include pre-processing, feature selection, and model selection. Many machine learning models developed by experts are designed manually and by trial and error. In other words, even experts need the time and resources to create good predictive machine learning models. The idea of automated machine learning (AutoML) is to automate a machine learning pipeline to release the burden of substantial development costs and manual processes. The algorithms leveraged in these systems have different hyper-parameters. On the other hand, different input datasets have various features. In both cases, the final performance of the model is closely related to the final selected configuration of features and hyper-parameters. That is why they are considered as crucial tasks in the AutoML. The challenges regarding the computationally expensive nature of tuning hyper-parameters and optimally selecting features create significant opportunities for filling the research gaps in the AutoML field. This dissertation explores how to select the features and tune the hyper-parameters of conventional machine learning algorithms efficiently and automatically. To address the challenges in the AutoML area, novel algorithms for hyper-parameter tuning and feature selection are proposed. The hyper-parameter tuning algorithm aims to provide the optimal set of hyper-parameters in three conventional machine learning models (Random Forest, XGBoost and Support Vector Machine) to obtain best scores regarding performance. On the other hand, the feature selection algorithm looks for the optimal subset of features to achieve the highest performance. Afterward, a hybrid framework is designed for both hyper-parameter tuning and feature selection. The proposed framework can discover close to the optimal configuration of features and hyper-parameters. The proposed framework includes the following components: (1) an automatic feature selection component based on artificial bee colony algorithms and machine learning training, and (2) an automatic hyper-parameter tuning component based on artificial bee colony algorithms and machine learning training for faster training and convergence of the learning models. The whole framework has been evaluated using four real-world datasets in different applications. This framework is an attempt to alleviate the challenges of hyper-parameter tuning and feature selection by using efficient algorithms. However, distributed processing, distributed learning, parallel computing, and other big data solutions are not taken into consideration in this framework

Hyperparameter Tuning for Machine and Deep Learning with R

This open access book provides a wealth of hands-on examples that illustrate how hyperparameter tuning can be applied in practice and gives deep insights into the working mechanisms of machine learning (ML) and deep learning (DL) methods. The aim of the book is to equip readers with the ability to achieve better results with significantly less time, costs, effort and resources using the methods described here. The case studies presented in this book can be run on a regular desktop or notebook computer. No high-performance computing facilities are required. The idea for the book originated in a study conducted by Bartz & Bartz GmbH for the Federal Statistical Office of Germany (Destatis). Building on that study, the book is addressed to practitioners in industry as well as researchers, teachers and students in academia. The content focuses on the hyperparameter tuning of ML and DL algorithms, and is divided into two main parts: theory (Part I) and application (Part II). Essential topics covered include: a survey of important model parameters; four parameter tuning studies and one extensive global parameter tuning study; statistical analysis of the performance of ML and DL methods based on severity; and a new, consensus-ranking-based way to aggregate and analyze results from multiple algorithms. The book presents analyses of more than 30 hyperparameters from six relevant ML and DL methods, and provides source code so that users can reproduce the results. Accordingly, it serves as a handbook and textbook alike

On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice

Machine learning algorithms have been used widely in various applications and areas. To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. It often requires deep knowledge of machine learning algorithms and appropriate hyper-parameter optimization techniques. Although several automatic optimization techniques exist, they have different strengths and drawbacks when applied to different types of problems. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization. This survey paper will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.Comment: 69 Pages, 10 tables, accepted in Neurocomputing, Elsevier. Github link: https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithm

Overcoming over–indebtedness with AI - A case study on the application of AutoML to research

Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced AnalyticsThis research examines how artificial intelligence may contribute to better understanding and overcoming over-indebtedness in contexts of high poverty risk. This study uses a field database of 1,654 over-indebted households to identify distinguishable clusters and to predict its risk factors. First, unsupervised machine learning generated three overindebtedness clusters: low-income (31.27%), low credit control (37.40%), and crisis-affected households (31.33%). These served as basis for a better understanding on the complex issue that is over-indebtedness. Second, a predictive model was developed to serve as a tool for policymakers and advisory services by streamlining the classification of overindebtedness profiles. On building such model, an AutoML approach was leveraged achieving performant results (92.1% accuracy score). Furthermore, within the AutoML framework, two techniques were employed, leading to a deeper discussion on the benefits and inner workings of such strategy. Ultimately, this research looks to contribute on three fronts: theoretical, by unfolding previously unexplored characteristics on the concept of over-indebtedness; methodological, by proposing AutoML as a powerful research tool accessible to investigators on many backgrounds; and social, by building real-world applications that aim at mitigating over-indebtedness and, consequently, poverty risk

Hyperparameter Tuning for Machine and Deep Learning with R

This open access book provides a wealth of hands-on examples that illustrate how hyperparameter tuning can be applied in practice and gives deep insights into the working mechanisms of machine learning (ML) and deep learning (DL) methods. The aim of the book is to equip readers with the ability to achieve better results with significantly less time, costs, effort and resources using the methods described here. The case studies presented in this book can be run on a regular desktop or notebook computer. No high-performance computing facilities are required. The idea for the book originated in a study conducted by Bartz & Bartz GmbH for the Federal Statistical Office of Germany (Destatis). Building on that study, the book is addressed to practitioners in industry as well as researchers, teachers and students in academia. The content focuses on the hyperparameter tuning of ML and DL algorithms, and is divided into two main parts: theory (Part I) and application (Part II). Essential topics covered include: a survey of important model parameters; four parameter tuning studies and one extensive global parameter tuning study; statistical analysis of the performance of ML and DL methods based on severity; and a new, consensus-ranking-based way to aggregate and analyze results from multiple algorithms. The book presents analyses of more than 30 hyperparameters from six relevant ML and DL methods, and provides source code so that users can reproduce the results. Accordingly, it serves as a handbook and textbook alike