59 research outputs found
Cooperative Coevolution for Non-Separable Large-Scale Black-Box Optimization: Convergence Analyses and Distributed Accelerations
Given the ubiquity of non-separable optimization problems in real worlds, in
this paper we analyze and extend the large-scale version of the well-known
cooperative coevolution (CC), a divide-and-conquer optimization framework, on
non-separable functions. First, we reveal empirical reasons of why
decomposition-based methods are preferred or not in practice on some
non-separable large-scale problems, which have not been clearly pointed out in
many previous CC papers. Then, we formalize CC to a continuous game model via
simplification, but without losing its essential property. Different from
previous evolutionary game theory for CC, our new model provides a much simpler
but useful viewpoint to analyze its convergence, since only the pure Nash
equilibrium concept is needed and more general fitness landscapes can be
explicitly considered. Based on convergence analyses, we propose a hierarchical
decomposition strategy for better generalization, as for any decomposition
there is a risk of getting trapped into a suboptimal Nash equilibrium. Finally,
we use powerful distributed computing to accelerate it under the multi-level
learning framework, which combines the fine-tuning ability from decomposition
with the invariance property of CMA-ES. Experiments on a set of
high-dimensional functions validate both its search performance and scalability
(w.r.t. CPU cores) on a clustering computing platform with 400 CPU cores
An approach to support generic topologies in distributed PSO algorithms in Spark
Particle Swarm Optimization (PSO) is a popular population-based search algorithm that has been applied to all kinds of complex optimization problems. Although the performance of the algorithm strongly depends on the social topology that determines the interaction between the particles during the search, current Metaheuristic Optimization Frameworks (MOFs) provide limited support for topologies. In this paper, we present an approach to support generic topologies in distributed PSO algorithms within a framework for the development and execution of populationbased metaheuristics in Spark, which is currently under development.Facultad de Informátic
On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
Machine learning algorithms have been used widely in various applications and
areas. To fit a machine learning model into different problems, its
hyper-parameters must be tuned. Selecting the best hyper-parameter
configuration for machine learning models has a direct impact on the model's
performance. It often requires deep knowledge of machine learning algorithms
and appropriate hyper-parameter optimization techniques. Although several
automatic optimization techniques exist, they have different strengths and
drawbacks when applied to different types of problems. In this paper,
optimizing the hyper-parameters of common machine learning models is studied.
We introduce several state-of-the-art optimization techniques and discuss how
to apply them to machine learning algorithms. Many available libraries and
frameworks developed for hyper-parameter optimization problems are provided,
and some open challenges of hyper-parameter optimization research are also
discussed in this paper. Moreover, experiments are conducted on benchmark
datasets to compare the performance of different optimization methods and
provide practical examples of hyper-parameter optimization. This survey paper
will help industrial users, data analysts, and researchers to better develop
machine learning models by identifying the proper hyper-parameter
configurations effectively.Comment: 69 Pages, 10 tables, accepted in Neurocomputing, Elsevier. Github
link:
https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithm
Recommended from our members
OptPlatform: metaheuristic optimisation framework for solving complex real-world problems
This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonWe optimise daily, whether that is planning a round trip that visits the most attractions within a given holiday budget or just taking a train instead of driving a car in a rush hour. Many problems, just like these, are solved by individuals as part of our daily schedule, and they are effortless and straightforward. If we now scale that to many individuals with many different schedules, like a school timetable, we get to a point where it is just not feasible or practical to solve by hand. In such instances, optimisation methods are used to obtain an optimal solution. In this thesis, a practical approach to optimisation has been taken by developing an optimisation platform with all the necessary tools to be used by practitioners who are not necessarily familiar with the subject of optimisation. First, a high-performance metaheuristic optimisation framework (MOF) called OptPlatform is implemented, and the versatility and performance are evaluated across multiple benchmarks and real-world optimisation problems. Results show that, compared to competing MOFs, the OptPlatform outperforms in both the solution quality and computation time. Second, the most suitable hardware platform for OptPlatform is determined by an in-depth analysis of Ant Colony Optimisation scaling across CPU, GPU and enterprise Xeon Phi. Contrary to the common benchmark problems used in the literature, the supply chain problem solved could not scale on GPUs. Third, a variety of metaheuristics are implemented into OptPlatform. Including, a new metaheuristic based on Imperialist Competitive Algorithm (ICA), called ICA with Independence and Constrained Assimilation (ICAwICA) is proposed. The ICAwICA was compared against two different types of benchmark problems, and results show the versatile application of the algorithm, matching and in some cases outperforming the custom-tuned approaches. Finally, essential MOF features like automatic algorithm selection and tuning, lacking on existing frameworks, are implemented in OptPlatform. Two novel approaches are proposed and compared to existing methods. Results indicate the superiority of the implemented tuning algorithms within constrained tuning budget environment
Short Papers of the 11th Conference on Cloud Computing Conference, Big Data & Emerging Topics (JCC-BD&ET 2023)
Compilación de los short papers presentados en las 11vas Jornadas de Cloud Computing, Big Data & Emerging Topics (JCC-BD&ET2023), llevadas a cabo en modalidad híbrida durante junio de 2023 y organizadas por el Instituto de Investigación en Informática LIDI (III-LIDI) y la Secretaría de Posgrado de la Facultad de Informática de la UNLP en colaboración con universidades de Argentina y del exterior.Facultad de Informátic
A Tent L\'evy Flying Sparrow Search Algorithm for Feature Selection: A COVID-19 Case Study
The "Curse of Dimensionality" induced by the rapid development of information
science, might have a negative impact when dealing with big datasets. In this
paper, we propose a variant of the sparrow search algorithm (SSA), called Tent
L\'evy flying sparrow search algorithm (TFSSA), and use it to select the best
subset of features in the packing pattern for classification purposes. SSA is a
recently proposed algorithm that has not been systematically applied to feature
selection problems. After verification by the CEC2020 benchmark function, TFSSA
is used to select the best feature combination to maximize classification
accuracy and minimize the number of selected features. The proposed TFSSA is
compared with nine algorithms in the literature. Nine evaluation metrics are
used to properly evaluate and compare the performance of these algorithms on
twenty-one datasets from the UCI repository. Furthermore, the approach is
applied to the coronavirus disease (COVID-19) dataset, yielding the best
average classification accuracy and the average number of feature selections,
respectively, of 93.47% and 2.1. Experimental results confirm the advantages of
the proposed algorithm in improving classification accuracy and reducing the
number of selected features compared to other wrapper-based algorithms
Advances in Artificial Intelligence: Models, Optimization, and Machine Learning
The present book contains all the articles accepted and published in the Special Issue “Advances in Artificial Intelligence: Models, Optimization, and Machine Learning” of the MDPI Mathematics journal, which covers a wide range of topics connected to the theory and applications of artificial intelligence and its subfields. These topics include, among others, deep learning and classic machine learning algorithms, neural modelling, architectures and learning algorithms, biologically inspired optimization algorithms, algorithms for autonomous driving, probabilistic models and Bayesian reasoning, intelligent agents and multiagent systems. We hope that the scientific results presented in this book will serve as valuable sources of documentation and inspiration for anyone willing to pursue research in artificial intelligence, machine learning and their widespread applications
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges
Most machine learning algorithms are configured by a set of hyperparameters whose values must be carefully chosen and which often considerably impact performance. To avoid a time-consuming and irreproducible manual process of trial-and-error to find well-performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods—for example, based on resampling error estimation for supervised machine learning—can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods, from simple techniques such as grid or random search to more advanced methods like evolution strategies, Bayesian optimization, Hyperband, and racing. This work gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with machine learning pipelines, runtime improvements, and parallelization. This article is categorized under: Algorithmic Development > Statistics Technologies > Machine Learning Technologies > Prediction
- …