Advances in optimisation algorithms and techniques for deep learning

In the last decade, deep learning(DL) has witnessed excellent performances on a variety of problems, including speech recognition, object recognition, detection, and natural language processing (NLP) among many others. Of these applications, one common challenge is to obtain ideal parameters during the training of the deep neural networks (DNN). These typical parameters are obtained by some optimisation techniques which have been studied extensively. These research have produced state-of-art(SOTA) results on speed and memory improvements for deep neural networks(NN) architectures. However, the SOTA optimisers have continued to be an active research area with no compilations of the existing optimisers reported in the literature. This paper provides an overview of the recent advances in optimisation algorithms and techniques used in DNN, highlighting the current SOTA optimisers, improvements made on these optimisation algorithms and techniques, alongside the trends in the development of optimisers used in training DL based models. The results of the search of the Scopus database for the optimisers in DL provides the articles reported as the summary of the DL optimisers. From what we can tell, there is no comprehensive compilation of the optimisation algorithms and techniques so far developed and used in DL research and applications, and this paper summarises these facts

Population-based JPEG Image Compression: Problem Re-Formulation

The JPEG standard is widely used in different image processing applications. One of the main components of the JPEG standard is the quantisation table (QT) since it plays a vital role in the image properties such as image quality and file size. In recent years, several efforts based on population-based metaheuristic (PBMH) algorithms have been performed to find the proper QT(s) for a specific image, although they do not take into consideration the user's opinion. Take an android developer as an example, who prefers a small-size image, while the optimisation process results in a high-quality image, leading to a huge file size. Another pitfall of the current works is a lack of comprehensive coverage, meaning that the QT(s) can not provide all possible combinations of file size and quality. Therefore, this paper aims to propose three distinct contributions. First, to include the user's opinion in the compression process, the file size of the output image can be controlled by a user in advance. Second, to tackle the lack of comprehensive coverage, we suggest a novel representation. Our proposed representation can not only provide more comprehensive coverage but also find the proper value for the quality factor for a specific image without any background knowledge. Both changes in representation and objective function are independent of the search strategies and can be used with any type of population-based metaheuristic (PBMH) algorithm. Therefore, as the third contribution, we also provide a comprehensive benchmark on 22 state-of-the-art and recently-introduced PBMH algorithms on our new formulation of JPEG image compression. Our extensive experiments on different benchmark images and in terms of different criteria show that our novel formulation for JPEG image compression can work effectively.Comment: 39 pages, this paper is submitted to the related journa

TRUSS STRUCTURE OPTIMIZATION BASED ON IMPROVED WOLF PACK ALGORITHM

Aiming at the optimization of truss structure, a wolf pack algorithm based on chaos and improved search strategy was proposed. The mathematical model of truss optimization was constructed, and the classical truss structure was optimized. The results were compared with those of other optimization algorithms. When selecting and updating the initial position of wolves, chaos idea was used to distribute the initial value evenly in the solution space; phase factor was introduced to optimize the formula of wolf detection; information interaction between wolves is increased and the number of runs is reduced. The numerical results show that the improved wolf pack algorithm has the characteristics of fewer parameters, simple programming, easy implementation, fast convergence speed, and can quickly find the optimal solution. It is suitable for the optimization design of the section size of space truss structures

The Application of Nature-inspired Metaheuristic Methods for Optimising Renewable Energy Problems and the Design of Water Distribution Networks

This work explores the technical challenges that emerge when applying bio-inspired optimisation methods to real-world engineering problems. A number of new heuristic algorithms were proposed and tested to deal with these challenges. The work is divided into three main dimensions: i) One of the most significant industrial optimisation problems is optimising renewable energy systems. Ocean wave energy is a promising technology for helping to meet future growth in global energy demand. However, the current technologies of wave energy converters (WECs) are not fully developed because of technical engineering and design challenges. This work proposes new hybrid heuristics consisting of cooperative coevolutionary frameworks and neuro-surrogate optimisation methods for optimising WECs problem in three domains, including position, control parameters, and geometric parameters. Our problem-specific algorithms perform better than existing approaches in terms of higher quality results and the speed of convergence. ii) The second part applies search methods to the optimization of energy output in wind farms. Wind energy has key advantages in terms of technological maturity, cost, and life-cycle greenhouse gas emissions. However, designing an accurate local wind speed and power prediction is challenging. We propose two models for wind speed and power forecasting for two wind farms located in Sweden and the Baltic Sea by a combination of recurrent neural networks and evolutionary search algorithms. The proposed models are superior to other applied machine learning methods. iii) Finally, we investigate the design of water distribution systems (WDS) as another challenging real-world optimisation problem. WDS optimisation is demanding because it has a high-dimensional discrete search space and complex constraints. A hybrid evolutionary algorithm is suggested for minimising the cost of various water distribution networks and for speeding up the convergence rate of search.Thesis (Ph.D.) -- University of Adelaide, School of Computer Science, 202

Feature selection using enhanced particle swarm optimisation for classification models.

In this research, we propose two Particle Swarm Optimisation (PSO) variants to undertake feature selection tasks. The aim is to overcome two major shortcomings of the original PSO model, i.e., premature convergence and weak exploitation around the near optimal solutions. The first proposed PSO variant incorporates four key operations, including a modified PSO operation with rectified personal and global best signals, spiral search based local exploitation, Gaussian distribution-based swarm leader enhancement, and mirroring and mutation operations for worst solution improvement. The second proposed PSO model enhances the first one through four new strategies, i.e., an adaptive exemplar breeding mechanism incorporating multiple optimal signals, nonlinear function oriented search coefficients, exponential and scattering schemes for swarm leader, and worst solution enhancement, respectively. In comparison with a set of 15 classical and advanced search methods, the proposed models illustrate statistical superiority for discriminative feature selection for a total of 13 data sets

Evolving machine learning and deep learning models using evolutionary algorithms

Despite the great success in data mining, machine learning and deep learning models are yet subject to material obstacles when tackling real-life challenges, such as feature selection, initialization sensitivity, as well as hyperparameter optimization. The prevalence of these obstacles has severely constrained conventional machine learning and deep learning methods from fulfilling their potentials. In this research, three evolving machine learning and one evolving deep learning models are proposed to eliminate above bottlenecks, i.e. improving model initialization, enhancing feature representation, as well as optimizing model configuration, respectively, through hybridization between the advanced evolutionary algorithms and the conventional ML and DL methods. Specifically, two Firefly Algorithm based evolutionary clustering models are proposed to optimize cluster centroids in K-means and overcome initialization sensitivity as well as local stagnation. Secondly, a Particle Swarm Optimization based evolving feature selection model is developed for automatic identification of the most effective feature subset and reduction of feature dimensionality for tackling classification problems. Lastly, a Grey Wolf Optimizer based evolving Convolutional Neural Network-Long Short-Term Memory method is devised for automatic generation of the optimal topological and learning configurations for Convolutional Neural Network-Long Short-Term Memory networks to undertake multivariate time series prediction problems. Moreover, a variety of tailored search strategies are proposed to eliminate the intrinsic limitations embedded in the search mechanisms of the three employed evolutionary algorithms, i.e. the dictation of the global best signal in Particle Swarm Optimization, the constraint of the diagonal movement in Firefly Algorithm, as well as the acute contraction of search territory in Grey Wolf Optimizer, respectively. The remedy strategies include the diversification of guiding signals, the adaptive nonlinear search parameters, the hybrid position updating mechanisms, as well as the enhancement of population leaders. As such, the enhanced Particle Swarm Optimization, Firefly Algorithm, and Grey Wolf Optimizer variants are more likely to attain global optimality on complex search landscapes embedded in data mining problems, owing to the elevated search diversity as well as the achievement of advanced trade-offs between exploration and exploitation