Advances in optimisation algorithms and techniques for deep learning

In the last decade, deep learning(DL) has witnessed excellent performances on a variety of problems, including speech recognition, object recognition, detection, and natural language processing (NLP) among many others. Of these applications, one common challenge is to obtain ideal parameters during the training of the deep neural networks (DNN). These typical parameters are obtained by some optimisation techniques which have been studied extensively. These research have produced state-of-art(SOTA) results on speed and memory improvements for deep neural networks(NN) architectures. However, the SOTA optimisers have continued to be an active research area with no compilations of the existing optimisers reported in the literature. This paper provides an overview of the recent advances in optimisation algorithms and techniques used in DNN, highlighting the current SOTA optimisers, improvements made on these optimisation algorithms and techniques, alongside the trends in the development of optimisers used in training DL based models. The results of the search of the Scopus database for the optimisers in DL provides the articles reported as the summary of the DL optimisers. From what we can tell, there is no comprehensive compilation of the optimisation algorithms and techniques so far developed and used in DL research and applications, and this paper summarises these facts

A Review on the Application of Natural Computing in Environmental Informatics

Natural computing offers new opportunities to understand, model and analyze the complexity of the physical and human-created environment. This paper examines the application of natural computing in environmental informatics, by investigating related work in this research field. Various nature-inspired techniques are presented, which have been employed to solve different relevant problems. Advantages and disadvantages of these techniques are discussed, together with analysis of how natural computing is generally used in environmental research.Comment: Proc. of EnviroInfo 201

Lost in optimisation of water distribution systems? A literature review of system design

This is the final version of the article. Available from MDPI via the DOI in this record.Optimisation of water distribution system design is a well-established research field, which has been extremely productive since the end of the 1980s. Its primary focus is to minimise the cost of a proposed pipe network infrastructure. This paper reviews in a systematic manner articles published over the past three decades, which are relevant to the design of new water distribution systems, and the strengthening, expansion and rehabilitation of existing water distribution systems, inclusive of design timing, parameter uncertainty, water quality, and operational considerations. It identifies trends and limits in the field, and provides future research directions. Exclusively, this review paper also contains comprehensive information from over one hundred and twenty publications in a tabular form, including optimisation model formulations, solution methodologies used, and other important details

Novel sampling techniques for reservoir history matching optimisation and uncertainty quantification in flow prediction

Modern reservoir management has an increasing focus on accurately predicting the likely range of field recoveries. A variety of assisted history matching techniques has been developed across the research community concerned with this topic. These techniques are based on obtaining multiple models that closely reproduce the historical flow behaviour of a reservoir. The set of resulted history matched models is then used to quantify uncertainty in predicting the future performance of the reservoir and providing economic evaluations for different field development strategies. The key step in this workflow is to employ algorithms that sample the parameter space in an efficient but appropriate manner. The algorithm choice has an impact on how fast a model is obtained and how well the model fits the production data. The sampling techniques that have been developed to date include, among others, gradient based methods, evolutionary algorithms, and ensemble Kalman filter (EnKF). This thesis has investigated and further developed the following sampling and inference techniques: Particle Swarm Optimisation (PSO), Hamiltonian Monte Carlo, and Population Markov Chain Monte Carlo. The inspected techniques have the capability of navigating the parameter space and producing history matched models that can be used to quantify the uncertainty in the forecasts in a faster and more reliable way. The analysis of these techniques, compared with Neighbourhood Algorithm (NA), has shown how the different techniques affect the predicted recovery from petroleum systems and the benefits of the developed methods over the NA. The history matching problem is multi-objective in nature, with the production data possibly consisting of multiple types, coming from different wells, and collected at different times. Multiple objectives can be constructed from these data and explicitly be optimised in the multi-objective scheme. The thesis has extended the PSO to handle multi-objective history matching problems in which a number of possible conflicting objectives must be satisfied simultaneously. The benefits and efficiency of innovative multi-objective particle swarm scheme (MOPSO) are demonstrated for synthetic reservoirs. It is demonstrated that the MOPSO procedure can provide a substantial improvement in finding a diverse set of good fitting models with a fewer number of very costly forward simulations runs than the standard single objective case, depending on how the objectives are constructed. The thesis has also shown how to tackle a large number of unknown parameters through the coupling of high performance global optimisation algorithms, such as PSO, with model reduction techniques such as kernel principal component analysis (PCA), for parameterising spatially correlated random fields. The results of the PSO-PCA coupling applied to a recent SPE benchmark history matching problem have demonstrated that the approach is indeed applicable for practical problems. A comparison of PSO with the EnKF data assimilation method has been carried out and has concluded that both methods have obtained comparable results on the example case. This point reinforces the need for using a range of assisted history matching algorithms for more confidence in predictions

Improved cuckoo search based neural network learning algorithms for data classification

Artificial Neural Networks (ANN) techniques, mostly Back-Propagation Neural Network (BPNN) algorithm has been used as a tool for recognizing a mapping function among a known set of input and output examples. These networks can be trained with gradient descent back propagation. The algorithm is not definite in finding the global minimum of the error function since gradient descent may get stuck in local minima, where it may stay indefinitely. Among the conventional methods, some researchers prefer Levenberg-Marquardt (LM) because of its convergence speed and performance. On the other hand, LM algorithms which are derivative based algorithms still face a risk of getting stuck in local minima. Recently, a novel meta-heuristic search technique called cuckoo search (CS) has gained a great deal of attention from researchers due to its efficient convergence towards optimal solution. But Cuckoo search is prone to less optimal solution during exploration and exploitation process due to large step lengths taken by CS due to Levy flight. It can also be used to improve the balance between exploration and exploitation of CS algorithm, and to increase the chances of the egg’s survival. This research proposed an improved CS called hybrid Accelerated Cuckoo Particle Swarm Optimization algorithm (HACPSO) with Accelerated particle Swarm Optimization (APSO) algorithm. In the proposed HACPSO algorithm, initially accelerated particle swarm optimization (APSO) algorithm searches within the search space and finds the best sub-search space, and then the CS selects the best nest by traversing the sub-search space. This exploration and exploitation method followed in the proposed HACPSO algorithm makes it to converge to global optima with more efficiency than the original Cuckoo Search (CS) algorithm. Finally, the proposed CS hybrid variants such as; HACPSO, HACPSO-BP, HACPSO-LM, CSBP, CSLM, CSERN, and CSLMERN are evaluated and compared with conventional Back propagation Neural Network (BPNN), Artificial Bee Colony Neural Network (ABCNN), Artificial Bee Colony Back propagation algorithm (ABC-BP), and Artificial Bee Colony Levenberg-Marquardt algorithm (ABC-LM). Specifically, 6 benchmark classification datasets are used for training the hybrid Artificial Neural Network algorithms. Overall from the simulation results, it is realized that the proposed CS based NN algorithms performs better than all other proposed and conventional models in terms of CPU Time, MSE, SD and accuracy

Uncertainty evaluation of reservoir simulation models using particle swarms and hierarchical clustering

History matching production data in finite difference reservoir simulation models has been and always will be a challenge for the industry. The principal hurdles that need to be overcome are finding a match in the first place and more importantly a set of matches that can capture the uncertainty range of the simulation model and to do this in as short a time as possible since the bottleneck in this process is the length of time taken to run the model. This study looks at the implementation of Particle Swarm Optimisation (PSO) in history matching finite difference simulation models. Particle Swarms are a class of evolutionary algorithms that have shown much promise over the last decade. This method draws parallels from the social interaction of swarms of bees, flocks of birds and shoals of fish. Essentially a swarm of agents are allowed to search the solution hyperspace keeping in memory each individual’s historical best position and iteratively improving the optimisation by the emergent interaction of the swarm. An intrinsic feature of PSO is its local search capability. A sequential niching variation of the PSO has been developed viz. Flexi-PSO that enhances the exploration and exploitation of the hyperspace and is capable of finding multiple minima. This new variation has been applied to history matching synthetic reservoir simulation models to find multiple distinct history 3 matches to try to capture the uncertainty range. Hierarchical clustering is then used to post-process the history match runs to reduce the size of the ensemble carried forward for prediction. The success of the uncertainty modelling exercise is then assessed by checking whether the production profile forecasts generated by the ensemble covers the truth case

An Evaluation of Performance Enhancements to Particle Swarm Optimisation on Real-World Data

Swarm Computation is a relatively new optimisation paradigm. The basic premise is to model the collective behaviour of self-organised natural phenomena such as swarms, flocks and shoals, in order to solve optimisation problems. Particle Swarm Optimisation (PSO) is a type of swarm computation inspired by bird flocks or swarms of bees by modelling their collective social influence as they search for optimal solutions. In many real-world applications of PSO, the algorithm is used as a data pre-processor for a neural network or similar post processing system, and is often extensively modified to suit the application. The thesis introduces techniques that allow unmodified PSO to be applied successfully to a range of problems, specifically three extensions to the basic PSO algorithm: solving optimisation problems by training a hyperspatial matrix, using a hierarchy of swarms to coordinate optimisation on several data sets simultaneously, and dynamic neighbourhood selection in swarms. Rather than working directly with candidate solutions to an optimisation problem, the PSO algorithm is adapted to train a matrix of weights, to produce a solution to the problem from the inputs. The search space is abstracted from the problem data. A single PSO swarm optimises a single data set and has difficulties where the data set comprises disjoint parts (such as time series data for different days). To address this problem, we introduce a hierarchy of swarms, where each child swarm optimises one section of the data set whose gbest particle is a member of the swarm above in the hierarchy. The parent swarm(s) coordinate their children and encourage more exploration of the solution space. We show that hierarchical swarms of this type perform better than single swarm PSO optimisers on the disjoint data sets used. PSO relies on interaction between particles within a neighbourhood to find good solutions. In many PSO variants, possible interactions are arbitrary and fixed on initialisation. Our third contribution is a dynamic neighbourhood selection: particles can modify their neighbourhood, based on the success of the candidate neighbour particle. As PSO is intended to reflect the social interaction of agents, this change significantly increases the ability of the swarm to find optimal solutions. Applied to real-world medical and cosmological data, this modification is and shows improvements over standard PSO approaches with fixed neighbourhoods

Particle Swarm Optimization

Particle swarm optimization (PSO) is a population based stochastic optimization technique influenced by the social behavior of bird flocking or fish schooling.PSO shares many similarities with evolutionary computation techniques such as Genetic Algorithms (GA). The system is initialized with a population of random solutions and searches for optima by updating generations. However, unlike GA, PSO has no evolution operators such as crossover and mutation. In PSO, the potential solutions, called particles, fly through the problem space by following the current optimum particles. This book represents the contributions of the top researchers in this field and will serve as a valuable tool for professionals in this interdisciplinary field