Metaheuristic design of feedforward neural networks: a review of two decades of research

Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era

Development of a sustainable groundwater management strategy and sequential compliance monitoring to control saltwater intrusion in coastal aquifers

The coastal areas of the world are characterized by high population densities, an abundance of food, and increased economic activities. These increasing human settlements, subsequent increases in agricultural developments and economic activities demand an increasing amount quantity of freshwater supplies to different sectors. Groundwater in coastal aquifers is one of the most important sources of freshwater supplies. Over exploitation of this coastal groundwater resource results in seawater intrusion and subsequent deterioration of groundwater quality in coastal aquifers. In addition, climate change induced sea level rise, in combination with the effect of excessive groundwater extraction, can accelerate the seawater intrusion. Adequate supply of good quality water to different sectors in coastal areas can be ensured by adoption of a proper management strategy for groundwater extraction. Optimal use of the coastal groundwater resource is one of the best management options, which can be achieved by employing a properly developed optimal groundwater extraction strategy. Coupled simulation-optimization (S-O) approaches are essential tools to obtain the optimal groundwater extraction patterns. This study proposes approaches for developing multiple objective management of coastal aquifers with the aid of barrier extraction wells as hydraulic control measure of saltwater intrusion in multilayered coastal aquifer systems. Therefore, two conflicting objectives of management policy are considered in this research, i.e. maximizing total groundwater extraction for advantageous purposes, and minimizing the total amount of water abstraction from barrier extraction wells. The study also proposes an adaptive management strategy for coastal aquifers by developing a three-dimensional (3-D) monitoring network design. The performance of the proposed methodologies is evaluated by using both an illustrative multilayered coastal aquifer system and a real life coastal aquifer study area. Coupled S-O approach is used as the basic tool to develop a saltwater intrusion management model to obtain the optimal groundwater extraction rates from a combination of feasible solutions on the Pareto optimal front. Simulation of saltwater intrusion processes requires solution of density dependent coupled flow and solute transport numerical simulation models that are computationally intensive. Therefore, computational efficiency in the coupled S-O approach is achieved by using an approximate emulator of the accompanying physical processes of coastal aquifers. These emulators, often known as surrogate models or meta-models, can replace the computationally intensive numerical simulation model in a coupled S-O approach for achieving computational efficiency. A number of meta-models have been developed and compared in this study for integration with the optimization algorithm in order to develop saltwater intrusion management model. Fuzzy Inference System (FIS), Adaptive Neuro Fuzzy Inference System (ANFIS), Multivariate Adaptive Regression Spline (MARS), and Gaussian Process Regression (GPR) based meta-models are developed in the present study for approximating coastal aquifer responses to groundwater extraction. Properly trained and tested meta-models are integrated with a Controlled Elitist Multiple Objective Genetic Algorithm (CEMOGA) within a coupled S-O approach. In each iteration of the optimization algorithm, the meta-models are used to compute the corresponding salinity concentrations for a set of candidate pumping patterns generated by the optimization algorithm. Upon convergence, the non-dominated global optimal solutions are obtained as the Pareto optimal front, which represents a trade-off between the two conflicting objectives of the pumping management problem. It is observed from the solutions of the meta-model based coupled S-O approach that the considered meta-models are capable of producing a Pareto optimal set of solutions quite accurately. However, each meta-modelling approach has distinct advantages over the others when utilized within the integrated S-O approach. Uncertainties in estimating complex flow and solute transport processes in coastal aquifers demand incorporation of the uncertainties related to some of the model parameters. Multidimensional heterogeneity of aquifer properties such as hydraulic conductivity, compressibility, and bulk density are considered as major sources of uncertainty in groundwater modelling system. Other sources of uncertainty are associated with spatial and temporal variability of hydrologic as well as human interventions, e.g. aquifer recharge and transient groundwater extraction patterns. Different realizations of these uncertain model parameters are obtained from different statistical distributions. FIS based meta-models are advanced to a Genetic Algorithm (GA) tuned hybrid FIS model (GA-FIS), to emulate physical processes of coastal aquifers and to evaluate responses of the coastal aquifers to groundwater extraction under groundwater parameter uncertainty. GA is used to tune the FIS parameters in order to obtain the optimal FIS structure. The GA-FIS models thus obtained are linked externally to the CEMOGA in order to derive an optimal pumping management strategy using the coupled S-O approach. The evaluation results show that the proposed saltwater intrusion management model is able to derive reliable optimal groundwater extraction strategies to control saltwater intrusion for the illustrative multilayered coastal aquifer system. The optimal management strategies obtained as solutions of GA-FIS based management models are shown to be reliable and accurate within the specified ranges of values for different realizations of uncertain groundwater parameters. One of the major concerns of the meta-model based integrated S-O approach is the uncertainty associated with the meta-model predictions. These prediction uncertainties, if not addressed properly, may propagate to the optimization procedures, and may deteriorate the optimality of the solutions. A standalone meta-model, when used within an optimal management model, may result in the optimization routine producing actually suboptimal solutions that may undermine the optimality of the groundwater extraction strategies. Therefore, this study proposes an ensemble approach to address the prediction uncertainties of meta-models. Ensemble is an approach to assimilate multiple similar or different algorithms or base learners (emulators). The basic idea of ensemble lies in developing a more reliable and robust prediction tool that incorporates each individual emulator's unique characteristic in order to predict future scenarios. Each individual member of the ensemble contains different input -output mapping functions. Based on their own mapping functions, these individual emulators provide varied predictions on the response variable. Therefore, the combined prediction of the ensemble is likely to be less biased and more robust, reliable, and accurate than that of any of the individual members of the ensemble. Performance of the ensemble meta-models is evaluated using an illustrative coastal aquifer study area. The results indicate that the meta-model based ensemble modelling approach is able to provide reliable solutions for a multilayered coastal aquifer management problem. Relative sea level rise, providing an additional saline water head at the seaside, has a significant impact on an increase in the salinization process of the coastal aquifers. Although excessive groundwater withdrawal is considered as the major cause of saltwater intrusion, relative sea level rise, in combination with the effect of excessive groundwater pumping, can exacerbate the already vulnerable coastal aquifers. This study incorporates the effects of relative sea level rise on the optimized groundwater extraction values for the specified management period. Variation of water concentrations in the tidal river and seasonal fluctuation of river water stage are also incorporated. Three meta-models are developed from the solution results of the numerical simulation model that simulates the coupled flow and solute transport processes in a coastal aquifer system. The results reveal that the proposed meta-models are capable of predicting density dependent coupled flow and solute transport patterns quite accurately. Based on the comparison results, the best meta-model is selected as a computationally cheap substitute of the simulation model in the coupled S-O based saltwater intrusion management model. The performance of the proposed methodology is evaluated for an illustrative multilayered coastal aquifer system in which the effect of climate change induced sea level rise is incorporated for the specified management period. The results show that the proposed saltwater intrusion management model provides acceptable, accurate, and reliable solutions while significantly improving computational efficiency in the coupled S-O methodology. The success of the developed management strategy largely depends on how accurately the prescribed management policy is implemented in real life situations. The actual implementation of a prescribed management strategy often differs from the prescribed planned strategy due to various uncertainties in predicting the consequences, as well as practical constraints, including noncompliance with the prescribed strategy. This results in actual consequences of a management strategy differing from the intended results. To bring the management consequences closer to the intended results, adaptive management strategies can be sequentially modified at different stages of the management horizon using feedback measurements from a deigned monitoring network. This feedback information can be the actual spatial and temporal concentrations resulting from the implementation of actual management strategy. Therefore, field-scale compliance of the developed coastal aquifer management strategy is a crucial aspect of an optimally designed groundwater extraction policy. A 3-D compliance monitoring network design methodology is proposed in this study in order to develop an adaptive and sequentially modified management policy, which aims to improve optimal and justifiable use of groundwater resources in coastal aquifers. In the first step, an ensemble meta-model based multiple objective prescriptive model is developed using a coupled S-O approach in order to derive a set of Pareto optimal groundwater extraction strategies. Prediction uncertainty of meta-models is addressed by utilizing a weighted average ensemble using Set Pair Analysis. In the second step, a monitoring network is designed for evaluating the compliance of the implemented strategies with the prescribed management goals due to possible uncertainties associated with field-scale application of the proposed management policy. Optimal monitoring locations are obtained by maximizing Shannon's entropy between the saltwater concentrations at the selected potential locations. Performance of the proposed 3-D sequential compliance monitoring network design is assessed for an illustrative multilayered coastal aquifer study area. The performance evaluations show that sequential improvements of optimal management strategy are possible by utilizing saltwater concentrations measurements at the proposed optimal compliance monitoring locations. The integrated S-O approach is used to develop a saltwater intrusion management model for a real world coastal aquifer system in the Barguna district of southern Bangladesh. The aquifer processes are simulated by using a 3-D finite element based combined flow and solute transport numerical code. The modelling and management of seawater intrusion processes are performed based on very limited hydrogeological data. The model is calibrated with respect to hydraulic heads for a period of five years from April 2010 to April 2014. The calibrated model is validated for the next three-year period from April 2015 to April 2017. The calibrated and partially validated model is then used within the integrated S-O approach to develop optimal groundwater abstraction patterns to control saltwater intrusion in the study area. Computational efficiency of the management model is achieved by using a MARS based meta-model approximately emulating the combined flow and solute transport processes of the study area. This limited evaluation demonstrates that a planned transient groundwater abstraction strategy, acquired as solution results of a meta-model based integrated S-O approach, is a useful management strategy for optimized water abstraction and saltwater intrusion control. This study shows the capability of the MARS meta-model based integrated S-O approach to solve real-life complex management problems in an efficient manner

Automatic Tuning of a Retina Model for a Cortical Visual Neuroprosthesis Using a Multi-Objective Optimization Genetic Algorithm

The retina is a very complex neural structure, which contains many different types of neurons interconnected with great precision, enabling sophisticated conditioning and coding of the visual information before it is passed via the optic nerve to higher visual centers. The encoding of visual information is one of the basic questions in visual and computational neuroscience and is also of seminal importance in the field of visual prostheses. In this framework, it is essential to have artificial retina systems to be able to function in a way as similar as possible to the biological retinas. This paper proposes an automatic evolutionary multi-objective strategy based on the NSGA-II algorithm for tuning retina models. Four metrics were adopted for guiding the algorithm in the search of those parameters that best approximate a synthetic retinal model output with real electrophysiological recordings. Results show that this procedure exhibits a high flexibility when different trade-offs has to be considered during the design of customized neuro prostheses

A neuroevolutionary approach to feature selection using multiobjective evolutionary algorithms

Feature selection plays a central role in predictive analysis where datasets have hundreds or thousands of variables available. It can also reduce the overall training time and the computational costs of the classifiers used. However, feature selection methods can be computationally intensive or dependent of human expertise to analyze data. This study proposes a neuroevolutionary approach which uses multiobjective evolutionary algorithms to optimize neural network parameters in order to find the best network able to identify the most important variables of analyzed data. Classification is done through a Support Vector Machine (SVM) classifier where specific parameters are also optimized. The method is applied to datasets with different number of features and classes.FCT - Fundação para a Ciência e Tecnologia in the scope of the projects: PEst-OE/EEI/UI0319/2014, UID/MAT/00013/2013, UID/CEC/ 00319/2019 and the European project MSCA-RISE-2015, NEWEX, with reference 734205

MEG: Multi-objective Ensemble Generation for Software Defect Prediction

Background: Defect Prediction research aims at assisting software engineers in the early identification of software defect during the development process. A variety of automated approaches, ranging from traditional classification models to more sophisticated learning approaches, have been explored to this end. Among these, recent studies have proposed the use of ensemble prediction models (i.e., aggregation of multiple base classifiers) to build more robust defect prediction models. / Aims: In this paper, we introduce a novel approach based on multi-objective evolutionary search to automatically generate defect prediction ensembles. Our proposal is not only novel with respect to the more general area of evolutionary generation of ensembles, but it also advances the state-of-the-art in the use of ensemble in defect prediction. / Method: We assess the effectiveness of our approach, dubbed as Multi-objective Ensemble Generation (MEG), by empirically benchmarking it with respect to the most related proposals we found in the literature on defect prediction ensembles and on multi-objective evolutionary ensembles (which, to the best of our knowledge, had never been previously applied to tackle defect prediction). / Result: Our results show that MEG is able to generate ensembles which produce similar or more accurate predictions than those achieved by all the other approaches considered in 73% of the cases (with favourable large effect sizes in 80% of them). / Conclusions: MEG is not only able to generate ensembles that yield more accurate defect predictions with respect to the benchmarks considered, but it also does it automatically, thus relieving the engineers from the burden of manual design and experimentation

Multi - objective cooperative neuro - evolution of recurrent neural networks for time series prediction

Cooperative coevolution is an evolutionary computation method which solves a problem by decomposing it into smaller subcomponents. Multi-objective optimization deals with conflicting objectives and produces multiple optimal solutions instead of a single global optimal solution. In previous work, a multi-objective cooperative co-evolutionary method was introduced for training feedforward neural networks on time series problems. In this paper, the same method is used for training recurrent neural networks. The proposed approach is tested on time series problems in which the different time-lags represent the different objectives. Multiple pre-processed datasets distinguished by their time-lags are used for training and testing. This results in the discovery of a single neural network that can correctly give predictions for data pre-processed using different time-lags. The method is tested on several benchmark time series problems on which it gives a competitive performance in comparison to the methods in the literature

AI-Assisted Discovery of Quantitative and Formal Models in Social Science

In social science, formal and quantitative models, such as ones describing economic growth and collective action, are used to formulate mechanistic explanations, provide predictions, and uncover questions about observed phenomena. Here, we demonstrate the use of a machine learning system to aid the discovery of symbolic models that capture nonlinear and dynamical relationships in social science datasets. By extending neuro-symbolic methods to find compact functions and differential equations in noisy and longitudinal data, we show that our system can be used to discover interpretable models from real-world data in economics and sociology. Augmenting existing workflows with symbolic regression can help uncover novel relationships and explore counterfactual models during the scientific process. We propose that this AI-assisted framework can bridge parametric and non-parametric models commonly employed in social science research by systematically exploring the space of nonlinear models and enabling fine-grained control over expressivity and interpretability.Comment: 19 pages, 4 figure

Ensemble of heterogeneous flexible neural trees using multiobjective genetic programming

Machine learning algorithms are inherently multiobjective in nature, where approximation error minimization and model's complexity simplification are two conflicting objectives. We proposed a multiobjective genetic programming (MOGP) for creating a heterogeneous flexible neural tree (HFNT), tree-like flexible feedforward neural network model. The functional heterogeneity in neural tree nodes was introduced to capture a better insight of data during learning because each input in a dataset possess different features. MOGP guided an initial HFNT population towards Pareto-optimal solutions, where the final population was used for making an ensemble system. A diversity index measure along with approximation error and complexity was introduced to maintain diversity among the candidates in the population. Hence, the ensemble was created by using accurate, structurally simple, and diverse candidates from MOGP final population. Differential evolution algorithm was applied to fine-tune the underlying parameters of the selected candidates. A comprehensive test over classification, regression, and time-series datasets proved the efficiency of the proposed algorithm over other available prediction methods. Moreover, the heterogeneous creation of HFNT proved to be efficient in making ensemble system from the final population

タイ国チャオプラヤ川流域の水資源管理に及ぼす気候変動の影響

京都大学0048新制・課程博士博士(工学)甲第22760号工博第4759号新制||工||1744(附属図書館)京都大学大学院工学研究科都市環境工学専攻(主査)教授 清水 芳久, 教授 田中 宏明, 教授 米田 稔学位規則第4条第1項該当Doctor of Philosophy (Engineering)Kyoto UniversityDFA