Metaheuristic design of feedforward neural networks: a review of two decades of research

Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era

Distributed Multi-Objective Evolutionary Algorithm For Dynamic Multi-Characteristic Social Networks Clustering

In this information era, social media and online social networks have become a huge data source. The social network perspective provides a clear way of analyzing the structure of whole social entities. These social media and online social networks are a virtual representation of real life as they represent real life relations between social actors (people). The primary focus of this study is to propose an algorithm and its implementation for clustering of multi-characteristic dynamic graphs in general, and multi-characteristic dynamic online social networks in specific. Social networks are typically stored as graph data (edges lists mostly), and dynamically changes with time either by expanding or shrinking. The topology of the graph data also changes along with the values for the relationships between nodes. Several algorithms were proposed for clustering, but only few of them deals with multi-characteristic and dynamic networks. Most of the proposed algorithms work for static networks or small networks and a very small number of algorithms work for huge and dynamic networks. In this study a practical algorithm is proposed which uses a combination of multi-objective evolutionary algorithms, distributed file systems and nested hybrid-indexing techniques to cluster the multi-characteristic dynamic huge social networks. The results of this work show a fast clustering system that is adaptive to dynamic interactions in social networks also provides a reliable distributed framework for BIG data analysi

Multiorder neurons for evolutionary higher-order clustering and growth

This letter proposes to use multiorder neurons for clustering irregularly shaped data arrangements. Multiorder neurons are an evolutionary extension of the use of higher-order neurons in clustering. Higher-order neurons parametrically model complex neuron shapes by replacing the classic synaptic weight by higher-order tensors. The multiorder neuron goes one step further and eliminates two problems associated with higher-order neurons. First, it uses evolutionary algorithms to select the best neuron order for a given problem. Second, it obtains more information about the underlying data distribution by identifying the correct order for a given cluster of patterns. Empirically we observed that when the correlation of clusters found with ground truth information is used in measuring clustering accuracy, the proposed evolutionary multiorder neurons method can be shown to outperform other related clustering methods. The simulation results from the Iris, Wine, and Glass data sets show significant improvement when compared to the results obtained using self-organizing maps and higher-order neurons. The letter also proposes an intuitive model by which multiorder neurons can be grown, thereby determining the number of clusters in data

Combinatorial optimization and metaheuristics

Today, combinatorial optimization is one of the youngest and most active areas of discrete mathematics. It is a branch of optimization in applied mathematics and computer science, related to operational research, algorithm theory and computational complexity theory. It sits at the intersection of several fields, including artificial intelligence, mathematics and software engineering. Its increasing interest arises for the fact that a large number of scientific and industrial problems can be formulated as abstract combinatorial optimization problems, through graphs and/or (integer) linear programs. Some of these problems have polynomial-time (“efficient”) algorithms, while most of them are NP-hard, i.e. it is not proved that they can be solved in polynomial-time. Mainly, it means that it is not possible to guarantee that an exact solution to the problem can be found and one has to settle for an approximate solution with known performance guarantees. Indeed, the goal of approximate methods is to find “quickly” (reasonable run-times), with “high” probability, provable “good” solutions (low error from the real optimal solution). In the last 20 years, a new kind of algorithm commonly called metaheuristics have emerged in this class, which basically try to combine heuristics in high level frameworks aimed at efficiently and effectively exploring the search space. This report briefly outlines the components, concepts, advantages and disadvantages of different metaheuristic approaches from a conceptual point of view, in order to analyze their similarities and differences. The two very significant forces of intensification and diversification, that mainly determine the behavior of a metaheuristic, will be pointed out. The report concludes by exploring the importance of hybridization and integration methods

Evolving interval-based representation for multiple classifier fusion.

Designing an ensemble of classifiers is one of the popular research topics in machine learning since it can give better results than using each constituent member. Furthermore, the performance of ensemble can be improved using selection or adaptation. In the former, the optimal set of base classifiers, meta-classifier, original features, or meta-data is selected to obtain a better ensemble than using the entire classifiers and features. In the latter, the base classifiers or combining algorithms working on the outputs of the base classifiers are made to adapt to a particular problem. The adaptation here means that the parameters of these algorithms are trained to be optimal for each problem. In this study, we propose a novel evolving combining algorithm using the adaptation approach for the ensemble systems. Instead of using numerical value when computing the representation for each class, we propose to use the interval-based representation for the class. The optimal value of the representation is found through Particle Swarm Optimization. During classification, a test instance is assigned to the class with the interval-based representation that is closest to the base classifiers’ prediction. Experiments conducted on a number of popular dataset confirmed that the proposed method is better than the well-known ensemble systems using Decision Template and Sum Rule as combiner, L2-loss Linear Support Vector Machine, Multiple Layer Neural Network, and the ensemble selection methods based on GA-Meta-data, META-DES, and ACO

Dynamic Facility Layout for Cellular and Reconfigurable Manufacturing using Dynamic Programming and Multi-Objective Metaheuristics

The facility layout problem is one of the most classical yet influential problems in the planning of production systems. A well-designed layout minimizes the material handling costs (MHC), personnel flow distances, work in process, and improves the performance of these systems in terms of operating costs and time. Because of this importance, facility layout has a rich literature in industrial engineering and operations research. Facility layout problems (FLPs) are generally concerned with positioning a set of facilities to satisfy some criteria or objectives under certain constraints. Traditional FLPs try to put facilities with the high material flow as close as possible to minimize the MHC. In static facility layout problems (SFLP), the product demands and mixes are considered deterministic parameters with constant values. The material flow between facilities is fixed over the planning horizon. However, in today’s market, manufacturing systems are constantly facing changes in product demands and mixes. These changes make it necessary to change the layout from one period to the other to be adapted to the changes. Consequently, there is a need for dynamic approaches of FLP that aim to generate layouts with high adaptation concerning changes in product demand and mix. This thesis focuses on studying the layout problems, with an emphasis on the changing environment of manufacturing systems. Despite the fact that designing layouts within the dynamic environment context is more realistic, the SFLP is observed to have been remained worthy to be analyzed. Hence, a math-heuristic approach is developed to solve an SFLP. To this aim, first, the facilities are grouped into many possible vertical clusters, second, the best combination of the generated clusters to be in the final layout are selected by solving a linear programming model, and finally, the selected clusters are sequenced within the shop floor. Although the presented math-heuristic approach is effective in solving SFLP, applying approaches to cope with the changing manufacturing environment is required. One of the most well-known approaches to deal with the changing manufacturing environment is the dynamic facility layout problem (DFLP). DFLP suits reconfigurable manufacturing systems since their machinery and material handling devices are reconfigurable to encounter the new necessities for the variations of product mix and demand. In DFLP, the planning horizon is divided into some periods. The goal is to find a layout for each period to minimize the total MHC for all periods and the total rearrangement costs between the periods. Dynamic programming (DP) has been known as one of the effective methods to optimize DFLP. In the DP method, all the possible layouts for every single period are generated and given to DP as its state-space. However, by increasing the number of facilities, it is impossible to give all the possible layouts to DP and only a restricted number of layouts should be fed to DP. This leads to ignoring some layouts and losing the optimality; to deal with this difficulty, an improved DP approach is proposed. It uses a hybrid metaheuristic algorithm to select the initial layouts for DP that lead to the best solution of DP for DFLP. The proposed approach includes two phases. In the first phase, a large set of layouts are generated through a heuristic method. In the second phase, a genetic algorithm (GA) is applied to search for the best subset of layouts to be given to DP. DP, improved by starting with the most promising initial layouts, is applied to find the multi-period layout. Finally, a tabu search algorithm is utilized for further improvement of the solution obtained by improved DP. Computational experiments show that improved DP provides more efficient solutions than DP approaches in the literature. The improved DP can efficiently solve DFLP and find the best layout for each period considering both material handling and layout rearrangement costs. However, rearrangement costs may include some unpredictable costs concerning interruption in production or moving of facilities. Therefore, in some cases, managerial decisions tend to avoid any rearrangements. To this aim, a semi-robust approach is developed to optimize an FLP in a cellular manufacturing system (CMS). In this approach, the pick-up/drop-off (P/D) points of the cells are changed to adapt the layout with changes in product demand and mix. This approach suits more a cellular flexible manufacturing system or a conventional system. A multi-objective nonlinear mixed-integer programming model is proposed to simultaneously search for the optimum number of cells, optimum allocation of facilities to cells, optimum intra- and inter-cellular layout design, and the optimum locations of the P/D points of the cells in each period. A modified non-dominated sorting genetic algorithm (MNSGA-II) enhanced by an improved non-dominated sorting strategy and a modified dynamic crowding distance procedure is used to find Pareto-optimal solutions. The computational experiments are carried out to show the effectiveness of the proposed MNSGA-II against other popular metaheuristic algorithms

An incremental approach to genetic algorithms based classification

Incremental learning has been widely addressed in the machine learning literature to cope with learning tasks where the learning environment is ever changing or training samples become available over time. However, most research work explores incremental learning with statistical algorithms or neural networks, rather than evolutionary algorithms. The work in this paper employs genetic algorithms (GAs) as basic learning algorithms for incremental learning within one or more classifier agents in a multi-agent environment. Four new approaches with different initialization schemes are proposed. They keep the old solutions and use an “integration” operation to integrate them with new elements to accommodate new attributes, while biased mutation and crossover operations are adopted to further evolve a reinforced solution. The simulation results on benchmark classification data sets show that the proposed approaches can deal with the arrival of new input attributes and integrate them with the original input space. It is also shown that the proposed approaches can be successfully used for incremental learning and improve classification rates as compared to the retraining GA. Possible applications for continuous incremental training and feature selection are also discussed

Mixture model with multiple allocations for clustering spatially correlated observations in the analysis of ChIP-Seq data

Model-based clustering is a technique widely used to group a collection of units into mutually exclusive groups. There are, however, situations in which an observation could in principle belong to more than one cluster. In the context of Next-Generation Sequencing (NGS) experiments, for example, the signal observed in the data might be produced by two (or more) different biological processes operating together and a gene could participate in both (or all) of them. We propose a novel approach to cluster NGS discrete data, coming from a ChIP-Seq experiment, with a mixture model, allowing each unit to belong potentially to more than one group: these multiple allocation clusters can be flexibly defined via a function combining the features of the original groups without introducing new parameters. The formulation naturally gives rise to a `zero-inflation group' in which values close to zero can be allocated, acting as a correction for the abundance of zeros that manifest in this type of data. We take into account the spatial dependency between observations, which is described through a latent Conditional Auto-Regressive process that can reflect different dependency patterns. We assess the performance of our model within a simulation environment and then we apply it to ChIP-seq real data.Comment: 25 pages; 3 tables, 6 figure

Incremental multiple objective genetic algorithms

This paper presents a new genetic algorithm approach to multi-objective optimization problemsIncremental Multiple Objective Genetic Algorithms (IMOGA). Different from conventional MOGA methods, it takes each objective into consideration incrementally. The whole evolution is divided into as many phases as the number of objectives, and one more objective is considered in each phase. Each phase is composed of two stages: first, an independent population is evolved to optimize one specific objective; second, the better-performing individuals from the evolved single-objective population and the multi-objective population evolved in the last phase are joined together by the operation of integration. The resulting population then becomes an initial multi-objective population, to which a multi-objective evolution based on the incremented objective set is applied. The experiment results show that, in most problems, the performance of IMOGA is better than that of three other MOGAs, NSGA-II, SPEA and PAES. IMOGA can find more solutions during the same time span, and the quality of solutions is better