Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions

Quantification of the stationary points and the associated basins of attraction of neural network loss surfaces is an important step towards a better understanding of neural network loss surfaces at large. This work proposes a novel method to visualise basins of attraction together with the associated stationary points via gradient-based random sampling. The proposed technique is used to perform an empirical study of the loss surfaces generated by two different error metrics: quadratic loss and entropic loss. The empirical observations confirm the theoretical hypothesis regarding the nature of neural network attraction basins. Entropic loss is shown to exhibit stronger gradients and fewer stationary points than quadratic loss, indicating that entropic loss has a more searchable landscape. Quadratic loss is shown to be more resilient to overfitting than entropic loss. Both losses are shown to exhibit local minima, but the number of local minima is shown to decrease with an increase in dimensionality. Thus, the proposed visualisation technique successfully captures the local minima properties exhibited by the neural network loss surfaces, and can be used for the purpose of fitness landscape analysis of neural networks.Comment: Preprint submitted to the Neural Networks journa

Robotic Architectures

In the development of mobile robotic systems, a robotic architecture plays a crucial role in interconnecting all the sub-systems and controlling the system. The design of robotic architectures for mobile autonomous robots is a challenging and complex task. With a number of existing architectures and tools to choose from, a review of the existing robotic architecture is essential. This paper surveys the different paradigms in robotic architectures. A classification of the existing robotic architectures and comparison of different proposals attributes and properties have been carried out. The paper also provides a view on the current state of designing robot architectures. It also proposes a conceptual model of a generalised robotic architecture for mobile autonomous robots.Defence Science Journal, 2010, 60(1), pp.15-22, DOI:http://dx.doi.org/10.14429/dsj.60.9

Particle swarm optimization with crossover : a review and empirical analysis

Since its inception in 1995, many improvements to the original particle swarm optimization (PSO) algorithm have been developed. This paper reviews one class of such PSO variations, i.e. PSO algorithms that make use of crossover operators. The review is supplemented with a more extensive sensitivity analysis of the crossover PSO algorithms than provided in the original publications. Two adaptations of a parent-centric crossover PSO algorithm are provided, resulting in improvements with respect to solution accuracy compared to the original parent-centric PSO algorithms. The paper then provides an extensive empirical analysis on a large benchmark of minimization problems, with the objective to identify those crossover PSO algorithms that perform best with respect to accuracy, success rate, and efficiency.http://link.springer.com/journal/104622017-02-20hb201

Empirical Loss Landscape Analysis of Neural Network Activation Functions

Activation functions play a significant role in neural network design by enabling non-linearity. The choice of activation function was previously shown to influence the properties of the resulting loss landscape. Understanding the relationship between activation functions and loss landscape properties is important for neural architecture and training algorithm design. This study empirically investigates neural network loss landscapes associated with hyperbolic tangent, rectified linear unit, and exponential linear unit activation functions. Rectified linear unit is shown to yield the most convex loss landscape, and exponential linear unit is shown to yield the least flat loss landscape, and to exhibit superior generalisation performance. The presence of wide and narrow valleys in the loss landscape is established for all activation functions, and the narrow valleys are shown to correlate with saturated neurons and implicitly regularised network configurations.Comment: Accepted for publication in Genetic and Evolutionary Computation Conference Companion, July 15--19, 2023, Lisbon, Portuga

Seeking multiple solutions:an updated survey on niching methods and their applications

Multi-Modal Optimization (MMO) aiming to locate multiple optimal (or near-optimal) solutions in a single simulation run has practical relevance to problem solving across many fields. Population-based meta-heuristics have been shown particularly effective in solving MMO problems, if equipped with specificallydesigned diversity-preserving mechanisms, commonly known as niching methods. This paper provides an updated survey on niching methods. The paper first revisits the fundamental concepts about niching and its most representative schemes, then reviews the most recent development of niching methods, including novel and hybrid methods, performance measures, and benchmarks for their assessment. Furthermore, the paper surveys previous attempts at leveraging the capabilities of niching to facilitate various optimization tasks (e.g., multi-objective and dynamic optimization) and machine learning tasks (e.g., clustering, feature selection, and learning ensembles). A list of successful applications of niching methods to real-world problems is presented to demonstrate the capabilities of niching methods in providing solutions that are difficult for other optimization methods to offer. The significant practical value of niching methods is clearly exemplified through these applications. Finally, the paper poses challenges and research questions on niching that are yet to be appropriately addressed. Providing answers to these questions is crucial before we can bring more fruitful benefits of niching to real-world problem solving

Training multi-agent teams from zero knowledge with the competitive coevolutionary team-based particle swarm optimiser

A new competitive coevolutionary team-based particle swarm optimiser (CCPSO(t)) algorithm is developed to train multi-agent teams from zero knowledge. The CCPSO(t) algorithm is applied to train a team of agents to play simple soccer. The algorithm uses the charged particle swarm optimiser in a competitive and cooperative coevolutionary training environment to train neural network controllers for the players. The CCPSO(t) algorithm makes use of the FIFA league ranking relative fitness function to gather detailed performance metrics from each game played. The training performance and convergence behaviour of the particle swarm is analysed. A hypothesis is presented that explains the lack of convergence in the particle swarms. After applying a clustering algorithm on the particle positions, a detailed visual and quantitative analysis of the player strategies is presented. The final results show that the CCPSO(t) algorithm is capable of evolving complex gameplay strategies for a complex non-deterministic game.http://link.springer.com/journal/5002017-02-28hb201

Set-based particle swarm optimization applied to the multidimensional knapsack problem

Particle swarm optimization algorithms have been successfully applied to discrete- valued optimization problems. However, in many cases the algorithms have been tailored specifically for the problem at hand. This paper proposes a generic set-based particle swarm optimization algorithm for use on discrete-valued optimization problems that can be formulated as set-based problems. A detailed sensitivity analysis of the parameters of the algorithm is conducted. The performance of the proposed algorithm is then compared against three other discrete particle swarm optimization algorithms from literature using the multidimensional knapsack problem, and is shown to statistically outperform the existing algorithms.http://www.springerlink.com/content/120597/?p=36e5205be3fa464a82382b977b16ece5&pi=2086hb201

Performance measures for dynamic multi-objective optimisation algorithms

When algorithms solve dynamic multi-objective optimisation problems (DMOOPs), performance measures are required to quantify the performance of the algorithm and to compare one algorithm’s performance against that of other algorithms. However, for dynamic multiobjective optimisation (DMOO) there are no standard performance measures. This article provides an overview of the performance measures that have been used so far. In addition, issues with performance measures that are currently being used in the DMOO literature are highlighted.http://www.elsevier.com/locate/insmv201

Transfer of Manure from Livestock Farms to Crop Fields as Fertilizer using an Ant Inspired Approach

Intensive livestock production might have a negative environmental impact, by producing large amounts of animal excrements, which, if not properly managed, can contaminate nearby water bodies with nutrient excess. However, if animal manure is exported to distant crop fields, to be used as organic fertilizer, pollution can be mitigated. It is a single-objective optimization problem, in regards to finding the best solution for the logistics process of satisfying nutrient crops needs by means of livestock manure. This paper proposes a dynamic approach to solve the problem, based on a decentralized nature-inspired cooperative technique, inspired by the foraging behavior of ants (AIA). Results provide important insights for policy-makers over the potential of using animal manure as fertilizer for crop fields, while AIA solves the problem effectively, in a fair way to the farmers and well balanced in terms of average transportation distances that need to be covered by each livestock farmer. Our work constitutes the first application of a decentralized AIA to this interesting real-world problem, in a domain where swarm intelligence methods are still under-exploited.Comment: Proc. of the XXIVth International Society for Photogrammetry and Remote Sensing (ISPRS) Congress, June 202