Symmetry Induction in Computational Intelligence

Symmetry has been a very useful tool to researchers in various scientific fields. At its most basic, symmetry refers to the invariance of an object to some transformation, or set of transformations. Usually one searches for, and uses information concerning an existing symmetry within given data, structure or concept to somehow improve algorithm performance or compress the search space. This thesis examines the effects of imposing or inducing symmetry on a search space. That is, the question being asked is whether only existing symmetries can be useful, or whether changing reference to an intuition-based definition of symmetry over the evaluation function can also be of use. Within the context of optimization, symmetry induction as defined in this thesis will have the effect of equating the evaluation of a set of given objects. Group theory is employed to explore possible symmetrical structures inherent in a search space. Additionally, conditions when the search space can have a symmetry induced on it are examined. The idea of a neighborhood structure then leads to the idea of opposition-based computing which aims to induce a symmetry of the evaluation function. In this context, the search space can be seen as having a symmetry imposed on it. To be useful, it is shown that an opposite map must be defined such that it equates elements of the search space which have a relatively large difference in their respective evaluations. Using this idea a general framework for employing opposition-based ideas is proposed. To show the efficacy of these ideas, the framework is applied to popular computational intelligence algorithms within the areas of Monte Carlo optimization, estimation of distribution and neural network learning. The first example application focuses on simulated annealing, a popular Monte Carlo optimization algorithm. At a given iteration, symmetry is induced on the system by considering opposite neighbors. Using this technique, a temporary symmetry over the neighborhood region is induced. This simple algorithm is benchmarked using common real optimization problems and compared against traditional simulated annealing as well as a randomized version. The results highlight improvements in accuracy, reliability and convergence rate. An application to image thresholding further confirms the results. Another example application, population-based incremental learning, is rooted in estimation of distribution algorithms. A major problem with these techniques is a rapid loss of diversity within the samples after a relatively low number of iterations. The opposite sample is introduced as a remedy to this problem. After proving an increased diversity, a new probability update procedure is designed. This opposition-based version of the algorithm is benchmarked using common binary optimization problems which have characteristics of deceptivity and attractive basins characteristic of difficult real world problems. Experiments reveal improvements in diversity, accuracy, reliability and convergence rate over the traditional approach. Ten instances of the traveling salesman problem and six image thresholding problems are used to further highlight the improvements. Finally, gradient-based learning for feedforward neural networks is improved using opposition-based ideas. The opposite transfer function is presented as a simple adaptive neuron which easily allows for efficiently jumping in weight space. It is shown that each possible opposite network represents a unique input-output mapping, each having an associated effect on the numerical conditioning of the network. Experiments confirm the potential of opposite networks during pre- and early training stages. A heuristic for efficiently selecting one opposite network per epoch is presented. Benchmarking focuses on common classification problems and reveals improvements in accuracy, reliability, convergence rate and generalization ability over common backpropagation variants. To further show the potential, the heuristic is applied to resilient propagation where similar improvements are also found

Hybrid algorithms for efficient Cholesky decomposition and matrix inverse using multicore CPUs with GPU accelerators

The use of linear algebra routines is fundamental to many areas of computational science, yet their implementation in software still forms the main computational bottleneck in many widely used algorithms. In machine learning and computational statistics, for example, the use of Gaussian distributions is ubiquitous, and routines for calculating the Cholesky decomposition, matrix inverse and matrix determinant must often be called many thousands of times for common algorithms, such as Markov chain Monte Carlo. These linear algebra routines consume most of the total computational time of a wide range of statistical methods, and any improvements in this area will therefore greatly increase the overall efficiency of algorithms used in many scientific application areas. The importance of linear algebra algorithms is clear from the substantial effort that has been invested over the last 25 years in producing low-level software libraries such as LAPACK, which generally optimise these linear algebra routines by breaking up a large problem into smaller problems that may be computed independently. The performance of such libraries is however strongly dependent on the specific hardware available. LAPACK was originally developed for single core processors with a memory hierarchy, whereas modern day computers often consist of mixed architectures, with large numbers of parallel cores and graphics processing units (GPU) being used alongside traditional CPUs. The challenge lies in making optimal use of these different types of computing units, which generally have very different processor speeds and types of memory. In this thesis we develop novel low-level algorithms that may be generally employed in blocked linear algebra routines, which automatically optimise themselves to take full advantage of the variety of heterogeneous architectures that may be available. We present a comparison of our methods with MAGMA, the state of the art open source implementation of LAPACK designed specifically for hybrid architectures, and demonstrate up to 400% increase in speed that may be obtained using our novel algorithms, specifically when running commonly used Cholesky matrix decomposition, matrix inverse and matrix determinant routines

Controlled particle systems for nonlinear filtering and global optimization

This thesis is concerned with the development and applications of controlled interacting particle systems for nonlinear filtering and global optimization problems. These problems are important in a number of engineering domains. In nonlinear filtering, there is a growing interest to develop geometric approaches for systems that evolve on matrix Lie groups. Examples include the problem of attitude estimation and motion tracking in aerospace engineering, robotics and computer vision. In global optimization, the challenges typically arise from the presence of a large number of local minimizers as well as the computational scalability of the solution. Gradient-free algorithms are attractive because in many practical situations, evaluating the gradient of the objective function may be computationally prohibitive. The thesis comprises two parts that are devoted to theory and applications, respectively. The theoretical part consists of three chapters that describe methods and algorithms for nonlinear filtering, global optimization, and numerical solutions of the Poisson equation that arise in both filtering and optimization. For the nonlinear filtering problem, the main contribution is to extend the feedback particle filter (FPF) algorithm to connected matrix Lie groups. In its general form, the FPF is shown to provide an intrinsic coordinate-free description of the filter that automatically satisfies the manifold constraint. The properties of the original (Euclidean) FPF, especially the gain-times-error feedback structure, are preserved in the generalization. For the global optimization problem, a controlled particle filter algorithm is introduced to numerically approximate a solution of the global optimization problem. The theoretical significance of this work comes from its variational aspects: (i) the proposed particle filter is a controlled interacting particle system where the control input represents the solution of a mean-field type optimal control problem; and (ii) the associated density transport is shown to be a gradient flow (steepest descent) for the optimal value function, with respect to the Kullback--Leibler divergence. For both the nonlinear filtering and optimization problems, the numerical implementation of the proposed algorithms require a solution of a Poisson equation. Two numerical algorithms are described for this purpose. In the Galerkin scheme, the gain function is approximated using a set of pre-defined basis functions; In the kernel-based scheme, a numerical solution is obtained by solving a certain fixed-point equation. Well-posedness results for the Poisson equation are also discussed. The second part of the thesis contains applications of the proposed algorithms to specific nonlinear filtering and optimization problems. The FPF is applied to the problem of attitude estimation - a nonlinear filtering problem on the Lie group SO(3). The formulae of the filter are described using both the rotation matrix and the quaternion coordinates. A comparison is provided between FPF and the several popular attitude filters including the multiplicative EKF, the invariant EKF, the unscented Kalman filter, the invariant ensemble Kalman filter and the bootstrap particle filter. Numerical simulations are presented to illustrate the comparison. As a practical application, experimental results for a motion tracking problem are presented. The objective is to estimate the attitude of a wrist-worn motion sensor based on the motion of the arm. In the presence of motion, considered here as the swinging motion of the arm, the observability of the sensor attitude is shown to improve. The estimation problem is mathematically formulated as a nonlinear filtering problem on the product Lie group SO(3)XSO(2), and experimental results are described using data from the gyroscope and the accelerometer installed on the sensor. For the global optimization problem, the proposed controlled particle filter is compared with several model-based algorithms that also employ probabilistic models to inform the search of the global minimizer. Examples of the model-based algorithms include the model reference adaptive search, the cross entropy, the model-based evolutionary optimization, and two algorithms based on bootstrap particle filtering. Performance comparisons are provided between the control-based and the sampling-based implementation. Results of Monte-Carlo simulations are described for several benchmark optimization problems

MS FT-2-2 7 Orthogonal polynomials and quadrature: Theory, computation, and applications

Quadrature rules find many applications in science and engineering. Their analysis is a classical area of applied mathematics and continues to attract considerable attention. This seminar brings together speakers with expertise in a large variety of quadrature rules. It is the aim of the seminar to provide an overview of recent developments in the analysis of quadrature rules. The computation of error estimates and novel applications also are described