Analysis of Different Types of Regret in Continuous Noisy Optimization

The performance measure of an algorithm is a crucial part of its analysis. The performance can be determined by the study on the convergence rate of the algorithm in question. It is necessary to study some (hopefully convergent) sequence that will measure how "good" is the approximated optimum compared to the real optimum. The concept of Regret is widely used in the bandit literature for assessing the performance of an algorithm. The same concept is also used in the framework of optimization algorithms, sometimes under other names or without a specific name. And the numerical evaluation of convergence rate of noisy algorithms often involves approximations of regrets. We discuss here two types of approximations of Simple Regret used in practice for the evaluation of algorithms for noisy optimization. We use specific algorithms of different nature and the noisy sphere function to show the following results. The approximation of Simple Regret, termed here Approximate Simple Regret, used in some optimization testbeds, fails to estimate the Simple Regret convergence rate. We also discuss a recent new approximation of Simple Regret, that we term Robust Simple Regret, and show its advantages and disadvantages.Comment: Genetic and Evolutionary Computation Conference 2016, Jul 2016, Denver, United States. 201

Parallel Deterministic and Stochastic Global Minimization of Functions with Very Many Minima

The optimization of three problems with high dimensionality and many local minima are investigated under five different optimization algorithms: DIRECT, simulated annealing, Spall’s SPSA algorithm, the KNITRO package, and QNSTOP, a new algorithm developed at Indiana University

Novel optimisation methods for numerical inverse problems

Inverse problems involve the determination of one or more unknown quantities usually appearing in the mathematical formulation of a physical problem. These unknown quantities may be boundary heat flux, various source terms, thermal and material properties, boundary shape and size. Solving inverse problems requires additional information through in-situ data measurements of the field variables of the physical problems. These problems are also ill-posed because the solution itself is sensitive to random errors in the measured input data. Regularisation techniques are usually used in order to deal with the instability of the solution. In the past decades, many methods based on the nonlinear least squares model, both deterministic (CGM) and stochastic (GA, PSO), have been investigated for numerical inverse problems. The goal of this thesis is to examine and explore new techniques for numerical inverse problems. The background theory of population-based heuristic algorithm known as quantum-behaved particle swarm optimisation (QPSO) is re-visited and examined. To enhance the global search ability of QPSO for complex multi-modal problems, several modifications to QPSO are proposed. These include perturbation operation, Gaussian mutation and ring topology model. Several parameter selection methods for these algorithms are proposed. Benchmark functions were used to test the performance of the modified algorithms. To address the high computational cost of complex engineering optimisation problems, two parallel models of the QPSO (master-slave, static subpopulation) were developed for different distributed systems. A hybrid method, which makes use of deterministic (CGM) and stochastic (QPSO) methods, is proposed to improve the estimated solution and the performance of the algorithms for solving the inverse problems. Finally, the proposed methods are used to solve typical problems as appeared in many research papers. The numerical results demonstrate the feasibility and efficiency of QPSO and the global search ability and stability of the modified versions of QPSO. Two novel methods of providing initial guess to CGM with approximated data from QPSO are also proposed for use in the hybrid method and were applied to estimate heat fluxes and boundary shapes. The simultaneous estimation of temperature dependent thermal conductivity and heat capacity was addressed by using QPSO with Gaussian mutation. This combination provides a stable algorithm even with noisy measurements. Comparison of the performance between different methods for solving inverse problems is also presented in this thesis

Generalized Simultaneous Perturbation-based Gradient Search with Reduced Estimator Bias

We present in this paper a family of generalized simultaneous perturbation-based gradient search (GSPGS) estimators that use noisy function measurements. The number of function measurements required by each estimator is guided by the desired level of accuracy. We first present in detail unbalanced generalized simultaneous perturbation stochastic approximation (GSPSA) estimators and later present the balanced versions (B-GSPSA) of these. We extend this idea further and present the generalized smoothed functional (GSF) and generalized random directions stochastic approximation (GRDSA) estimators, respectively, as well as their balanced variants. We show that estimators within any specified class requiring more number of function measurements result in lower estimator bias. We present a detailed analysis of both the asymptotic and non-asymptotic convergence of the resulting stochastic approximation schemes. We further present a series of experimental results with the various GSPGS estimators on the Rastrigin and quadratic function objectives. Our experiments are seen to validate our theoretical findings.Comment: The material in this paper was presented in part at the Conference on Information Sciences and Systems (CISS) in March 202

Guaranteed Non-Orthogonal Tensor Decomposition via Alternating Rank-11 Updates

In this paper, we provide local and global convergence guarantees for recovering CP (Candecomp/Parafac) tensor decomposition. The main step of the proposed algorithm is a simple alternating rank-$1$ update which is the alternating version of the tensor power iteration adapted for asymmetric tensors. Local convergence guarantees are established for third order tensors of rank $k$ in $d$ dimensions, when $k=o \bigl( d^{1.5} \bigr)$ and the tensor components are incoherent. Thus, we can recover overcomplete tensor decomposition. We also strengthen the results to global convergence guarantees under stricter rank condition $k \le \beta d$ (for arbitrary constant $\beta > 1$) through a simple initialization procedure where the algorithm is initialized by top singular vectors of random tensor slices. Furthermore, the approximate local convergence guarantees for $p$-th order tensors are also provided under rank condition $k=o \bigl( d^{p/2} \bigr)$. The guarantees also include tight perturbation analysis given noisy tensor.Comment: We have added an additional sub-algorithm to remove the (approximate) residual error left after the tensor power iteratio