Canonical Duality-Triality Theory: Bridge Between Nonconvex Analysis/Mechanics and Global Optimization in Complex Systems

Canonical duality-triality is a breakthrough methodological theory, which can be used not only for modeling complex systems within a unified framework, but also for solving a wide class of challenging problems from real-world applications. This paper presents a brief review on this theory, its philosophical origin, physics foundation, and mathematical statements in both finite and infinite dimensional spaces, with emphasizing on its role for bridging the gap between nonconvex analysis/mechanics and global optimization. Special attentions are paid on unified understanding the fundamental difficulties in large deformation mechanics, bifurcation/chaos in nonlinear science, and the NP-hard problems in global optimization, as well as the theorems, methods, and algorithms for solving these challenging problems. Misunderstandings and confusions on some basic concepts, such as objectivity, nonlinearity, Lagrangian, and generalized convexities are discussed and classified. Breakthrough from recent challenges and conceptual mistakes by M. Voisei, C. Zalinescu and his co-worker are addressed. Some open problems and future works in global optimization and nonconvex mechanics are proposed.Comment: 43 pages, 4 figures. appears in Mathematics and Mechanics of Solids, 201

Particle Swarm Optimization: A survey of historical and recent developments with hybridization perspectives

Particle Swarm Optimization (PSO) is a metaheuristic global optimization paradigm that has gained prominence in the last two decades due to its ease of application in unsupervised, complex multidimensional problems which cannot be solved using traditional deterministic algorithms. The canonical particle swarm optimizer is based on the flocking behavior and social co-operation of birds and fish schools and draws heavily from the evolutionary behavior of these organisms. This paper serves to provide a thorough survey of the PSO algorithm with special emphasis on the development, deployment and improvements of its most basic as well as some of the state-of-the-art implementations. Concepts and directions on choosing the inertia weight, constriction factor, cognition and social weights and perspectives on convergence, parallelization, elitism, niching and discrete optimization as well as neighborhood topologies are outlined. Hybridization attempts with other evolutionary and swarm paradigms in selected applications are covered and an up-to-date review is put forward for the interested reader.Comment: 34 pages, 7 table

Convergence analysis of beetle antennae search algorithm and its applications

The beetle antennae search algorithm was recently proposed and investigated for solving global optimization problems. Although the performance of the algorithm and its variants were shown to be better than some existing meta-heuristic algorithms, there is still a lack of convergence analysis. In this paper, we provide theoretical analysis on the convergence of the beetle antennae search algorithm. We test the performance of the BAS algorithm via some representative benchmark functions. Meanwhile, some applications of the BAS algorithm are also presented.Comment: n

A Computation Offloading Incentive Mechanism with Delay and Cost Constraints under 5G Satellite-ground IoV architecture

The 5G Internet of Vehicles has become a new paradigm alongside the growing popularity and variety of computation-intensive applications with high requirements for computational resources and analysis capabilities. Existing network architectures and resource management mechanisms may not sufficiently guarantee satisfactory Quality of Experience and network efficiency, mainly suffering from coverage limitation of Road Side Units, insufficient resources, and unsatisfactory computational capabilities of onboard equipment, frequently changing network topology, and ineffective resource management schemes. To meet the demands of such applications, in this article, we first propose a novel architecture by integrating the satellite network with 5G cloud-enabled Internet of Vehicles to efficiently support seamless coverage and global resource management. A incentive mechanism based joint optimization problem of opportunistic computation offloading under delay and cost constraints is established under the aforementioned framework, in which a vehicular user can either significantly reduce the application completion time by offloading workloads to several nearby vehicles through opportunistic vehicle-to-vehicle channels while effectively controlling the cost or protect its own profit by providing compensated computing service. As the optimization problem is non-convex and NP-hard, simulated annealing based on the Markov Chain Monte Carlo as well as the metropolis algorithm is applied to solve the optimization problem, which can efficaciously obtain both high-quality and cost-effective approximations of global optimal solutions. The effectiveness of the proposed mechanism is corroborated through simulation results

Generalized Eigenvalue Problems with Specified Eigenvalues

We consider the distance from a (square or rectangular) matrix pencil to the nearest matrix pencil in 2-norm that has a set of specified eigenvalues. We derive a singular value optimization characterization for this problem and illustrate its usefulness for two applications. First, the characterization yields a singular value formula for determining the nearest pencil whose eigenvalues lie in a specified region in the complex plane. For instance, this enables the numerical computation of the nearest stable descriptor system in control theory. Second, the characterization partially solves the problem posed in [Boutry et al. 2005] regarding the distance from a general rectangular pencil to the nearest pencil with a complete set of eigenvalues. The involved singular value optimization problems are solved by means of BFGS and Lipschitz-based global optimization algorithms.Comment: 23 pages with 3 pdf figure

Filled function method for nonlinear equations

AbstractSystems of nonlinear equations are ubiquitous in engineering, physics and mechanics, and have myriad applications. Generally, they are very difficult to solve. In this paper, we will present a filled function method to solve nonlinear systems. We will first convert the nonlinear systems into equivalent global optimization problems with the property: x∗ is a global minimizer if and only if its function value is zero. A filled function method is proposed to solve the converted global optimization problem. Numerical examples are presented to illustrate our new techniques

Porcupine Neural Networks: (Almost) All Local Optima are Global

Neural networks have been used prominently in several machine learning and statistics applications. In general, the underlying optimization of neural networks is non-convex which makes their performance analysis challenging. In this paper, we take a novel approach to this problem by asking whether one can constrain neural network weights to make its optimization landscape have good theoretical properties while at the same time, be a good approximation for the unconstrained one. For two-layer neural networks, we provide affirmative answers to these questions by introducing Porcupine Neural Networks (PNNs) whose weight vectors are constrained to lie over a finite set of lines. We show that most local optima of PNN optimizations are global while we have a characterization of regions where bad local optimizers may exist. Moreover, our theoretical and empirical results suggest that an unconstrained neural network can be approximated using a polynomially-large PNN

A Distributed Hierarchical SGD Algorithm with Sparse Global Reduction

Reducing communication in training large-scale machine learning applications on distributed platform is still a big challenge. To address this issue, we propose a distributed hierarchical averaging stochastic gradient descent (Hier-AVG) algorithm with infrequent global reduction by introducing local reduction. As a general type of parallel SGD, Hier-AVG can reproduce several popular synchronous parallel SGD variants by adjusting its parameters. We show that Hier-AVG with infrequent global reduction can still achieve standard convergence rate for non-convex optimization problems. In addition, we show that more frequent local averaging with more participants involved can lead to faster training convergence. By comparing Hier-AVG with another popular distributed training algorithm K-AVG, we show that through deploying local averaging with fewer number of global averaging, Hier-AVG can still achieve comparable training speed while frequently get better test accuracy. This indicates that local averaging can serve as an alternative remedy to effectively reduce communication overhead when the number of learners is large. Experimental results of Hier-AVG with several state-of-the-art deep neural nets on CIFAR-10 and IMAGENET-1K are presented to validate our analysis and show its superiority.Comment: 38 page

A Practical Approach to Quasi-convex Optimization

A new and simple method for quasi-convex optimization is introduced from which its various applications can be derived. Especially, a global optimum under constrains can be approximated for all continuous functions.Comment: 8 pages, 5 figure

Rational Optimization using Sum-of-Squares Techniques

Motivated by many control applications, this paper deals with the global solutions of unconstrained optimization problems. First, a simple SOS method is presented to find the infimum of a polynomial, which can be handled efficiently using the relevant software tools. The main idea of this method is to introduce a perturbation variable whose approaching to zero results in a solution with any arbitrary precision. The proposed technique is then extended to the case of rational functions. The primary advantages of this approach over the existing ones are its simplicity and capability of treating problems for which the existing methods are not efficient, as demonstrated in three numerical examples