A stopping criterion for multi-objective optimization evolutionary algorithms

This Paper Puts Forward A Comprehensive Study Of The Design Of Global Stopping Criteria For Multi-Objective Optimization. In This Study We Propose A Global Stopping Criterion, Which Is Terms As Mgbm After The Authors Surnames. Mgbm Combines A Novel Progress Indicator, Called Mutual Domination Rate (Mdr) Indicator, With A Simplified Kalman Filter, Which Is Used For Evidence-Gathering Purposes. The Mdr Indicator, Which Is Also Introduced, Is A Special-Purpose Progress Indicator Designed For The Purpose Of Stopping A Multi-Objective Optimization. As Part Of The Paper We Describe The Criterion From A Theoretical Perspective And Examine Its Performance On A Number Of Test Problems. We Also Compare This Method With Similar Approaches To The Issue. The Results Of These Experiments Suggest That Mgbm Is A Valid And Accurate Approach. (C) 2016 Elsevier Inc. All Rights Reserved.This work was funded in part by CNPq BJT Project 407851/2012-7 and CNPq PVE Project 314017/2013-

CoCoA: A General Framework for Communication-Efficient Distributed Optimization

The scale of modern datasets necessitates the development of efficient distributed optimization methods for machine learning. We present a general-purpose framework for distributed computing environments, CoCoA, that has an efficient communication scheme and is applicable to a wide variety of problems in machine learning and signal processing. We extend the framework to cover general non-strongly-convex regularizers, including L1-regularized problems like lasso, sparse logistic regression, and elastic net regularization, and show how earlier work can be derived as a special case. We provide convergence guarantees for the class of convex regularized loss minimization objectives, leveraging a novel approach in handling non-strongly-convex regularizers and non-smooth loss functions. The resulting framework has markedly improved performance over state-of-the-art methods, as we illustrate with an extensive set of experiments on real distributed datasets

Scalable multi-objective optimization

This thesis is concerned with the three open in multi-objective optimization: (i) the development of strategies for dealing with problems with many objective functions; (ii) the comprehension and solution of the model-building issues of current MOEDAs, and; (iii) the formulation of stopping criteria for multi-objective optimizers. We argue about what elements of MOEDAs should be modified in order to achieve a substantial improvement on their performance and scalability. However, in order to supply a solid ground for that discussion, some other elements are to be discussed as well. In particular, this thesis: sketches the supporting theoretical corpus and the fundamentals of MOEA and MOEDA algorithms; analyzes the scalability issue of MOEAs from both theoretical and experimental points of view; discusses the possible directions of improvement for MOEAs’ scalability, presenting the current trends of research; gives reasons of why EDAs can be used as a foundation for achieving a sizable improvement with regard to the scalability issue; examines the model-building issue in depth, hypothesizing on how it affects MOEDAs performance; proposes a novel model-building algorithm, the model-building growing neural gas (MBGNG), which fulfill the requirements for a new approach derived from the previous debate, and; introduces a novel MOEDA, the multi-objective neural EDA, that is constructed using MB-GNG as foundation. The formulation of an strategy for stopping multi-objective optimizers became obvious and necessary as this thesis was developed. The lack of an adequate stopping criterion made the rendered any experimentation that had to do with many objectives a rather cumbersome task. That is why it was compulsory to deal with this issue in order to proceed with further studies. In this regard, the thesis: provides an updated and exhaustive state-of-the-art of this matter; examines the properties and characteristics that a given stopping criterion should exhibit; puts forward a new stopping criterion, denominated MGBM, after the authors last names, that has a small computational footprint, and; experimentally validates MGBM in a set of experiments. Theoretical discussions and algorithm proposals are experimentally contrasted with current state-of-the-art approaches when required. --------------------------------------------------------------------------------------------------------------------------------------------------------------------------Muchas actividades humanas están relacionadas con la elaboración de artefactos cuyas características, organización y/o costes de producción, etc., se deben ajustar en la manera más eficiente posible. Este hecho ha creado la necesidad de tener herramientas matemáticas y computacionales capaces de tratar estos problemas, lo cual ha impulsado el desarrollo de distintas áreas de investigación interrelacionadas, como, por ejemplo, la optimización, programación matemática, investigación de operaciones, etc. El concepto de optimización se puede formular en términos matemáticos como el proceso de buscar una o más soluciones factibles que se correspondan con los valores extremos de una o varias funciones. La mayor parte de los problemas de optimización reales implican la optimización de más de una función a la vez. Esta clase de problemas se conoce como problemas de optimización multi-objetivo (POM). Existe una clase de POM que es particularmente atractivo debido a su complejidad inherente: los denominados problemas de muchos objetivos. Estos son problemas con un número relativamente elevado de funciones objetivo. Numerosos experimentos han mostrado que los métodos “tradicionales” no logran un desempeño adecuado debido a la relación intensamente exponencial entre la dimensión del conjunto objetivo y la cantidad de recursos requeridos para resolver el problema correctamente. Estos problemas tienen una naturaleza poco intuitiva y, en particular, sus soluciones son difíciles de visualizar por un tomador de decisiones humano. Sin embargo, son bastante comunes en la práctica (Stewart et al., 2008). La optimización multi-objetivo ha recibido una importante atención por parte de la comunidad dedicada a los algoritmos evolutivos (Coello Coello et al., 2007). Sin embargo, se ha hecho patente la necesidad de buscar alternativas para poder tratar con los problemas de muchos objetivos. Los algoritmos de estimación de distribución (EDAs, por sus siglas en inglés) (Lozano et al., 2006) son buenos candidatos para esa tarea. Estos algoritmos se han presentado como una revolución en el campo de la computación evolutiva. Ellos sustituyen la aplicación de operadores inspirados en la selección natural por la síntesis de un modelo estadístico. Este modelo es muestreado para generar nuevos elementos y así proseguir con la búsqueda de soluciones. Sin embargo, los EDAs multi-objetivo (MOEDAs) no han logrado cumplir las expectativas creadas a priori. El leit motif de esta tesis se puede resumir en que la causa principal del bajo rendimiento MOEDAs se debe a los algoritmos de aprendizaje automático que se aplican en la construcción de modelos estadísticos. Los trabajos existentes hasta el momento han tomado una aproximación de “caja negra” al problema de la construcción de modelos. Por esa razón, se aplican métodos de aprendizaje automático ya existentes sin modificación alguna, sin percatarse que el problema de la construcción de modelos para EDAs tiene unos requisitos propios que en varios casos son contradictorios con el contexto original de aplicación de los mencionados algoritmos. En particular, hay propiedades compartidas por la mayoría de los enfoques de aprendizaje automático que podrían evitar la obtención de una mejora sustancial en el resultado de los MOEDAs. Ellas son: el tratamiento incorrecto de los valores atípicos (outliers) en el conjunto de datos; tendencia a la pérdida de la diversidad de la población, y; exceso de esfuerzo computacional dedicado a la búsqueda de un modelo óptimo. Estos problemas, aunque ya están presentes en los EDAs de un solo objetivo, se hacen patentes al escalar a problemas de varios objetivos y, en particular, a muchos objetivos. Además, con el aumento de la cantidad de objetivos con frecuencia esta situación se ve agravada por las consecuencias de la “maldición de la dimensionalidad”. La cuestión de los valores atípicos en los datos es un buen ejemplo de como la comunidad no ha notado esta diferencia. En el contexto tradicional del aprendizaje automático los valores extremos son considerados como datos ruidosos o irrelevantes y, por tanto, deben ser evitados. Sin embargo, los valores atípicos en los datos de la construcción de modelos representan las regiones recién descubiertas o soluciones candidatas del conjunto de decisión y por lo tanto deben ser explorados. En este caso, los casos aislados debe ser al menos igualmente representados por el modelo con respecto a los que están formando grupos. Sobre la base de estos razonamientos se estructuran los principales resultados obtenidos en el desarrollo de la tesis. A continuación se enumeran brevemente los mismos mencionando las referencias principales de los mismos. Comprensión del problema de la construcción de modelos en MOEDAs (Martí et al., 2010a, 2008b, 2009c). Se analiza que los EDAs han asumido incorrectamente que la construcción de modelos es un problema tradicional de aprendizaje automático. En el trabajo se muestra experimentalmente la hipótesis. Growing Neural Gas: una alternativa viable para construcción de modelos (Martí et al., 2008c). Se propone el Model-Building Growing Neural Gas network (MB-GNG), una modificación de las redes neuronales tipo Growing Neural Gas. MB-GNG tiene las propiedades requeridas para tratar correctamente la construcción de modelos. MONEDA: mejorando el desempeño de los MOEDAs (Martí et al., 2008a, 2009b, 2010c). El Multi-objective Optimization Neural EDA (MONEDA) fue ideado con el fin de hacer frente a los problemas arriba descritos de los MOEDAs y, por lo tanto, mejorar la escalabilidad de los MOEDAs. MONEDA utiliza MB-GNG para la construcción de modelos. Gracias a su algoritmo específico de construcción de modelos, la preservación de las élites de individuos de la población y su mecanismo de sustitución de individuos MONEDA es escalable capaz de resolver POMs continuos de muchos objetivos con un mejor desepeño que algoritmos similares a un coste computacional menor. Esta propuesta fue nominada a mejor trabajo en GECCO’2008. MONEDA en problemas de alta complejidad (Martí et al., 2009d). En este caso se lleva a cabo una amplia experimentación para comprender como las características de MONEDA provocan una mejora en el desempeño del algoritmo, y si sus resultados mejoran los obtenidos de otros enfoques. Se tratan problemas de alta complejidad. Estos experimentos demostraron que MONEDA produce resultados sustancialmente mejores que los algoritmos similares a una menor coste computacional. Nuevos paradigmas de aprendizaje: MARTEDA (Martí et al., 2010d). Si bien MB-GNG y MONEDA mostraron que la vía del tratamiento correcto de la construcción de modelos era una de las formas de obtener mejores resultados, ellos no evadían por completo el punto esencial: el paradigma de aprendizaje empleado. Al combinar un paradigma de aprendizaje automático alternativo, en particular, la Teoría de Resonancia Adaptativa, se trata a este asunto desde su raíz. En este respecto se han obtenido algunos resultados preliminares alentadores. Criterios de parada y convergencia (Martí et al., 2007, 2009a, 2010e). Con la realización de los experimentos anteriores nos percatamos de la falta de de un criterio de parada adecuado y que esta es un área inexplorada en el ámbito de la investigación en algoritmos evolutivos multi-objectivo. Abordamos esta cuestión proponiendo una serie de criterios de parada que se han demostrado efectivos en problemas sintéticos y del mundo real

Computing Volumes and Convex Hulls: Variations and Extensions

Geometric techniques are frequently utilized to analyze and reason about multi-dimensional data. When confronted with large quantities of such data, simplifying geometric statistics or summaries are often a necessary first step. In this thesis, we make contributions to two such fundamental concepts of computational geometry: Klee's Measure and Convex Hulls. The former is concerned with computing the total volume occupied by a set of overlapping rectangular boxes in d-dimensional space, while the latter is concerned with identifying extreme vertices in a multi-dimensional set of points. Both problems are frequently used to analyze optimal solutions to multi-objective optimization problems: a variant of Klee's problem called the Hypervolume Indicator gives a quantitative measure for the quality of a discrete Pareto Optimal set, while the Convex Hull represents the subset of solutions that are optimal with respect to at least one linear optimization function.In the first part of the thesis, we investigate several practical and natural variations of Klee's Measure Problem. We develop a specialized algorithm for a specific case of Klee's problem called the “grounded” case, which also solves the Hypervolume Indicator problem faster than any earlier solution for certain dimensions. Next, we extend Klee's problem to an uncertainty setting where the existence of the input boxes are defined probabilistically, and study computing the expectation of the volume. Additionally, we develop efficient algorithms for a discrete version of the problem, where the volume of a box is redefined to be the cardinality of its overlap with a given point set.The second part of the thesis investigates the convex hull problem on uncertain input. To this extent, we examine two probabilistic uncertainty models for point sets. The first model incorporates uncertainty in the existence of the input points. The second model extends the first one by incorporating locational uncertainty. For both models, we study the problem of computing the probability that a given point is contained in the convex hull of the uncertain points. We also consider the problem of finding the most likely convex hull, i.e., the mode of the convex hull random variable

Doctor of Philosophy

dissertationThe contributions of this dissertation are centered around designing new algorithms in the general area of sublinear algorithms such as streaming, core sets and sublinear verification, with a special interest in problems arising from data analysis including data summarization, clustering, matrix problems and massive graphs. In the first part, we focus on summaries and coresets, which are among the main techniques for designing sublinear algorithms for massive data sets. We initiate the study of coresets for uncertain data and study coresets for various types of range counting queries on uncertain data. We focus mainly on the indecisive model of locational uncertainty since it comes up frequently in real-world applications when multiple readings of the same object are made. In this model, each uncertain point has a probability density describing its location, defined as $k$ distinct locations. Our goal is to construct a subset of the uncertain points, including their locational uncertainty, so that range counting queries can be answered by examining only this subset. For each type of query we provide coreset constructions with approximation-size trade-offs. We show that random sampling can be used to construct each type of coreset, and we also provide significantly improved bounds using discrepancy-based techniques on axis-aligned range queries. In the second part, we focus on designing sublinear-space algorithms for approximate computations on massive graphs. In particular, we consider graph MAXCUT and correlation clustering problems and develop sampling based approaches to construct truly sublinear ($o(n)$) sized coresets for graphs that have polynomial (i.e., $n^{\delta}$ for any $\delta >0$) average degree. Our technique is based on analyzing properties of random induced subprograms of the linear program formulations of the problems. We demonstrate this technique with two examples. Firstly, we present a sublinear sized core set to approximate the value of the MAX CUT in a graph to a $(1+\epsilon)$ factor. To the best of our knowledge, all the known methods in this regime rely crucially on near-regularity assumptions. Secondly, we apply the same framework to construct a sublinear-sized coreset for correlation clustering. Our coreset construction also suggests 2-pass streaming algorithms for computing the MAX CUT and correlation clustering objective values which are left as future work at the time of writing this dissertation. Finally, we focus on streaming verification algorithms as another model for designing sublinear algorithms. We give the first polylog space and sublinear (in number of edges) communication protocols for any streaming verification problems in graphs. We present efficient streaming interactive proofs that can verify maximum matching exactly. Our results cover all flavors of matchings (bipartite/ nonbipartite and weighted). In addition, we also present streaming verifiers for approximate metric TSP and exact triangle counting, as well as for graph primitives such as the number of connected components, bipartiteness, minimum spanning tree and connectivity. In particular, these are the first results for weighted matchings and for metric TSP in any streaming verification model. Our streaming verifiers use only polylogarithmic space while exchanging only polylogarithmic communication with the prover in addition to the output size of the relevant solution. We also initiate a study of streaming interactive proofs (SIPs) for problems in data analysis and present efficient SIPs for some fundamental problems. We present protocols for clustering and shape fitting including minimum enclosing ball (MEB), width of a point set, $k$-centers and $k$-slab problem. We also present protocols for fundamental matrix analysis problems: We provide an improved protocol for rectangular matrix problems, which in turn can be used to verify $k$ (approximate) eigenvectors of an $n \times n$ integer matrix $A$. In general our solutions use polylogarithmic rounds of communication and polylogarithmic total communication and verifier space

Almost Symmetries and the Unit Commitment Problem

This thesis explores two main topics. The first is almost symmetry detection on graphs. The presence of symmetry in combinatorial optimization problems has long been considered an anathema, but in the past decade considerable progress has been made. Modern integer and constraint programming solvers have automatic symmetry detection built-in to either exploit or avoid symmetric regions of the search space. Automatic symmetry detection generally works by converting the input problem to a graph which is in exact correspondence with the problem formulation. Symmetry can then be detected on this graph using one of the excellent existing algorithms; these are also the symmetries of the problem formulation.The motivation for detecting almost symmetries on graphs is that almost symmetries in an integer program can force the solver to explore nearly symmetric regions of the search space. Because of the known correspondence between integer programming formulations and graphs, this is a first step toward detecting almost symmetries in integer programming formulations. Though we are only able to compute almost symmetries for graphs of modest size, the results indicate that almost symmetry is definitely present in some real-world combinatorial structures, and likely warrants further investigation.The second topic explored in this thesis is integer programming formulations for the unit commitment problem. The unit commitment problem involves scheduling power generators to meet anticipated energy demand while minimizing total system operation cost. Today, practitioners usually formulate and solve unit commitment as a large-scale mixed integer linear program.The original intent of this project was to bring the analysis of almost symmetries to the unit commitment problem. Two power generators are almost symmetric in the unit commitment problem if they have almost identical parameters. Along the way, however, new formulations for power generators were discovered that warranted a thorough investigation of their own. Chapters 4 and 5 are a result of this research.Thus this work makes three contributions to the unit commitment problem: a convex hull description for a power generator accommodating many types of constraints, an improved formulation for time-dependent start-up costs, and an exact symmetry reduction technique via reformulation

Analysis of Biochemical Reaction Networks using Tropical and Polyhedral Geometry Methods

The field of systems biology makes an attempt to realise various biological functions and processes as the emergent properties of the underlying biochemical network model. The area of computational systems biology deals with the computational methods to compute such properties. In this context, the thesis primarily discusses novel computational methods to compute the emergent properties as well as to recognize the essence in complex network models. The computational methods described in the thesis are based on the computer algebra techniques, namely tropical geometry and extreme currents. Tropical geometry is based on ideas of dominance of monomials appearing in a system of differential equations, which are often used to describe the dynamics of the network model. In such differential equation based models, tropical geometry deals with identification of the metastable regimes, defined as low dimensional regions of the phase space close to which the dynamics is much slower compared to the rest of the phase space. The application of such properties in model reduction and symbolic dynamics are demonstrated in the network models obtained from a public database namely Biomodels. Extreme currents are limiting edges of the convex polyhedrons describing the admissible fluxes in biochemical networks, which are helpful to decompose a biochemical network into a set of irreducible pathways. The pathways are shown to be associated with given clinical outcomes thereby providing some mechanistic insights associated with the clinical phenotypes. Similar to the tropical geometry, the method based on extreme currents is evaluated on the network models derived from a public database namely KEGG. Therefore, this thesis makes an attempt to explain the emergent properties of the network model by determining extreme currents or metastable regimes. Additionally, their applicability in the real world network models are discussed