Hybrid-Parallel Parameter Estimation for Frequentist and Bayesian Models

Distributed algorithms in machine learning follow two main flavors: horizontal partitioning, where the data is distributed across multiple slaves and vertical partitioning, where the model parameters are partitioned across multiple machines. The main drawback of the former strategy is that the model parameters need to be replicated on every machine. This is problematic when the number of parameters is very large, and hence cannot fit in a single machine. This drawback of the latter strategy is that the data needs to be replicated on each machine, thus failing to scale to massive datasets.The goal of this thesis is to achieve the best of both worlds by partitioning both - the data as well as the model parameters, thus enabling the training of more sophisticated models on massive datasets. In order to do so, we exploit a structure that is observed in several machine learning models, which we term as \textit{Double-Separability}. Double-Separability basically means that the objective function of the model can be decomposed into independent sub-functions which can be computed independently. For distributed machine learning, this implies that both data and model parameters can partitioned across machines and stochastic updates for parameters can be carried out independently and without any locking. Furthermore, double-separability naturally lends itself to developing efficient asynchronous algorithms which enable computation and communication to happen in parallel, offering further speedup. Some machine learning models such as Matrix Factorization directly exhibit double-separability in their objective function, however the majority of models do not. My work explores techniques to reformulate the objective function of such models to cast them into double-separable form. Often this involves introducing additional auxiliary variables that have nice interpretations. In this direction, I have developed Hybrid Parallel algorithms for machine learning tasks that include {\it Latent Collaborative Retrieval}, {\it Multinomial Logistic Regression}, {\it Variational Inference for Mixture of Exponential Families} and {\it Factorization Machines}. The software resulting from this work are available for public use under an open-source license

Global search algorithms for geometric and multiplicative optimization problems

Orientador: Paulo Augusto Valente FerreiraTese (doutorado) - Universidade Estadual de Campinas, Faculdade de Engenharia Eletrica e de ComputaçãoResumo: Nesta tese são propostos novos algoritmos de otimização baseados na busca global para duas importantes classes de problemas de programação não-linear: problemas geométricos, nos quais as funções envolvidas são descritas por somas de polinômios generalizados, e problemas de programação multiplicativa convexa, os quais, por sua vez, apresentam funções objetivos e/ou restrições expressas como produtos de funções convexas. Uma abordagem multiobjetivo para problemas geométricos posinomiais, que admitem reformulações convexas, é apresentada. Para problemas geométricos signomiais, que não possuem reformulações convexas conhecidas, propõe-se incorporar um procedimento de busca local a um algoritmo branch-and-bound, visando acelerar a convergência deste tipo de algoritmo. Elementos de análise convexa e programação multiobjetivo são usados para abordar problemas de programação multiplicativa quando estes apresentam produtos e somas de produtos de funções convexas positivas nas suas funções objetivos. Um mínimo global para o primeiro caso é obtido como o limite das soluções de uma seqüência de minimizações quase-côncavas sobre politopos, resolvidas eficientemente por meio de enumeração de vértices. Um mínimo global para o segundo caso é obtido como o limite das soluções de uma seqüência de problemas quadráticos indefinidos com características especiais, resolvidos por enumeração de restrições. O desempenho computacional dos algoritmos propostos nesta tese é avaliado por meio de problemas-testes e comparado com algoritmos alternativos existentes na literaturaAbstract: In this thesis new optimization algorithms based on global search are proposed for two important classes of nonlinear programming problems: geometric problems, in which the functions involved are described by a sum of generalized polynomials, and convex multiplicative problems, in which, in turn, objective functions and/or constraints are expressed as a product of convex functions. A multiobjective approach for posinomial geometric problems, which admit convex reformulations, is presented. As convex reformulations for signomial geometric problems are unknown, a local search procedure with the purpose of speeding up the convergence of branchand-bound algorithms is proposed. Elements of convex analysis and multiobjective programming are used for dealing with multiplicative programming problems presenting products and sums of products of positive convex functions in their objective functions. A global minimum in the first case is obtained as the limit of a sequence of quasi-concave minimizations on polytopes, efficiently solved by vertex enumeration. A global minimum for the second case is obtained as the limit of a sequence of special indefinite quadratic problems, solved by constraint enumeration. The computational performance of the algorithms proposed in this thesis has been evaluated by means of test problems and compared with alternate algorithms from the literatureDoutoradoAutomaçãoDoutor em Engenharia Elétric

Algorithmes coniques pour la minimisation quasiconcave

Borne inférieur simpliciale -- Borne inférieur double-simpliciable -- Subdivisions -- Algorithmes coniques -- Minimisation de la somme d'un produit de fonctions convexes et d'une fonction convexe additionnelle sur un ensemble convexe