87 research outputs found
Bregman Voronoi Diagrams: Properties, Algorithms and Applications
The Voronoi diagram of a finite set of objects is a fundamental geometric
structure that subdivides the embedding space into regions, each region
consisting of the points that are closer to a given object than to the others.
We may define many variants of Voronoi diagrams depending on the class of
objects, the distance functions and the embedding space. In this paper, we
investigate a framework for defining and building Voronoi diagrams for a broad
class of distance functions called Bregman divergences. Bregman divergences
include not only the traditional (squared) Euclidean distance but also various
divergence measures based on entropic functions. Accordingly, Bregman Voronoi
diagrams allow to define information-theoretic Voronoi diagrams in statistical
parametric spaces based on the relative entropy of distributions. We define
several types of Bregman diagrams, establish correspondences between those
diagrams (using the Legendre transformation), and show how to compute them
efficiently. We also introduce extensions of these diagrams, e.g. k-order and
k-bag Bregman Voronoi diagrams, and introduce Bregman triangulations of a set
of points and their connexion with Bregman Voronoi diagrams. We show that these
triangulations capture many of the properties of the celebrated Delaunay
triangulation. Finally, we give some applications of Bregman Voronoi diagrams
which are of interest in the context of computational geometry and machine
learning.Comment: Extend the proceedings abstract of SODA 2007 (46 pages, 15 figures
Bregman Voronoi diagrams
A preliminary version appeared in the 18th ACM-SIAM Symposium on Discrete Algorithms, pp. 746- 755, 2007International audienceThe Voronoi diagram of a finite set of objects is a fundamental geometric structure that subdivides the embedding space into regions, each region consisting of the points that are closer to a given object than to the others. We may define various variants of Voronoi diagrams depending on the class of objects, the distance function and the embedding space. In this paper, we investigate a framework for defining and building Voronoi diagrams for a broad class of distance functions called Bregman divergences. Bregman divergences include not only the traditional (squared) Euclidean distance but also various divergence measures based on entropic functions. Accordingly, Bregman Voronoi diagrams allow one to define information-theoretic Voronoi diagrams in sta- tistical parametric spaces based on the relative entropy of distributions. We define several types of Bregman diagrams, establish correspondences between those diagrams (using the Legendre transformation), and show how to compute them efficiently. We also introduce extensions of these diagrams, e.g. k-order and k-bag Bregman Voronoi diagrams, and introduce Bregman triangulations of a set of points and their connection with Bregman Voronoi diagrams. We show that these triangulations capture many of the properties of the celebrated Delaunay triangulation
Optimal interval clustering: Application to Bregman clustering and statistical mixture learning
We present a generic dynamic programming method to compute the optimal
clustering of scalar elements into pairwise disjoint intervals. This
case includes 1D Euclidean -means, -medoids, -medians, -centers,
etc. We extend the method to incorporate cluster size constraints and show how
to choose the appropriate by model selection. Finally, we illustrate and
refine the method on two case studies: Bregman clustering and statistical
mixture learning maximizing the complete likelihood.Comment: 10 pages, 3 figure
Guaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalities
Information-theoretic measures such as the entropy, cross-entropy and the
Kullback-Leibler divergence between two mixture models is a core primitive in
many signal processing tasks. Since the Kullback-Leibler divergence of mixtures
provably does not admit a closed-form formula, it is in practice either
estimated using costly Monte-Carlo stochastic integration, approximated, or
bounded using various techniques. We present a fast and generic method that
builds algorithmically closed-form lower and upper bounds on the entropy, the
cross-entropy and the Kullback-Leibler divergence of mixtures. We illustrate
the versatile method by reporting on our experiments for approximating the
Kullback-Leibler divergence between univariate exponential mixtures, Gaussian
mixtures, Rayleigh mixtures, and Gamma mixtures.Comment: 20 pages, 3 figure
-MLE: A fast algorithm for learning statistical mixture models
We describe -MLE, a fast and efficient local search algorithm for learning
finite statistical mixtures of exponential families such as Gaussian mixture
models. Mixture models are traditionally learned using the
expectation-maximization (EM) soft clustering technique that monotonically
increases the incomplete (expected complete) likelihood. Given prescribed
mixture weights, the hard clustering -MLE algorithm iteratively assigns data
to the most likely weighted component and update the component models using
Maximum Likelihood Estimators (MLEs). Using the duality between exponential
families and Bregman divergences, we prove that the local convergence of the
complete likelihood of -MLE follows directly from the convergence of a dual
additively weighted Bregman hard clustering. The inner loop of -MLE can be
implemented using any -means heuristic like the celebrated Lloyd's batched
or Hartigan's greedy swap updates. We then show how to update the mixture
weights by minimizing a cross-entropy criterion that implies to update weights
by taking the relative proportion of cluster points, and reiterate the mixture
parameter update and mixture weight update processes until convergence. Hard EM
is interpreted as a special case of -MLE when both the component update and
the weight update are performed successively in the inner loop. To initialize
-MLE, we propose -MLE++, a careful initialization of -MLE guaranteeing
probabilistically a global bound on the best possible complete likelihood.Comment: 31 pages, Extend preliminary paper presented at IEEE ICASSP 201
The Bregman chord divergence
Distances are fundamental primitives whose choice significantly impacts the
performances of algorithms in machine learning and signal processing. However
selecting the most appropriate distance for a given task is an endeavor.
Instead of testing one by one the entries of an ever-expanding dictionary of
{\em ad hoc} distances, one rather prefers to consider parametric classes of
distances that are exhaustively characterized by axioms derived from first
principles. Bregman divergences are such a class. However fine-tuning a Bregman
divergence is delicate since it requires to smoothly adjust a functional
generator. In this work, we propose an extension of Bregman divergences called
the Bregman chord divergences. This new class of distances does not require
gradient calculations, uses two scalar parameters that can be easily tailored
in applications, and generalizes asymptotically Bregman divergences.Comment: 10 page
- …