178 research outputs found
Graph Laplacians and their convergence on random neighborhood graphs
Given a sample from a probability measure with support on a submanifold in
Euclidean space one can construct a neighborhood graph which can be seen as an
approximation of the submanifold. The graph Laplacian of such a graph is used
in several machine learning methods like semi-supervised learning,
dimensionality reduction and clustering. In this paper we determine the
pointwise limit of three different graph Laplacians used in the literature as
the sample size increases and the neighborhood size approaches zero. We show
that for a uniform measure on the submanifold all graph Laplacians have the
same limit up to constants. However in the case of a non-uniform measure on the
submanifold only the so called random walk graph Laplacian converges to the
weighted Laplace-Beltrami operator.Comment: Improved presentation, typos corrected, to appear in JML
Exploring Data Geometry for Continual Learning
Continual learning aims to efficiently learn from a non-stationary stream of
data while avoiding forgetting the knowledge of old data. In many practical
applications, data complies with non-Euclidean geometry. As such, the commonly
used Euclidean space cannot gracefully capture non-Euclidean geometric
structures of data, leading to inferior results. In this paper, we study
continual learning from a novel perspective by exploring data geometry for the
non-stationary stream of data. Our method dynamically expands the geometry of
the underlying space to match growing geometric structures induced by new data,
and prevents forgetting by keeping geometric structures of old data into
account. In doing so, making use of the mixed curvature space, we propose an
incremental search scheme, through which the growing geometric structures are
encoded. Then, we introduce an angular-regularization loss and a
neighbor-robustness loss to train the model, capable of penalizing the change
of global geometric structures and local geometric structures. Experiments show
that our method achieves better performance than baseline methods designed in
Euclidean space.Comment: CVPR 202
A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold
Although Deep Learning (DL) has achieved success in complex Artificial
Intelligence (AI) tasks, it suffers from various notorious problems (e.g.,
feature redundancy, and vanishing or exploding gradients), since updating
parameters in Euclidean space cannot fully exploit the geometric structure of
the solution space. As a promising alternative solution, Riemannian-based DL
uses geometric optimization to update parameters on Riemannian manifolds and
can leverage the underlying geometric information. Accordingly, this article
presents a comprehensive survey of applying geometric optimization in DL. At
first, this article introduces the basic procedure of the geometric
optimization, including various geometric optimizers and some concepts of
Riemannian manifold. Subsequently, this article investigates the application of
geometric optimization in different DL networks in various AI tasks, e.g.,
convolution neural network, recurrent neural network, transfer learning, and
optimal transport. Additionally, typical public toolboxes that implement
optimization on manifold are also discussed. Finally, this article makes a
performance comparison between different deep geometric optimization methods
under image recognition scenarios.Comment: 41 page
Compactness and isotopy finiteness for submanifolds with uniformly bounded geometric curvature energies
In this paper, we establish compactness for various geometric curvature
energies including integral Menger curvature, and tangent-point repulsive
potentials, defined a priori on the class of compact, embedded -dimensional
Lipschitz submanifolds in . It turns out that due to a
smoothing effect any sequence of submanifolds with uniformly bounded energy
contains a subsequence converging in to a limit submanifold.
This result has two applications. The first one is an isotopy finiteness
theorem: there are only finitely many isotopy types of such submanifolds below
a given energy value, and we provide explicit bounds on the number of isotopy
types in terms of the respective energy. The second one is the lower
semicontinuity - with respect to Hausdorff-convergence of submanifolds - of all
geometric curvature energies under consideration, which can be used to minimise
each of these energies within prescribed isotopy classes.Comment: 44 pages, 5 figure
Global rates of convergence for nonconvex optimization on manifolds
We consider the minimization of a cost function on a manifold using
Riemannian gradient descent and Riemannian trust regions (RTR). We focus on
satisfying necessary optimality conditions within a tolerance .
Specifically, we show that, under Lipschitz-type assumptions on the pullbacks
of to the tangent spaces of , both of these algorithms produce points
with Riemannian gradient smaller than in
iterations. Furthermore, RTR returns a point where also the Riemannian
Hessian's least eigenvalue is larger than in
iterations. There are no assumptions on initialization.
The rates match their (sharp) unconstrained counterparts as a function of the
accuracy (up to constants) and hence are sharp in that sense.
These are the first deterministic results for global rates of convergence to
approximate first- and second-order Karush-Kuhn-Tucker points on manifolds.
They apply in particular for optimization constrained to compact submanifolds
of , under simpler assumptions.Comment: 33 pages, IMA Journal of Numerical Analysis, 201
Manifold interpolation and model reduction
One approach to parametric and adaptive model reduction is via the
interpolation of orthogonal bases, subspaces or positive definite system
matrices. In all these cases, the sampled inputs stem from matrix sets that
feature a geometric structure and thus form so-called matrix manifolds. This
work will be featured as a chapter in the upcoming Handbook on Model Order
Reduction (P. Benner, S. Grivet-Talocia, A. Quarteroni, G. Rozza, W.H.A.
Schilders, L.M. Silveira, eds, to appear on DE GRUYTER) and reviews the
numerical treatment of the most important matrix manifolds that arise in the
context of model reduction. Moreover, the principal approaches to data
interpolation and Taylor-like extrapolation on matrix manifolds are outlined
and complemented by algorithms in pseudo-code.Comment: 37 pages, 4 figures, featured chapter of upcoming "Handbook on Model
Order Reduction
- …