5,227 research outputs found
Principal Manifolds and Nonlinear Dimension Reduction via Local Tangent Space Alignment
Nonlinear manifold learning from unorganized data points is a very
challenging unsupervised learning and data visualization problem with a great
variety of applications. In this paper we present a new algorithm for manifold
learning and nonlinear dimension reduction. Based on a set of unorganized data
points sampled with noise from the manifold, we represent the local geometry of
the manifold using tangent spaces learned by fitting an affine subspace in a
neighborhood of each data point. Those tangent spaces are aligned to give the
internal global coordinates of the data points with respect to the underlying
manifold by way of a partial eigendecomposition of the neighborhood connection
matrix. We present a careful error analysis of our algorithm and show that the
reconstruction errors are of second-order accuracy. We illustrate our algorithm
using curves and surfaces both in
2D/3D and higher dimensional Euclidean spaces, and 64-by-64 pixel face images
with various pose and lighting conditions. We also address several theoretical
and algorithmic issues for further research and improvements
Adaptive Affinity Matrix for Unsupervised Metric Learning
Spectral clustering is one of the most popular clustering approaches with the
capability to handle some challenging clustering problems. Most spectral
clustering methods provide a nonlinear map from the data manifold to a
subspace. Only a little work focuses on the explicit linear map which can be
viewed as the unsupervised distance metric learning. In practice, the selection
of the affinity matrix exhibits a tremendous impact on the unsupervised
learning. While much success of affinity learning has been achieved in recent
years, some issues such as noise reduction remain to be addressed. In this
paper, we propose a novel method, dubbed Adaptive Affinity Matrix (AdaAM), to
learn an adaptive affinity matrix and derive a distance metric from the
affinity. We assume the affinity matrix to be positive semidefinite with
ability to quantify the pairwise dissimilarity. Our method is based on posing
the optimization of objective function as a spectral decomposition problem. We
yield the affinity from both the original data distribution and the widely-used
heat kernel. The provided matrix can be regarded as the optimal representation
of pairwise relationship on the manifold. Extensive experiments on a number of
real-world data sets show the effectiveness and efficiency of AdaAM
High Dimensional Nonlinear Learning using Local Coordinate Coding
This paper introduces a new method for semi-supervised learning on high
dimensional nonlinear manifolds, which includes a phase of unsupervised basis
learning and a phase of supervised function learning. The learned bases provide
a set of anchor points to form a local coordinate system, such that each data
point on the manifold can be locally approximated by a linear combination
of its nearby anchor points, with the linear weights offering a
local-coordinate coding of . We show that a high dimensional nonlinear
function can be approximated by a global linear function with respect to this
coding scheme, and the approximation quality is ensured by the locality of such
coding. The method turns a difficult nonlinear learning problem into a simple
global linear learning problem, which overcomes some drawbacks of traditional
local learning methods. The work also gives a theoretical justification to the
empirical success of some biologically-inspired models using sparse coding of
sensory data, since a local coding scheme must be sufficiently sparse. However,
sparsity does not always satisfy locality conditions, and can thus possibly
lead to suboptimal results. The properties and performances of the method are
empirically verified on synthetic data, handwritten digit classification, and
object recognition tasks
Geometric Numerical Integration of the Assignment Flow
The assignment flow is a smooth dynamical system that evolves on an
elementary statistical manifold and performs contextual data labeling on a
graph. We derive and introduce the linear assignment flow that evolves
nonlinearly on the manifold, but is governed by a linear ODE on the tangent
space. Various numerical schemes adapted to the mathematical structure of these
two models are designed and studied, for the geometric numerical integration of
both flows: embedded Runge-Kutta-Munthe-Kaas schemes for the nonlinear flow,
adaptive Runge-Kutta schemes and exponential integrators for the linear flow.
All algorithms are parameter free, except for setting a tolerance value that
specifies adaptive step size selection by monitoring the local integration
error, or fixing the dimension of the Krylov subspace approximation. These
algorithms provide a basis for applying the assignment flow to machine learning
scenarios beyond supervised labeling, including unsupervised labeling and
learning from controlled assignment flows
Missing Value Imputation With Unsupervised Backpropagation
Many data mining and data analysis techniques operate on dense matrices or
complete tables of data. Real-world data sets, however, often contain unknown
values. Even many classification algorithms that are designed to operate with
missing values still exhibit deteriorated accuracy. One approach to handling
missing values is to fill in (impute) the missing values. In this paper, we
present a technique for unsupervised learning called Unsupervised
Backpropagation (UBP), which trains a multi-layer perceptron to fit to the
manifold sampled by a set of observed point-vectors. We evaluate UBP with the
task of imputing missing values in datasets, and show that UBP is able to
predict missing values with significantly lower sum-squared error than other
collaborative filtering and imputation techniques. We also demonstrate with 24
datasets and 9 supervised learning algorithms that classification accuracy is
usually higher when randomly-withheld values are imputed using UBP, rather than
with other methods
Nonlinear Supervised Dimensionality Reduction via Smooth Regular Embeddings
The recovery of the intrinsic geometric structures of data collections is an
important problem in data analysis. Supervised extensions of several manifold
learning approaches have been proposed in the recent years. Meanwhile, existing
methods primarily focus on the embedding of the training data, and the
generalization of the embedding to initially unseen test data is rather
ignored. In this work, we build on recent theoretical results on the
generalization performance of supervised manifold learning algorithms.
Motivated by these performance bounds, we propose a supervised manifold
learning method that computes a nonlinear embedding while constructing a smooth
and regular interpolation function that extends the embedding to the whole data
space in order to achieve satisfactory generalization. The embedding and the
interpolator are jointly learnt such that the Lipschitz regularity of the
interpolator is imposed while ensuring the separation between different
classes. Experimental results on several image data sets show that the proposed
method outperforms traditional classifiers and the supervised dimensionality
reduction algorithms in comparison in terms of classification accuracy in most
settings
Explore intrinsic geometry of sleep dynamics and predict sleep stage by unsupervised learning techniques
We propose a novel unsupervised approach for sleep dynamics exploration and
automatic annotation by combining modern harmonic analysis tools. Specifically,
we apply diffusion-based algorithms, diffusion map (DM) and alternating
diffusion (AD) algorithms, to reconstruct the intrinsic geometry of sleep
dynamics by reorganizing the spectral information of an electroencephalogram
(EEG) extracted from a nonlinear-type time frequency analysis tool, the
synchrosqueezing transform (SST). The visualization is achieved by the
nonlinear dimension reduction properties of DM and AD. Moreover, the
reconstructed nonlinear geometric structure of the sleep dynamics allows us to
achieve the automatic annotation purpose. The hidden Markov model is trained to
predict the sleep stage. The prediction performance is validated on a publicly
available benchmark database, Physionet Sleep-EDF [extended] SC* and ST*, with
the leave-one-subject-out cross validation. The overall accuracy and macro F1
achieve 82:57% and 76% in Sleep-EDF SC* and 77.01% and 71:53% in Sleep-EDF ST*,
which is compatible with the state-of-the-art results by supervised
learning-based algorithms. The results suggest the potential of the proposed
algorithm for clinical applications.Comment: 41 pages, 21 figures. arXiv admin note: text overlap with
arXiv:1803.0171
Unsupervised Manifold Clustering of Topological Phononics
Classification of topological phononics is challenging due to the lack of
universal topological invariants and the randomness of structure patterns.
Here, we show the unsupervised manifold learning for clustering topological
phononics without any priori knowledge, neither topological invariants nor
supervised trainings, even when systems are imperfect or disordered. This is
achieved by exploiting the real-space projection operator about finite phononic
lattices to describe the correlation between oscillators. We exemplify the
efficient unsupervised manifold clustering in typical phononic systems,
including one-dimensional Su-Schrieffer-Heeger-type phononic chain with random
couplings, amorphous phononic topological insulators, higher-order phononic
topological states and non-Hermitian phononic chain with random dissipations.
The results would inspire more efforts on applications of unsupervised machine
learning for topological phononic devices and beyond.Comment: 6 pages, 4 figure
Dimensionality Reduction on SPD Manifolds: The Emergence of Geometry-Aware Methods
Representing images and videos with Symmetric Positive Definite (SPD)
matrices, and considering the Riemannian geometry of the resulting space, has
been shown to yield high discriminative power in many visual recognition tasks.
Unfortunately, computation on the Riemannian manifold of SPD matrices
-especially of high-dimensional ones- comes at a high cost that limits the
applicability of existing techniques. In this paper, we introduce algorithms
able to handle high-dimensional SPD matrices by constructing a
lower-dimensional SPD manifold. To this end, we propose to model the mapping
from the high-dimensional SPD manifold to the low-dimensional one with an
orthonormal projection. This lets us formulate dimensionality reduction as the
problem of finding a projection that yields a low-dimensional manifold either
with maximum discriminative power in the supervised scenario, or with maximum
variance of the data in the unsupervised one. We show that learning can be
expressed as an optimization problem on a Grassmann manifold and discuss fast
solutions for special cases. Our evaluation on several classification tasks
evidences that our approach leads to a significant accuracy gain over
state-of-the-art methods.Comment: arXiv admin note: text overlap with arXiv:1407.112
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Despite the widespread practical success of deep learning methods, our
theoretical understanding of the dynamics of learning in deep neural networks
remains quite sparse. We attempt to bridge the gap between the theory and
practice of deep learning by systematically analyzing learning dynamics for the
restricted case of deep linear neural networks. Despite the linearity of their
input-output map, such networks have nonlinear gradient descent dynamics on
weights that change with the addition of each new hidden layer. We show that
deep linear networks exhibit nonlinear learning phenomena similar to those seen
in simulations of nonlinear networks, including long plateaus followed by rapid
transitions to lower error solutions, and faster convergence from greedy
unsupervised pretraining initial conditions than from random initial
conditions. We provide an analytical description of these phenomena by finding
new exact solutions to the nonlinear dynamics of deep learning. Our theoretical
analysis also reveals the surprising finding that as the depth of a network
approaches infinity, learning speed can nevertheless remain finite: for a
special class of initial conditions on the weights, very deep networks incur
only a finite, depth independent, delay in learning speed relative to shallow
networks. We show that, under certain conditions on the training data,
unsupervised pretraining can find this special class of initial conditions,
while scaled random Gaussian initializations cannot. We further exhibit a new
class of random orthogonal initial conditions on weights that, like
unsupervised pre-training, enjoys depth independent learning times. We further
show that these initial conditions also lead to faithful propagation of
gradients even in deep nonlinear networks, as long as they operate in a special
regime known as the edge of chaos.Comment: Submission to ICLR2014. Revised based on reviewer feedbac
- …