28,581 research outputs found
General Tensor Decomposition, Moment Matrices and Applications
SubmittedInternational audienceThe tensor decomposition addressed in this paper may be seen as a generalisation of Singular Value Decomposition of matrices. We consider general multilinear and multihomogeneous tensors. We show how to reduce the problem to a truncated moment matrix problem and give a new criterion for flat extension of Quasi-Hankel matrices. We connect this criterion to the commutation characterisation of border bases. A new algorithm is described. It applies for general multihomogeneous tensors, extending the approach of J.J. Sylvester to binary forms. An example illustrates the algebraic operations involved in this approach and how the decomposition can be recovered from eigenvector computation
Training Input-Output Recurrent Neural Networks through Spectral Methods
We consider the problem of training input-output recurrent neural networks
(RNN) for sequence labeling tasks. We propose a novel spectral approach for
learning the network parameters. It is based on decomposition of the
cross-moment tensor between the output and a non-linear transformation of the
input, based on score functions. We guarantee consistent learning with
polynomial sample and computational complexity under transparent conditions
such as non-degeneracy of model parameters, polynomial activations for the
neurons, and a Markovian evolution of the input sequence. We also extend our
results to Bidirectional RNN which uses both previous and future information to
output the label at each time point, and is employed in many NLP tasks such as
POS tagging
Sample Complexity Analysis for Learning Overcomplete Latent Variable Models through Tensor Methods
We provide guarantees for learning latent variable models emphasizing on the
overcomplete regime, where the dimensionality of the latent space can exceed
the observed dimensionality. In particular, we consider multiview mixtures,
spherical Gaussian mixtures, ICA, and sparse coding models. We provide tight
concentration bounds for empirical moments through novel covering arguments. We
analyze parameter recovery through a simple tensor power update algorithm. In
the semi-supervised setting, we exploit the label or prior information to get a
rough estimate of the model parameters, and then refine it using the tensor
method on unlabeled samples. We establish that learning is possible when the
number of components scales as , where is the observed
dimension, and is the order of the observed moment employed in the tensor
method. Our concentration bound analysis also leads to minimax sample
complexity for semi-supervised learning of spherical Gaussian mixtures. In the
unsupervised setting, we use a simple initialization algorithm based on SVD of
the tensor slices, and provide guarantees under the stricter condition that
(where constant can be larger than ), where the
tensor method recovers the components under a polynomial running time (and
exponential in ). Our analysis establishes that a wide range of
overcomplete latent variable models can be learned efficiently with low
computational and sample complexity through tensor decomposition methods.Comment: Title change
Symmetric tensor decomposition
We present an algorithm for decomposing a symmetric tensor, of dimension n
and order d as a sum of rank-1 symmetric tensors, extending the algorithm of
Sylvester devised in 1886 for binary forms. We recall the correspondence
between the decomposition of a homogeneous polynomial in n variables of total
degree d as a sum of powers of linear forms (Waring's problem), incidence
properties on secant varieties of the Veronese Variety and the representation
of linear forms as a linear combination of evaluations at distinct points. Then
we reformulate Sylvester's approach from the dual point of view. Exploiting
this duality, we propose necessary and sufficient conditions for the existence
of such a decomposition of a given rank, using the properties of Hankel (and
quasi-Hankel) matrices, derived from multivariate polynomials and normal form
computations. This leads to the resolution of polynomial equations of small
degree in non-generic cases. We propose a new algorithm for symmetric tensor
decomposition, based on this characterization and on linear algebra
computations with these Hankel matrices. The impact of this contribution is
two-fold. First it permits an efficient computation of the decomposition of any
tensor of sub-generic rank, as opposed to widely used iterative algorithms with
unproved global convergence (e.g. Alternate Least Squares or gradient
descents). Second, it gives tools for understanding uniqueness conditions, and
for detecting the rank
Fourier PCA and Robust Tensor Decomposition
Fourier PCA is Principal Component Analysis of a matrix obtained from higher
order derivatives of the logarithm of the Fourier transform of a
distribution.We make this method algorithmic by developing a tensor
decomposition method for a pair of tensors sharing the same vectors in rank-
decompositions. Our main application is the first provably polynomial-time
algorithm for underdetermined ICA, i.e., learning an matrix
from observations where is drawn from an unknown product
distribution with arbitrary non-Gaussian components. The number of component
distributions can be arbitrarily higher than the dimension and the
columns of only need to satisfy a natural and efficiently verifiable
nondegeneracy condition. As a second application, we give an alternative
algorithm for learning mixtures of spherical Gaussians with linearly
independent means. These results also hold in the presence of Gaussian noise.Comment: Extensively revised; details added; minor errors corrected;
exposition improve
Hankel Tensors: Associated Hankel Matrices and Vandermonde Decomposition
Hankel tensors arise from applications such as signal processing. In this
paper, we make an initial study on Hankel tensors. For each Hankel tensor, we
associate it with a Hankel matrix and a higher order two-dimensional symmetric
tensor, which we call the associated plane tensor. If the associated Hankel
matrix is positive semi-definite, we call such a Hankel tensor a strong Hankel
tensor. We show that an order -dimensional tensor is a Hankel tensor if
and only if it has a Vandermonde decomposition. We call a Hankel tensor a
complete Hankel tensor if it has a Vandermonde decomposition with positive
coefficients. We prove that if a Hankel tensor is copositive or an even order
Hankel tensor is positive semi-definite, then the associated plane tensor is
copositive or positive semi-definite, respectively. We show that even order
strong and complete Hankel tensors are positive semi-definite, the Hadamard
product of two strong Hankel tensors is a strong Hankel tensor, and the
Hadamard product of two complete Hankel tensors is a complete Hankel tensor. We
show that all the H-eigenvalue of a complete Hankel tensors (maybe of odd
order) are nonnegative. We give some upper bounds and lower bounds for the
smallest and the largest Z-eigenvalues of a Hankel tensor, respectively.
Further questions on Hankel tensors are raised
Tensor decompositions for learning latent variable models
This work considers a computationally and statistically efficient parameter
estimation method for a wide class of latent variable models---including
Gaussian mixture models, hidden Markov models, and latent Dirichlet
allocation---which exploits a certain tensor structure in their low-order
observable moments (typically, of second- and third-order). Specifically,
parameter estimation is reduced to the problem of extracting a certain
(orthogonal) decomposition of a symmetric tensor derived from the moments; this
decomposition can be viewed as a natural generalization of the singular value
decomposition for matrices. Although tensor decompositions are generally
intractable to compute, the decomposition of these specially structured tensors
can be efficiently obtained by a variety of approaches, including power
iterations and maximization approaches (similar to the case of matrices). A
detailed analysis of a robust tensor power method is provided, establishing an
analogue of Wedin's perturbation theorem for the singular vectors of matrices.
This implies a robust and computationally tractable estimation approach for
several popular latent variable models
- …