20,472 research outputs found
Efficient Algorithms and Error Analysis for the Modified Nystrom Method
Many kernel methods suffer from high time and space complexities and are thus
prohibitive in big-data applications. To tackle the computational challenge,
the Nystr\"om method has been extensively used to reduce time and space
complexities by sacrificing some accuracy. The Nystr\"om method speedups
computation by constructing an approximation of the kernel matrix using only a
few columns of the matrix. Recently, a variant of the Nystr\"om method called
the modified Nystr\"om method has demonstrated significant improvement over the
standard Nystr\"om method in approximation accuracy, both theoretically and
empirically.
In this paper, we propose two algorithms that make the modified Nystr\"om
method practical. First, we devise a simple column selection algorithm with a
provable error bound. Our algorithm is more efficient and easier to implement
than and nearly as accurate as the state-of-the-art algorithm. Second, with the
selected columns at hand, we propose an algorithm that computes the
approximation in lower time complexity than the approach in the previous work.
Furthermore, we prove that the modified Nystr\"om method is exact under certain
conditions, and we establish a lower error bound for the modified Nystr\"om
method.Comment: 9-page paper plus appendix. In Proceedings of the 17th International
Conference on Artificial Intelligence and Statistics (AISTATS) 2014,
Reykjavik, Iceland. JMLR: W&CP volume 3
Optimal CUR Matrix Decompositions
The CUR decomposition of an matrix finds an
matrix with a subset of columns of together with an matrix with a subset of rows of as well as a
low-rank matrix such that the matrix approximates the matrix
that is, , where
denotes the Frobenius norm and is the best matrix
of rank constructed via the SVD. We present input-sparsity-time and
deterministic algorithms for constructing such a CUR decomposition where
and and rank. Up to constant
factors, our algorithms are simultaneously optimal in and rank.Comment: small revision in lemma 4.
Improving CUR Matrix Decomposition and the Nystr\"{o}m Approximation via Adaptive Sampling
The CUR matrix decomposition and the Nystr\"{o}m approximation are two
important low-rank matrix approximation techniques. The Nystr\"{o}m method
approximates a symmetric positive semidefinite matrix in terms of a small
number of its columns, while CUR approximates an arbitrary data matrix by a
small number of its columns and rows. Thus, CUR decomposition can be regarded
as an extension of the Nystr\"{o}m approximation.
In this paper we establish a more general error bound for the adaptive
column/row sampling algorithm, based on which we propose more accurate CUR and
Nystr\"{o}m algorithms with expected relative-error bounds. The proposed CUR
and Nystr\"{o}m algorithms also have low time complexity and can avoid
maintaining the whole data matrix in RAM. In addition, we give theoretical
analysis for the lower error bounds of the standard Nystr\"{o}m method and the
ensemble Nystr\"{o}m method. The main theoretical results established in this
paper are novel, and our analysis makes no special assumption on the data
matrices
On sparse representations of linear operators and the approximation of matrix products
Thus far, sparse representations have been exploited largely in the context
of robustly estimating functions in a noisy environment from a few
measurements. In this context, the existence of a basis in which the signal
class under consideration is sparse is used to decrease the number of necessary
measurements while controlling the approximation error. In this paper, we
instead focus on applications in numerical analysis, by way of sparse
representations of linear operators with the objective of minimizing the number
of operations needed to perform basic operations (here, multiplication) on
these operators. We represent a linear operator by a sum of rank-one operators,
and show how a sparse representation that guarantees a low approximation error
for the product can be obtained from analyzing an induced quadratic form. This
construction in turn yields new algorithms for computing approximate matrix
products.Comment: 6 pages, 3 figures; presented at the 42nd Annual Conference on
Information Sciences and Systems (CISS 2008
Matrix Completion from Non-Uniformly Sampled Entries
In this paper, we consider matrix completion from non-uniformly sampled
entries including fully observed and partially observed columns. Specifically,
we assume that a small number of columns are randomly selected and fully
observed, and each remaining column is partially observed with uniform
sampling. To recover the unknown matrix, we first recover its column space from
the fully observed columns. Then, for each partially observed column, we
recover it by finding a vector which lies in the recovered column space and
consists of the observed entries. When the unknown matrix is
low-rank, we show that our algorithm can exactly recover it from merely
entries, where is the rank of the matrix. Furthermore,
for a noisy low-rank matrix, our algorithm computes a low-rank approximation of
the unknown matrix and enjoys an additive error bound measured by Frobenius
norm. Experimental results on synthetic datasets verify our theoretical claims
and demonstrate the effectiveness of our proposed algorithm
Matrix-free construction of HSS representation using adaptive randomized sampling
We present new algorithms for the randomized construction of hierarchically
semi-separable matrices, addressing several practical issues. The HSS
construction algorithms use a partially matrix-free, adaptive randomized
projection scheme to determine the maximum off-diagonal block rank. We develop
both relative and absolute stopping criteria to determine the minimum dimension
of the random projection matrix that is sufficient for the desired accuracy.
Two strategies are discussed to adaptively enlarge the random sample matrix:
repeated doubling of the number of random vectors, and iteratively incrementing
the number of random vectors by a fixed number. The relative and absolute
stopping criteria are based on probabilistic bounds for the Frobenius norm of
the random projection of the Hankel blocks of the input matrix. We discuss
parallel implementation and computation and communication cost of both
variants. Parallel numerical results for a range of applications, including
boundary element method matrices and quantum chemistry Toeplitz matrices, show
the effectiveness, scalability and numerical robustness of the proposed
algorithms.Comment: 24 pages, 4 figures, 20 reference
Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions
Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets. This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed—either explicitly or
implicitly—to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, robustness, and/or speed. These claims are supported by extensive numerical experiments and a detailed error analysis. The specific benefits of randomized techniques depend on the computational environment. Consider the model problem of finding the k dominant components of the singular value decomposition of an m × n matrix. (i) For a dense input matrix, randomized algorithms require O(mn log(k))
floating-point operations (flops) in contrast to O(mnk) for classical algorithms. (ii) For a sparse input matrix, the flop count matches classical Krylov subspace methods, but the randomized approach is more robust and can easily be reorganized to exploit multiprocessor architectures. (iii) For a matrix that is too large to fit in fast memory, the randomized techniques require only a constant number of passes over the data, as opposed to O(k) passes for classical algorithms. In fact, it is sometimes possible to perform matrix approximation with a single pass over the data
Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions
Low-rank matrix approximations, such as the truncated singular value
decomposition and the rank-revealing QR decomposition, play a central role in
data analysis and scientific computing. This work surveys and extends recent
research which demonstrates that randomization offers a powerful tool for
performing low-rank matrix approximation. These techniques exploit modern
computational architectures more fully than classical methods and open the
possibility of dealing with truly massive data sets.
This paper presents a modular framework for constructing randomized
algorithms that compute partial matrix decompositions. These methods use random
sampling to identify a subspace that captures most of the action of a matrix.
The input matrix is then compressed---either explicitly or implicitly---to this
subspace, and the reduced matrix is manipulated deterministically to obtain the
desired low-rank factorization. In many cases, this approach beats its
classical competitors in terms of accuracy, speed, and robustness. These claims
are supported by extensive numerical experiments and a detailed error analysis
Fast Multipole Method as a Matrix-Free Hierarchical Low-Rank Approximation
There has been a large increase in the amount of work on hierarchical
low-rank approximation methods, where the interest is shared by multiple
communities that previously did not intersect. This objective of this article
is two-fold; to provide a thorough review of the recent advancements in this
field from both analytical and algebraic perspectives, and to present a
comparative benchmark of two highly optimized implementations of contrasting
methods for some simple yet representative test cases. We categorize the recent
advances in this field from the perspective of compute-memory tradeoff, which
has not been considered in much detail in this area. Benchmark tests reveal
that there is a large difference in the memory consumption and performance
between the different methods.Comment: 19 pages, 6 figure
Near-Optimal Column-Based Matrix Reconstruction
We consider low-rank reconstruction of a matrix using its columns and we
present asymptotically optimal algorithms for both spectral norm and Frobenius
norm reconstruction. The main tools we introduce to obtain our r esults are:
(i) the use of fast approximate SVD-like decompositions for column
reconstruction, and (ii) two deter ministic algorithms for selecting rows from
matrices with orthonormal columns, building upon the sparse represen tation
theorem for decompositions of the identity that appeared in \cite{BSS09}.Comment: SIAM Journal on Computing (SICOMP), invited to special issue of FOCS
201
- …