517 research outputs found
Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives
Part 2 of this monograph builds on the introduction to tensor networks and
their operations presented in Part 1. It focuses on tensor network models for
super-compressed higher-order representation of data/parameters and related
cost functions, while providing an outline of their applications in machine
learning and data analytics. A particular emphasis is on the tensor train (TT)
and Hierarchical Tucker (HT) decompositions, and their physically meaningful
interpretations which reflect the scalability of the tensor network approach.
Through a graphical approach, we also elucidate how, by virtue of the
underlying low-rank tensor approximations and sophisticated contractions of
core tensors, tensor networks have the ability to perform distributed
computations on otherwise prohibitively large volumes of data/parameters,
thereby alleviating or even eliminating the curse of dimensionality. The
usefulness of this concept is illustrated over a number of applied areas,
including generalized regression and classification (support tensor machines,
canonical correlation analysis, higher order partial least squares),
generalized eigenvalue decomposition, Riemannian optimization, and in the
optimization of deep neural networks. Part 1 and Part 2 of this work can be
used either as stand-alone separate texts, or indeed as a conjoint
comprehensive review of the exciting field of low-rank tensor networks and
tensor decompositions.Comment: 232 page
Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives
Part 2 of this monograph builds on the introduction to tensor networks and
their operations presented in Part 1. It focuses on tensor network models for
super-compressed higher-order representation of data/parameters and related
cost functions, while providing an outline of their applications in machine
learning and data analytics. A particular emphasis is on the tensor train (TT)
and Hierarchical Tucker (HT) decompositions, and their physically meaningful
interpretations which reflect the scalability of the tensor network approach.
Through a graphical approach, we also elucidate how, by virtue of the
underlying low-rank tensor approximations and sophisticated contractions of
core tensors, tensor networks have the ability to perform distributed
computations on otherwise prohibitively large volumes of data/parameters,
thereby alleviating or even eliminating the curse of dimensionality. The
usefulness of this concept is illustrated over a number of applied areas,
including generalized regression and classification (support tensor machines,
canonical correlation analysis, higher order partial least squares),
generalized eigenvalue decomposition, Riemannian optimization, and in the
optimization of deep neural networks. Part 1 and Part 2 of this work can be
used either as stand-alone separate texts, or indeed as a conjoint
comprehensive review of the exciting field of low-rank tensor networks and
tensor decompositions.Comment: 232 page
An improved Newton iteration for the generalized inverse of a matrix, with applications
The purpose here is to clarify and illustrate the potential for the use of variants of Newton's method of solving problems of practical interest on highly personal computers. The authors show how to accelerate the method substantially and how to modify it successfully to cope with ill-conditioned matrices. The authors conclude that Newton's method can be of value for some interesting computations, especially in parallel and other computing environments in which matrix products are especially easy to work with
Recommended from our members
Structured Sub-Nyquist Sampling with Applications in Compressive Toeplitz Covariance Estimation, Super-Resolution and Phase Retrieval
Sub-Nyquist sampling has received a huge amount of interest in the past decade. In classical compressed sensing theory, if the measurement procedure satisfies a particular condition known as Restricted Isometry Property (RIP), we can achieve stable recovery of signals of low-dimensional intrinsic structures with an order-wise optimal sample size. Such low-dimensional structures include sparse and low rank for both vector and matrix cases. The main drawback of conventional compressed sensing theory is that random measurements are required to ensure the RIP property. However, in many applications such as imaging and array signal processing, applying independent random measurements may not be practical as the systems are deterministic. Moreover, random measurements based compressed sensing always exploits convex programs for signal recovery even in the noiseless case, and solving those programs is computationally intensive if the ambient dimension is large, especially in the matrix case. The main contribution of this dissertation is that we propose a deterministic sub-Nyquist sampling framework for compressing the structured signal and come up with computationally efficient algorithms. Besides widely studied sparse and low-rank structures, we particularly focus on the cases that the signals of interest are stationary or the measurements are of Fourier type. The key difference between our work from classical compressed sensing theory is that we explicitly exploit the second-order statistics of the signals, and study the equivalent quadratic measurement model in the correlation domain. The essential observation made in this dissertation is that a difference/sum coarray structure will arise from the quadratic model if the measurements are of Fourier type. With these observations, we are able to achieve a better compression rate for covariance estimation, identify more sources in array signal processing or recover the signals of larger sparsity. In this dissertation, we will first study the problem of Toeplitz covariance estimation. In particular, we will show how to achieve an order-wise optimal compression rate using the idea of sparse arrays in both general and low-rank cases. Then, an analysis framework of super-resolution with positivity constraint is established. We will present fundamental robustness guarantees, efficient algorithms and applications in practices. Next, we will study the problem of phase-retrieval for which we successfully apply the sparse array ideas by fully exploiting the quadratic measurement model. We achieve near-optimal sample complexity for both sparse and general cases with practical Fourier measurements and provide efficient and deterministic recovery algorithms. In the end, we will further elaborate on the essential role of non-negative constraint in underdetermined inverse problems. In particular, we will analyze the nonlinear co-array interpolation problem and develop a universal upper bound of the interpolation error. Bilinear problem with non-negative constraint will be considered next and the exact characterization of the ambiguous solutions will be established for the first time in literature. At last, we will show how to apply the nested array idea to solve real problems such as Kriging. Using spatial correlation information, we are able to have a stable estimate of the field of interest with fewer sensors than classic methodologies. Extensive numerical experiments are implemented to demonstrate our theoretical claims
Accelerating Cosmic Microwave Background map-making procedure through preconditioning
Estimation of the sky signal from sequences of time ordered data is one of
the key steps in Cosmic Microwave Background (CMB) data analysis, commonly
referred to as the map-making problem. Some of the most popular and general
methods proposed for this problem involve solving generalised least squares
(GLS) equations with non-diagonal noise weights given by a block-diagonal
matrix with Toeplitz blocks. In this work we study new map-making solvers
potentially suitable for applications to the largest anticipated data sets.
They are based on iterative conjugate gradient (CG) approaches enhanced with
novel, parallel, two-level preconditioners. We apply the proposed solvers to
examples of simulated non-polarised and polarised CMB observations, and a set
of idealised scanning strategies with sky coverage ranging from nearly a full
sky down to small sky patches. We discuss in detail their implementation for
massively parallel computational platforms and their performance for a broad
range of parameters characterising the simulated data sets. We find that our
best new solver can outperform carefully-optimised standard solvers used today
by a factor of as much as 5 in terms of the convergence rate and a factor of up
to in terms of the time to solution, and to do so without significantly
increasing the memory consumption and the volume of inter-processor
communication. The performance of the new algorithms is also found to be more
stable and robust, and less dependent on specific characteristics of the
analysed data set. We therefore conclude that the proposed approaches are well
suited to address successfully challenges posed by new and forthcoming CMB data
sets.Comment: 19 pages // Final version submitted to A&
Accelerating Cosmic Microwave Background map-making procedure through preconditioning
Estimation of the sky signal from sequences of time ordered data is one of
the key steps in Cosmic Microwave Background (CMB) data analysis, commonly
referred to as the map-making problem. Some of the most popular and general
methods proposed for this problem involve solving generalised least squares
(GLS) equations with non-diagonal noise weights given by a block-diagonal
matrix with Toeplitz blocks. In this work we study new map-making solvers
potentially suitable for applications to the largest anticipated data sets.
They are based on iterative conjugate gradient (CG) approaches enhanced with
novel, parallel, two-level preconditioners. We apply the proposed solvers to
examples of simulated non-polarised and polarised CMB observations, and a set
of idealised scanning strategies with sky coverage ranging from nearly a full
sky down to small sky patches. We discuss in detail their implementation for
massively parallel computational platforms and their performance for a broad
range of parameters characterising the simulated data sets. We find that our
best new solver can outperform carefully-optimised standard solvers used today
by a factor of as much as 5 in terms of the convergence rate and a factor of up
to in terms of the time to solution, and to do so without significantly
increasing the memory consumption and the volume of inter-processor
communication. The performance of the new algorithms is also found to be more
stable and robust, and less dependent on specific characteristics of the
analysed data set. We therefore conclude that the proposed approaches are well
suited to address successfully challenges posed by new and forthcoming CMB data
sets.Comment: 19 pages // Final version submitted to A&
Massively parallel Poisson and QR factorization solvers
AbstractThe paper brings a massively parallel Poisson solver for rectangle domain and parallel algorithms for computation of QR factorization of a dense matrix A by means of Householder reflections and Givens rotations. The computer model under consideration is a SIMD mesh-connected toroidal n × n processor array.The Dirichlet problem is replaced by its finite-difference analog on an M × N (M + 1, N are powers of two) grid. The algorithm is composed of parallel fast sine transform and cyclic odd-even reduction blocks and runs in a fully parallel fashion. Its computational complexity is O(M N log Ln2), where L = max(M + 1, N). A parallel proposal of QR factorization by the Householder method zeros all subdiagonal elements in each column and updates all elements of the given submatrix in parallel. For the second method with Givens rotations, the parallel scheme of the Sameh and Kuck was chosen where the disjoint rotations can be computed simultaneously.The algorithms were coded in MPF and MPL parallel programming languages and results of computational experiments on the MasPar MP-1 system are also presented
- …