Estimating the trace of the matrix inverse by interpolating from the diagonal of an approximate inverse

A number of applications require the computation of the trace of a matrix that is implicitly available through a function. A common example of a function is the inverse of a large, sparse matrix, which is the focus of this paper. When the evaluation of the function is expensive, the task is computationally challenging because the standard approach is based on a Monte Carlo method which converges slowly. We present a different approach that exploits the pattern correlation, if present, between the diagonal of the inverse of the matrix and the diagonal of some approximate inverse that can be computed inexpensively. We leverage various sampling and fitting techniques to fit the diagonal of the approximation to the diagonal of the inverse. Depending on the quality of the approximate inverse, our method may serve as a standalone kernel for providing a fast trace estimate with a small number of samples. Furthermore, the method can be used as a variance reduction method for Monte Carlo in some cases. This is decided dynamically by our algorithm. An extensive set of experiments with various technique combinations on several matrices from some real applications demonstrate the potential of our method. (C) 2016 Published by Elsevier Inc

Algorithms for Large Scale Problems in Eigenvalue and Svd Computations and in Big Data Applications

As ”big data” has increasing influence on our daily life and research activities, it poses significant challenges on various research areas. Some applications often demand a fast solution of large, sparse eigenvalue and singular value problems; In other applications, extracting knowledge from large-scale data requires many techniques such as statistical calculations, data mining, and high performance computing. In this dissertation, we develop efficient and robust iterative methods and software for the computation of eigenvalue and singular values. We also develop practical numerical and data mining techniques to estimate the trace of a function of a large, sparse matrix and to detect in real-time blob-filaments in fusion plasma on extremely large parallel computers. In the first work, we propose a hybrid two stage SVD method for efficiently and accurately computing a few extreme singular triplets, especially the ones corresponding to the smallest singular values. The first stage achieves fast convergence while the second achieves the final accuracy. Furthermore, we develop a high-performance preconditioned SVD software based on the proposed method on top of the state-of-the-art eigensolver PRIMME. The method can be used with or without preconditioning, on parallel computers, and is superior to other state-of-the-art SVD methods in both efficiency and robustness. In the second study, we provide insights and develop practical algorithms to accomplish efficient and accurate computation of interior eigenpairs using refined projection techniques in non-Krylov iterative methods. By analyzing different implementations of the refined projection, we propose a new hybrid method to efficiently find interior eigenpairs without compromising accuracy. Our numerical experiments illustrate the efficiency and robustness of the proposed method. In the third work, we present a novel method to estimate the trace of matrix inverse that exploits the pattern correlation between the diagonal of the inverse of the matrix and that of some approximate inverse. We leverage various sampling and fitting techniques to fit the diagonal of the approximation to that of the inverse. Our method may serve as a standalone kernel for providing a fast trace estimate or as a variance reduction method for Monte Carlo in some cases. An extensive set of experiments demonstrate the potential of our method. In the fourth study, we provide first results on applying outlier detection techniques to effectively tackle the fusion blob detection problem on extremely large parallel machines. We present a real-time region outlier detection algorithm to efficiently find and track blobs in fusion experiments and simulations. Our experiments demonstrated we can achieve linear time speedup up to 1024 MPI processes and complete blob detection in two or three milliseconds

Computing Singular Values of Large Matrices with an Inverse-Free Preconditioned Krylov Subspace Method

We present an efficient algorithm for computing a few extreme singular values of a large sparse m×n matrix C. Our algorithm is based on reformulating the singular value problem as an eigenvalue problem for CTC. To address the clustering of the singular values, we develop an inverse-free preconditioned Krylov subspace method to accelerate convergence. We consider preconditioning that is based on robust incomplete factorizations, and we discuss various implementation issues. Extensive numerical tests are presented to demonstrate efficiency and robustness of the new algorithm

Singular Value Computation and Subspace Clustering

In this dissertation we discuss two problems. In the first part, we consider the problem of computing a few extreme eigenvalues of a symmetric definite generalized eigenvalue problem or a few extreme singular values of a large and sparse matrix. The standard method of choice of computing a few extreme eigenvalues of a large symmetric matrix is the Lanczos or the implicitly restarted Lanczos method. These methods usually employ a shift-and-invert transformation to accelerate the speed of convergence, which is not practical for truly large problems. With this in mind, Golub and Ye proposes an inverse-free preconditioned Krylov subspace method, which uses preconditioning instead of shift-and-invert to accelerate the convergence. To compute several eigenvalues, Wielandt is used in a straightforward manner. However, the Wielandt deflation alters the structure of the problem and may cause some difficulties in certain applications such as the singular value computations. So we first propose to consider a deflation by restriction method for the inverse-free Krylov subspace method. We generalize the original convergence theory for the inverse-free preconditioned Krylov subspace method to justify this deflation scheme. We next extend the inverse-free Krylov subspace method with deflation by restriction to the singular value problem. We consider preconditioning based on robust incomplete factorization to accelerate the convergence. Numerical examples are provided to demonstrate efficiency and robustness of the new algorithm. In the second part of this thesis, we consider the so-called subspace clustering problem, which aims for extracting a multi-subspace structure from a collection of points lying in a high-dimensional space. Recently, methods based on self expressiveness property (SEP) such as Sparse Subspace Clustering and Low Rank Representations have been shown to enjoy superior performances than other methods. However, methods with SEP may result in representations that are not amenable to clustering through graph partitioning. We propose a method where the points are expressed in terms of an orthonormal basis. The orthonormal basis is optimally chosen in the sense that the representation of all points is sparsest. Numerical results are given to illustrate the effectiveness and efficiency of this method

Applications of low-rank approximation: complex networks and inverse problems

The use of low-rank approximation is crucial when one is interested in solving problems of large dimension. In this case, the matrix with reduced rank can be obtained starting from the singular value decomposition considering only the largest components. This thesis describes how the use of the low-rank approximation can be applied both in the analysis of complex networks and in the solution of inverse problems. In the first case, it will be explained how to identify the most important nodes or how to determine the ease of traveling between them in large-scale networks that arise in many applications. The use of low-rank approximation is presented both for undirected and directed networks, whose adjacency matrices are symmetric and nonsymmetric, respectively. As a second application, we propose how to identify inhomogeneities in the ground or the presence of conductive substances. This survey is addressed with the aid of electromagnetic induction measurements taken with a ground conductivity meter. Starting from electromagnetic data collected by this device, the electrical conductivity profile of the soil is reconstructed with the aid of a regularized damped Gauss{Newton method. The inversion method is based on the low-rank approximation of the Jacobian of the function to be inverted

Many-Body Entanglement in Classical & Quantum Simulators

Entanglement is not only the key resource for many quantum technologies, but essential in understanding the structure of many-body quantum matter. At the interface of these two crucial areas are simulators, controlled systems capable of mimicking physical models that might escape analytical tractability. Traditionally, these simulations have been performed classically, where recent advancements such as tensor-networks have made explicit the limitation entanglement places on scalability. Increasingly however, analog quantum simulators are expected to yield deep insight into complex systems. This thesis advances the field in across various interconnected fronts. Firstly, we introduce schemes for verifying and distributing entanglement in a quantum dot simulator, tailored to specific experimental constraints. We then confirm that quantum dot simulators would be natural candidates for simulating many-body localization (MBL) - a recently emerged phenomenon that seems to evade traditional statistical mechanics. Following on from that, we investigate MBL from an entanglement perspective, shedding new light on the nature of the transition to it from a ergodic regime. As part of that investigation we make use of the logarithmic negativity, an entanglement measure applicable to many-body mixed states. In order to tie back into quantum simulators, we then propose an experimental scheme to measure the logarithmic negativity in realistic many-body settings. This method uses choice measurements on three or more copies of a mixed state along with machine learning techniques. We also introduce a fast method for computing many-body entanglement in classical simulations that significantly increases the size of system addressable. Finally, we introduce quimb, an open-source library for interactive but efficient quantum information and many-body calculations. It contains general purpose tensor-network support alongside other novel algorithms

Order reduction of semilinear differential matrix and tensor equations

In this thesis, we are interested in approximating, by model order reduction, the solution to large-scale matrix- or tensor-valued semilinear Ordinary Differential Equations (ODEs). Under specific hypotheses on the linear operators and the considered domain, these ODEs often stem from the space discretization on a tensor basis of semilinear Partial Differential Equations (PDEs) with a dimension greater than or equal to two. The bulk of this thesis is devoted to the case where the discrete system is a matrix equation. We consider separately the cases of general Lipschitz continuous nonlinear functions and the Differential Riccati Equation (DRE) with a quadratic nonlinear term. In both settings, we construct a pair of left-right approximation spaces that leads to a reduced semilinear matrix differential equation with the same structure as the original problem, which can be more rapidly integrated with matrix-oriented integrators. For the DRE, under certain assumptions on the data, we show that a reduction process onto rational Krylov subspaces obtains significant computational and memory savings as opposed to current approaches. In the more general setting, a challenging difference lies in selecting and constructing the two approximation bases to handle the nonlinear term effectively. In addition, the nonlinear term also needs to be approximated for efficiency. To this end, in the framework of the Proper Orthogonal Decomposition (POD) methodology and the Discrete Empirical Interpolation Method (DEIM), we derive a novel matrix-oriented reduction process leading to a practical, structure-aware low order approximation of the original problem. In the final part of the thesis, we consider the multidimensional setting. Here we extend the matrix-oriented POD-DEIM algorithm to the tensor setting and illustrate how we can apply it to systems of such equations. Moreover, we discuss how to integrate the reduced-order model and, in particular, how to solve the resulting tensor-valued linear systems

A Wiener Chaos Based Approach to Stability Analysis of Stochastic Shear Flows

As the aviation industry expands, consuming oil reserves, generating carbon dioxide gas and adding to environmental concerns, there is an increasing need for drag reduction technology. The ability to maintain a laminar flow promises significant reductions in drag, with economic and environmental benefits. Whilst development of flow control technology has gained interest, few studies investigate the impacts that uncertainty, in flow properties, can have on flow stability. Inclusion of uncertainty, inherent in all physical systems, facilitates a more realistic analysis, and is therefore central to this research. To this end, we study the stability of stochastic shear flows, and adopt a framework based upon the Wiener Chaos expansion for efficient numerical computations. We explore the stability of stochastic Poiseuille, Couette and Blasius boundary layer type base flows, presenting stochastic results for both the modal and non modal problem, contrasting with the deterministic case and identifying the responsible flow characteristics. From a numerical perspective we show that the Wiener Chaos expansion offers a highly efficient framework for the study of relatively low dimensional stochastic flow problems, whilst Monte Carlo methods remain superior in higher dimensions. Further, we demonstrate that a Gaussian auto-covariance provides a suitable model for the stochasticity present in typical wind tunnel tests, at least in the case of a Blasius boundary layer. From a physical perspective we demonstrate that it is neither the number of inflection points in a defect, nor the input variance attributed to a defect, that influences the variance in stability characteristics for Poiseuille flow, but the shape/symmetry of the defect. Conversely, we show the symmetry of defects to be less important in the case of the Blasius boundary layer, where we find that defects which increase curvature in the vicinity of the critical point generally reduce stability. In addition, we show that defects which enhance gradients in the outer regions of a boundary layer can excite centre modes with the potential to significantly impact neutral curves. Such effects can lead to the development of an additional lobe at lower wave-numbers, can be related to jet flows, and can significantly reduce the critical Reynolds number.EPSR

Algorithmic advances in learning from large dimensional matrices and scientific data

University of Minnesota Ph.D. dissertation.May 2018. Major: Computer Science. Advisor: Yousef Saad. 1 computer file (PDF); xi, 196 pages.This thesis is devoted to answering a range of questions in machine learning and data analysis related to large dimensional matrices and scientific data. Two key research objectives connect the different parts of the thesis: (a) development of fast, efficient, and scalable algorithms for machine learning which handle large matrices and high dimensional data; and (b) design of learning algorithms for scientific data applications. The work combines ideas from multiple, often non-traditional, fields leading to new algorithms, new theory, and new insights in different applications. The first of the three parts of this thesis explores numerical linear algebra tools to develop efficient algorithms for machine learning with reduced computation cost and improved scalability. Here, we first develop inexpensive algorithms combining various ideas from linear algebra and approximation theory for matrix spectrum related problems such as numerical rank estimation, matrix function trace estimation including log-determinants, Schatten norms, and other spectral sums. We also propose a new method which simultaneously estimates the dimension of the dominant subspace of covariance matrices and obtains an approximation to the subspace. Next, we consider matrix approximation problems such as low rank approximation, column subset selection, and graph sparsification. We present a new approach based on multilevel coarsening to compute these approximations for large sparse matrices and graphs. Lastly, on the linear algebra front, we devise a novel algorithm based on rank shrinkage for the dictionary learning problem, learning a small set of dictionary columns which best represent the given data. The second part of this thesis focuses on exploring novel non-traditional applications of information theory and codes, particularly in solving problems related to machine learning and high dimensional data analysis. Here, we first propose new matrix sketching methods using codes for obtaining low rank approximations of matrices and solving least squares regression problems. Next, we demonstrate that codewords from certain coding scheme perform exceptionally well for the group testing problem. Lastly, we present a novel machine learning application for coding theory, that of solving large scale multilabel classification problems. We propose a new algorithm for multilabel classification which is based on group testing and codes. The algorithm has a simple inexpensive prediction method, and the error correction capabilities of codes are exploited for the first time to correct prediction errors. The third part of the thesis focuses on devising robust and stable learning algorithms, which yield results that are interpretable from specific scientific application viewpoint. We present Union of Intersections (UoI), a flexible, modular, and scalable framework for statistical-machine learning problems. We then adapt this framework to develop new algorithms for matrix decomposition problems such as nonnegative matrix factorization (NMF) and CUR decomposition. We apply these new methods to data from Neuroscience applications in order to obtain insights into the functionality of the brain. Finally, we consider the application of material informatics, learning from materials data. Here, we deploy regression techniques on materials data to predict physical properties of materials