리만 최적화와 그래프 신경망에 기반한 저 랭크 행렬완성 알고리듬에 관한 연구

학위논문(박사)--서울대학교 대학원 :공과대학 전기·정보공학부,2020. 2. 심병효.최근, 일부의 관측치로부터 행렬의 모든 원소들을 복원하는 방법으로 저 랭크 행렬 완성 (LRMC)이 많은 주목을 받고 있다. LRMC는 추천 시스템, 위상 복원, 사물 인터넷 지역화, 영상 잡음 제거, 밀리미터 웨이브 통 신등을 포함한 다양한 응용분야에서 사용되고 있다. 본 논문에서는 LRMC에 대해 연구하여 LRMC의 가능성과 한계에 대한 더 나은 이해를 할 수 있도록 기존 결과들을 구조적이고 접근 가능한 방식으로 분류한다. 구체적으로, 최신 LRMC 기법들을 두 가지 범주로 분류한 다음 각각 의범주를 분석한다. 특히, 행렬의 고유한 성질과 같은 LRMC 기법을 사용 할때 고려해야 할 사항들을 분석한다. 기존의 LRMC 기법은 가우시안 랜 덤행렬과 같은 일반적인 상황에서 성공적이었으나 많은 실제 상황에서 는복원하고자 하는 저 랭크 행렬이 그래프 구조 또는 다양체 구조와 같은 비유클리드 구조를 가질 수 있다. 본 논문에서는 실제 응용에서 LRMC의 성능을 향상시키기 위해 이 런추가적인 구조가 활용될 수 있음을 보인다. 특히, 사물 인터넷 네트워 크지역화를 위한 유클리드 거리 행렬 완성 알고리듬을 제안한다. 유클리 드거리 행렬을 낮은 랭크를 갖는 양의 준정부호 행렬의 함수로 표현한다. 이러한 양의 준정부호 행렬들의 집합은 미분이 잘 정의되어 있는 리 만다양체를 형성하므로 유클리드 공간에서의 알고리듬을 적당히 변형하 여LRMC에 사용할 수 있다. LRMC를 위해 우리는 켤레 기울기를 활용 한리만 다양체에서의 지역화 (LRM-CG)라 불리는 변경된 켤레 기울기 기 반알고리듬을 제안한다. 제안하는 LRM-CG 알고리듬은 관측된 쌍 거리 가특이값에 의해 오염되는 시나리오로 쉽게 확장 될 수 있음을 보인다. 실제로 특이값을 희소 행렬로 모델링 한 다음 특이값 행렬을 규제 항으 로LRMC에 추가함으로써 특이값을 효과적으로 제어 할 수 있다. 분석을 통 해LRM-CG 알고리듬이 확장된 Wolfe 조건 아래 원래 유클리드 거리 행렬 에선형적으로 수렴하는 것을 보인다. 모의 실험을 통해 LRM-CG와 확 장버전이 유클리드 거리 행렬을 복구하는 데 효과적임을 보인다. 또한, 그래프 모델을 사용하여 표현될 수 있는 저 랭크 행렬 복원을 위 한그래프 신경망 (GNN) 기반 기법을 제안한다. 그래프 신경망 기반의 LRM C(GNN-LRMC)라 불리는 기법은 복원하고자 하는 행렬의 그래프 영 역특징들을 추출하기 위해 변형된 합성곱 연산을 사용한다. 이렇게 추출 된특징들을 GNN의 학습 과정에 활용하여 행렬의 원소들을 복원할 수 있다. 합성 및 실제 데이터를 사용한 모의 실험을 통하여 제안하는 GNN -LRMC의 우수한 복구 성능을 보였다.In recent years, low-rank matrix completion (LRMC) has received much attention as a paradigm to recover the unknown entries of a matrix from partial observations. It has a wide range of applications in many areas, including recommendation system, phase retrieval, IoT localization, image denoising, milimeter wave (mmWave) communication, to name just a few. In this dissertation, we present a comprehensive overview of low-rank matrix completion. In order to have better view, insight, and understanding of potentials and limitations of LRMC, we present early scattered results in a structured and accessible way. To be specific, we classify the state-of-the-art LRMC techniques into two main categories and then explain each category in detail. We further discuss issues to be considered, including intrinsic properties required for the matrix recovery, when one would like to use LRMC techniques. However, conventional LRMC techniques have been most successful on a general setting of the low-rank matrix, say, Gaussian random matrix. In many practical situations, the desired low rank matrix might have an underlying non-Euclidean structure, such as graph or manifold structure. In our work, we show that such additional data structures can be exploited to improve the recovery performance of LRMC in real-life applications. In particular, we propose a Euclidean distance matrix completion algorithm for internet of things (IoT) network localization. In our approach, we express the Euclidean distance matrix as a function of the low rank positive semidefinite (PSD) matrix. Since the set of these PSD matrices forms a Riemannian manifold in which the notation of differentiability can be defined, we can recycle, after a proper modification, an algorithm in the Euclidean space. In order to solve the low-rank matrix completion, we propose a modified conjugate gradient algorithm, referred to as localization in Riemannian manifold using conjugate gradient (LRM-CG). We also show that the proposed LRM-CG algorithm can be easily extended to the scenario in which the observed pairwise distances are contaminated by the outliers. In fact, by modeling outliers as a sparse matrix and then adding a regularization term of the outlier matrix into the low-rank matrix completion problem, we can effectively control the outliers. From the convergence analysis, we show that LRM-CG converges linearly to the original Euclidean distance matrix under the extended Wolfes conditions. From the numerical experiments, we demonstrate that LRM-CG as well as its extended version is effective in recovering the Euclidean distance matrix. In order to solve the LRMC problem in which the desired low-rank matrix can be expressed using a graph model, we also propose a graph neural network (GNN) scheme. Our approach, referred to as graph neural network-based low-rank matrix completion (GNN-LRMC), is to use a modified convolution operation to extract the features across the graph domain. The feature data enable the training process of the proposed GNN to reconstruct the unknown entries and also optimize the graph model of the desired low-rank matrix. We demonstrate the reconstruction performance of the proposed GNN-LRMC using synthetic and real-life datasets.Abstract i Contents iii List of Tables vii List of Figures viii 1 Introduction 2 1.1 Motivation 2 1.2 Outline of the dissertation 5 2 Low-Rank Matrix Completion 6 2.1 LRMC Applications 6 2.1.1 Recommendation system 6 2.1.2 Phase retrieval 8 2.1.3 Localization in IoT networks 8 2.1.4 Image compression and restoration 10 2.1.5 Massive multiple-input multiple-output (MIMO) 12 2.1.6 Millimeter wave (mmWave) communication 12 2.2 Intrinsic Properties of LRMC 13 2.2.1 Sparsity of Observed Entries 13 2.2.2 Coherence 18 2.3 Rank Minimization Problem 22 2.4 LRMC Algorithms Without the Rank Information 25 2.4.1 Nuclear Norm Minimization (NNM) 25 2.4.2 Singular Value Thresholding (SVT) 28 2.4.3 Iteratively Reweighted Least Squares (IRLS) Minimization 31 2.5 LRMC Algorithms Using Rank Information 32 2.5.1 Greedy Techniques 34 2.5.2 Alternating Minimization Techniques 37 2.5.3 Optimization over Smooth Riemannian Manifold 39 2.5.4 Truncated NNM 41 2.6 Performance Guarantee 44 2.7 Empirical Performance Evaluation 46 2.8 Choosing the Right Matrix Completion Algorithms 55 3 IoT Localization Via LRMC 56 3.1 Problem Model 57 3.2 Optimization over Riemannian Manifold 61 3.3 Localization in Riemannian Manifold Using Conjugate Gradient (LRMCG) 66 3.4 Computational Complexity 71 3.5 Recovery Condition Analysis 73 3.5.1 Convergence of LRM-CG at Sampled Entries 73 3.5.2 Exact Recovery of Euclidean Distance Matrices 79 3.5.3 Discussion on A3 86 4 Extended LRM-CG for The Outlier Problem 92 4.1 Problem Model 94 4.2 Extended LRM-CG 94 4.3 Numerical Evaluation 97 4.3.1 Simulation Setting 98 4.3.2 Convergence Efficiency 99 4.3.3 Performance Evaluation 99 4.3.4 Outlier Problem 107 4.3.5 Real Data 107 5 LRMC Via Graph Neural Network 112 5.1 Graph Model 116 5.2 Proposed GNN-LRMC 116 5.2.1 Adaptive Model 119 5.2.2 Multilayer GNN 119 5.2.3 Output Model 122 5.2.4 Training Cost Function 123 5.3 Numerical Evaluation 123 6 Conculsion 127 A Proof of Lemma 6 129 B Proof of Theorem 7 131 C Proof of Lemma 8 134 D Proof of Theorem 9 136 E Proof of Lemma 10 140 F Proof of Lemma 12 141 G Proof of Lemma 13 142 H Proof of Lemma 14 144 I Proof of Lemma 15 146 J Proof of Lemma 17 151 K Proof of Lemma 19 154 L Proof of Lemma 20 156 M Proof of Lemma 21 158 Abstract (In Korean) 173 Acknowlegement 175Docto

Higher-Rank Numerical Ranges and Compression Problems

We consider higher-rank versions of the standard numerical range for matrices. A central motivation for this investigation comes from quantum error correction. We develop the basic structure theory for the higher-rank numerical ranges, and give a complete description in the Hermitian case. We also consider associated projection compression problems.Comment: 14 pages, 3 figures, to appear in Linear Algebra and its Application

Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions

Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets. This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed—either explicitly or implicitly—to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, robustness, and/or speed. These claims are supported by extensive numerical experiments and a detailed error analysis. The specific benefits of randomized techniques depend on the computational environment. Consider the model problem of finding the k dominant components of the singular value decomposition of an m × n matrix. (i) For a dense input matrix, randomized algorithms require O(mn log(k)) floating-point operations (flops) in contrast to O(mnk) for classical algorithms. (ii) For a sparse input matrix, the flop count matches classical Krylov subspace methods, but the randomized approach is more robust and can easily be reorganized to exploit multiprocessor architectures. (iii) For a matrix that is too large to fit in fast memory, the randomized techniques require only a constant number of passes over the data, as opposed to O(k) passes for classical algorithms. In fact, it is sometimes possible to perform matrix approximation with a single pass over the data

Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions

Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets. This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed---either explicitly or implicitly---to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, speed, and robustness. These claims are supported by extensive numerical experiments and a detailed error analysis

Higher rank numerical ranges and low rank perturbations of quantum channels

For a positive integer $k$, the rank-$k$ numerical range $\Lambda_k(A)$ of an operator $A$ acting on a Hilbert space \cH of dimension at least $k$ is the set of scalars $\lambda$ such that $PAP = \lambda P$ for some rank $k$ orthogonal projection $P$. In this paper, a close connection between low rank perturbation of an operator $A$ and $\Lambda_k(A)$ is established. In particular, for $1 \le r < k$ it is shown that $\Lambda_k(A) \subseteq \Lambda_{k-r}(A+F)$ for any operator $F$ with \rank (F) \le r. In quantum computing, this result implies that a quantum channel with a $k$-dimensional error correcting code under a perturbation of rank $\le r$ will still have a $(k-r)$-dimensional error correcting code. Moreover, it is shown that if $A$ is normal or if the dimension of $A$ is finite, then $\Lambda_k(A)$ can be obtained as the intersection of $\Lambda_{k-r}(A+F)$ for a collection of rank $r$ operators $F$. Examples are given to show that the result fails if $A$ is a general operator. The closure and the interior of the convex set $\Lambda_k(A)$ are completely determined. Analogous results are obtained for $\Lambda_\infty(A)$ defined as the set of scalars $\lambda$ such that $PAP = \lambda P$ for an infinite rank orthogonal projection $P$. It is shown that $\Lambda_\infty(A)$ is the intersection of all $\Lambda_k(A)$ for $k = 1, 2, >...$. If $A - \mu I$ is not compact for any \mu \in \IC, then the closure and the interior of $\Lambda_\infty(A)$ coincide with those of the essential numerical range of $A$. The situation for the special case when $A-\mu I$ is compact for some \mu \in \IC is also studied.Comment: 21 page

Canonical forms, higher rank numerical range, convexity, totally isotropic subspace, matrix equations

Results on matrix canonical forms are used to give a complete description of the higher rank numerical range of matrices arising from the study of quantum error correction. It is shown that the set can be obtained as the intersection of closed half planes (of complex numbers). As a result, it is always a convex set in $\mathcal C$. Moreover, the higher rank numerical range of a normal matrix is a convex polygon determined by the eigenvalues. These two consequences confirm the conjectures of Choi et al. on the subject. In addition, the results are used to derive a formula for the optimal upper bound for the dimension of a totally isotropic subspace of a square matrix, and verify the solvability of certain matrix equations.Comment: 10 pages. To appear in Proceedings of the American Mathematical Societ