Learning Combinatorial Embedding Networks for Deep Graph Matching

Graph matching refers to finding node correspondence between graphs, such that the corresponding node and edge's affinity can be maximized. In addition with its NP-completeness nature, another important challenge is effective modeling of the node-wise and structure-wise affinity across graphs and the resulting objective, to guide the matching procedure effectively finding the true matching against noises. To this end, this paper devises an end-to-end differentiable deep network pipeline to learn the affinity for graph matching. It involves a supervised permutation loss regarding with node correspondence to capture the combinatorial nature for graph matching. Meanwhile deep graph embedding models are adopted to parameterize both intra-graph and cross-graph affinity functions, instead of the traditional shallow and simple parametric forms e.g. a Gaussian kernel. The embedding can also effectively capture the higher-order structure beyond second-order edges. The permutation loss model is agnostic to the number of nodes, and the embedding model is shared among nodes such that the network allows for varying numbers of nodes in graphs for training and inference. Moreover, our network is class-agnostic with some generalization capability across different categories. All these features are welcomed for real-world applications. Experiments show its superiority against state-of-the-art graph matching learning methods.Comment: ICCV2019 oral. Code available at https://github.com/Thinklab-SJTU/PCA-G

컴퓨터비전을 위한 그래프정합과 고차그래프정합: 새로운 알고리즘과 분석에 관한 연구

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2016. 8. 이경무.Establishing image feature correspondences is fundamental problem in computer vision and machine learning research fields. Myriad of graph matching algorithms have been proposed to tackle this problem by regarding correspondence problem as a graph matching problem. However, the graph matching problem is challenging since there are various types of noises in real world scenarioe.g., non-rigid motion, view-point change, and background clutter. The objective of this dissertation is to propose robust graph matching algorithms for feature correspondence task in computer vision and to investigate an effective graph matching strategy. For the purpose, at first, two robust simulation based graph matching algorithms are introduced: the one is based on Random Walks simulation and the other is based on Markov Chain Monte Carlo sampling simulation. Secondly, two different graph matching formulations and their transformal relation are studied since equivalence between two formulations are not well studied in graph matching fields. It is demonstrated that conventional graph matching algorithms can solve both types of formulations by proposing conversion principle between two formulations. Finally, these whole statements are extended into hypergraph matching problem by introducing two robust hypergraph matching algorithms which are based on Random Walks and Markov Chain Monte Carlo, by relating two different hypergraph matching formulations, and by reinterpreting previous hypergraph matching algorithms into their counterpart formulations. Throughout chapters in this dissertation, comparative and extensive experiments verify characteristics of formulations, transformal relations, and algorithms. Synthetic graph matching problems as well as real image feature correspondence problems are performed in various and severe noise conditions.Chapter 1 Introduction 1 1.1 Graph Matching Problem 1 1.1.1 Graph Matching for Computer Vision 1 1.1.2 Graph Matching Formulation 2 1.1.3 Extension to Hypergraph Matching 5 1.2 Outline of Dissertation 6 Chapter 2 Graph Matching via Random Walks 9 2.1 Introduction 9 2.1.1 Related Works 10 2.2 Problem Formulation 12 2.2.1 Graph Matching Formulation 12 2.2.2 Hypergraphs Matching Formulation 13 2.3 Graph Matching via Random Walks 16 2.3.1 Random Walks for Graph Matching 16 2.3.2 Reweighting Jumps for Graph Matching 19 2.4 Hypergraph Matching via Random Walks 22 2.4.1 Hypergraph Random Walks 22 2.4.2 Reweighting Jumps for Hypergraph Matching 23 2.5 Experiments 26 2.5.1 Random Graph Matching 27 2.5.2 Synthetic Point Matching 34 2.5.3 Image Sequence Matching 37 2.5.4 Image Feature Matching 39 2.6 Conclusion 44 Chapter 3 Graph Matching via Markov Chain Monte Carlo 45 3.1 Introduction 45 3.2 Graph Matching Formulation 47 3.3 Algorithm 49 3.3.1 State Transition 49 3.3.2 Energy Formulation 49 3.3.3 Data-Driven Proposal 51 3.4 Hypergraph Extension 53 3.4.1 Hypergraph Matching Problem 53 3.4.2 Energy Formulation & Data-Driven Proposal 54 3.5 Experiment 54 3.5.1 Random Graph Matching Problem 54 3.5.2 Random Hypergraph Matching Problem 58 3.6 Conclusion 59 Chapter 4 Graph and Hypergraph Matching Revisited 63 4.1 Introduction 63 4.2 Related Works 65 4.3 Two Types of Formulations 66 4.3.1 Adjacency-based Formulation 67 4.3.2 Affinity-based Formulation 69 4.3.3 Relation between Two Formulations 70 4.4 Affinity Measures 72 4.5 Existing Methods & Re-interpretations 74 4.5.1 Spectral Matching 74 4.5.2 Integer Projected Fixed Point 75 4.5.3 Reweighted Random Walks Matching 76 4.5.4 Factorized Graph Matching 77 4.6 High-order Methods & Reinterpretations 78 4.6.1 Hypergraph Matching by Zass and Shashua 81 4.6.2 SVD-based Hypergraph Matching 82 4.6.3 Tensor Power Iteration based Hypergraph Matching 82 4.6.4 Reweighted Random Walks for Hypergraph Matching 83 4.6.5 Discrete Hypergraph Matching 85 4.7 Experiments & Comparison 85 4.8 Conclusion 102 Chapter 5 Conclusion 105 5.1 Summary and Contribution of Dissertation 105 5.2 Future Works 107 Bibliography 109 국문 초록 117Docto

Adaptively Transforming Graph Matching

Recently, many graph matching methods that incorporate pairwise constraint and that can be formulated as a quadratic assignment problem (QAP) have been proposed. Although these methods demonstrate promising results for the graph matching problem, they have high complexity in space or time. In this paper, we introduce an adaptively transforming graph matching (ATGM) method from the perspective of functional representation. More precisely, under a transformation formulation, we aim to match two graphs by minimizing the discrepancy between the original graph and the transformed graph. With a linear representation map of the transformation, the pairwise edge attributes of graphs are explicitly represented by unary node attributes, which enables us to reduce the space and time complexity significantly. Due to an efficient Frank-Wolfe method-based optimization strategy, we can handle graphs with hundreds and thousands of nodes within an acceptable amount of time. Meanwhile, because transformation map can preserve graph structures, a domain adaptation-based strategy is proposed to remove the outliers. The experimental results demonstrate that our proposed method outperforms the state-of-the-art graph matching algorithms

Fast multi-image matching via density-based clustering

We consider the problem of finding consistent matches across multiple images. Previous state-of-the-art solutions use constraints on cycles of matches together with convex optimization, leading to computationally intensive iterative algorithms. In this paper, we propose a clustering-based formulation. We first rigorously show its equivalence with the previous one, and then propose QuickMatch, a novel algorithm that identifies multi-image matches from a density function in feature space. We use the density to order the points in a tree, and then extract the matches by breaking this tree using feature distances and measures of distinctiveness. Our algorithm outperforms previous state-of-the-art methods (such as MatchALS) in accuracy, and it is significantly faster (up to 62 times faster on some bechmarks), and can scale to large datasets (with more than twenty thousands features).Accepted manuscriptSupporting documentatio

Virtual Line Descriptor and Semi-Local Matching Method for Reliable Feature Correspondence

International audienceFinding reliable correspondences between sets of feature points in two images remains challenging in case of ambiguities or strong transformations. In this paper, we define a photometric descriptor for virtual lines that join neighbouring feature points. We show that it can be used in the second-order term of existing graph matchers to significantly improve their accuracy. We also define a semi-local matching method based on this descriptor. We show that it is robust to strong transformations and more accurate than existing graph matchers for scenes with significant occlusions, including for very low inlier rates. Used as a preprocessor to filter outliers from match candidates, it significantly improves the robustness of RANSAC and reduces camera calibration errors