Abnormality detection using graph matching for multi-task dynamics of autonomous systems

Self-learning abilities in autonomous systems are essential to improve their situational awareness and detection of normal/abnormal situations. In this work, we propose a graph matching technique for activity detection in autonomous agents by using the Gromov-Wasserstein framework. A clustering approach is used to discretise continuous agents' states related to a specific task into a set of nodes with similar objectives. Additionally, a probabilistic transition matrix between nodes is used as edges weights to build a graph. In this paper, we extract an abnormal area based on a sub-graph that encodes the differences between coupled of activities. Such sub-graph is obtained by applying a threshold on the optimal transport matrix, which is obtained through the graph matching procedure. The obtained results are evaluated through experiments performed by a robot in a simulated environment and by a real autonomous vehicle moving within a University Campus

Robust Point Cloud Registration Framework Based on Deep Graph Matching(TPAMI Version)

3D point cloud registration is a fundamental problem in computer vision and robotics. Recently, learning-based point cloud registration methods have made great progress. However, these methods are sensitive to outliers, which lead to more incorrect correspondences. In this paper, we propose a novel deep graph matching-based framework for point cloud registration. Specifically, we first transform point clouds into graphs and extract deep features for each point. Then, we develop a module based on deep graph matching to calculate a soft correspondence matrix. By using graph matching, not only the local geometry of each point but also its structure and topology in a larger range are considered in establishing correspondences, so that more correct correspondences are found. We train the network with a loss directly defined on the correspondences, and in the test stage the soft correspondences are transformed into hard one-to-one correspondences so that registration can be performed by a correspondence-based solver. Furthermore, we introduce a transformer-based method to generate edges for graph construction, which further improves the quality of the correspondences. Extensive experiments on object-level and scene-level benchmark datasets show that the proposed method achieves state-of-the-art performance. The code is available at: \href{https://github.com/fukexue/RGM}{https://github.com/fukexue/RGM}.Comment: accepted by TPAMI 2022. arXiv admin note: substantial text overlap with arXiv:2103.0425

Learning Cross-modal Context Graph for Visual Grounding

Visual grounding is a ubiquitous building block in many vision-language tasks and yet remains challenging due to large variations in visual and linguistic features of grounding entities, strong context effect and the resulting semantic ambiguities. Prior works typically focus on learning representations of individual phrases with limited context information. To address their limitations, this paper proposes a language-guided graph representation to capture the global context of grounding entities and their relations, and develop a cross-modal graph matching strategy for the multiple-phrase visual grounding task. In particular, we introduce a modular graph neural network to compute context-aware representations of phrases and object proposals respectively via message propagation, followed by a graph-based matching module to generate globally consistent localization of grounding phrases. We train the entire graph neural network jointly in a two-stage strategy and evaluate it on the Flickr30K Entities benchmark. Extensive experiments show that our method outperforms the prior state of the arts by a sizable margin, evidencing the efficacy of our grounding framework. Code is available at "https://github.com/youngfly11/LCMCG-PyTorch".Comment: AAAI-202

Graph Matching via Sequential Monte Carlo

International audienceGraph matching is a powerful tool for computer vision and machine learning. In this paper, a novel approach to graph matching is developed based on the sequential Monte Carlo framework. By constructing a sequence of intermediate target distributions, the proposed algorithm sequentially performs a sampling and importance resampling to maximize the graph matching objective. Through the sequential sampling procedure, the algorithm effectively collects potential matches under one-to-one matching constraints to avoid the adverse effect of outliers and deformation. Experimental evaluations on synthetic graphs and real images demonstrate its higher robustness to deformation and outliers

Designing Machine Learning Models for Graph Analytics

University of Technology Sydney. Faculty of Engineering and Information Technology.With growing popularity of the machine learning methods, there have been a great number of machine learning methods proposed for graph analytics. In this thesis, we design three machine learning based models for the popular graph analysis tasks such as node classification, graph interaction prediction and subgraph matching. Firstly, we design a binarized graph neural network to efficiently obtain the vector representations for vertices and graphs. Recently, there have been some breakthroughs in graph analysis by applying the Graph Neural Networks (GNNs). However, the parameters of the network and the embedding of nodes are represented in real-valued matrices in existing GNN-based approaches which may limit the efficiency and scalability of these models. This motivates us to develop a binarized graph neural network to learn the binary representations of the nodes with binary network parameters following the GNN-based paradigm. Our proposed method can be seamlessly integrated into the existing GNN-based embedding approaches to binarize the model parameters and learn the compact embedding. Secondly, we design a graph of graphs neural network for entity interaction prediction, and then extend the model to support the graph classification task with more expressive representations. Entity interaction prediction is essential in many important applications, which can be quite challenging when there are two types of graphs are involved: local graphs for structured entities and a global graph for the interactions between structured entities. We observe that existing works cannot properly exploit the unique graph of graphs structure. In this thesis, we propose a Graph of Graphs Neural Network, namely GoGNN, which extracts the features of the given graph in a hierarchical way. Based on GoGNN, we further propose a Powerful Graph Of graphs neural Network, namely PGON, which has 3-Weisfeiler-Lehman expressive power and can be used to handle the graph classification task. Thirdly, we design a reinforcement learning based query vertex ordering model for subgraph matching. Subgraph matching is a fundamental problem in graph analytics. Instead generating the matching order with heuristics, our model could capture and make full use of the graph information, and thus determine the query vertex order with the adaptive learning-based rule that could significantly reduce the number of redundant enumerations. With the help of the reinforcement learning framework, our model could consider the long-term benefits during order generation. Extensive experiments on real-life datasets indicate the efficiency and effectiveness of our proposed models in the corresponding graph analytic tasks