75 research outputs found
LIPIcs, Volume 251, ITCS 2023, Complete Volume
LIPIcs, Volume 251, ITCS 2023, Complete Volum
Recommended from our members
Foundations of Node Representation Learning
Low-dimensional node representations, also called node embeddings, are a cornerstone in the modeling and analysis of complex networks. In recent years, advances in deep learning have spurred development of novel neural network-inspired methods for learning node representations which have largely surpassed classical \u27spectral\u27 embeddings in performance. Yet little work asks the central questions of this thesis: Why do these novel deep methods outperform their classical predecessors, and what are their limitations?
We pursue several paths to answering these questions. To further our understanding of deep embedding methods, we explore their relationship with spectral methods, which are better understood, and show that some popular deep methods are equivalent to spectral methods in a certain natural limit. We also introduce the problem of inverting node embeddings in order to probe what information they contain. Further, we propose a simple, non-deep method for node representation learning, and find it to often be competitive with modern deep graph networks in downstream performance.
To better understand the limitations of node embeddings, we prove some upper and lower bounds on their capabilities. Most notably, we prove that node embeddings are capable of exact low-dimensional representation of networks with bounded max degree or arboricity, and we further show that a simple algorithm can find such exact embeddings for real-world networks. By contrast, we also prove inherent bounds on random graph models, including those derived from node embeddings, to capture key structural properties of networks without simply memorizing a given graph
Geometric Learning on Graph Structured Data
Graphs provide a ubiquitous and universal data structure that can be applied in many domains such as social networks, biology, chemistry, physics, and computer science. In this thesis we focus on two fundamental paradigms in graph learning: representation learning and similarity learning over graph-structured data. Graph representation learning aims to learn embeddings for nodes by integrating topological and feature information of a graph. Graph similarity learning brings into play with similarity functions that allow to compute similarity between pairs of graphs in a vector space. We address several challenging issues in these two paradigms, designing powerful, yet efficient and theoretical guaranteed machine learning models that can leverage rich topological structural properties of real-world graphs.
This thesis is structured into two parts. In the first part of the thesis, we will present how to develop powerful Graph Neural Networks (GNNs) for graph representation learning from three different perspectives: (1) spatial GNNs, (2) spectral GNNs, and (3) diffusion GNNs. We will discuss the model architecture, representational power, and convergence properties of these GNN models. Specifically, we first study how to develop expressive, yet efficient and simple message-passing aggregation schemes that can go beyond the Weisfeiler-Leman test (1-WL). We propose a generalized message-passing framework by incorporating graph structural properties into an aggregation scheme. Then, we introduce a new local isomorphism hierarchy on neighborhood subgraphs. We further develop a novel neural model, namely GraphSNN, and theoretically prove that this model is more expressive than the 1-WL test. After that, we study how to build an effective and efficient graph convolution model with spectral graph filters. In this study, we propose a spectral GNN model, called DFNets, which incorporates a novel spectral graph filter, namely feedback-looped filters. As a result, this model can provide better localization on neighborhood while achieving fast convergence and linear memory requirements. Finally, we study how to capture the rich topological information of a graph using graph diffusion. We propose a novel GNN architecture with dynamic PageRank, based on a learnable transition matrix. We explore two variants of this GNN architecture: forward-euler solution and invariable feature solution, and theoretically prove that our forward-euler GNN architecture is guaranteed with the convergence to a stationary distribution.
In the second part of this thesis, we will introduce a new optimal transport distance metric on graphs in a regularized learning framework for graph kernels. This optimal transport distance metric can preserve both local and global structures between graphs during the transport, in addition to preserving features and their local variations. Furthermore, we propose two strongly convex regularization terms to theoretically guarantee the convergence and numerical stability in finding an optimal assignment between graphs. One regularization term is used to regularize a Wasserstein distance between graphs in the same ground space. This helps to preserve the local clustering structure on graphs by relaxing the optimal transport problem to be a cluster-to-cluster assignment between locally connected vertices. The other regularization term is used to regularize a Gromov-Wasserstein distance between graphs across different ground spaces based on degree-entropy KL divergence. This helps to improve the matching robustness of an optimal alignment to preserve the global connectivity structure of graphs. We have evaluated our optimal transport-based graph kernel using different benchmark tasks. The experimental results show that our models considerably outperform all the state-of-the-art methods in all benchmark tasks
Revisiting Adversarial Attacks on Graph Neural Networks for Graph Classification
Graph neural networks (GNNs) have achieved tremendous success in the task of
graph classification and its diverse downstream real-world applications.
Despite the huge success in learning graph representations, current GNN models
have demonstrated their vulnerability to potentially existent adversarial
examples on graph-structured data. Existing approaches are either limited to
structure attacks or restricted to local information, urging for the design of
a more general attack framework on graph classification, which faces
significant challenges due to the complexity of generating local-node-level
adversarial examples using the global-graph-level information. To address this
"global-to-local" attack challenge, we present a novel and general framework to
generate adversarial examples via manipulating graph structure and node
features. Specifically, we make use of Graph Class Activation Mapping and its
variant to produce node-level importance corresponding to the graph
classification task. Then through a heuristic design of algorithms, we can
perform both feature and structure attacks under unnoticeable perturbation
budgets with the help of both node-level and subgraph-level importance.
Experiments towards attacking four state-of-the-art graph classification models
on six real-world benchmarks verify the flexibility and effectiveness of our
framework.Comment: 13 pages, 7 figure
LIPIcs, Volume 261, ICALP 2023, Complete Volume
LIPIcs, Volume 261, ICALP 2023, Complete Volum
Towards Long-Tailed Recognition for Graph Classification via Collaborative Experts
Graph classification, aiming at learning the graph-level representations for
effective class assignments, has received outstanding achievements, which
heavily relies on high-quality datasets that have balanced class distribution.
In fact, most real-world graph data naturally presents a long-tailed form,
where the head classes occupy much more samples than the tail classes, it thus
is essential to study the graph-level classification over long-tailed data
while still remaining largely unexplored. However, most existing long-tailed
learning methods in visions fail to jointly optimize the representation
learning and classifier training, as well as neglect the mining of the
hard-to-classify classes. Directly applying existing methods to graphs may lead
to sub-optimal performance, since the model trained on graphs would be more
sensitive to the long-tailed distribution due to the complex topological
characteristics. Hence, in this paper, we propose a novel long-tailed
graph-level classification framework via Collaborative Multi-expert Learning
(CoMe) to tackle the problem. To equilibrate the contributions of head and tail
classes, we first develop balanced contrastive learning from the view of
representation learning, and then design an individual-expert classifier
training based on hard class mining. In addition, we execute gated fusion and
disentangled knowledge distillation among the multiple experts to promote the
collaboration in a multi-expert framework. Comprehensive experiments are
performed on seven widely-used benchmark datasets to demonstrate the
superiority of our method CoMe over state-of-the-art baselines.Comment: Accepted by IEEE Transactions on Big Data (TBD 2024
Label Deconvolution for Node Representation Learning on Large-scale Attributed Graphs against Learning Bias
Node representation learning on attributed graphs -- whose nodes are
associated with rich attributes (e.g., texts and protein sequences) -- plays a
crucial role in many important downstream tasks. To encode the attributes and
graph structures simultaneously, recent studies integrate pre-trained models
with graph neural networks (GNNs), where pre-trained models serve as node
encoders (NEs) to encode the attributes. As jointly training large NEs and GNNs
on large-scale graphs suffers from severe scalability issues, many methods
propose to train NEs and GNNs separately. Consequently, they do not take
feature convolutions in GNNs into consideration in the training phase of NEs,
leading to a significant learning bias from that by the joint training. To
address this challenge, we propose an efficient label regularization technique,
namely Label Deconvolution (LD), to alleviate the learning bias by a novel and
highly scalable approximation to the inverse mapping of GNNs. The inverse
mapping leads to an objective function that is equivalent to that by the joint
training, while it can effectively incorporate GNNs in the training phase of
NEs against the learning bias. More importantly, we show that LD converges to
the optimal objective function values by thejoint training under mild
assumptions. Experiments demonstrate LD significantly outperforms
state-of-the-art methods on Open Graph Benchmark datasets
Multimedia Big Data Analytics and Fusion for Data Science
Title from PDF of title page, viewed May 24, 2023Dissertation advisor: Shu-Ching ChenVitaIncludes bibliographical references (pages 178-212)Dissertation (Ph.D.)--Department of Computer Science and Electrical Engineering. University of Missouri--Kansas City, 2023Big data is becoming increasingly prevalent in people's everyday lives due to the enormous quantity of data generated from social and economic activities worldwide. As a result, extensive research has been undertaken to support the big data revolution. However, as data grows in volume, traditional data analytic methods face various challenges—especially when raw data comes in multiple forms and formats. This dissertation proposes a multimodal big data analytics and fusion framework that addresses several challenges in data science for handling and learning from multimodal big data.
The proposed framework addresses issues during a standard data science project workflow, including data fusion, spatio-temporal deep feature extraction, and model training optimization strategy. First, a hierarchical graph fusion network is presented to capture the inter-modality correlations among modalities. The network hierarchy models the modality-wise combinations with gradually increased complexity to explore all n-modality interactions. Next, an adaptive spatio-temporal graph network is proposed to capture the hidden patterns from spatio-temporal data. It exploits local and global node correlations by improving the pre-defined graph Laplacian and automatically generates the graph adjacency matrix based on a data-driven method. In addition, a dynamic multi-task learning method is introduced to optimize the model training progress by dynamically adjusting the loss weights assigned to each task. It systematically monitors the sample-level prediction errors, task-level weight parameter changing rate, and iteration-level total loss to adjust the weight balance among tasks. The proposed framework has been evaluated on various datasets, including disaster event videos, social media, traffic flow, and other public datasets.Introduction -- Related work -- Overview of the framework -- Dynamic multi-task learning -- Hierarchical graph fusion -- Spatio-temporal graph network -- Conclusions and future wor
Knowledge extraction from unstructured data
Data availability is becoming more essential, considering the current growth of web-based data. The data available on the web are represented as unstructured, semi-structured, or structured data. In order to make the web-based data available for several Natural Language Processing or Data Mining tasks, the data needs to be presented as machine-readable data in a structured format. Thus, techniques for addressing the problem of capturing knowledge from unstructured data sources are needed. Knowledge extraction methods are used by the research communities to address this problem; methods that are able to capture knowledge in a natural language text and map the extracted knowledge to existing knowledge presented in knowledge graphs (KGs). These knowledge extraction methods include Named-entity recognition, Named-entity Disambiguation, Relation Recognition, and Relation Linking. This thesis addresses the problem of extracting knowledge over unstructured data and discovering patterns in the extracted knowledge. We devise a rule-based approach for entity and relation recognition and linking. The defined approach effectively maps entities and relations within a text to their resources in a target KG. Additionally, it overcomes the challenges of recognizing and linking entities and relations to a specific KG by employing devised catalogs of linguistic and domain-specific rules that state the criteria to recognize entities in a sentence of a particular language, and a deductive database that encodes knowledge in community-maintained KGs. Moreover, we define a Neuro-symbolic approach for the tasks of knowledge extraction in encyclopedic and domain-specific domains; it combines symbolic and sub-symbolic components to overcome the challenges of entity recognition and linking and the limitation of the availability of training data while maintaining the accuracy of recognizing and linking entities. Additionally, we present a context-aware framework for unveiling semantically related posts in a corpus; it is a knowledge-driven framework that retrieves associated posts effectively. We cast the problem of unveiling semantically related posts in a corpus into the Vertex Coloring Problem. We evaluate the performance of our techniques on several benchmarks related to various domains for knowledge extraction tasks. Furthermore, we apply these methods in real-world scenarios from national and international projects. The outcomes show that our techniques are able to effectively extract knowledge encoded in unstructured data and discover patterns over the extracted knowledge presented as machine-readable data. More importantly, the evaluation results provide evidence to the effectiveness of combining the reasoning capacity of the symbolic frameworks with the power of pattern recognition and classification of sub-symbolic models
A Survey on Explainability of Graph Neural Networks
Graph neural networks (GNNs) are powerful graph-based deep-learning models
that have gained significant attention and demonstrated remarkable performance
in various domains, including natural language processing, drug discovery, and
recommendation systems. However, combining feature information and
combinatorial graph structures has led to complex non-linear GNN models.
Consequently, this has increased the challenges of understanding the workings
of GNNs and the underlying reasons behind their predictions. To address this,
numerous explainability methods have been proposed to shed light on the inner
mechanism of the GNNs. Explainable GNNs improve their security and enhance
trust in their recommendations. This survey aims to provide a comprehensive
overview of the existing explainability techniques for GNNs. We create a novel
taxonomy and hierarchy to categorize these methods based on their objective and
methodology. We also discuss the strengths, limitations, and application
scenarios of each category. Furthermore, we highlight the key evaluation
metrics and datasets commonly used to assess the explainability of GNNs. This
survey aims to assist researchers and practitioners in understanding the
existing landscape of explainability methods, identifying gaps, and fostering
further advancements in interpretable graph-based machine learning.Comment: submitted to Bulletin of the IEEE Computer Society Technical
Committee on Data Engineerin
- …