19 research outputs found
Continual Graph Learning
Various real-world graphs grow with time, necessitating the research of continual graph learning (CGL), which aims to accommodate new tasks over newly emerged graph data while maintaining the model performance over existing tasks.
First, we study the CGL task configurations in different application scenarios and develop a comprehensive Continual Graph Learning Benchmark (CGLB).
CGLB contains comprehensive CGL tasks under various experimental settings, as well as a toolkit for developing CGL techniques.
Second, we developed a series of CGL techniques: 1) Hierarchical Prototype Networks (HPNs), 2) Sparsified Subgraph Memory (SSM), and 3) Subgraph Episodic Memory (SEM).
Hierarchical Prototype Networks (HPNs) is designed to extract basic shareable features and store them into prototypes. In this way, the forgetting problem can be alleviated by knowledge sharing and independently updated prototypes.
Next, SSM is a memory-replay based CGL technique, which stores a set of representative historical data from previous tasks to replay while learning new tasks. While topological information is critical in characterizing graph data, existing memory replay based CGL techniques only store individual nodes for replay and do not consider the topological information due to the memory explosion problem.
To this end, SSM is designed to sparsify the selected computation graphs into fixed size before storing them into the memory.
In this way, we can significantly reduce the memory consumption of a computation subgraph, and for the first time enable GNNs to utilize the explicit topological information for memory replay. Based on SSM, we developed the SEM, which adopts graph Ricci-curvature as the criteria during the computation subgraph sparsification.
Finally, in experiments, we study various real-world graph data including social network, citation network, product co-purchasing network, scene graph, and molecule graphs
Self-Supervised Continual Graph Learning in Adaptive Riemannian Spaces
Continual graph learning routinely finds its role in a variety of real-world
applications where the graph data with different tasks come sequentially.
Despite the success of prior works, it still faces great challenges. On the one
hand, existing methods work with the zero-curvature Euclidean space, and
largely ignore the fact that curvature varies over the coming graph sequence.
On the other hand, continual learners in the literature rely on abundant
labels, but labeling graph in practice is particularly hard especially for the
continuously emerging graphs on-the-fly. To address the aforementioned
challenges, we propose to explore a challenging yet practical problem, the
self-supervised continual graph learning in adaptive Riemannian spaces. In this
paper, we propose a novel self-supervised Riemannian Graph Continual Learner
(RieGrace). In RieGrace, we first design an Adaptive Riemannian GCN (AdaRGCN),
a unified GCN coupled with a neural curvature adapter, so that Riemannian space
is shaped by the learnt curvature adaptive to each graph. Then, we present a
Label-free Lorentz Distillation approach, in which we create teacher-student
AdaRGCN for the graph sequence. The student successively performs
intra-distillation from itself and inter-distillation from the teacher so as to
consolidate knowledge without catastrophic forgetting. In particular, we
propose a theoretically grounded Generalized Lorentz Projection for the
contrastive distillation in Riemannian space. Extensive experiments on the
benchmark datasets show the superiority of RieGrace, and additionally, we
investigate on how curvature changes over the graph sequence.Comment: Accepted by AAAI 2023 (Main Track), 9 pages, 4 figure
A Topology-aware Graph Coarsening Framework for Continual Graph Learning
Continual learning on graphs tackles the problem of training a graph neural
network (GNN) where graph data arrive in a streaming fashion and the model
tends to forget knowledge from previous tasks when updating with new data.
Traditional continual learning strategies such as Experience Replay can be
adapted to streaming graphs, however, these methods often face challenges such
as inefficiency in preserving graph topology and incapability of capturing the
correlation between old and new tasks. To address these challenges, we propose
TA, a (t)opology-(a)ware graph (co)arsening and (co)ntinual
learning framework that stores information from previous tasks as a reduced
graph. At each time period, this reduced graph expands by combining with a new
graph and aligning shared nodes, and then it undergoes a "zoom out" process by
reduction to maintain a stable size. We design a graph coarsening algorithm
based on node representation proximities to efficiently reduce a graph and
preserve topological information. We empirically demonstrate the learning
process on the reduced graph can approximate that of the original graph. Our
experiments validate the effectiveness of the proposed framework on three
real-world datasets using different backbone GNN models
Continual Learning on Dynamic Graphs via Parameter Isolation
Many real-world graph learning tasks require handling dynamic graphs where
new nodes and edges emerge. Dynamic graph learning methods commonly suffer from
the catastrophic forgetting problem, where knowledge learned for previous
graphs is overwritten by updates for new graphs. To alleviate the problem,
continual graph learning methods are proposed. However, existing continual
graph learning methods aim to learn new patterns and maintain old ones with the
same set of parameters of fixed size, and thus face a fundamental tradeoff
between both goals. In this paper, we propose Parameter Isolation GNN (PI-GNN)
for continual learning on dynamic graphs that circumvents the tradeoff via
parameter isolation and expansion. Our motivation lies in that different
parameters contribute to learning different graph patterns. Based on the idea,
we expand model parameters to continually learn emerging graph patterns.
Meanwhile, to effectively preserve knowledge for unaffected patterns, we find
parameters that correspond to them via optimization and freeze them to prevent
them from being rewritten. Experiments on eight real-world datasets corroborate
the effectiveness of PI-GNN compared to state-of-the-art baselines
Overcoming Catastrophic Forgetting in Graph Neural Networks with Experience Replay
Graph Neural Networks (GNNs) have recently received significant research
attention due to their superior performance on a variety of graph-related
learning tasks. Most of the current works focus on either static or dynamic
graph settings, addressing a single particular task, e.g., node/graph
classification, link prediction. In this work, we investigate the question: can
GNNs be applied to continuously learning a sequence of tasks? Towards that, we
explore the Continual Graph Learning (CGL) paradigm and present the Experience
Replay based framework ER-GNN for CGL to alleviate the catastrophic forgetting
problem in existing GNNs. ER-GNN stores knowledge from previous tasks as
experiences and replays them when learning new tasks to mitigate the
catastrophic forgetting issue. We propose three experience node selection
strategies: mean of feature, coverage maximization, and influence maximization,
to guide the process of selecting experience nodes. Extensive experiments on
three benchmark datasets demonstrate the effectiveness of our ER-GNN and shed
light on the incremental graph (non-Euclidean) structure learning.Comment: 9 pages, 7 figure
Graph Relation Aware Continual Learning
Continual graph learning (CGL) studies the problem of learning from an
infinite stream of graph data, consolidating historical knowledge, and
generalizing it to the future task. At once, only current graph data are
available. Although some recent attempts have been made to handle this task, we
still face two potential challenges: 1) most of existing works only manipulate
on the intermediate graph embedding and ignore intrinsic properties of graphs.
It is non-trivial to differentiate the transferred information across graphs.
2) recent attempts take a parameter-sharing policy to transfer knowledge across
time steps or progressively expand new architecture given shifted graph
distribution. Learning a single model could loss discriminative information for
each graph task while the model expansion scheme suffers from high model
complexity. In this paper, we point out that latent relations behind graph
edges can be attributed as an invariant factor for the evolving graphs and the
statistical information of latent relations evolves. Motivated by this, we
design a relation-aware adaptive model, dubbed as RAM-CG, that consists of a
relation-discovery modular to explore latent relations behind edges and a
task-awareness masking classifier to accounts for the shifted. Extensive
experiments show that RAM-CG provides significant 2.2%, 6.9% and 6.6% accuracy
improvements over the state-of-the-art results on CitationNet, OGBN-arxiv and
TWITCH dataset, respective
Towards Robust Graph Incremental Learning on Evolving Graphs
Incremental learning is a machine learning approach that involves training a
model on a sequence of tasks, rather than all tasks at once. This ability to
learn incrementally from a stream of tasks is crucial for many real-world
applications. However, incremental learning is a challenging problem on
graph-structured data, as many graph-related problems involve prediction tasks
for each individual node, known as Node-wise Graph Incremental Learning (NGIL).
This introduces non-independent and non-identically distributed characteristics
in the sample data generation process, making it difficult to maintain the
performance of the model as new tasks are added. In this paper, we focus on the
inductive NGIL problem, which accounts for the evolution of graph structure
(structural shift) induced by emerging tasks. We provide a formal formulation
and analysis of the problem, and propose a novel regularization-based technique
called Structural-Shift-Risk-Mitigation (SSRM) to mitigate the impact of the
structural shift on catastrophic forgetting of the inductive NGIL problem. We
show that the structural shift can lead to a shift in the input distribution
for the existing tasks, and further lead to an increased risk of catastrophic
forgetting. Through comprehensive empirical studies with several benchmark
datasets, we demonstrate that our proposed method,
Structural-Shift-Risk-Mitigation (SSRM), is flexible and easy to adapt to
improve the performance of state-of-the-art GNN incremental learning frameworks
in the inductive setting
CaseGNN: Graph Neural Networks for Legal Case Retrieval with Text-Attributed Graphs
Legal case retrieval is an information retrieval task in the legal domain,
which aims to retrieve relevant cases with a given query case. Recent research
of legal case retrieval mainly relies on traditional bag-of-words models and
language models. Although these methods have achieved significant improvement
in retrieval accuracy, there are still two challenges: (1) Legal structural
information neglect. Previous neural legal case retrieval models mostly encode
the unstructured raw text of case into a case representation, which causes the
lack of important legal structural information in a case and leads to poor case
representation; (2) Lengthy legal text limitation. When using the powerful
BERT-based models, there is a limit of input text lengths, which inevitably
requires to shorten the input via truncation or division with a loss of legal
context information. In this paper, a graph neural networks-based legal case
retrieval model, CaseGNN, is developed to tackle these challenges. To
effectively utilise the legal structural information during encoding, a case is
firstly converted into a Text-Attributed Case Graph (TACG), followed by a
designed Edge Graph Attention Layer and a readout function to obtain the case
graph representation. The CaseGNN model is optimised with a carefully designed
contrastive loss with easy and hard negative sampling. Since the text
attributes in the case graph come from individual sentences, the restriction of
using language models is further avoided without losing the legal context.
Extensive experiments have been conducted on two benchmarks from COLIEE 2022
and COLIEE 2023, which demonstrate that CaseGNN outperforms other
state-of-the-art legal case retrieval methods. The code has been released on
https://github.com/yanran-tang/CaseGNN