40 research outputs found
MolCPT: Molecule Continuous Prompt Tuning to Generalize Molecular Representation Learning
Molecular representation learning is crucial for the problem of molecular
property prediction, where graph neural networks (GNNs) serve as an effective
solution due to their structure modeling capabilities. Since labeled data is
often scarce and expensive to obtain, it is a great challenge for GNNs to
generalize in the extensive molecular space. Recently, the training paradigm of
"pre-train, fine-tune" has been leveraged to improve the generalization
capabilities of GNNs. It uses self-supervised information to pre-train the GNN,
and then performs fine-tuning to optimize the downstream task with just a few
labels. However, pre-training does not always yield statistically significant
improvement, especially for self-supervised learning with random structural
masking. In fact, the molecular structure is characterized by motif subgraphs,
which are frequently occurring and influence molecular properties. To leverage
the task-related motifs, we propose a novel paradigm of "pre-train, prompt,
fine-tune" for molecular representation learning, named molecule continuous
prompt tuning (MolCPT). MolCPT defines a motif prompting function that uses the
pre-trained model to project the standalone input into an expressive prompt.
The prompt effectively augments the molecular graph with meaningful motifs in
the continuous representation space; this provides more structural patterns to
aid the downstream classifier in identifying molecular properties. Extensive
experiments on several benchmark datasets show that MolCPT efficiently
generalizes pre-trained GNNs for molecular property prediction, with or without
a few fine-tuning steps
Marginal Nodes Matter: Towards Structure Fairness in Graphs
In social network, a person located at the periphery region (marginal node)
is likely to be treated unfairly when compared with the persons at the center.
While existing fairness works on graphs mainly focus on protecting sensitive
attributes (e.g., age and gender), the fairness incurred by the graph structure
should also be given attention. On the other hand, the information aggregation
mechanism of graph neural networks amplifies such structure unfairness, as
marginal nodes are often far away from other nodes. In this paper, we focus on
novel fairness incurred by the graph structure on graph neural networks, named
\emph{structure fairness}. Specifically, we first analyzed multiple graphs and
observed that marginal nodes in graphs have a worse performance of downstream
tasks than others in graph neural networks. Motivated by the observation, we
propose \textbf{S}tructural \textbf{Fair} \textbf{G}raph \textbf{N}eural
\textbf{N}etwork (SFairGNN), which combines neighborhood expansion based
structure debiasing with hop-aware attentive information aggregation to achieve
structure fairness. Our experiments show \SFairGNN can significantly improve
structure fairness while maintaining overall performance in the downstream
tasks.Comment: SIGKDD Explorations (To Appear
TinyKG: Memory-Efficient Training Framework for Knowledge Graph Neural Recommender Systems
There has been an explosion of interest in designing various Knowledge Graph
Neural Networks (KGNNs), which achieve state-of-the-art performance and provide
great explainability for recommendation. The promising performance is mainly
resulting from their capability of capturing high-order proximity messages over
the knowledge graphs. However, training KGNNs at scale is challenging due to
the high memory usage. In the forward pass, the automatic differentiation
engines (\textsl{e.g.}, TensorFlow/PyTorch) generally need to cache all
intermediate activation maps in order to compute gradients in the backward
pass, which leads to a large GPU memory footprint. Existing work solves this
problem by utilizing multi-GPU distributed frameworks. Nonetheless, this poses
a practical challenge when seeking to deploy KGNNs in memory-constrained
environments, especially for industry-scale graphs.
Here we present TinyKG, a memory-efficient GPU-based training framework for
KGNNs for the tasks of recommendation. Specifically, TinyKG uses exact
activations in the forward pass while storing a quantized version of
activations in the GPU buffers. During the backward pass, these low-precision
activations are dequantized back to full-precision tensors, in order to compute
gradients. To reduce the quantization errors, TinyKG applies a simple yet
effective quantization algorithm to compress the activations, which ensures
unbiasedness with low variance. As such, the training memory footprint of KGNNs
is largely reduced with negligible accuracy loss. To evaluate the performance
of our TinyKG, we conduct comprehensive experiments on real-world datasets. We
found that our TinyKG with INT2 quantization aggressively reduces the memory
footprint of activation maps with , only with loss in accuracy,
allowing us to deploy KGNNs on memory-constrained devices
Hessian-aware Quantized Node Embeddings for Recommendation
Graph Neural Networks (GNNs) have achieved state-of-the-art performance in
recommender systems. Nevertheless, the process of searching and ranking from a
large item corpus usually requires high latency, which limits the widespread
deployment of GNNs in industry-scale applications. To address this issue, many
methods compress user/item representations into the binary embedding space to
reduce space requirements and accelerate inference. Also, they use the
Straight-through Estimator (STE) to prevent vanishing gradients during
back-propagation. However, the STE often causes the gradient mismatch problem,
leading to sub-optimal results.
In this work, we present the Hessian-aware Quantized GNN (HQ-GNN) as an
effective solution for discrete representations of users/items that enable fast
retrieval. HQ-GNN is composed of two components: a GNN encoder for learning
continuous node embeddings and a quantized module for compressing
full-precision embeddings into low-bit ones. Consequently, HQ-GNN benefits from
both lower memory requirements and faster inference speeds compared to vanilla
GNNs. To address the gradient mismatch problem in STE, we further consider the
quantized errors and its second-order derivatives for better stability. The
experimental results on several large-scale datasets show that HQ-GNN achieves
a good balance between latency and performance