14 research outputs found
DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation
Talking head synthesis is a promising approach for the video production
industry. Recently, a lot of effort has been devoted in this research area to
improve the generation quality or enhance the model generalization. However,
there are few works able to address both issues simultaneously, which is
essential for practical applications. To this end, in this paper, we turn
attention to the emerging powerful Latent Diffusion Models, and model the
Talking head generation as an audio-driven temporally coherent denoising
process (DiffTalk). More specifically, instead of employing audio signals as
the single driving factor, we investigate the control mechanism of the talking
face, and incorporate reference face images and landmarks as conditions for
personality-aware generalized synthesis. In this way, the proposed DiffTalk is
capable of producing high-quality talking head videos in synchronization with
the source audio, and more importantly, it can be naturally generalized across
different identities without any further fine-tuning. Additionally, our
DiffTalk can be gracefully tailored for higher-resolution synthesis with
negligible extra computational cost. Extensive experiments show that the
proposed DiffTalk efficiently synthesizes high-fidelity audio-driven talking
head videos for generalized novel identities. For more video results, please
refer to \url{https://sstzal.github.io/DiffTalk/}.Comment: Project page https://sstzal.github.io/DiffTalk
What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders
The last years have witnessed the emergence of a promising self-supervised
learning strategy, referred to as masked autoencoding. However, there is a lack
of theoretical understanding of how masking matters on graph autoencoders
(GAEs). In this work, we present masked graph autoencoder (MaskGAE), a
self-supervised learning framework for graph-structured data. Different from
standard GAEs, MaskGAE adopts masked graph modeling (MGM) as a principled
pretext task - masking a portion of edges and attempting to reconstruct the
missing part with partially visible, unmasked graph structure. To understand
whether MGM can help GAEs learn better representations, we provide both
theoretical and empirical evidence to comprehensively justify the benefits of
this pretext task. Theoretically, we establish close connections between GAEs
and contrastive learning, showing that MGM significantly improves the
self-supervised learning scheme of GAEs. Empirically, we conduct extensive
experiments on a variety of graph benchmarks, demonstrating the superiority of
MaskGAE over several state-of-the-arts on both link prediction and node
classification tasks.Comment: KDD 2023 research track. Code available at
https://github.com/EdisonLeeeee/MaskGA
Language Agents for Detecting Implicit Stereotypes in Text-to-image Models at Scale
The recent surge in the research of diffusion models has accelerated the
adoption of text-to-image models in various Artificial Intelligence Generated
Content (AIGC) commercial products. While these exceptional AIGC products are
gaining increasing recognition and sparking enthusiasm among consumers, the
questions regarding whether, when, and how these models might unintentionally
reinforce existing societal stereotypes remain largely unaddressed. Motivated
by recent advancements in language agents, here we introduce a novel agent
architecture tailored for stereotype detection in text-to-image models. This
versatile agent architecture is capable of accommodating free-form detection
tasks and can autonomously invoke various tools to facilitate the entire
process, from generating corresponding instructions and images, to detecting
stereotypes. We build the stereotype-relevant benchmark based on multiple
open-text datasets, and apply this architecture to commercial products and
popular open source text-to-image models. We find that these models often
display serious stereotypes when it comes to certain prompts about personal
characteristics, social cultural context and crime-related aspects. In summary,
these empirical findings underscore the pervasive existence of stereotypes
across social dimensions, including gender, race, and religion, which not only
validate the effectiveness of our proposed approach, but also emphasize the
critical necessity of addressing potential ethical risks in the burgeoning
realm of AIGC. As AIGC continues its rapid expansion trajectory, with new
models and plugins emerging daily in staggering numbers, the challenge lies in
the timely detection and mitigation of potential biases within these models
LasTGL: An Industrial Framework for Large-Scale Temporal Graph Learning
Over the past few years, graph neural networks (GNNs) have become powerful
and practical tools for learning on (static) graph-structure data. However,
many real-world applications, such as social networks and e-commerce, involve
temporal graphs where nodes and edges are dynamically evolving. Temporal graph
neural networks (TGNNs) have progressively emerged as an extension of GNNs to
address time-evolving graphs and have gradually become a trending research
topic in both academics and industry. Advancing research and application in
such an emerging field necessitates the development of new tools to compose
TGNN models and unify their different schemes for dealing with temporal graphs.
In this work, we introduce LasTGL, an industrial framework that integrates
unified and extensible implementations of common temporal graph learning
algorithms for various advanced tasks. The purpose of LasTGL is to provide the
essential building blocks for solving temporal graph learning tasks, focusing
on the guiding principles of user-friendliness and quick prototyping on which
PyTorch is based. In particular, LasTGL provides comprehensive temporal graph
datasets, TGNN models and utilities along with well-documented tutorials,
making it suitable for both absolute beginners and expert deep learning
practitioners alike.Comment: Preprint; Work in progres
HeteroNet: Heterophily-aware Representation Learning on Heterogenerous Graphs
Real-world graphs are typically complex, exhibiting heterogeneity in the
global structure, as well as strong heterophily within local neighborhoods.
While a growing body of literature has revealed the limitations of common graph
neural networks (GNNs) in handling homogeneous graphs with heterophily, little
work has been conducted on investigating the heterophily properties in the
context of heterogeneous graphs. To bridge this research gap, we identify the
heterophily in heterogeneous graphs using metapaths and propose two practical
metrics to quantitatively describe the levels of heterophily. Through in-depth
investigations on several real-world heterogeneous graphs exhibiting varying
levels of heterophily, we have observed that heterogeneous graph neural
networks (HGNNs), which inherit many mechanisms from GNNs designed for
homogeneous graphs, fail to generalize to heterogeneous graphs with heterophily
or low level of homophily. To address the challenge, we present HeteroNet,
a heterophily-aware HGNN that incorporates both masked metapath prediction and
masked label prediction tasks to effectively and flexibly handle both
homophilic and heterophilic heterogeneous graphs. We evaluate the performance
of HeteroNet on five real-world heterogeneous graph benchmarks with varying
levels of heterophily. The results demonstrate that HeteroNet outperforms
strong baselines in the semi-supervised node classification task, providing
valuable insights into effectively handling more complex heterogeneous graphs.Comment: Preprin
GUARD: Graph Universal Adversarial Defense
Graph convolutional networks (GCNs) have been shown to be vulnerable to small
adversarial perturbations, which becomes a severe threat and largely limits
their applications in security-critical scenarios. To mitigate such a threat,
considerable research efforts have been devoted to increasing the robustness of
GCNs against adversarial attacks. However, current defense approaches are
typically designed to prevent GCNs from untargeted adversarial attacks and
focus on overall performance, making it challenging to protect important local
nodes from more powerful targeted adversarial attacks. Additionally, a
trade-off between robustness and performance is often made in existing
research. Such limitations highlight the need for developing an effective and
efficient approach that can defend local nodes against targeted attacks,
without compromising the overall performance of GCNs. In this work, we present
a simple yet effective method, named Graph Universal Adversarial Defense
(GUARD). Unlike previous works, GUARD protects each individual node from
attacks with a universal defensive patch, which is generated once and can be
applied to any node (node-agnostic) in a graph. GUARD is fast, straightforward
to implement without any change to network architecture nor any additional
parameters, and is broadly applicable to any GCNs. Extensive experiments on
four benchmark datasets demonstrate that GUARD significantly improves
robustness for several established GCNs against multiple adversarial attacks
and outperforms state-of-the-art defense methods by large margins.Comment: Accepted by CIKM 2023. Code is publicly available at
https://github.com/EdisonLeeeee/GUAR
Scaling Up Dynamic Graph Representation Learning via Spiking Neural Networks
Recent years have seen a surge in research on dynamic graph representation learning, which aims to model temporal graphs that are dynamic and evolving constantly over time. However, current work typically models graph dynamics with recurrent neural networks (RNNs), making them suffer seriously from computation and memory overheads on large temporal graphs. So far, scalability of dynamic graph representation learning on large temporal graphs remains one of the major challenges. In this paper, we present a scalable framework, namely SpikeNet, to efficiently capture the temporal and structural patterns of temporal graphs. We explore a new direction in that we can capture the evolving dynamics of temporal graphs with spiking neural networks (SNNs) instead of RNNs. As a low-power alternative to RNNs, SNNs explicitly model graph dynamics as spike trains of neuron populations and enable spike-based propagation in an efficient way. Experiments on three large real-world temporal graph datasets demonstrate that SpikeNet outperforms strong baselines on the temporal node classification task with lower computational costs. Particularly, SpikeNet generalizes to a large temporal graph (2.7M nodes and 13.9M edges) with significantly fewer parameters and computation overheads