Self-supervised representation learning on text-attributed graphs, which aims
to create expressive and generalizable representations for various downstream
tasks, has received increasing research attention lately. However, existing
methods either struggle to capture the full extent of structural context
information or rely on task-specific training labels, which largely hampers
their effectiveness and generalizability in practice. To solve the problem of
self-supervised representation learning on text-attributed graphs, we develop a
novel Graph-Centric Language model -- GRENADE. Specifically, GRENADE exploits
the synergistic effect of both pre-trained language model and graph neural
network by optimizing with two specialized self-supervised learning algorithms:
graph-centric contrastive learning and graph-centric knowledge alignment. The
proposed graph-centric self-supervised learning algorithms effectively help
GRENADE to capture informative textual semantics as well as structural context
information on text-attributed graphs. Through extensive experiments, GRENADE
shows its superiority over state-of-the-art methods. Implementation is
available at \url{https://github.com/bigheiniu/GRENADE}.Comment: Findings of EMNLP 202