Graph Neural Networks for Natural Language Processing

By constructing graph-structured data from the input data, Graph Neural Network (GNN) enhances the performance of numerous Natural Language Processing (NLP) tasks. In this thesis, we mainly focus on two aspects of NLP: text classification and knowledge graph completion. TextGCN shows excellent performance in text classification by leveraging the graph structure of the entire corpus without using any external resources, especially under a limited labelled data setting. Two questions are explored: (1) Under the transductive semi-supervised setting, how to utilize the documents better and learn the complex relationship between nodes. (2) How to transform TextGCN into an inductive model and also reduce the time and space complexity? In detail, firstly, a comprehensive analysis was conducted on TextGCN and its variants. Secondly, we propose ME-GCN, a novel method for text classification that utilizes multi-dimensional edge features in a graph neural network (GNN) for the first time. It uses the corpus-trained word and document-based edge features for semi-supervised classification and has been shown to be effective through experiments on benchmark datasets under the limited labelled data setting. Thirdly, InducT-GCN, an inductive framework for GCN-based text classification that does not require additional resources is introduced. The framework introduces a novel approach to make transductive GCN-based text classification models inductive, improving performance and reducing time and space complexity. Most existing work for Temporal Knowledge Graph Completion (TKGC) overlooks the significance of explicit temporal information and fails to skip irrelevant snapshots based on the entity-related relation in the query. To address this, we introduced Re-Temp (Relation-Aware Temporal Representation Learning), a model that leverages explicit temporal embedding and a skip information flow after each timestamp to eliminate unnecessary information for prediction

Structure Learning for Headline Generation

Headline generation is an important problem in natural language processing, which aims to describe a document by a compact and informative headline. Some recent successes on this task have been achieved by advanced graph-based neural models, which marry the representational power of deep neural networks with the structural modeling ability of the relational sentence graphs. The advantages of graph-based neural models over traditional Seq2Seq models lie in that they can encode long-distance relationship between sentences beyond the surface linear structure. However, since documents are typically weakly-structured data, modern graph-based neural models usually rely on manually designed rules or some heuristics to construct the sentence graph a prior. This may largely limit the power and increase the cost of the graph-based methods. In this paper, therefore, we propose to incorporate structure learning into the graph-based neural models for headline generation. That is, we want to automatically learn the sentence graph using a data-driven way, so that we can unveil the document structure flexibly without prior heuristics or rules. To achieve this goal, we employ a deep & wide network to encode rich relational information between sentences for the sentence graph learning. For the deep component, we leverage neural matching models, either representation-focused or interaction-focused model, to learn semantic similarity between sentences. For the wide component, we encode a variety of discourse relations between sentences. A Graph Convolutional Network (GCN) is then applied over the sentence graph to generate high-level relational representations for headline generation. The whole model could be optimized end-to-end so that the structure and representation could be learned jointly. Empirical studies show that our model can significantly outperform the state-of-the-art headline generation models