Guiding Abstractive Summarization using Structural Information

Abstractive summarization takes a set of sentences from a source document and reproduces its salient information using the summarizer's own words into a summary. Produced summaries may contain novel words and have different grammatical structures from the source document. In a sense, abstractive summarization is closer to how a human summarizes, yet it is also more difficult to automate since it requires a full understanding of the natural language. However, with the inception of deep learning, many new summarization systems achieved improved automatic and manual evaluation scores. One prominent deep learning model is the sequence-to-sequence model with an attention-based mechanism. Moreover, the advent of pre-trained language models over a huge set of unlabeled data further improved the performance of a summarization system. However, with all the said improvements, abstractive summarization is still adversely affected by hallucination and disfluency. Furthermore, all these recent works that used a seq2seq model require a large dataset since the underlying neural network easily overfits on a small dataset resulting in a poor approximation and high variance outputs. The problem is that these large datasets often came with only a single reference summary for each source document despite that it is known that human annotators are subject to a certain degree of subjectivity when writing a summary. We addressed the first problem by using a mechanism where the model uses a guidance signal to control what tokens are to be generated. A guidance signal can be defined as different types of signals that are fed into the model in addition to the source document where a commonly used one is structural information from the source document. Recent approaches showed good results using this approach, however, they were using a joint-training approach for the guiding mechanism, in other words, the model needs to be re-trained if a different guidance signal is used which is costly. We propose approaches that work without re-training and therefore are more flexible with regards to the guidance signal source and also computationally cheaper. We performed two different experiments where the first one is a novel guided mechanism that extends previous work on abstractive summarization using Abstract Meaning Representation (AMR) with a neural language generation stage which we guide using side information. Results showed that our approach improves over a strong baseline by 2 ROUGE-2 points. The second experiment is a guided key-phrase extractor for more informative summarization. This experiment showed mixed results, but we provide an analysis of the negative and positive output examples. The second problem was addressed by our proposed manual evaluation framework called Highlight-based Reference-less Evaluation Summarization (HighRES). The proposed framework avoids reference bias and provides absolute instead of ranked evaluation of the systems. To validate our approach we employed crowd-workers to augment with highlights on the eXtreme SUMmarization (XSUM) dataset which is a highly abstractive summarization dataset. We then compared two abstractive systems (Pointer Generator and T-Conv) to demonstrate our approach. Results showed that HighRES improves inter-annotator agreement in comparison to using the source document directly, while it also emphasizes differences among systems that would be ignored under other evaluation approaches. Our work also produces annotated dataset which gives more understanding on how humans select salient information from the source document

Towards generic relation extraction

A vast amount of usable electronic data is in the form of unstructured text. The relation extraction task aims to identify useful information in text (e.g., PersonW works for OrganisationX, GeneY encodes ProteinZ) and recode it in a format such as a relational database that can be more effectively used for querying and automated reasoning. However, adapting conventional relation extraction systems to new domains or tasks requires significant effort from annotators and developers. Furthermore, previous adaptation approaches based on bootstrapping start from example instances of the target relations, thus requiring that the correct relation type schema be known in advance. Generic relation extraction (GRE) addresses the adaptation problem by applying generic techniques that achieve comparable accuracy when transferred, without modification of model parameters, across domains and tasks. Previous work on GRE has relied extensively on various lexical and shallow syntactic indicators. I present new state-of-the-art models for GRE that incorporate governordependency information. I also introduce a dimensionality reduction step into the GRE relation characterisation sub-task, which serves to capture latent semantic information and leads to significant improvements over an unreduced model. Comparison of dimensionality reduction techniques suggests that latent Dirichlet allocation (LDA) – a probabilistic generative approach – successfully incorporates a larger and more interdependent feature set than a model based on singular value decomposition (SVD) and performs as well as or better than SVD on all experimental settings. Finally, I will introduce multi-document summarisation as an extrinsic test bed for GRE and present results which demonstrate that the relative performance of GRE models is consistent across tasks and that the GRE-based representation leads to significant improvements over a standard baseline from the literature. Taken together, the experimental results 1) show that GRE can be improved using dependency parsing and dimensionality reduction, 2) demonstrate the utility of GRE for the content selection step of extractive summarisation and 3) validate the GRE claim of modification-free adaptation for the first time with respect to both domain and task. This thesis also introduces data sets derived from publicly available corpora for the purpose of rigorous intrinsic evaluation in the news and biomedical domains