Guiding Principles for Participatory Design-inspired Natural Language Processing

We introduce 9 guiding principles 1 to integrate Participatory Design (PD) methods in the development of Natural Language Processing (NLP) systems. The adoption of PD methods by NLP will help to alleviate issues concerning the development of more democratic, fairer, less-biased technologies to process natural language data. This short paper is the outcome of an ongoing dialogue between designers and NLP experts and adopts a non-standard format following previous work by Traum (2000); Bender (2013); Abzianidze and Bos (2019). Every section is a guiding principle. While principles 1-3 illustrate assumptions and methods that inform community-based PD practices , we used two fictional design scenarios (Encinas and Blythe, 2018), which build on top of situations familiar to the authors, to elicit the identification of the other 6. Principles 4-6 describes the impact of PD methods on the design of NLP systems, targeting two critical aspects: data collection & annotation , and the deployment & evaluation. Finally, principles 7-9 guide a new reflexivity of the NLP research with respect to its context, actors and participants, and aims. We hope this guide will offer inspiration and a road-map to develop a new generation of PD-inspired NLP

Neural architectures for open-type relation argument extraction

In this work, we focus on the task of open-type relation argument extraction (ORAE): given a corpus, a query entity Q, and a knowledge base relation (e.g., “Q authored notable work with title X”), the model has to extract an argument of non-standard entity type (entities that cannot be extracted by a standard named entity tagger, for example, X: the title of a book or a work of art) from the corpus. We develop and compare a wide range of neural models for this task yielding large improvements over a strong baseline obtained with a neural question answering system. The impact of different sentence encoding architectures and answer extraction methods is systematically compared. An encoder based on gated recurrent units combined with a conditional random fields tagger yields the best results. We release a data set to train and evaluate ORAE, based on Wikidata and obtained by distant supervision

FBK’s Neural Machine Translation Systems for IWSLT 2016

In this paper, we describe FBK’s neural machine translation (NMT) systems submitted at the International Workshop on Spoken Language Translation (IWSLT) 2016. The systems are based on the state-of-the-art NMT architecture that is equipped with a bi-directional encoder and an attention mechanism in the decoder. They leverage linguistic information such as lemmas and part-of-speech tags of the source words in the form of additional factors along with the words. We compare performances of word and subword NMT systems along with different optimizers. Further, we explore different ensemble techniques to leverage multiple models within the same and across different networks. Several reranking methods are also explored. Our submissions cover all directions of the MSLT task, as well as en-{de, fr} and {de, fr}-en directions of TED. Compared to previously published best results on the TED 2014 test set, our models achieve comparable results on en-de and surpass them on en-fr (+2 BLEU) and fr-en (+7.7 BLEU) language pairs

Cross Target Generalization for Stance Detection

Stance detection is a popular NLP task which consists in automatically inferring the opinion expressed in a text with respect to a given target. Cross-target generalization is a known problem in stance detection, where systems tend to perform poorly when exposed to targets unseen during training.  Given that data annotation is expensive and time-consuming, finding ways to leverage other sources of knowledge to improve cross-target stance detection can offer great benefits. In this thesis, I suggest to improve the robustness of cross-target stance detection in three settings.First, I explore weak supervision through synthetically annotated samples as a means to provide knowledge about unseen targets to a stance detection system. To this end, I design a simple and inexpensive framework and show experimentally that integrating synthetic data is helpful for cross-target generalization. Secondly, I investigate cross-genre stance detection, where knowledge from annotated tweets is leveraged to improve news stance detection on targets unseen during training. Due to their peculiar stylistic characteristics, transferring knowledge between samples belonging to different genres is non-trivial. To allow the model to capture the useful stance-specific features, I propose to treat the task adversarially. Thirdly, I study multi-modality as a means to enhance cross-target generalization. Specifically, I design a robust multi-task BERT-based architecture that combines textual input with high-frequency intra-day time series from stock market prices. I show experimentally and through detailed result analysis that the proposed system benefits from financial information, and achieves state-of-the-art results: this demonstrates that the combination of multiple input signals is effective for cross-target stance detection, and opens interesting research directions for future work. In addition, I created the first multi-task, multi-genre and multi-modal resource for stance detection. It provides two aligned textual signals, composed of carefully selected and expert-annotated tweets and news articles; moreover, it contains aligned financial signal in the form of fine-grained intra-day stock market prices variations. This large and integrated resource provides a comprehensive framework for robust training and fair model evaluation of the above-mentioned algorithms. I released the entire resource for future research

Towards Automatic Fake News Detection: Cross-Level Stance Detection in News Articles

In this paper, we propose to adapt the four-staged pipeline proposed by Zubiaga et al. (2018) for the Rumor Verification task to the problem of Fake News Detection. We show that the recently released FNC-1 corpus covers two of its steps, namely the Tracking and the Stance Detection task. We identify asymmetry in length in the input to be a key characteristic of the latter step, when adapted to the framework of Fake News Detection, and propose to handle it as a specific type of CrossLevel Stance Detection. Inspired by theories from the field of Journalism Studies, we implement and test two architectures to successfully model the internal structure of an article and its interactions with a claim.The first author (CC) would like to thank the Siemens Machine Intelligence Group (CT RDA BAM MIC-DE, Munich) and the NERC DREAM CDT (grant no. 1945246) for partially funding this work. The third author (NC) is grateful for support from the UK EPSRC (grant no. EP/MOO5089/1

Will-They-Won't-They: A Very Large Dataset for Stance Detection on Twitter

We present a new challenging stance detection dataset, called Will-They-Won’t-They (WT--WT), which contains 51,284 tweets in English, making it by far the largest available dataset of the type. All the annotations are carried out by experts; therefore, the dataset constitutes a high-quality and reliable benchmark for future research in stance detection. Our experiments with a wide range of recent state-of-the-art stance detection systems show that the dataset poses a strong challenge to existing models in this domain.Keynes Fund, Cambridg