12 research outputs found
A Good Sample is Hard to Find: Noise Injection Sampling and Self-Training for Neural Language Generation Models
Deep neural networks (DNN) are quickly becoming the de facto standard
modeling method for many natural language generation (NLG) tasks. In order for
such models to truly be useful, they must be capable of correctly generating
utterances for novel meaning representations (MRs) at test time. In practice,
even sophisticated DNNs with various forms of semantic control frequently fail
to generate utterances faithful to the input MR. In this paper, we propose an
architecture agnostic self-training method to sample novel MR/text utterance
pairs to augment the original training data. Remarkably, after training on the
augmented data, even simple encoder-decoder models with greedy decoding are
capable of generating semantically correct utterances that are as good as
state-of-the-art outputs in both automatic and human evaluations of quality.Comment: Accepted as a long paper at INLG 201
Automatic Ground Truth Expansion for Timeline Evaluation
The development of automatic systems that can produce timeline summaries by filtering high-volume streams of text documents, retaining only those that are relevant to a particular information need (e.g. topic or event), remains a very challenging task. To advance the field of automatic timeline generation, robust and reproducible evaluation methodologies are needed. To this end, several evaluation metrics and labeling methodologies have recently been developed - focusing on information nugget or cluster-based ground truth representations, respectively. These methodologies rely on human assessors manually mapping timeline items (e.g. tweets) to an explicit representation of what information a 'good' summary should contain. However, while these evaluation methodologies produce reusable ground truth labels, prior works have reported cases where such labels fail to accurately estimate the performance of new timeline generation systems due to label incompleteness. In this paper, we first quantify the extent to which timeline summary ground truth labels fail to generalize to new summarization systems, then we propose and evaluate new automatic solutions to this issue. In particular, using a depooling methodology over 21 systems and across three high-volume datasets, we quantify the degree of system ranking error caused by excluding those systems when labeling. We show that when considering lower-effectiveness systems, the test collections are robust (the likelihood of systems being miss-ranked is low). However, we show that the risk of systems being miss-ranked increases as the effectiveness of systems held-out from the pool increases. To reduce the risk of miss-ranking systems, we also propose two different automatic ground truth label expansion techniques. Our results show that our proposed expansion techniques can be effective for increasing the robustness of the TREC-TS test collections, markedly reducing the number of miss-rankings by up to 50% on average among the scenarios tested
Multimodal Social Media Analysis for Gang Violence Prevention
Gang violence is a severe issue in major cities across the U.S. and recent
studies [Patton et al. 2017] have found evidence of social media communications
that can be linked to such violence in communities with high rates of exposure
to gang activity. In this paper we partnered computer scientists with social
work researchers, who have domain expertise in gang violence, to analyze how
public tweets with images posted by youth who mention gang associations on
Twitter can be leveraged to automatically detect psychosocial factors and
conditions that could potentially assist social workers and violence outreach
workers in prevention and early intervention programs. To this end, we
developed a rigorous methodology for collecting and annotating tweets. We
gathered 1,851 tweets and accompanying annotations related to visual concepts
and the psychosocial codes: aggression, loss, and substance use. These codes
are relevant to social work interventions, as they represent possible pathways
to violence on social media. We compare various methods for classifying tweets
into these three classes, using only the text of the tweet, only the image of
the tweet, or both modalities as input to the classifier. In particular, we
analyze the usefulness of mid-level visual concepts and the role of different
modalities for this tweet classification task. Our experiments show that
individually, text information dominates classification performance of the loss
class, while image information dominates the aggression and substance use
classes. Our multimodal approach provides a very promising improvement (18%
relative in mean average precision) over the best single modality approach.
Finally, we also illustrate the complexity of understanding social media data
and elaborate on open challenges