1,502 research outputs found
A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
Referring expressions are natural language constructions used to identify
particular objects within a scene. In this paper, we propose a unified
framework for the tasks of referring expression comprehension and generation.
Our model is composed of three modules: speaker, listener, and reinforcer. The
speaker generates referring expressions, the listener comprehends referring
expressions, and the reinforcer introduces a reward function to guide sampling
of more discriminative expressions. The listener-speaker modules are trained
jointly in an end-to-end learning framework, allowing the modules to be aware
of one another during learning while also benefiting from the discriminative
reinforcer's feedback. We demonstrate that this unified framework and training
achieves state-of-the-art results for both comprehension and generation on
three referring expression datasets. Project and demo page:
https://vision.cs.unc.edu/referComment: Some typo fixed; comprehension results on refcocog updated; more
human evaluation results adde
SCOPIC Design and Overview
National Foreign Language Resource Cente
Social Cognition Parallax Interview Corpus (SCOPIC)
National Foreign Language Resource Cente
Computational Sociolinguistics: A Survey
Language is a social phenomenon and variation is inherent to its social
nature. Recently, there has been a surge of interest within the computational
linguistics (CL) community in the social dimension of language. In this article
we present a survey of the emerging field of "Computational Sociolinguistics"
that reflects this increased interest. We aim to provide a comprehensive
overview of CL research on sociolinguistic themes, featuring topics such as the
relation between language and social identity, language use in social
interaction and multilingual communication. Moreover, we demonstrate the
potential for synergy between the research communities involved, by showing how
the large-scale data-driven methods that are widely used in CL can complement
existing sociolinguistic studies, and how sociolinguistics can inform and
challenge the methods and assumptions employed in CL studies. We hope to convey
the possible benefits of a closer collaboration between the two communities and
conclude with a discussion of open challenges.Comment: To appear in Computational Linguistics. Accepted for publication:
18th February, 201
Linguistically Aided Speaker Diarization Using Speaker Role Information
Speaker diarization relies on the assumption that speech segments
corresponding to a particular speaker are concentrated in a specific region of
the speaker space; a region which represents that speaker's identity. These
identities are not known a priori, so a clustering algorithm is typically
employed, which is traditionally based solely on audio. Under noisy conditions,
however, such an approach poses the risk of generating unreliable speaker
clusters. In this work we aim to utilize linguistic information as a
supplemental modality to identify the various speakers in a more robust way. We
are focused on conversational scenarios where the speakers assume distinct
roles and are expected to follow different linguistic patterns. This distinct
linguistic variability can be exploited to help us construct the speaker
identities. That way, we are able to boost the diarization performance by
converting the clustering task to a classification one. The proposed method is
applied in real-world dyadic psychotherapy interactions between a provider and
a patient and demonstrated to show improved results.Comment: from v1: restructured Introduction and Background, added experimental
results with ASR text and language-only baselin
HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognition
Emotion recognition in conversations is challenging due to the multi-modal
nature of the emotion expression. We propose a hierarchical cross-attention
model (HCAM) approach to multi-modal emotion recognition using a combination of
recurrent and co-attention neural network models. The input to the model
consists of two modalities, i) audio data, processed through a learnable
wav2vec approach and, ii) text data represented using a bidirectional encoder
representations from transformers (BERT) model. The audio and text
representations are processed using a set of bi-directional recurrent neural
network layers with self-attention that converts each utterance in a given
conversation to a fixed dimensional embedding. In order to incorporate
contextual knowledge and the information across the two modalities, the audio
and text embeddings are combined using a co-attention layer that attempts to
weigh the utterance level embeddings relevant to the task of emotion
recognition. The neural network parameters in the audio layers, text layers as
well as the multi-modal co-attention layers, are hierarchically trained for the
emotion classification task. We perform experiments on three established
datasets namely, IEMOCAP, MELD and CMU-MOSI, where we illustrate that the
proposed model improves significantly over other benchmarks and helps achieve
state-of-art results on all these datasets.Comment: 11 pages, 6 figure
- …