16,432 research outputs found
Cross-modality Data Augmentation for End-to-End Sign Language Translation
End-to-end sign language translation (SLT) aims to convert sign language
videos into spoken language texts directly without intermediate
representations. It has been a challenging task due to the modality gap between
sign videos and texts and the data scarcity of labeled data. To tackle these
challenges, we propose a novel Cross-modality Data Augmentation (XmDA)
framework to transfer the powerful gloss-to-text translation capabilities to
end-to-end sign language translation (i.e. video-to-text) by exploiting pseudo
gloss-text pairs from the sign gloss translation model. Specifically, XmDA
consists of two key components, namely, cross-modality mix-up and
cross-modality knowledge distillation. The former explicitly encourages the
alignment between sign video features and gloss embeddings to bridge the
modality gap. The latter utilizes the generation knowledge from gloss-to-text
teacher models to guide the spoken language text generation. Experimental
results on two widely used SLT datasets, i.e., PHOENIX-2014T and CSL-Daily,
demonstrate that the proposed XmDA framework significantly and consistently
outperforms the baseline models. Extensive analyses confirm our claim that XmDA
enhances spoken language text generation by reducing the representation
distance between videos and texts, as well as improving the processing of
low-frequency words and long sentences.Comment: Accepted to Findings EMNLP 202
SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmark
In this paper, we present SignAvatars, the first large-scale multi-prompt 3D
sign language (SL) motion dataset designed to bridge the communication gap for
hearing-impaired individuals. While there has been an exponentially growing
number of research regarding digital communication, the majority of existing
communication technologies primarily cater to spoken or written languages,
instead of SL, the essential communication method for hearing-impaired
communities. Existing SL datasets, dictionaries, and sign language production
(SLP) methods are typically limited to 2D as the annotating 3D models and
avatars for SL is usually an entirely manual and labor-intensive process
conducted by SL experts, often resulting in unnatural avatars. In response to
these challenges, we compile and curate the SignAvatars dataset, which
comprises 70,000 videos from 153 signers, totaling 8.34 million frames,
covering both isolated signs and continuous, co-articulated signs, with
multiple prompts including HamNoSys, spoken language, and words. To yield 3D
holistic annotations, including meshes and biomechanically-valid poses of body,
hands, and face, as well as 2D and 3D keypoints, we introduce an automated
annotation pipeline operating on our large corpus of SL videos. SignAvatars
facilitates various tasks such as 3D sign language recognition (SLR) and the
novel 3D SL production (SLP) from diverse inputs like text scripts, individual
words, and HamNoSys notation. Hence, to evaluate the potential of SignAvatars,
we further propose a unified benchmark of 3D SL holistic motion production. We
believe that this work is a significant step forward towards bringing the
digital world to the hearing-impaired communities. Our project page is at
https://signavatars.github.io/Comment: 9 pages; Project page available at https://signavatars.github.io
Artificial Intelligence for Multimedia Signal Processing
Artificial intelligence technologies are also actively applied to broadcasting and multimedia processing technologies. A lot of research has been conducted in a wide variety of fields, such as content creation, transmission, and security, and these attempts have been made in the past two to three years to improve image, video, speech, and other data compression efficiency in areas related to MPEG media processing technology. Additionally, technologies such as media creation, processing, editing, and creating scenarios are very important areas of research in multimedia processing and engineering. This book contains a collection of some topics broadly across advanced computational intelligence algorithms and technologies for emerging multimedia signal processing as: Computer vision field, speech/sound/text processing, and content analysis/information mining
λ₯λ¬λ κΈ°λ° μμ± λͺ¨λΈμ μ΄μ©ν μμ°μ΄μ²λ¦¬ λ°μ΄ν° μ¦κ° κΈ°λ²
νμλ
Όλ¬Έ(λ°μ¬)--μμΈλνκ΅ λνμ :곡과λν μ»΄ν¨ν°κ³΅νλΆ,2020. 2. μ΄μꡬ.Recent advances in generation capability of deep learning models have spurred interest in utilizing deep generative models for unsupervised generative data augmentation (GDA). Generative data augmentation aims to improve the performance of a downstream machine learning model by augmenting the original dataset with samples generated from a deep latent variable model. This data augmentation approach is attractive to the natural language processing community, because (1) there is a shortage of text augmentation techniques that require little supervision and (2) resource scarcity being prevalent. In this dissertation, we explore the feasibility of exploiting deep latent variable models for data augmentation on three NLP tasks: sentence classification, spoken language understanding (SLU) and dialogue state tracking (DST), represent NLP tasks of various complexities and properties -- SLU requires multi-task learning of text classification and sequence tagging, while DST requires the understanding of hierarchical and recurrent data structures. For each of the three tasks, we propose a task-specific latent variable model based on conditional, hierarchical and sequential variational autoencoders (VAE) for multi-modal joint modeling of linguistic features and the relevant annotations. We conduct extensive experiments to statistically justify our hypothesis that deep generative data augmentation is beneficial for all subject tasks. Our experiments show that deep generative data augmentation is effective for the select tasks, supporting the idea that the technique can potentially be utilized for other range of NLP tasks. Ablation and qualitative studies reveal deeper insight into the underlying mechanisms of generative data augmentation. As a secondary contribution, we also shed light onto the recurring posterior collapse phenomenon in autoregressive VAEs and, subsequently, propose novel techniques to reduce the model risk, which is crucial for proper training of complex VAE models, enabling them to synthesize better samples for data augmentation. In summary, this work intends to demonstrate and analyze the effectiveness of unsupervised generative data augmentation in NLP. Ultimately, our approach enables standardized adoption of generative data augmentation, which can be applied orthogonally to existing regularization techniques.μ΅κ·Ό λ₯λ¬λ κΈ°λ° μμ± λͺ¨λΈμ κΈκ²©ν λ°μ μΌλ‘ μ΄λ₯Ό μ΄μ©ν μμ± κΈ°λ° λ°μ΄ν° μ¦κ° κΈ°λ²(generative data augmentation, GDA)μ μ€ν κ°λ₯μ±μ λν κΈ°λκ° μ»€μ§κ³ μλ€. μμ± κΈ°λ° λ°μ΄ν° μ¦κ° κΈ°λ²μ λ₯λ¬λ κΈ°λ° μ μ¬λ³μ λͺ¨λΈμμ μμ± λ μνμ μλ³Έ λ°μ΄ν°μ
μ μΆκ°νμ¬ μ°κ΄λ νμ€ν¬μ μ±λ₯μ ν₯μμν€λ κΈ°μ μ μλ―Ένλ€. λ°λΌμ μμ± κΈ°λ° λ°μ΄ν° μ¦κ° κΈ°λ²μ λ°μ΄ν° 곡κ°μμ μ΄λ€μ§λ μ κ·ν κΈ°μ μ ν ννλ‘ κ°μ£Όλ μ μλ€. μ΄λ¬ν λ₯λ¬λ κΈ°λ° μμ± λͺ¨λΈμ μλ‘μ΄ νμ© κ°λ₯μ±μ μμ°μ΄μ²λ¦¬ λΆμΌμμ λμ± μ€μνκ² λΆκ°λλ μ΄μ λ (1) λ²μ© κ°λ₯ν ν
μ€νΈ λ°μ΄ν° μ¦κ° κΈ°μ μ λΆμ¬μ (2) ν
μ€νΈ λ°μ΄ν°μ ν¬μμ±μ 극볡ν μ μλ λμμ΄ νμνκΈ° λλ¬Έμ΄λ€. λ¬Έμ μ 볡μ‘λμ νΉμ§μ κ³¨κ³ λ£¨ μ±μ§νκΈ° μν΄ λ³Έ λ
Όλ¬Έμμλ ν
μ€νΈ λΆλ₯(text classification), μμ°¨μ λ μ΄λΈλ§κ³Ό λ©ν°νμ€νΉ κΈ°μ μ΄ νμν λ°ν μ΄ν΄(spoken language understanding, SLU), κ³μΈ΅μ μ΄λ©° μ¬κ·μ μΈ λ°μ΄ν° ꡬ쑰μ λν κ³ λ €κ° νμν λν μν μΆμ (dialogue state tracking, DST) λ± μΈ κ°μ§ λ¬Έμ μμ λ₯λ¬λ κΈ°λ° μμ± λͺ¨λΈμ νμ©ν λ°μ΄ν° μ¦κ° κΈ°λ²μ νλΉμ±μ λν΄ λ€λ£¬λ€. λ³Έ μ°κ΅¬μμλ 쑰건λΆ, κ³μΈ΅μ λ° μμ°¨μ variational autoencoder (VAE)μ κΈ°λ°νμ¬ κ° μμ°μ΄μ²λ¦¬ λ¬Έμ μ νΉνλ ν
μ€νΈ λ° μ°κ΄ λΆμ°© μ 보λ₯Ό λμμ μμ±νλ νΉμ λ₯λ¬λ μμ± λͺ¨λΈλ€μ μ μνκ³ , λ€μν νλ₯ λͺ¨λΈκ³Ό λ°μ΄ν°μ
μ λ€λ£¨λ λ± ν λμ μ€νμ ν΅ν΄ λ₯ μμ± λͺ¨λΈ κΈ°λ° λ°μ΄ν° μ¦κ° κΈ°λ²μ ν¨κ³Όλ₯Ό ν΅κ³μ μΌλ‘ μ
μ¦νμλ€. λΆμμ μ°κ΅¬μμλ μκΈ°νκ·μ (autoregressive) VAEμμ λΉλ²ν λ°μνλ posterior collapse λ¬Έμ μ λν΄ νꡬνκ³ , ν΄λΉ λ¬Έμ λ₯Ό μνν μ μλ μ κ· λ°©μλ μ μνλ€. ν΄λΉ λ°©λ²μ μμ±μ λ°μ΄ν° μ¦κ°μ νμν 볡μ‘ν VAE λͺ¨λΈμ μ μ©νμμ λ, μμ± λͺ¨λΈμ μμ± μ§μ΄ ν₯μλμ΄ λ°μ΄ν° μ¦κ° ν¨κ³Όμλ κΈμ μ μΈ μν₯μ λ―ΈμΉ μ μμμ κ²μ¦νμλ€. λ³Έ λ
Όλ¬Έμ ν΅ν΄ μμ°μ΄μ²λ¦¬ λΆμΌμμ κΈ°μ‘΄ μ κ·ν κΈ°λ²κ³Ό λ³ν μ μ© κ°λ₯ν λΉμ§λ ννμ λ°μ΄ν° μ¦κ° κΈ°λ²μ νμ€νλ₯Ό κΈ°λν΄ λ³Ό μ μλ€.1 Introduction 1
1.1 Motivation 1
1.2 Dissertation Overview 6
2 Background and Related Work 8
2.1 Deep Latent Variable Models 8
2.1.1 Variational Autoencoder (VAE) 10
2.1.2 Deep Generative Models and Text Generation 12
2.2 Data Augmentation 12
2.2.1 General Description 13
2.2.2 Categorization of Data Augmentation 14
2.2.3 Theoretical Explanations 21
2.3 Summary 24
3 Basic Task: Text Classi cation 25
3.1 Introduction 25
3.2 Our Approach 28
3.2.1 Proposed Models 28
3.2.2 Training with I-VAE 29
3.3 Experiments 31
3.3.1 Datasets 32
3.3.2 Experimental Settings 33
3.3.3 Implementation Details 34
3.3.4 Data Augmentation Results 36
3.3.5 Ablation Studies 39
3.3.6 Qualitative Analysis 40
3.4 Summary 45
4 Multi-task Learning: Spoken Language Understanding 46
4.1 Introduction 46
4.2 Related Work 48
4.3 Model Description 48
4.3.1 Framework Formulation 48
4.3.2 Joint Generative Model 49
4.4 Experiments 56
4.4.1 Datasets 56
4.4.2 Experimental Settings 57
4.4.3 Generative Data Augmentation Results 61
4.4.4 Comparison to Other State-of-the-art Results 63
4.4.5 Ablation Studies 63
4.5 Summary 67
5 Complex Data: Dialogue State Tracking 68
5.1 Introduction 68
5.2 Background and Related Work 70
5.2.1 Task-oriented Dialogue 70
5.2.2 Dialogue State Tracking 72
5.2.3 Conversation Modeling 72
5.3 Variational Hierarchical Dialogue Autoencoder (VHDA) 73
5.3.1 Notations 73
5.3.2 Variational Hierarchical Conversational RNN 74
5.3.3 Proposed Model 75
5.3.4 Posterior Collapse 82
5.4 Experimental Results 84
5.4.1 Experimental Settings 84
5.4.2 Data Augmentation Results 90
5.4.3 Intrinsic Evaluation - Language Evaluation 94
5.4.4 Qualitative Results 95
5.5 Summary 101
6 Conclusion 103
6.1 Summary 103
6.2 Limitations 104
6.3 Future Work 105Docto
- β¦