2 research outputs found
λ₯λ¬λ κΈ°λ° μμ± λͺ¨λΈμ μ΄μ©ν μμ°μ΄μ²λ¦¬ λ°μ΄ν° μ¦κ° κΈ°λ²
νμλ
Όλ¬Έ(λ°μ¬)--μμΈλνκ΅ λνμ :곡과λν μ»΄ν¨ν°κ³΅νλΆ,2020. 2. μ΄μꡬ.Recent advances in generation capability of deep learning models have spurred interest in utilizing deep generative models for unsupervised generative data augmentation (GDA). Generative data augmentation aims to improve the performance of a downstream machine learning model by augmenting the original dataset with samples generated from a deep latent variable model. This data augmentation approach is attractive to the natural language processing community, because (1) there is a shortage of text augmentation techniques that require little supervision and (2) resource scarcity being prevalent. In this dissertation, we explore the feasibility of exploiting deep latent variable models for data augmentation on three NLP tasks: sentence classification, spoken language understanding (SLU) and dialogue state tracking (DST), represent NLP tasks of various complexities and properties -- SLU requires multi-task learning of text classification and sequence tagging, while DST requires the understanding of hierarchical and recurrent data structures. For each of the three tasks, we propose a task-specific latent variable model based on conditional, hierarchical and sequential variational autoencoders (VAE) for multi-modal joint modeling of linguistic features and the relevant annotations. We conduct extensive experiments to statistically justify our hypothesis that deep generative data augmentation is beneficial for all subject tasks. Our experiments show that deep generative data augmentation is effective for the select tasks, supporting the idea that the technique can potentially be utilized for other range of NLP tasks. Ablation and qualitative studies reveal deeper insight into the underlying mechanisms of generative data augmentation. As a secondary contribution, we also shed light onto the recurring posterior collapse phenomenon in autoregressive VAEs and, subsequently, propose novel techniques to reduce the model risk, which is crucial for proper training of complex VAE models, enabling them to synthesize better samples for data augmentation. In summary, this work intends to demonstrate and analyze the effectiveness of unsupervised generative data augmentation in NLP. Ultimately, our approach enables standardized adoption of generative data augmentation, which can be applied orthogonally to existing regularization techniques.μ΅κ·Ό λ₯λ¬λ κΈ°λ° μμ± λͺ¨λΈμ κΈκ²©ν λ°μ μΌλ‘ μ΄λ₯Ό μ΄μ©ν μμ± κΈ°λ° λ°μ΄ν° μ¦κ° κΈ°λ²(generative data augmentation, GDA)μ μ€ν κ°λ₯μ±μ λν κΈ°λκ° μ»€μ§κ³ μλ€. μμ± κΈ°λ° λ°μ΄ν° μ¦κ° κΈ°λ²μ λ₯λ¬λ κΈ°λ° μ μ¬λ³μ λͺ¨λΈμμ μμ± λ μνμ μλ³Έ λ°μ΄ν°μ
μ μΆκ°νμ¬ μ°κ΄λ νμ€ν¬μ μ±λ₯μ ν₯μμν€λ κΈ°μ μ μλ―Ένλ€. λ°λΌμ μμ± κΈ°λ° λ°μ΄ν° μ¦κ° κΈ°λ²μ λ°μ΄ν° 곡κ°μμ μ΄λ€μ§λ μ κ·ν κΈ°μ μ ν ννλ‘ κ°μ£Όλ μ μλ€. μ΄λ¬ν λ₯λ¬λ κΈ°λ° μμ± λͺ¨λΈμ μλ‘μ΄ νμ© κ°λ₯μ±μ μμ°μ΄μ²λ¦¬ λΆμΌμμ λμ± μ€μνκ² λΆκ°λλ μ΄μ λ (1) λ²μ© κ°λ₯ν ν
μ€νΈ λ°μ΄ν° μ¦κ° κΈ°μ μ λΆμ¬μ (2) ν
μ€νΈ λ°μ΄ν°μ ν¬μμ±μ 극볡ν μ μλ λμμ΄ νμνκΈ° λλ¬Έμ΄λ€. λ¬Έμ μ 볡μ‘λμ νΉμ§μ κ³¨κ³ λ£¨ μ±μ§νκΈ° μν΄ λ³Έ λ
Όλ¬Έμμλ ν
μ€νΈ λΆλ₯(text classification), μμ°¨μ λ μ΄λΈλ§κ³Ό λ©ν°νμ€νΉ κΈ°μ μ΄ νμν λ°ν μ΄ν΄(spoken language understanding, SLU), κ³μΈ΅μ μ΄λ©° μ¬κ·μ μΈ λ°μ΄ν° ꡬ쑰μ λν κ³ λ €κ° νμν λν μν μΆμ (dialogue state tracking, DST) λ± μΈ κ°μ§ λ¬Έμ μμ λ₯λ¬λ κΈ°λ° μμ± λͺ¨λΈμ νμ©ν λ°μ΄ν° μ¦κ° κΈ°λ²μ νλΉμ±μ λν΄ λ€λ£¬λ€. λ³Έ μ°κ΅¬μμλ 쑰건λΆ, κ³μΈ΅μ λ° μμ°¨μ variational autoencoder (VAE)μ κΈ°λ°νμ¬ κ° μμ°μ΄μ²λ¦¬ λ¬Έμ μ νΉνλ ν
μ€νΈ λ° μ°κ΄ λΆμ°© μ 보λ₯Ό λμμ μμ±νλ νΉμ λ₯λ¬λ μμ± λͺ¨λΈλ€μ μ μνκ³ , λ€μν νλ₯ λͺ¨λΈκ³Ό λ°μ΄ν°μ
μ λ€λ£¨λ λ± ν λμ μ€νμ ν΅ν΄ λ₯ μμ± λͺ¨λΈ κΈ°λ° λ°μ΄ν° μ¦κ° κΈ°λ²μ ν¨κ³Όλ₯Ό ν΅κ³μ μΌλ‘ μ
μ¦νμλ€. λΆμμ μ°κ΅¬μμλ μκΈ°νκ·μ (autoregressive) VAEμμ λΉλ²ν λ°μνλ posterior collapse λ¬Έμ μ λν΄ νꡬνκ³ , ν΄λΉ λ¬Έμ λ₯Ό μνν μ μλ μ κ· λ°©μλ μ μνλ€. ν΄λΉ λ°©λ²μ μμ±μ λ°μ΄ν° μ¦κ°μ νμν 볡μ‘ν VAE λͺ¨λΈμ μ μ©νμμ λ, μμ± λͺ¨λΈμ μμ± μ§μ΄ ν₯μλμ΄ λ°μ΄ν° μ¦κ° ν¨κ³Όμλ κΈμ μ μΈ μν₯μ λ―ΈμΉ μ μμμ κ²μ¦νμλ€. λ³Έ λ
Όλ¬Έμ ν΅ν΄ μμ°μ΄μ²λ¦¬ λΆμΌμμ κΈ°μ‘΄ μ κ·ν κΈ°λ²κ³Ό λ³ν μ μ© κ°λ₯ν λΉμ§λ ννμ λ°μ΄ν° μ¦κ° κΈ°λ²μ νμ€νλ₯Ό κΈ°λν΄ λ³Ό μ μλ€.1 Introduction 1
1.1 Motivation 1
1.2 Dissertation Overview 6
2 Background and Related Work 8
2.1 Deep Latent Variable Models 8
2.1.1 Variational Autoencoder (VAE) 10
2.1.2 Deep Generative Models and Text Generation 12
2.2 Data Augmentation 12
2.2.1 General Description 13
2.2.2 Categorization of Data Augmentation 14
2.2.3 Theoretical Explanations 21
2.3 Summary 24
3 Basic Task: Text Classi cation 25
3.1 Introduction 25
3.2 Our Approach 28
3.2.1 Proposed Models 28
3.2.2 Training with I-VAE 29
3.3 Experiments 31
3.3.1 Datasets 32
3.3.2 Experimental Settings 33
3.3.3 Implementation Details 34
3.3.4 Data Augmentation Results 36
3.3.5 Ablation Studies 39
3.3.6 Qualitative Analysis 40
3.4 Summary 45
4 Multi-task Learning: Spoken Language Understanding 46
4.1 Introduction 46
4.2 Related Work 48
4.3 Model Description 48
4.3.1 Framework Formulation 48
4.3.2 Joint Generative Model 49
4.4 Experiments 56
4.4.1 Datasets 56
4.4.2 Experimental Settings 57
4.4.3 Generative Data Augmentation Results 61
4.4.4 Comparison to Other State-of-the-art Results 63
4.4.5 Ablation Studies 63
4.5 Summary 67
5 Complex Data: Dialogue State Tracking 68
5.1 Introduction 68
5.2 Background and Related Work 70
5.2.1 Task-oriented Dialogue 70
5.2.2 Dialogue State Tracking 72
5.2.3 Conversation Modeling 72
5.3 Variational Hierarchical Dialogue Autoencoder (VHDA) 73
5.3.1 Notations 73
5.3.2 Variational Hierarchical Conversational RNN 74
5.3.3 Proposed Model 75
5.3.4 Posterior Collapse 82
5.4 Experimental Results 84
5.4.1 Experimental Settings 84
5.4.2 Data Augmentation Results 90
5.4.3 Intrinsic Evaluation - Language Evaluation 94
5.4.4 Qualitative Results 95
5.5 Summary 101
6 Conclusion 103
6.1 Summary 103
6.2 Limitations 104
6.3 Future Work 105Docto
On the Combination of Game-Theoretic Learning and Multi Model Adaptive Filters
This paper casts coordination of a team of robots within the framework of game theoretic learning algorithms. In particular a novel variant of fictitious play is proposed, by considering multi-model adaptive filters as a method to estimate other playersβ strategies. The proposed algorithm can be used as a coordination mechanism between players when they should take decisions under uncertainty. Each player chooses an action after taking into account the actions of the other players and also the uncertainty. Uncertainty can occur either in terms of noisy observations or various types of other players. In addition, in contrast to other game-theoretic and heuristic algorithms for distributed optimisation, it is not necessary to find the optimal parameters a priori. Various parameter values can be used initially as inputs to different models. Therefore, the resulting decisions will be aggregate results of all the parameter values. Simulations are used to test the performance of the proposed methodology against other game-theoretic learning algorithms.</p