Open House, ISM in Tachikawa, 2017.6.16統計数理研究所オープンハウス（立川）、H29.6.16ポスター発

Tomoko Matsui

松井 知子

English

RISM (Repository of the Institute of Statistical Mathematics) / 統計数理研究所 学術研究リポジトリ

統計的機械学習による音声認識研究松井 知子 モデリング研究系 教授2017年6月16日 統計数理研究所 オープンハウス本研究室では統計的学習機械を用いて、音声/音楽/画像/SNSなどを処理する方法について研究しています。具体的にはカーネルマシン、ブースティング、協調フィルタリングの手法を用いて、1. 音声・話者認識2. 音楽情報処理3. 画像識別4. SNS解析5. WEBユーザビリティ評価6. 都市インテリジェンス などの研究課題に取り組んでいます。カーネルマシンブースティング協調フィルタリング都市インテリジェンス画像情報処理音情報処理SNS• 統計科学を用いて、– データから、内在する数学的な構造を発見する。– その数学的な構造に基づいて、予測や判別などの情報処理を行う。• 帰納的アプローチv.s. • 自然科学でよく見られる演繹的アプローチ– 仮説をたて、推論し、実験的または理論的に検証する。• カーネルマシン– 自動的な特徴（/モデル）選択機構を含む。– 非線形の扱いに優れている。– サポートベクターマシン（SVM）、罰金付ロジスティック回帰マシン• いろいろな確率モデルによる方法– 混合ガウス分布モデル– 隠れマルコフモデル• ガウス過程状態空間モデル など本研究室では統計的機械学習とその応用研究に興味のある学生さんを募集しています！【Generalized distillation frameworkを用いた音声認識】【概要】【統計的機械学習】共同研究者： Konstantin Markov (会津大学）Generalized Distillation or 	teacheror	studentstudentHinton’s Distillation▪ Training many different models on same training data:▪ Improves the performance, but▪ Makes the whole model big and unsuitable in practice.▪ How to train single, small model with simlar performance?▪ Use the output of the big model as “soft” targets for the small model – Model Compression (Caruana, 2006).▪ When references, i.e. “hard” targets, are available (Hinton, 2015):▪ Combine the “soft” and “hard” targets and control the softness of the “soft” targets.▪ The big model is called teacher and the small one – student.▪ Given the c-class classification task with training data { "# , %# }#'() ~+) ", % ,			"#∈ ℝ/, %# ∈ ℚ1, where ℚ1 is a space of c-dimensional probability vectors, teacher training is to find:23 = arg	min;∈ℱ= () ∑ ? %# , @ 2 "# + Ω( 2)#'( )where @() is a softmax, ?() is the loss, and Ω() is a regularizer.▪ Then, for the student we have:2E = arg	min;∈ℱF () ∑ 1 − I ? %# ,@ 2 "# + I? J# , @ 2 "#)#'(where J# = @(;= KLM ) ∈ ℚ1 and N > 0 controls the smoothness.Vapnik’s Privileged Information ▪ Often during training some additional information is available which is not accessible during test time. Given training data{ "#,"#∗, &# }#()* ~,* ", "#∗, & 	▪ How to leverage this information to make better model?▪ The naïve way – estimate the mapping " .→"∗ and generate "∗during testing.▪ Vapnik’s way (restricted to SVMs):▪ Similarity Control (Vapnik, 2009). Implemented in SVM+ objective. ▪ Knowledge transfer (Vapnik, 2015). Train 01 on { "#∗, &# }#()* and use it during the training of 02 on { "#, &# }#()* .Generalized Distillation ▪ Combination of Hinton’s distillation and Vapnik’s privileged information approaches (Lopez-Pas, 2016).▪ Three step process. Given training data { "#,"#∗, &# }#()*1. Learn teacher +, ∈ ℱ, using { "#∗, &# }#()* ;2. Compute teacher “soft” labels /# = 1 23 45∗6 for some temperature 7;3. Learn student +8 ∈ ℱ8 using both { "# ,/# }#()* and { "# ,&# }#()* , distillation objective and imitation parameter 9 ∈ 0,1 .▪ Generalized distillation reduces to:▪ Hinton’s distillation when "# = "#∗.▪ Vapnik’s method when "#∗ is privileged description of "#.Application in Speech Recognition▪ Features for ASR:▪ Spectrum based – MFCC, FBANK, etc.▪ Main features, widely used.▪ Easy to obtain.▪ Highly variable.▪ Affected by noise, etc.▪ Articulatory movements based.▪ Not affected by noise.▪ Less variable.▪ Difficult to obtain – EMA, X-rays, MRI.▪ Impractical for real time ASR.Articulatory and Spectrum Feature Integration ▪ Feature based.▪ Articulatory Inversion.▪ Most popular approach.▪ Mapping with various regression techniques.▪ Model based.▪ More difficult.▪ HMM/BN (Markov, 2006)▪ Generalized Distillation (this work).ACOARTModelACOTraining TestingART*map generateGMM-HMM versus DNN-HMM AMs DNN distillationtrainingExperiments ▪ Database.▪ University of Wisconsin X-ray micro-beam database (XRMB).▪ Consists of simultaneously recorded acoustic and articulatory measurements from 47 American English speakers.▪ Features▪ Acoustic – MFCC (39 dim.)▪ Articulatory – Displacement of 8 articulatory points (16 dim.)▪ All feature vectors normalized and synchronized.▪ Training procedure1. Train conventional GMM-HMM model using both acoustic and articulatoryfeatures. 2. Perform forced alignment to obtain DNN “hard” targets.3. Train teacher DNN using both acoustic and articulatory features.4. Train student DNN using acoustic features only and guided by the “teacher”.▪ Testing procedure1. Use student DNN with acoustic features only to obtain HMM state probabilities.2. Use standard HMM decoding (Viterbi) to obtain recognition result.▪ Evaluation metric – Phoneme Error Rate (PER)Results ▪ DNN parameters:▪ Feed Forward.▪ Input widow – 17 frames.▪ Activation – ReLU.▪ Dropout – 40%.▪ Teacher DNN▪ 5 layers▪ 3073 nodes.▪ Student DNN▪ 4 layers▪ 2048 nodes.18By DNN type: Feed Forwardvs. RecurrentBy integration type:Feature basedvs.Model based7.176.743.544.555.566.577.58Fea-based Model-basedFeed-Forward DNNPER5.826.113.544.555.566.577.58Fea-based Model-basedRecurrent DNNPERConclusions ▪ Generalized distillation:▪ Is an effective method for model based integration of information unavailable at testing time.▪ Allows smaller student models (4 layers / 2048 nodes) to reach performance close to bigger teacher models (5 layers / 3072 nodes).▪ DNN structure:▪ Recurrent DNNs outperform Feed-Forward DNNs in the ASR task since they better model long-term temporal dependencies.▪ Time complexity of Recurrent DNNs is higher than Feed-Forward DNNs.▪ Integration approach:▪ Model based and Feature based integration achieve comparable results.▪ Feature based integration requires higher computational power.Generalized Distillation[D. D. Lopez-Paz, L. Bottou, B. Scho ̈lkopf, and V. Vapnik, “Unifying distillation and privileged information,” ICLR, 2016] DNN Distillation Training and Testing Training Testing

統計的機械学習による音声認識研究

https://ismrepo.ism.ac.jp/?action=repository_action_common_download&item_id=33712&item_no=1&attribute_id=17&file_no=1

統計的機械学習による音声認識研究

Abstract

Similar works

Full text

Available Versions

RISM (Repository of the Institute of Statistical Mathematics) / 統計数理研究所学術研究リポジトリ

統計的機械学習による音声認識研究

Abstract

Similar works

Full text

Available Versions

RISM (Repository of the Institute of Statistical Mathematics) / 統計数理研究所 学術研究リポジトリ

RISM (Repository of the Institute of Statistical Mathematics) / 統計数理研究所学術研究リポジトリ