181 research outputs found
Contextual Phonetic Pretraining for End-to-end Utterance-level Language and Speaker Recognition
Pretrained contextual word representations in NLP have greatly improved
performance on various downstream tasks. For speech, we propose contextual
frame representations that capture phonetic information at the acoustic frame
level and can be used for utterance-level language, speaker, and speech
recognition. These representations come from the frame-wise intermediate
representations of an end-to-end, self-attentive ASR model (SAN-CTC) on spoken
utterances. We first train the model on the Fisher English corpus with
context-independent phoneme labels, then use its representations at inference
time as features for task-specific models on the NIST LRE07 closed-set language
recognition task and a Fisher speaker recognition task, giving significant
improvements over the state-of-the-art on both (e.g., language EER of 4.68% on
3sec utterances, 23% relative reduction in speaker EER). Results remain
competitive when using a novel dilated convolutional model for language
recognition, or when ASR pretraining is done with character labels only.Comment: submitted to INTERSPEECH 201
Multitask Learning from Augmented Auxiliary Data for Improving Speech Emotion Recognition
Despite the recent progress in speech emotion recognition (SER),
state-of-the-art systems lack generalisation across different conditions. A key
underlying reason for poor generalisation is the scarcity of emotion datasets,
which is a significant roadblock to designing robust machine learning (ML)
models. Recent works in SER focus on utilising multitask learning (MTL) methods
to improve generalisation by learning shared representations. However, most of
these studies propose MTL solutions with the requirement of meta labels for
auxiliary tasks, which limits the training of SER systems. This paper proposes
an MTL framework (MTL-AUG) that learns generalised representations from
augmented data. We utilise augmentation-type classification and unsupervised
reconstruction as auxiliary tasks, which allow training SER systems on
augmented data without requiring any meta labels for auxiliary tasks. The
semi-supervised nature of MTL-AUG allows for the exploitation of the abundant
unlabelled data to further boost the performance of SER. We comprehensively
evaluate the proposed framework in the following settings: (1) within corpus,
(2) cross-corpus and cross-language, (3) noisy speech, (4) and adversarial
attacks. Our evaluations using the widely used IEMOCAP, MSP-IMPROV, and EMODB
datasets show improved results compared to existing state-of-the-art methods.Comment: Under review IEEE Transactions on Affective Computin
Syväoppiminen puhutun kielen tunnistamisessa
This thesis applies deep learning based classification techniques to identify natural languages from speech. The primary motivation behind this thesis is to implement accurate techniques for segmenting multimedia materials by the languages spoken in them.
Several existing state-of-the-art, deep learning based approaches are discussed and a subset of the discussed approaches are selected for quantitative experimentation. The selected model architectures are trained on several well-known spoken language identification datasets containing several different languages. Segmentation granularity varies between models, some supporting input audio lengths of 0.2 seconds, while others require 10 second long input to make a language decision.
Results from the thesis experiments show that an unsupervised representation of acoustic units, produced by a deep sequence-to-sequence auto encoder, cannot reach the language identification performance of a supervised representation, produced by a multilingual phoneme recognizer. Contrary to most existing results, in this thesis, acoustic-phonetic language classifiers trained on labeled spectral representations outperform phonotactic classifiers trained on bottleneck features of a multilingual phoneme recognizer. More work is required, using transcribed datasets and automatic speech recognition techniques, to investigate why phoneme embeddings did not outperform simple, labeled spectral features.
While an accurate online language segmentation tool for multimedia materials could not be constructed, the work completed in this thesis provides several insights for building feasible, modern spoken language identification systems. As a side-product of the experiments performed during this thesis, a free open source spoken language identification software library called "lidbox" was developed, allowing future experiments to begin where the experiments of this thesis end.Tämä diplomityö keskittyy soveltamaan syviä neuroverkkomalleja luonnollisten kielien automaattiseen tunnistamiseen puheesta. Tämän työn ensisijainen tavoite on toteuttaa tarkka menetelmä multimediamateriaalien ositteluun niissä esiintyvien puhuttujen kielien perusteella.
Työssä tarkastellaan useampaa jo olemassa olevaa neuroverkkoihin perustuvaa lähestymistapaa, joista valitaan alijoukko tarkempaan tarkasteluun, kvantitatiivisten kokeiden suorittamiseksi. Valitut malliarkkitehtuurit koulutetaan käyttäen eri puhetietokantoja, sisältäen useampia eri kieliä. Kieliosittelun hienojakoisuus vaihtelee käytettyjen mallien mukaan, 0,2 sekunnista 10 sekuntiin, riippuen kuinka pitkän aikaikkunan perusteella malli pystyy tuottamaan kieliennusteen.
Diplomityön aikana suoritetut kokeet osoittavat, että sekvenssiautoenkoodaajalla ohjaamattomasti löydetty puheen diskreetti akustinen esitysmuoto ei ole riittävä kielen tunnistamista varten, verrattuna foneemitunnistimen tuottamaan, ohjatusti opetettuun foneemiesitysmuotoon. Tässä työssä havaittiin, että akustisfoneettiset kielentunnistusmallit saavuttavat korkeamman kielentunnistustarkkuuden kuin foneemiesitysmuotoa käyttävät kielentunnistusmallit, mikä eroaa monista kirjallisuudessa esitetyistä tuloksista. Diplomityön tutkimuksia on jatkettava, esimerkiksi litteroituja puhetietokantoja ja puheentunnistusmenetelmiä käyttäen, jotta pystyttäisiin selittämään miksi foneemimallin tuottamalla esitysmuodolla ei saatu parempia tuloksia kuin yksinkertaisemmalla, taajuusspektrin esitysmuodolla.
Tämän työn aikana puhutun kielen tunnistaminen osoittautui huomattavasti haasteellisemmaksi kuin mitä työn alussa oli arvioitu, eikä työn aikana onnistuttu toteuttamaan tarpeeksi tarkkaa multimediamateriaalien kielienosittelumenetelmää. Tästä huolimatta, työssä esitetyt lähestymistavat tarjoavat toimivia käytännön menetelmiä puhutun kielen tunnistamiseen tarkoitettujen, modernien järjestelmien rakentamiseksi. Tämän diplomityön sivutuotteena syntyi myös puhutun kielen tunnistamiseen tarkoitettu avoimen lähdekoodin kirjasto nimeltä "lidbox", jonka ansiosta tämän työn kvantitatiivisia kokeita voi jatkaa siitä, mihin ne tämän työn päätteeksi jäivät
An Efficient Temporary Deepfake Location Approach Based Embeddings for Partially Spoofed Audio Detection
Partially spoofed audio detection is a challenging task, lying in the need to
accurately locate the authenticity of audio at the frame level. To address this
issue, we propose a fine-grained partially spoofed audio detection method,
namely Temporal Deepfake Location (TDL), which can effectively capture
information of both features and locations. Specifically, our approach involves
two novel parts: embedding similarity module and temporal convolution
operation. To enhance the identification between the real and fake features,
the embedding similarity module is designed to generate an embedding space that
can separate the real frames from fake frames. To effectively concentrate on
the position information, temporal convolution operation is proposed to
calculate the frame-specific similarities among neighboring frames, and
dynamically select informative neighbors to convolution. Extensive experiments
show that our method outperform baseline models in ASVspoof2019 Partial Spoof
dataset and demonstrate superior performance even in the crossdataset scenario.
The code is released online.Comment: Submitted to ICASSP 202
- …