Contextual Language Model Adaptation for Conversational Agents

Statistical language models (LM) play a key role in Automatic Speech Recognition (ASR) systems used by conversational agents. These ASR systems should provide a high accuracy under a variety of speaking styles, domains, vocabulary and argots. In this paper, we present a DNN-based method to adapt the LM to each user-agent interaction based on generalized contextual information, by predicting an optimal, context-dependent set of LM interpolation weights. We show that this framework for contextual adaptation provides accuracy improvements under different possible mixture LM partitions that are relevant for both (1) Goal-oriented conversational agents where it's natural to partition the data by the requested application and for (2) Non-goal oriented conversational agents where the data can be partitioned using topic labels that come from predictions of a topic classifier. We obtain a relative WER improvement of 3% with a 1-pass decoding strategy and 6% in a 2-pass decoding framework, over an unadapted model. We also show up to a 15% relative improvement in recognizing named entities which is of significant value for conversational ASR systems.Comment: Interspeech 2018 (accepted

Movie Description

Audio Description (AD) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed ADs, which are temporally aligned to full length movies. In addition we also collected and aligned movie scripts used in prior work and compare the two sources of descriptions. In total the Large Scale Movie Description Challenge (LSMDC) contains a parallel corpus of 118,114 sentences and video clips from 202 movies. First we characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing ADs to scripts, we find that ADs are indeed more visual and describe precisely what is shown rather than what should happen according to the scripts created prior to movie production. Furthermore, we present and compare the results of several teams who participated in a challenge organized in the context of the workshop "Describing and Understanding Video & The Large Scale Movie Description Challenge (LSMDC)", at ICCV 2015

“¿Triste estás? I don’t know nan molla”: Multilingual pop song fandubs by @miree_music

Fandubbing, or dubbing made by fans of any audiovisual product, is a lin- guistically and technologically sophisticated enterprise enacted by many devoted fans. This study presents the case of Miree, a 24-year-old fandubber with more than 1 million subscribers on YouTube and more than 300 multi- lingual fandubbed songs. Using a qualitative-interpretive approach, we con- ducted an in-depth interview with Miree and analyzed her top 30 videos by views to reveal how Miree performed fandubbing, how she expressed her fan identity through fandubbing, and which were some of the implications of fandubbing for language learning. Results show that Miree realized both interlinguistic genuine fandubbing and intralinguistic parodic fandubbing, strategically adopting translanguaging to orchestrate a multimodal perfor- mance, engage her fanbase, and activate several informal language learning opportunities and contexts afforded by fandubbing.The study was partly supported by the publicly funded research project ForVid: Video as a language learning format in and outside the classroom (RT2018-100790-B-100; 2019–2021), ‘Research Challenges’ R+D+i Projects, Ministry of Science and Innovation, Spain, and by the Fundamental Research Funds for the Central Universities (2020QD036; China)

The development of ‘drama’ in YouTube discourse

This thesis presents a systematic discourse analysis of sustained antagonistic debate—called 'drama'—on the video-sharing website, YouTube. Following a two-year observation of a YouTube community of practice discussing Christianity and atheism, 20 video 'pages' (including talk from videos and text comments) from a drama event were identified and transcribed, producing a 86,859 word corpus comprising 136 minutes of video talk and 1,738 comments. Using metaphor-led discourse analysis (Cameron & Maslen, 2010b) of the total corpus, metaphor vehicles were identified, coded, and grouped by semantic and narrative relationships to identify systematic use and trace the development of discourse activity. Close discourse analysis of a subset of the corpus was then employed to investigate membership categorisation (Housley & Fitzgerald, 2002), impoliteness (Culpeper, 2011), and positioning (Harré & van Langenhove, 1998), providing a systematic description of different factors contributing to the emergence of 'drama'. Analysis shows that 'drama' developed when negative views of one user's impolite words exposed the different expectations of other users about acceptable YouTube interaction. Hyperbolic, metaphorical language derived from the Bible and narratives about tragic historical events often exaggerated, escalated, and extended negative evaluations of others. Categories like 'Christian' were used dynamically to connect impolite words and actions of individuals to social groups, thereby also extending negative evaluations. With implications for understanding 'flaming' and transgression of social norms in web 2.0 environments, this thesis concludes that inflammatory language led to 'drama' because: (1) users had diverse expectations about social interaction and organisation, (2) users drew upon the Bible's moral authority to support opposing actions, and (3) the online platform's technical features afforded immediate reactions to non-present others. The 'drama' then developed when users' responses to one another created both additional topics for antagonistic debate and more disagreement about which words and actions were acceptable

Semi-Supervised Acoustic Model Training by Discriminative Data Selection from Multiple ASR Systems' Hypotheses

While the performance of ASR systems depends on the size of the training data, it is very costly to prepare accurate and faithful transcripts. In this paper, we investigate a semisupervised training scheme, which takes the advantage of huge quantities of unlabeled video lecture archive, particularly for the deep neural network (DNN) acoustic model. In the proposed method, we obtain ASR hypotheses by complementary GMM-and DNN-based ASR systems. Then, a set of CRF-based classifiers is trained to select the correct hypotheses and verify the selected data. The proposed hypothesis combination shows higher quality compared with the conventional system combination method (ROVER). Moreover, compared with the conventional data selection based on confidence measure score, our method is demonstrated more effective for filtering usable data. Significant improvement in the ASR accuracy is achieved over the baseline system and in comparison with the models trained with the conventional system combination and data selection methods

Picture the Music: Performing Arts Library Planning with Photo Elicitation

Photo elicitation, a form of ethnographic journaling, provided insights into university music and dance student needs in library and campus spaces and services. In this case study, subjects took a photo for each of twenty prompts related to their daily lives as students and performing artists, then discussed their own photos in a one-hour individual interview. Researchers qualitatively analyzed the gathered data. This article reports findings related to: discovering and obtaining music and dance works, study spaces and sound levels, forces of habit and the implications for student library use, and library-related findings regarding practice rooms and classrooms

CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines

Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research

Streaming Automatic Speech Recognition with Hybrid Architectures and Deep Neural Network Models

Tesis por compendio[ES] Durante la última década, los medios de comunicación han experimentado una revolución, alejándose de la televisión convencional hacia las plataformas de contenido bajo demanda. Además, esta revolución no ha cambiado solamente la manera en la que nos entretenemos, si no también la manera en la que aprendemos. En este sentido, las plataformas de contenido educativo bajo demanda también han proliferado para proporcionar recursos educativos de diversos tipos. Estas nuevas vías de distribución de contenido han llegado con nuevos requisitos para mejorar la accesibilidad, en particular las relacionadas con las dificultades de audición y las barreras lingüísticas. Aquí radica la oportunidad para el reconocimiento automático del habla (RAH) para cumplir estos requisitos, proporcionando subtitulado automático de alta calidad. Este subtitulado proporciona una base sólida para reducir esta brecha de accesibilidad, especialmente para contenido en directo o streaming. Estos sistemas de streaming deben trabajar bajo estrictas condiciones de tiempo real, proporcionando la subtitulación tan rápido como sea posible, trabajando con un contexto limitado. Sin embargo, esta limitación puede conllevar una degradación de la calidad cuando se compara con los sistemas para contenido en diferido u offline. Esta tesis propone un sistema de RAH en streaming con baja latencia, con una calidad similar a un sistema offline. Concretamente, este trabajo describe el camino seguido desde el sistema offline híbrido inicial hasta el eficiente sistema final de reconocimiento en streaming. El primer paso es la adaptación del sistema para efectuar una sola iteración de reconocimiento haciendo uso de modelos de lenguaje estado del arte basados en redes neuronales. En los sistemas basados en múltiples iteraciones estos modelos son relegados a una segunda (o posterior) iteración por su gran coste computacional. Tras adaptar el modelo de lenguaje, el modelo acústico basado en redes neuronales también tiene que adaptarse para trabajar con un contexto limitado. La integración y la adaptación de estos modelos es ampliamente descrita en esta tesis, evaluando el sistema RAH resultante, completamente adaptado para streaming, en conjuntos de datos académicos extensamente utilizados y desafiantes tareas basadas en contenidos audiovisuales reales. Como resultado, el sistema proporciona bajas tasas de error con un reducido tiempo de respuesta, comparables al sistema offline.[CA] Durant l'última dècada, els mitjans de comunicació han experimentat una revolució, allunyant-se de la televisió convencional cap a les plataformes de contingut sota demanda. A més a més, aquesta revolució no ha canviat només la manera en la que ens entretenim, si no també la manera en la que aprenem. En aquest sentit, les plataformes de contingut educatiu sota demanda també han proliferat pera proporcionar recursos educatius de diversos tipus. Aquestes noves vies de distribució de contingut han arribat amb nous requisits per a millorar l'accessibilitat, en particular les relacionades amb les dificultats d'audició i les barreres lingüístiques. Aquí radica l'oportunitat per al reconeixement automàtic de la parla (RAH) per a complir aquests requisits, proporcionant subtitulat automàtic d'alta qualitat. Aquest subtitulat proporciona una base sòlida per a reduir aquesta bretxa d'accessibilitat, especialment per a contingut en directe o streaming. Aquests sistemes han de treballar sota estrictes condicions de temps real, proporcionant la subtitulació tan ràpid com sigui possible, treballant en un context limitat. Aquesta limitació, però, pot comportar una degradació de la qualitat quan es compara amb els sistemes per a contingut en diferit o offline. Aquesta tesi proposa un sistema de RAH en streaming amb baixa latència, amb una qualitat similar a un sistema offline. Concretament, aquest treball descriu el camí seguit des del sistema offline híbrid inicial fins l'eficient sistema final de reconeixement en streaming. El primer pas és l'adaptació del sistema per a efectuar una sola iteració de reconeixement fent servir els models de llenguatge de l'estat de l'art basat en xarxes neuronals. En els sistemes basats en múltiples iteracions aquests models son relegades a una segona (o posterior) iteració pel seu gran cost computacional. Un cop el model de llenguatge s'ha adaptat, el model acústic basat en xarxes neuronals també s'ha d'adaptar per a treballar amb un context limitat. La integració i l'adaptació d'aquests models és àmpliament descrita en aquesta tesi, avaluant el sistema RAH resultant, completament adaptat per streaming, en conjunts de dades acadèmiques àmpliament utilitzades i desafiants tasques basades en continguts audiovisuals reals. Com a resultat, el sistema proporciona baixes taxes d'error amb un reduït temps de resposta, comparables al sistema offline.[EN] Over the last decade, the media have experienced a revolution, turning away from the conventional TV in favor of on-demand platforms. In addition, this media revolution not only changed the way entertainment is conceived but also how learning is conducted. Indeed, on-demand educational platforms have also proliferated and are now providing educational resources on diverse topics. These new ways to distribute content have come along with requirements to improve accessibility, particularly related to hearing difficulties and language barriers. Here is the opportunity for automatic speech recognition (ASR) to comply with these requirements by providing high-quality automatic captioning. Automatic captioning provides a sound basis for diminishing the accessibility gap, especially for live or streaming content. To this end, streaming ASR must work under strict real-time conditions, providing captions as fast as possible, and working with limited context. However, this limited context usually leads to a quality degradation as compared to the pre-recorded or offline content. This thesis is aimed at developing low-latency streaming ASR with a quality similar to offline ASR. More precisely, it describes the path followed from an initial hybrid offline system to an efficient streaming-adapted system. The first step is to perform a single recognition pass using a state-of-the-art neural network-based language model. In conventional multi-pass systems, this model is often deferred to the second or later pass due to its computational complexity. As with the language model, the neural-based acoustic model is also properly adapted to work with limited context. The adaptation and integration of these models is thoroughly described and assessed using fully-fledged streaming systems on well-known academic and challenging real-world benchmarks. In brief, it is shown that the proposed adaptation of the language and acoustic models allows the streaming-adapted system to reach the accuracy of the initial offline system with low latency.Jorge Cano, J. (2022). Streaming Automatic Speech Recognition with Hybrid Architectures and Deep Neural Network Models [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/191001Compendi

Computer-Assisted Interpreting Tools (CAI) and options for automation with Automatic Speech Recognition

In recent years, several studies have indicated interpreters resist adopting new technologies. Yet, such technologies have enabled the development of several tools to help those professionals. In this paper, using bibliographical and documental research, we briefly analyse the tools cited by several authors to identify which ones remain up to date and available on the market. Following that, we present concepts about automation, and observe the usage of automatic speech recognition (ASR), while analysing its potential benefits and the current level of maturity of such an approach, especially regarding Computer-Assisted Interpreting (CAI) tools. The goal of this paper is to present the community of interpreters and researchers with a view of the state of the art in technology for interpreting as well as some future perspectives for this area

Complexity in Second Language Study Emotions

This book offers a socially situated view of the emergence of emotionality for additional language (L2) learners in classroom interaction in Japan. Grounded in a complexity perspective, the author argues that emotions need to be studied as they are dynamically experienced and understood in all of their multidimensional colors by individuals (in interaction). Via practitioner research, Sampson applies a small-lens focus, interweaving experiential and discursive data, offering possibilities for exploring, interpreting and representing the lived experience of L2 study emotions in a more holistic yet detailed, social yet individual fashion. Amidst the currently expanding interest in L2 study emotions, the book presents a strong case for the benefits of locating interpretations of the emergence of L2 study emotions back into situated, dynamic, social context. Sampson’s work will be of interest to students and researchers in second language acquisition and L2 learning psychology