Acoustic characterization of speech rhythm: going beyond metrics with recurrent neural networks

Languages have long been described according to their perceived rhythmic attributes. The associated typologies are of interest in psycholinguistics as they partly predict newborns' abilities to discriminate between languages and provide insights into how adult listeners process non-native languages. Despite the relative success of rhythm metrics in supporting the existence of linguistic rhythmic classes, quantitative studies have yet to capture the full complexity of temporal regularities associated with speech rhythm. We argue that deep learning offers a powerful pattern-recognition approach to advance the characterization of the acoustic bases of speech rhythm. To explore this hypothesis, we trained a medium-sized recurrent neural network on a language identification task over a large database of speech recordings in 21 languages. The network had access to the amplitude envelopes and a variable identifying the voiced segments, assuming that this signal would poorly convey phonetic information but preserve prosodic features. The network was able to identify the language of 10-second recordings in 40% of the cases, and the language was in the top-3 guesses in two-thirds of the cases. Visualization methods show that representations built from the network activations are consistent with speech rhythm typologies, although the resulting maps are more complex than two separated clusters between stress and syllable-timed languages. We further analyzed the model by identifying correlations between network activations and known speech rhythm metrics. The findings illustrate the potential of deep learning tools to advance our understanding of speech rhythm through the identification and exploration of linguistically relevant acoustic feature spaces.Comment: 15 pages, 7 figure

An exploration of the rhythm of Malay

In recent years there has been a surge of interest in speech rhythm. However we still lack a clear understanding of the nature of rhythm and rhythmic differences across languages. Various metrics have been proposed as means for measuring rhythm on the phonetic level and making typological comparisons between languages (Ramus et al, 1999; Grabe & Low, 2002; Dellwo, 2006) but the debate is ongoing on the extent to which these metrics capture the rhythmic basis of speech (Arvaniti, 2009; Fletcher, in press). Furthermore, cross linguistic studies of rhythm have covered a relatively small number of languages and research on previously unclassified languages is necessary to fully develop the typology of rhythm. This study examines the rhythmic features of Malay, for which, to date, relatively little work has been carried out on aspects rhythm and timing. The material for the analysis comprised 10 sentences produced by 20 speakers of standard Malay (10 males and 10 females). The recordings were first analysed using rhythm metrics proposed by Ramus et. al (1999) and Grabe & Low (2002). These metrics (∆C, %V, rPVI, nPVI) are based on durational measurements of vocalic and consonantal intervals. The results indicated that Malay clustered with other so-called syllable-timed languages like French and Spanish on the basis of all metrics. However, underlying the overall findings for these metrics there was a large degree of variability in values across speakers and sentences, with some speakers having values in the range typical of stressed-timed languages like English. Further analysis has been carried out in light of Fletcher’s (in press) argument that measurements based on duration do not wholly reflect speech rhythm as there are many other factors that can influence values of consonantal and vocalic intervals, and Arvaniti’s (2009) suggestion that other features of speech should also be considered in description of rhythm to discover what contributes to listeners’ perception of regularity. Spectrographic analysis of the Malay recordings brought to light two parameters that displayed consistency and regularity for all speakers and sentences: the duration of individual vowels and the duration of intervals between intensity minima. This poster presents the results of these investigations and points to connections between the features which seem to be consistently regulated in the timing of Malay connected speech and aspects of Malay phonology. The results are discussed in light of current debate on the descriptions of rhythm

Vocal Synthetics: Designing for an Adaptable Singing Synthesizer

Technological music tools such as digital audio workstations and electronic music instruments have enabled musicians without formal training to create music that is heard by millions of people. The automation by software and hardware can create compelling productions without limitations from performance ability. However, the automation of vocals is particularly difficult because beyond pitch and timbre, the vocalization of language requires additional parameters for control. As the production of a vocal synthesizer and its vocal palettes is complex, the current market sees these difficulties represented through products that have limited voices and do not adapt to vocal trends. This project demonstrates a tool that allows producers to use a simple typing interface for the input of words, allowing the output to be integrated and controlled by modern digital audio workstations. Using a machine learning solution, the tool is not dependent on large stores of audio data once a model is trained and since it contains a simple method to create new voices, it can keep up with evolving musical trends and vocal styles. The aim is to bring the human voice into the realm of digital music production enabling a music maker to include a large range of vocal styles within their production tool set. This paper outlines the design and development of the tool and culminates in a piece of music that illustrates the value of applying design thinking research strategies to an artistic and technical challenge

Contribution of prosodic timing patterns into perceived foreign accent

Polyanskaya L. Contribution of prosodic timing patterns into perceived foreign accent. Bielefeld: Universitätsbibliothek Bielefeld; 2015.Wenn Menschen eine Fremdsprache lernen, verbessert sich mit den allgemeinen Fortschritten in deren Beherrschung auch die Kontrolle der Timingmechanismen beim Sprechen in der Fremdsprache. Ziel dieser Arbeit ist es herauszufinden, ob diese Veränderungen perzeptiv relevant sind, und ob die verbesserte Kontrolle der Timingmechanismen bei fortgeschrittenen Sprachlernern deren wahrgenommenen Akzent reduziert. Sprachspezifische Unterschiede in prosodischen Timingmustern sind gut dokumentiert. So weisen etwa die Dauern von vokalischen und konsonantischen Intervallen in den Sprachen, die traditionell als betonungszählend klassifiziert werden, eine höhere Variabilität auf als in Sprachen, die traditionell als silbenzählend klassifiziert werden. Silbenzählende Sprachen weisen außerdem eine höhere Sprechrate auf als betonungszählende Sprachen. Darüber hinaus zeigen Untersuchungen zu verschiedenen Sprachen, dass Nichtmuttersprachler eine geringere Sprechrate und niedrigere Variabilität im Timing gesprochener Äußerungen aufweisen als Muttersprachler. Diese Unterschiede beeinflussen die Verständlichkeit von gesprochenen Äußerungen von Nichtmuttersprachlern sowie deren wahrgenommenen fremdsprachlichen Akzent (FA). Allerdings sind die Geschwindigkeit – gemessen in sprachlichen Intervallen pro Zeiteinheit – und die Variabilität der Dauern dieser Intervalle in gesprochenen Äußerungen miteinander korreliert: Je höher die Sprechgeschwindigkeit ist, desto geringer ist die Variabilität der Intervalldauern. Dies wirkt sich auch in der Wahrnehmung aus. Daraus ergibt sich die Frage, in welchem Maß beide Faktoren zur Wahrnehmung eines FA bei Nichtmuttersprachlern beitragen. Um diese Frage zu beantworten, müssen beide Faktoren isoliert betrachtet werden. Tempo und Timingvariabilität beim Sprechen einer Fremdsprache erhöhen sich im Verlauf von deren Erwerb, unabhängig davon, ob sich Mutter- und Fremdsprache (im Folgenden: L1 und L2) hinsichtlich ihrer Timingcharakteristika unterscheiden. Der Grad dieser Veränderung sollte folglich auch die Stärke des wahrgenommenen FA reflektieren. Wenn die Timingunterschiede perzeptiv relevant sind, sollten Äußerungen, deren Timingmuster der eines fortgeschrittenen L2-Lerners entsprechen, als schwächer akzentuiert wahrgenommen werden als solche, deren Timingmuster denen eines Anfängers entsprechen, auch wenn spektrale und intonatorische Unterschiede eliminiert werden. Dabei wird die Frage zu klären sein, in welchem Maß die beiden Faktoren Tempo und Timingvariabilität den wahrgenommenen FA beeinflussen. Grundannahme dieser Arbeit ist, dass der Einfluss der Variabilität geringer ist, wenn L1 und L2 ähnliche Timingcharakteristika haben. In dieser Arbeit werden die Timingmuster von deutschen und französischen Lernern des Englischen hinsichtlich ihres Einflusses auf den wahrgenommenen FA untersucht, wobei zusätzlich jeweils Anfänger und fortgeschrittene Lerner getestet werden. Die Timingcharakteristika des Deutschen ähneln denen des Englischen, während sich das Französische in dieser Hinsicht deutlich vom Englischen unterscheidet. Daraus ergeben sich zwei Hypothesen: (1) Im Englischen fortgeschrittener deutscher Lerner (gegenüber Anfängern) äußert sich die Reduktion des wahrgenommenen FA stärker in einer Erhöhung der Sprechrate; (2) Bei französischen Englischlernern spielt die Timingvariabilität eine größere Rolle als die Veränderung der Sprechrate im Verlauf des Spracherwerbs. Diese Hypothesen wurden anhand von vier Forschungsfragen überprüft: 1. Nehmen Muttersprachler der Zielsprache (Englisch) Unterschiede im Timing gesprochener Äußerungen zwischen Anfängern und fortgeschrittenen Englischlernern wahr? 2. Korreliert die Reduktion des wahrgenommenen FA mit den Veränderungen der Timingmuster im Verlauf des L2-Erwerbs? 3. Welche Anteile haben die einzelnen Faktoren Sprechtempo und Timingvariabilität am wahrgenommenen FA? 4. Zeigen sich hinsichtlich der separaten Anteile von Sprechtempo und Timingvariabilität am wahrgenommenen FA Unterschiede zwischen Lernern mit typologisch unterschiedlichen Muttersprachen? In dieser Arbeit wird über die Ergebnisse von drei Wahrnehmungsexperimenten berichtet, die zur Beantwortung der Forschungsfragen durchgeführt wurden. Die Arbeit ist wie folgt strukturiert: Im ersten Kapitel werden der theoretische Hintergrund vorgestellt und die Arbeitshypothesen erläutert. Das Kapitel beginnt mit einer Definition des Begriffs „FA“ und einer Diskussion der wichtigsten Faktoren, die zur Wahrnehmung des FA beitragen. Dabei wird auch ein kurzer Überblick über Modelle des Zweitspracherwerbs gegeben. Weiterhin werden segmentale und prosodische Unterschiede zwischen L1 und L2 und deren Einfluss auf den wahrgenommenen FA diskutiert, und es wird erörtert, wie diese Unterschiede in verschiedenen Modellen des Zweitspracherwerbs erklärt werden. Zudem wird es auch auf die Frage eingegangen, ob segmentale oder prosodische Faktoren einen größeren Einfluss auf den wahrgenommenen FA haben. Schließlich folgt eine Diskussion des Prosodiebegriffs, unter Einbeziehung der Subsysteme Betonung, Intonation und Timing. Im Besonderen wird auf Timingmuster eingegangen, wobei der Fokus darauf liegt, wie Timing in gesprochener Sprache gemessen werden kann, wie Sprechrate und Timingvariabilität zusammenhängen, und wie Timingunterschiede zwischen Muttersprachlern und Sprachlernern die Wahrnehmung von FA beeinflussen. Anschließend an diese Diskussion wird die oben eingeführten Arbeitshypothesen formuliert und motiviert. In den Kapiteln 3, 4 und 5 werden die einzelnen Wahrnehmungsexperimente beschrieben, im letzten Kapitel zusammengefasst und diskutiert. Die Ergebnisse der Experimente stützen die oben eingeführte Hypothese und können wie folgt zusammengefasst werden: 1. Muttersprachler nehmen die Unterschiede zwischen den Timingmustern in den L2-Produktionen fortgeschrittener und weniger fortgeschrittener Sprachlerner wahr. Sie tendieren jedoch dazu, Unterschiede in der Variabilität in Klassifikationsaufgaben und in nichtsprachlichen Stimuli zu ignorieren. Je natürlicher und sprachähnlicher die Stimuli sind, desto stärker werden Unterschiede in der Timingvariabilität wahrgenommen. 2. Die Stärke des wahrgenommenen FA korreliert, wie vorhergesagt, mit Veränderungen in Sprechtempo und Timingvariabilität, die mit steigender Kompetenz in der L2 einhergehen. Fortgeschrittenere Sprecher sprechen schneller und mit höherer Variabilität von sowohl Vokal- als auch Silbendauern. Äußerungen mit höherem Sprechtempo und höherer Timingvariabilität werden von Muttersprachlern des Englischen als weniger stark akzentuiert wahrgenommen. 3. Der kombinierte Beitrag von Sprechtempo und Timingvariabilität zum wahrgenommenen FA ist größer als die Summe der Effekte beider Faktoren in Isolation. Experimente, in denen jeweils einer der beiden Faktoren kontrolliert wird, zeigen, dass beide zum wahrgenommenen FA beitragen. 4. Die relative Gewichtung beider Faktoren hängt davon ab, ob L1 und L2 hinsichtlich ihrer Timingcharakteristika ähnlich oder verschieden sind. Wenn sich L1 und L2, wie im Fall von Französich und Englisch, stark unterscheiden, ist der Beitrag der Variabilität größer; wenn sich L1 und L2 hinsichtlich ihrer Timingcharakteristika ähneln – wie im Fall von Deutsch und Englisch – spielt das Sprechtempo für die Wahrnehmung des FA die wichtigere Rolle

Auditory-Motor Rhythms and Speech Processing in French and German Listeners

Moving to a speech rhythm can enhance verbal processing in the listener by increasing temporal expectancies (Falk and Dalla Bella, 2016). Here we tested whether this hypothesis holds for prosodically diverse languages such as German (a lexical stress-language) and French (a non-stress language). Moreover, we examined the relation between motor performance and the benefits for verbal processing as a function of language. Sixty-four participants, 32 German and 32 French native speakers detected subtle word changes in accented positions in metrically structured sentences to which they previously tapped with their index finger. Before each sentence, they were cued by a metronome to tap either congruently (i.e., to accented syllables) or incongruently (i.e., to non-accented parts) to the following speech stimulus. Both French and German speakers detected words better when cued to tap congruently compared to incongruent tapping. Detection performance was predicted by participants' motor performance in the non-verbal cueing phase. Moreover, tapping rate while participants tapped to speech predicted detection differently for the two language groups, in particular in the incongruent tapping condition. We discuss our findings in light of the rhythmic differences of both languages and with respect to recent theories of expectancy-driven and multisensory speech processing

A computational model for studying L1’s effect on L2 speech learning

abstract: Much evidence has shown that first language (L1) plays an important role in the formation of L2 phonological system during second language (L2) learning process. This combines with the fact that different L1s have distinct phonological patterns to indicate the diverse L2 speech learning outcomes for speakers from different L1 backgrounds. This dissertation hypothesizes that phonological distances between accented speech and speakers' L1 speech are also correlated with perceived accentedness, and the correlations are negative for some phonological properties. Moreover, contrastive phonological distinctions between L1s and L2 will manifest themselves in the accented speech produced by speaker from these L1s. To test the hypotheses, this study comes up with a computational model to analyze the accented speech properties in both segmental (short-term speech measurements on short-segment or phoneme level) and suprasegmental (long-term speech measurements on word, long-segment, or sentence level) feature space. The benefit of using a computational model is that it enables quantitative analysis of L1's effect on accent in terms of different phonological properties. The core parts of this computational model are feature extraction schemes to extract pronunciation and prosody representation of accented speech based on existing techniques in speech processing field. Correlation analysis on both segmental and suprasegmental feature space is conducted to look into the relationship between acoustic measurements related to L1s and perceived accentedness across several L1s. Multiple regression analysis is employed to investigate how the L1's effect impacts the perception of foreign accent, and how accented speech produced by speakers from different L1s behaves distinctly on segmental and suprasegmental feature spaces. Results unveil the potential application of the methodology in this study to provide quantitative analysis of accented speech, and extend current studies in L2 speech learning theory to large scale. Practically, this study further shows that the computational model proposed in this study can benefit automatic accentedness evaluation system by adding features related to speakers' L1s.Dissertation/ThesisDoctoral Dissertation Speech and Hearing Science 201

Effects of timing on users' perceived control when interacting with intelligent systems

This research relates to the usability of mixed-initiative interaction systems, in which actions can be initiated either through a choice by the user or through intelligent decisions taken by the system. The key issue addressed here is how to preserve the user's perceived control ("sense of agency'') when the control of the interaction is being transferred between the system and the user in a back-and-forth manner. Previous research in social psychology and cognitive neuroscience suggests timing is a factor that can influence perceived control in such back-and-forth interactions. This dissertation explores the hypothesis that in mixed-initiative interaction, a predictable interaction rhythm can preserve the user's sense of control and enhance their experience during a task (e.g. higher confidence in task performance, stronger temporal alignment, lower perceived levels of stress and effort), whereas irregular interaction timing can have the opposite effect. Three controlled experiments compare alternative rhythmic strategies when users interact with simple visual stimuli, simple auditory stimuli, and a more realistic assisted text labelling task. The results of all three experiments support the hypothesis that a predictable interaction rhythm is beneficial in a range of interaction modalities and applications. This research contributes to the field of human-computer interaction (HCI) in four ways. Firstly, it builds novel connections between existing theories in cognitive neuroscience, social psychology and HCI, highlighting how rhythmic temporal structures can be beneficial to the user's experience: particularly, their sense of control. Secondly, it establishes timing as a crucial design resource for mixed-initiative interaction, and provides empirical evidence of how the user's perceived control and other task experiences (such as reported levels of confidence, stress and effort) can be influenced by the manipulation of timing. Thirdly, it provides quantitative measures for the user's entrainment behaviours that are applicable to a wide range of interaction timescales. Lastly, it contextualises the design of timing in a realistic application scenario and offers insights to the design of general end-user automation and decision support tools.Cambridge Commonwealth European and International Trust Scholarship Cambridge Philosophical Society research studentship Gonville & Caius College hardship grant China Scholarship Council (CSC) (The agreement with CSC has been officially cancelled on 28 December 2017.

The analysis of breathing and rhythm in speech

Speech rhythm can be described as the temporal patterning by which speech events, such as vocalic onsets, occur. Despite efforts to quantify and model speech rhythm across languages, it remains a scientifically enigmatic aspect of prosody. For instance, one challenge lies in determining how to best quantify and analyse speech rhythm. Techniques range from manual phonetic annotation to the automatic extraction of acoustic features. It is currently unclear how closely these differing approaches correspond to one another. Moreover, the primary means of speech rhythm research has been the analysis of the acoustic signal only. Investigations of speech rhythm may instead benefit from a range of complementary measures, including physiological recordings, such as of respiratory effort. This thesis therefore combines acoustic recording with inductive plethysmography (breath belts) to capture temporal characteristics of speech and speech breathing rhythms. The first part examines the performance of existing phonetic and algorithmic techniques for acoustic prosodic analysis in a new corpus of rhythmically diverse English and Mandarin speech. The second part addresses the need for an automatic speech breathing annotation technique by developing a novel function that is robust to the noisy plethysmography typical of spontaneous, naturalistic speech production. These methods are then applied in the following section to the analysis of English speech and speech breathing in a second, larger corpus. Finally, behavioural experiments were conducted to investigate listeners' perception of speech breathing using a novel gap detection task. The thesis establishes the feasibility, as well as limits, of automatic methods in comparison to manual annotation. In the speech breathing corpus analysis, they help show that speakers maintain a normative, yet contextually adaptive breathing style during speech. The perception experiments in turn demonstrate that listeners are sensitive to the violation of these speech breathing norms, even if unconsciously so. The thesis concludes by underscoring breathing as a necessary, yet often overlooked, component in speech rhythm planning and production