Emotion-aware voice interfaces based on speech signal processing

Voice interfaces (VIs) will become increasingly widespread in current daily lives as AI techniques progress. VIs can be incorporated into smart devices like smartphones, as well as integrated into autos, home automation systems, computer operating systems, and home appliances, among other things. Current speech interfaces, however, are unaware of users’ emotional states and hence cannot support real communication. To overcome these limitations, it is necessary to implement emotional awareness in future VIs. This thesis focuses on how speech signal processing (SSP) and speech emotion recognition (SER) can enable VIs to gain emotional awareness. Following an explanation of what emotion is and how neural networks are implemented, this thesis presents the results of several user studies and surveys. Emotions are complicated, and they are typically characterized using category and dimensional models. They can be expressed verbally or nonverbally. Although existing voice interfaces are unaware of users’ emotional states and cannot support natural conversations, it is possible to perceive users’ emotions by speech based on SSP in future VIs. One section of this thesis, based on SSP, investigates mental restorative effects on humans and their measures from speech signals. SSP is less intrusive and more accessible than traditional measures such as attention scales or response tests, and it can provide a reliable assessment for attention and mental restoration. SSP can be implemented into future VIs and utilized in future HCI user research. The thesis then moves on to present a novel attention neural network based on sparse correlation features. The detection accuracy of emotions in the continuous speech was demonstrated in a user study utilizing recordings from a real classroom. In this section, a promising result will be shown. In SER research, it is unknown if existing emotion detection methods detect acted emotions or the genuine emotion of the speaker. Another section of this thesis is concerned with humans’ ability to act on their emotions. In a user study, participants were instructed to imitate five fundamental emotions. The results revealed that they struggled with this task; nevertheless, certain emotions were easier to replicate than others. A further study concern is how VIs should respond to users’ emotions if SER techniques are implemented in VIs and can recognize users’ emotions. The thesis includes research on ways for dealing with the emotions of users. In a user study, users were instructed to make sad, angry, and terrified VI avatars happy and were asked if they would like to be treated the same way if the situation were reversed. According to the results, the majority of participants tended to respond to these unpleasant emotions with neutral emotion, but there is a difference among genders in emotion selection. For a human-centered design approach, it is important to understand what the users’ preferences for future VIs are. In three distinct cultures, a questionnaire-based survey on users’ attitudes and preferences for emotion-aware VIs was conducted. It was discovered that there are almost no gender differences. Cluster analysis found that there are three fundamental user types that exist in all cultures: Enthusiasts, Pragmatists, and Sceptics. As a result, future VI development should consider diverse sorts of consumers. In conclusion, future VIs systems should be designed for various sorts of users as well as be able to detect the users’ disguised or actual emotions using SER and SSP technologies. Furthermore, many other applications, such as restorative effects assessments, can be included in the VIs system

Bilingual first language acquisition in Malay and English : a morphological and suprasegmental study in the development of plural expressions in a bilingual child

This thesis investigates the development of plural marking in a child raised in Malay and English simultaneously, from the morphological and prosodic perspective. For the morphological plural development, the child’s plural acquisition is analysed within the Processability Theory (PT) framework de Bot (1992) de Bot (1992) thus widening PT’s typological range of application to a language such as Malay, which belongs to the Austronesian family (Dryer & Haspelmath, 2013). PT has been tested for morphological development in L2 English (Di Biase, Kawaguchi, & Yamaguchi, 2015; Johnston, 2000) and several typologically different languages as well as bilingual first language acquisition (BFLA) such as Japanese-English (Itani-Adams, 2013). However, PT has not been empirically tested for any language of the Austronesian family nor in a Bilingual First Language Acquisition (BFLA) constellation involving Malay and English. The Malay-English language pair is interesting because of the remarkably different linguistic mechanisms used for encoding plurality in the two languages; morphologically, Malay marks plurality through distinct forms of reduplication such as rumah-rumah ‘houses’, buah-buahan’ (plural form of buah ‘fruit’) and bukit-bukau ‘hills’ (Sew, 2007). In contrast, English uses morphological inflections -s suffixed to the stem, e.g., cat/cats, dog/dogs, book/books (Carstairs-McCarthy, 2002). Malay reduplication, as previously shown, involves more than a single word, however, functionally speaking it is equivalent to one word plus a marker of plurality. Thus, prosodic mechanisms play a crucial role in distinguishing between mere repetition and grammatical reduplication in Malay (Gil, 2005). Since plurality is expressed very differently in each language, this study investigates how a bilingual child develops simultaneously two grammatical systems. The participant in this research is a female child named Rina, who was raised in Malay-English environment from birth. This investigation comprises of two parts; first is the longitudinal investigation of her plural acquisition from age 2;10 to 3;10. During this period, Rina was living in Australia, where the environmentally predominant language was English. The second complementary part is an investigation of Rina’s plural marking systems at age 4;8 when she had returned to Malaysia, where the predominant environmental language was Malay. For the longitudinal study, the database for the analyses was obtained from separate Malay and English recording sessions, which were conducted weekly from age 2;10 to 3;10. Likewise, the data for Rina’s plural expression at 4;8 was also obtained from separate Malay and English environment recordings. For the morphological plural development, results indicate that Rina developed two different systems to mark plurality in Malay and English. Her plural marking developed in the sequence predicted by PT. However, though she clearly distinguished the two languages, bidirectional influences from English to Malay and Malay to English were found in the corpus, both in the longitudinal study as well as at age 4;8. In the longitudinal study, it was found that in expressing plurals in Malay and English, Rina used various linguistic devices: one of the predominant strategies she employed in both languages was iteration, a strategy in which Rina expressed more than one objects by repeating the lexical item according to the number of individuated entities (hence four cats would be expressed as cat cat cat cat). Reduplication, the target grammatical Malay plural, only emerged at 3;8. Thus, we examine the prosodic development of the child’s iteration up till the emergence of reduplication. Findings indicate that the development from iteration to reduplication is gradual; the main acoustic correlate that she employed during the longitudinal study was final-syllable lengthening. She only began differentiating various prosodic mechanisms (such as pausing, duration and pitch) to distinguish repetition and reduplication in her plural marking at age 4;8. This study offers a new perspective on the interplay between the two languages in the early stages of grammatical development in a bilingual child. The specific features of plurality in Malay and English and how they develop in the bilingual child may shed light on the applicability of PT to BFLA. Also, the link between the child’s morphological development and prosodic mechanisms show that in acquiring the prosodic structures of reduplication, Rina creates partial and increasingly specific analyses of the grammatical forms, gradually approaching the conventional adult form

Language-Learner Computer Interactions

This book focuses on learner-computer interactions (LCI) in second language learning environments drawing largely on sociocultural theories of language development. It brings together a rich and varied range of theoretical discussions and applications in order to illustrate the way in which LCI can enrich our comprehension of technology-mediated communication, hence enhancing learners’ digital literacy skills. The book is based on the premise that, in order to fully understand the nature of language and literacy development in digital spaces, researchers and practitioners in linguistics, sciences and engineering need to borrow from each others’ theoretical and practical toolkits. In light of this premise, themes include such aspects as educational ergonomics, affordances, complex systems learning, learner personas and corpora, while also describing such data collecting tools as video screen capture devices, eye-tracking or intelligent learning tutoring systems

Multimodal Shared-Control Interaction for Mobile Robots in AAL Environments

This dissertation investigates the design, development and implementation of cognitively adequate, safe and robust, spatially-related, multimodal interaction between human operators and mobile robots in Ambient Assisted Living environments both from the theoretical and practical perspectives. By focusing on different aspects of the concept Interaction, the essential contribution of this dissertation is divided into three main research packages; namely, Formal Interaction, Spatial Interaction and Multimodal Interaction in AAL. As the principle package, in Formal Interaction, research effort is dedicated to developing a formal language based interaction modelling and management solution process and a unified dialogue modelling approach. This package aims to enable a robust, flexible, and context-sensitive, yet formally controllable and tractable interaction. This type of interaction can be used to support the interaction management of any complex interactive systems, including the ones covered in the other two research packages. In the second research package, Spatial Interaction, a general qualitative spatial knowledge based multi-level conceptual model is developed and proposed. The goal is to support a spatially-related interaction in human-robot collaborative navigation. With a model-based computational framework, the proposed conceptual model has been implemented and integrated into a practical interactive system which has been evaluated by empirical studies. It has been particularly tested with respect to a set of high-level and model-based conceptual strategies for resolving the frequent spatially-related communication problems in human-robot interaction. Last but not least, in Multimodal Interaction in AAL, attention is drawn to design, development and implementation of multimodal interaction for elderly persons. In this elderly-friendly scenario, ageing-related characteristics are carefully considered for an effective and efficient interaction. Moreover, a standard model based empirical framework for evaluating multimodal interaction is provided. This framework was especially applied to evaluate a minutely developed and systematically improved elderly-friendly multimodal interactive system through a series of empirical studies with groups of elderly persons

The Language of Jokes in the Digital Age

In this accessible book, Delia Chiaro provides a fresh overview of the language of jokes in a globalized and digitalized world. The book shows how, while on the one hand the lingua-cultural nuts and bolts of jokes have remained unchanged over time, on the other, the time-space compression brought about by modern technology has generated new settings and new ways of joking and playing with language. The Language of Jokes in the Digital Age covers a wide range of settings from social networks, e-mails and memes, to more traditional fields of film and TV (especially sitcoms and game shows) and advertising. Chiaro’s consideration of the increasingly virtual context of jokes delights with both up-to-date examples and frequent reference to the most central theories of comedy. This lively book will be essential reading for any student or researcher working in the area of language and humour and will be of interest to those in language and media and sociolinguistics

Practical, appropriate, empirically-validated guidelines for designing educational games

There has recently been a great deal of interest in the potential of computer games to function as innovative educational tools. However, there is very little evidence of games fulfilling that potential. Indeed, the process of merging the disparate goals of education and games design appears problematic, and there are currently no practical guidelines for how to do so in a coherent manner. In this paper, we describe the successful, empirically validated teaching methods developed by behavioural psychologists and point out how they are uniquely suited to take advantage of the benefits that games offer to education. We conclude by proposing some practical steps for designing educational games, based on the techniques of Applied Behaviour Analysis. It is intended that this paper can both focus educational games designers on the features of games that are genuinely useful for education, and also introduce a successful form of teaching that this audience may not yet be familiar with

The Language of Jokes in the Digital Age

In this accessible book, Delia Chiaro provides a fresh overview of the language of jokes in a globalized and digitalized world. The book shows how, while on the one hand the lingua-cultural nuts and bolts of jokes have remained unchanged over time, on the other, the time-space compression brought about by modern technology has generated new settings and new ways of joking and playing with language. The Language of Jokes in the Digital Age covers a wide range of settings from social networks, e-mails and memes, to more traditional fields of film and TV (especially sitcoms and game shows) and advertising. Chiaro’s consideration of the increasingly virtual context of jokes delights with both up-to-date examples and frequent reference to the most central theories of comedy. This lively book will be essential reading for any student or researcher working in the area of language and humour and will be of interest to those in language and media and sociolinguistics