An End-to-End Conversational Style Matching Agent

We present an end-to-end voice-based conversational agent that is able to engage in naturalistic multi-turn dialogue and align with the interlocutor's conversational style. The system uses a series of deep neural network components for speech recognition, dialogue generation, prosodic analysis and speech synthesis to generate language and prosodic expression with qualities that match those of the user. We conducted a user study (N=30) in which participants talked with the agent for 15 to 20 minutes, resulting in over 8 hours of natural interaction data. Users with high consideration conversational styles reported the agent to be more trustworthy when it matched their conversational style. Whereas, users with high involvement conversational styles were indifferent. Finally, we provide design guidelines for multi-turn dialogue interactions using conversational style adaptation

Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour

Rapport, the close and harmonious relationship in which interaction partners are "in sync" with each other, was shown to result in smoother social interactions, improved collaboration, and improved interpersonal outcomes. In this work, we are first to investigate automatic prediction of low rapport during natural interactions within small groups. This task is challenging given that rapport only manifests in subtle non-verbal signals that are, in addition, subject to influences of group dynamics as well as inter-personal idiosyncrasies. We record videos of unscripted discussions of three to four people using a multi-view camera system and microphones. We analyse a rich set of non-verbal signals for rapport detection, namely facial expressions, hand motion, gaze, speaker turns, and speech prosody. Using facial features, we can detect low rapport with an average precision of 0.7 (chance level at 0.25), while incorporating prior knowledge of participants' personalities can even achieve early prediction without a drop in performance. We further provide a detailed analysis of different feature sets and the amount of information contained in different temporal segments of the interactions.Comment: 12 pages, 6 figure

It’s a long way to Monte-Carlo: probabilistic display in GPS navigation

We present a mobile, GPS-based multimodal navigation system, equipped with inertial control that allows users to explore and navigate through an augmented physical space, incorporating and displaying the uncertainty resulting from inaccurate sensing and unknown user intentions. The system propagates uncertainty appropriately via Monte Carlo sampling and predicts at a user-controllable time horizon. Control of the Monte Carlo exploration is entirely tilt-based. The system output is displayed both visually and in audio. Audio is rendered via granular synthesis to accurately display the probability of the user reaching targets in the space. We also demonstrate the use of uncertain prediction in a trajectory following task, where a section of music is modulated according to the changing predictions of user position with respect to the target trajectory. We show that appropriate display of the full distribution of potential future users positions with respect to sites-of-interest can improve the quality of interaction over a simplistic interpretation of the sensed data

Evaluating the Effectiveness of a Goal-Directed Intervention on the Social Interaction of Children with Neurodevelopmental Disabilities

This study sought to address social interaction needs of children with neurodevelopmental disabilities through an eight week intervention focusing on social skills at Spaulding Youth Center (SYC), a residential facility in Northeast United States. The study implemented a randomized control design, with a control and intervention group, totaling n=19 students. Results have implications for future practice

Designing Teaching Materials for the Development of Conversation. The use of films in online environments.

Not all scholars are aware that successful foreign language teaching must be based on authentic language input, so that the proficiency level reached is appropriate to the initial goal set for our target cohort. Most textbooks haven’t got enough material with authentic or natural language. After all, we still expect students to reproduce grammatically correct sentences following a specific well-known grammatical rule. That is the main reason why the language we find in these books is rather “artificial”, something we cannot consider to be “daily language”. Teaching oral skills in a foreign language is a complex issue. For this reason, we propose film dialogues to be a more natural model of conversation than textbook dialogues. This paper is based on the most popular theories of spoken discourse analysis (Austin (1962), Searle (1969), Schmidt and Richards, (1980)), focusing on students’ familiarity with speech acts as essential components for them to be communicatively appropriate. Likewise, teachers’ updating is essential since they have to master both verbal and non-verbal language, discourse theories and their application to the teaching context. Thus, students have the opportunity to get more in touch with natural conversation. This proposal is focused on naturally occurring conversation and virtual learning. Methodology suggested is as follows: the teacher selects extracts leading to the subsequent design of activities. Later on, students will watch them, paying attention only to very specific aspects of language and culture previously discussed. It is understood that, if students are not fully guided or watch the whole film, they get lost or learn erratically. The aim of this type of activity is to reinforce students’ awareness of how native speakers behave and interact in their daily lives. A foreign language is a constantly changing activity, determined by many factors, which cannot be taught unless it is offered in the right context and through the right medium

Automated Gaze-Based Mind Wandering Detection during Computerized Learning in Classrooms

We investigate the use of commercial off-the-shelf (COTS) eye-trackers to automatically detect mind wandering—a phenomenon involving a shift in attention from task-related to task-unrelated thoughts—during computerized learning. Study 1 (N = 135 high-school students) tested the feasibility of COTS eye tracking while students learn biology with an intelligent tutoring system called GuruTutor in their classroom. We could successfully track eye gaze in 75% (both eyes tracked) and 95% (one eye tracked) of the cases for 85% of the sessions where gaze was successfully recorded. In Study 2, we used this data to build automated student-independent detectors of mind wandering, obtaining accuracies (mind wandering F1 = 0.59) substantially better than chance (F1 = 0.24). Study 3 investigated context-generalizability of mind wandering detectors, finding that models trained on data collected in a controlled laboratory more successfully generalized to the classroom than the reverse. Study 4 investigated gaze- and video- based mind wandering detection, finding that gaze-based detection was superior and multimodal detection yielded an improvement in limited circumstances. We tested live mind wandering detection on a new sample of 39 students in Study 5 and found that detection accuracy (mind wandering F1 = 0.40) was considerably above chance (F1 = 0.24), albeit lower than offline detection accuracy from Study 1 (F1 = 0.59), a finding attributable to handling of missing data. We discuss our next steps towards developing gaze-based attention-aware learning technologies to increase engagement and learning by combating mind wandering in classroom contexts

Incorporating android conversational agents in m-learning apps

Smart Mobile Devices Have Fostered New Learning Scenarios That Demand Sophisticated Interfaces. Multimodal Conversational Agents Have Became A Strong Alternative To Develop Human-Machine Interfaces That Provide A More Engaging And Human-Like Relationship Between Students And The System. The Main Developers Of Operating Systems For Such Devices Have Provided Application Programming Interfaces For Developers To Implement Their Own Applications, Including Different Solutions For Developing Graphical Interfaces, Sensor Control And Voice Interaction. Despite The Usefulness Of Such Resources, There Are No Strategies Defined For Coupling The Multimodal Interface With The Possibilities That These Devices Offer To Enhance Mobile Educative Apps With Intelligent Communicative Capabilities And Adaptation To The User Needs. In This Paper, We Present A Practical M-Learning Application That Integrates Features Of Android Application Programming Interfaces On A Modular Architecture That Emphasizes Interaction Management And Context-Awareness To Foster User-Adaptively, Robustness And Maintainability.This work was supported in part by Projects MINECO TEC2012-37832-C02-01, CICYT TEC2011-28626-C02-02, CAM CONTEXTS (S2009/TIC-1485

A multimodal perspective on modality in the English language classroom

This thesis is a study of an engage, study, activate (ESA) lesson of teaching modals of present deduction. The lesson has been taken from a published English language teaching course book and is typical of the way modal forms are presented to teach epistemic modality in many commercially produced English language teaching course books. I argue that for cognitive, social, linguistic and procedural reasons the linguistic forms and structures presented in the lesson are not straightforwardly transferred to the activate stage of the lesson. Using insights from spoken language corpora I carry out a comparative analysis with the modal forms presented in the course book. I then explore the notion of ‘context’ and drawing on systemic functional grammar discuss how modal forms function in discourse to realise interpersonal relations. Moving my research to the English language classroom I collect ethnographic classroom data and using social semiotic multimodality as an analytical framework I explore learner interaction to uncover the communicative resources learners use to express epistemic modality in a discussion activity from the same lesson. My analysis reveals that the modal structures in the course book differ to some extent from spoken language corpora. It shows that the course book offers no instruction on the interpersonal dimension of modality and thus how speakers use signals of modality to position themselves interpersonally vis-à-vis their interlocutors. The data collected from the English language class reveals that during the lesson learners communicate modality through modes of communication such as eye gaze, gesture and posture in addition to spoken language. Again drawing from systemic functional grammar I explain how these modes have the potential to express interpersonal meaning and thus highlight that meaning is communicated through modal ensembles. Based on these findings I propose a number of teaching strategies to raise awareness of the interpersonal function of modality in multimodal discourse, and for the use of language corpora to better inform teaching materials on selections of modality