Toward a social signaling framework : activity and emphasis in speech

Thesis (M. Eng.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2006.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Includes bibliographical references (p. 67-70).Language is not the only form of verbal communication. Loudness, pitch, speaking rate, and other non-linguistic speech features are crucial aspects of human spoken interaction. In this thesis, we separate these speech features into two categories -- vocal Activity and vocal Emphasis -- and propose a framework for classifying high-level social behavior according to those metrics. We present experiments showing that non-linguistic speech analysis alone can account for appreciable portions of social phenomena. We report statistically significant results in measuring the persuasiveness of pitches, the effectiveness of customer service representatives, and the severity of depression. Effect sizes of these studies explain up to 60% of the sample variances and yield binary decision accuracies nearing 90%.by William T. Stoltzman.M.Eng

Voice Command Controller

Signal processing technology has been strongly developed and it has attracted interest from scientists and engineers around the world from the last decade. Speech synthesis and speech recognition are particular topic in the field that have been widely used and developed in many different area such as business, controlling, education and entertainment. The project\u27s main objective is to study and develop an application program with the Speech SDK through design and implementation of Tele-Control system based on the commercial product of National Semiconductor: Carrier-Current Transceiver (LM 1893) and Speech development kit (Speech SDK4.0) from Microsoft Corporation. The project is suitable to be used in restricted areas where space, wiring, decoration and signal interference are issues of concerned. Speech SDK is an interesting and useful tool in helping develop a Voice application programs. In this project, the user can use voice command interact with the control program to control a remote device. In conjunction with hardware modification, extra function can be added to the program such as controlling camera, video capture and position control buttons on the environment map, the project will be suitable for security purposes

Computational modeling of turn-taking dynamics in spoken conversations

The study of human interaction dynamics has been at the center for multiple research disciplines in- cluding computer and social sciences, conversational analysis and psychology, for over decades. Recent interest has been shown with the aim of designing computational models to improve human-machine interaction system as well as support humans in their decision-making process. Turn-taking is one of the key aspects of conversational dynamics in dyadic conversations and is an integral part of human- human, and human-machine interaction systems. It is used for discourse organization of a conversation by means of explicit phrasing, intonation, and pausing, and it involves intricate timing. In verbal (e.g., telephone) conversation, the turn transitions are facilitated by inter- and intra- speaker silences and over- laps. In early research of turn-taking in the speech community, the studies include durational aspects of turns, cues for turn yielding intention and lastly designing turn transition modeling for spoken dia- log agents. Compared to the studies of turn transitions very few works have been done for classifying overlap discourse, especially the competitive act of overlaps and function of silences. Given the limitations of the current state-of-the-art, this dissertation focuses on two aspects of con- versational dynamics: 1) design automated computational models for analyzing turn-taking behavior in a dyadic conversation, 2) predict the outcome of the conversations, i.e., observed user satisfaction, using turn-taking descriptors, and later these two aspects are used to design a conversational profile for each speaker using turn-taking behavior and the outcome of the conversations. The analysis, experiments, and evaluation has been done on a large dataset of Italian call-center spoken conversations where customers and agents are engaged in real problem-solving tasks. Towards solving our research goal, the challenges include automatically segmenting and aligning speakers’ channel from the speech signal, identifying and labeling the turn-types and its functional aspects. The task becomes more challenging due to the presence of overlapping speech. To model turn- taking behavior, the intension behind these overlapping turns needed to be considered. However, among all, the most critical question is how to model observed user satisfaction in a dyadic conversation and what properties of turn-taking behavior can be used to represent and predict the outcome. Thus, the computational models for analyzing turn-taking dynamics, in this dissertation includes au- tomatic segmenting and labeling turn types, categorization of competitive vs non-competitive overlaps, silences (e.g., lapse, pauses) and functions of turns in terms of dialog acts. The novel contributions of the work presented here are to 1. design of a fully automated turn segmentation and labeling (e.g., agent vs customer’s turn, lapse within the speaker, and overlap) system. 2. the design of annotation guidelines for segmenting and annotating the speech overlaps with the competitive and non-competitive labels. 3. demonstrate how different channels of information such as acoustic, linguistic, and psycholin- guistic feature sets perform in the classification of competitive vs non-competitive overlaps. 4. study the role of speakers and context (i.e., agents’ and customers’ speech) for conveying the information of competitiveness for each individual feature set and their combinations. 5. investigate the function of long silences towards the information flow in a dyadic conversation. The extracted turn-taking cues is then used to automatically predict the outcome of the conversation, which is modeled from continuous manifestations of emotion. The contributions include 1. modeling the state of the observed user satisfaction in terms of the final emotional manifestation of the customer (i.e., user). 2. analysis and modeling turn-taking properties to display how each turn type influence the user satisfaction. 3. study of how turn-taking behavior changes within each emotional state. Based on the studies conducted in this work, it is demonstrated that turn-taking behavior, specially competitiveness of overlaps, is more than just an organizational tool in daily human interactions. It represents the beneficial information and contains the power to predict the outcome of the conversation in terms of satisfaction vs not-satisfaction. Combining the turn-taking behavior and the outcome of the conversation, the final and resultant goal is to design a conversational profile for each speaker. Such profiled information not only facilitate domain experts but also would be useful to the call center agent in real time. These systems are fully automated and no human intervention is required. The findings are po- tentially relevant to the research of overlapping speech and automatic analysis of human-human and human-machine interactions

How to write health dialog for a talking computer

AbstractAutomated dialogue systems delivered over the telephone offer a promising approach to delivering health-related interventions to populations of individuals at low-cost. Over the past two decades, an automated telephone system called Telephone-Linked Care or TLC has been successfully designed and evaluated by the authors and their colleagues. This work has resulted in over twenty systems for various health-related conditions and lifestyle behaviors. This paper describes our approach to developing and writing dialogue for these automated telephone systems, including determining the program objectives, defining the target population, and selecting a theory of behavior change to guide the intervention. Both macro and micro issues are considered in constructing dialogue systems that are engaging for the target population, easy to use, and effective at promoting positive health behaviors and outcomes

Vocal Interactivity in-and-between Humans, Animals, and Robots

Almost all animals exploit vocal signals for a range of ecologically motivated purposes: detecting predators/prey and marking territory, expressing emotions, establishing social relations, and sharing information. Whether it is a bird raising an alarm, a whale calling to potential partners, a dog responding to human commands, a parent reading a story with a child, or a business-person accessing stock prices using Siri, vocalization provides a valuable communication channel through which behavior may be coordinated and controlled, and information may be distributed and acquired. Indeed, the ubiquity of vocal interaction has led to research across an extremely diverse array of fields, from assessing animal welfare, to understanding the precursors of human language, to developing voice-based human–machine interaction. Opportunities for cross-fertilization between these fields abound; for example, using artificial cognitive agents to investigate contemporary theories of language grounding, using machine learning to analyze different habitats or adding vocal expressivity to the next generation of language-enabled autonomous social agents. However, much of the research is conducted within well-defined disciplinary boundaries, and many fundamental issues remain. This paper attempts to redress the balance by presenting a comparative review of vocal interaction within-and-between humans, animals, and artificial agents (such as robots), and it identifies a rich set of open research questions that may benefit from an interdisciplinary analysis

Managing distress over time in psychotherapy : guiding the client in and through intense emotional work

Clients who seek psychotherapeutic treatment have had personal experiences involving some form of distress. Although research has shown that the client's ability to experience and express painful emotions during therapy can have a therapeutic benefit, it has also been argued that displaying distress may convey a form of helplessness and vulnerability, and thus, clients may be reluctant to cast themselves in this light. Using the methods of conversation analysis, this paper explores how a client's upsetting experience is managed over the course of a single session of client-centered therapy. The main analytic focus will be on (1) the different therapist practices used to orient to the client's distress, (2) the varying forms of client opposition to the therapist's attempts to work with the distress, and (3) the context sensitivity of orienting to distress and how certain practices may be uniquely shaped by what had occurred in prior talk. It was found that, whereas certain types of therapist responses tended to be endorsed by the client, others were forcefully rejected as inappropriate displays of understanding or empathy. By focusing on repeated sequential episodes over time in which a client conveys distress, followed by the therapist's response, this paper sheds light on the interactional trajectory through which a client and therapist are able to resolve impasses to emotional exploration and to successfully secure extended and intense emotional work