Producing Acoustic-Prosodic Entrainment in a Robotic Learning Companion to Build Learner Rapport

abstract: With advances in automatic speech recognition, spoken dialogue systems are assuming increasingly social roles. There is a growing need for these systems to be socially responsive, capable of building rapport with users. In human-human interactions, rapport is critical to patient-doctor communication, conflict resolution, educational interactions, and social engagement. Rapport between people promotes successful collaboration, motivation, and task success. Dialogue systems which can build rapport with their user may produce similar effects, personalizing interactions to create better outcomes. This dissertation focuses on how dialogue systems can build rapport utilizing acoustic-prosodic entrainment. Acoustic-prosodic entrainment occurs when individuals adapt their acoustic-prosodic features of speech, such as tone of voice or loudness, to one another over the course of a conversation. Correlated with liking and task success, a dialogue system which entrains may enhance rapport. Entrainment, however, is very challenging to model. People entrain on different features in many ways and how to design entrainment to build rapport is unclear. The first goal of this dissertation is to explore how acoustic-prosodic entrainment can be modeled to build rapport. Towards this goal, this work presents a series of studies comparing, evaluating, and iterating on the design of entrainment, motivated and informed by human-human dialogue. These models of entrainment are implemented in the dialogue system of a robotic learning companion. Learning companions are educational agents that engage students socially to increase motivation and facilitate learning. As a learning companion’s ability to be socially responsive increases, so do vital learning outcomes. A second goal of this dissertation is to explore the effects of entrainment on concrete outcomes such as learning in interactions with robotic learning companions. This dissertation results in contributions both technical and theoretical. Technical contributions include a robust and modular dialogue system capable of producing prosodic entrainment and other socially-responsive behavior. One of the first systems of its kind, the results demonstrate that an entraining, social learning companion can positively build rapport and increase learning. This dissertation provides support for exploring phenomena like entrainment to enhance factors such as rapport and learning and provides a platform with which to explore these phenomena in future work.Dissertation/ThesisDoctoral Dissertation Computer Science 201

From Instructionism to Constructionism: the role of tinkering in educational technology

https://www.ester.ee/record=b5258038*es

MS. AN (Meeting Students’ Academic Needs): A Socially Adaptive Robot Tutor For Student Engagement In Math Education

This research presents a new, socially adaptive robot tutor, Ms. An (Meeting Students’ Academic Needs). The goal of this research was to use a decision tree model to develop a socially adaptive robot tutor that predicted and responded to student emotion and performance to actively engage students in mathematics education. The novelty of this multi-disciplinary project is the combination of the fields of HRI, AI, and education. In this study we 1) implemented a decision tree model to classify student emotion and performance for use in adaptive robot tutoring-an approach not applied to educational robotics; 2) presented an intuitive interface for seamless robot operation by novice users; and 3) applied direct human teaching methods (guided practice and progress monitoring) for a robot tutor to engage students in mathematics education. Twenty 4th and 5th grade students in rural South Carolina participated in a between subjects study with two conditions: A) with a non-adaptive robot (control group); and B) with a socially adaptive robot (adaptive group). Students engaged in two one-on-one tutoring sessions to practice multiplication per the South Carolina 4th and 5th grade mathematics state standards. Although our decision tree models were not very predictive, the results gave answers to our current questions and clarity for future directions. Our adaptive strategies to engage students academically were effective. Further, all students enjoyed working with the robot and we did not see a difference in emotional engagement across the two groups. This study offered insight for developing a socially adaptive robot tutor to engage students academically and emotionally while practicing multiplication. Results from this study will inform the human-robot interaction (HRI) and artificial intelligence (AI) communities on best practices and techniques within the scope of this wor

Learning instructional communication skills in peer collaborative problem solving : a case of moving referent.

The present work is an attempt to combine two traditions of communication study: referential communication approach and sociolinguistic approach. The purpose was to examine how children ages 5 and 7 years learn to give instructions to each other in a peer collaborative problem solving situation. In an effort to identify interaction patterns and possible developmental progressions, various coding and categorization schemes were developed to analyze the processes of: negotiation of themes, establishing a common perspective toward the task, co-constructing messages and shared names. A comparison was made between the children\u27s development of spatial terms for a stationary referent and a moving referent. The analysis shows that 7-year-olds shared themes more actively, using explicit means, compared to 5-year-olds. The older children\u27s instructions were more informative and made in the task-appropriate referential perspective. For the purpose of establishing shared names, the older children engaged themselves in the naming process less often because they used names that can be more easily shared. The development of spatial terms for a moving referent seems to lag behind the development of those for a stationary referent. The children gradually learned, across ages and sessions, to participate to maximize the team effectiveness. Finally, dynamic changes in instructional messages were analyzed using a mode of graphic representation

STRUGGLING STUDENTS BECOME SUCCESSFUL ART MENTORS: A MIXED METHOD MULTI-CASE STUDY

In the era of high stakes testing and teacher accountability, schools in the US continue the fight to prevent high school drop-outs and meet the needs of academically struggling students. Both the visual arts and mentoring programs have been shown to improve academic performance and emotional well-being. This study examined the impact of an art mentoring program on academically struggling high school students who became art mentors to elementary students. It considered the impact being an art mentor would have on academic success, delinquent behavior, and self-concept. Results suggest a positive relationship between being an art mentor and academic performance as defined through a .5 average positive increase in GPA and positive changes noted by classroom teachers. Results did not reveal a significant change in delinquent behavior or self-concept and suggest the need for further research in those areas

Cooperative design of a cross-age tutoring system based on a social networking platform

In South Africa, many young children from poor social and economic backgrounds are cared for at home by parents or guardians who are themselves illiterate. This leads to poor educational outcomes later in life. Yet there are many privileged teenagers with access to mobile technologies who spend a greater portion of their spare time interacting on ubiquitous social media platforms. This presents an opportunity whereby the poor educational outcomes referred to previously could be addressed by applying a technology solution providing social media-based homework support by privileged teenagers to underprivileged younger children. However, most applications designed for use by children are designed by adults, with little understanding of the user requirements of the target end users. This research explores the following question: How can a cross-age tutoring system be designed for implementation on a social networking platform to support numeracy and literacy skill acquisition? The main contribution of this research was the definition of the Cooperative design by Children for Children (CD2C) Design Framework, a blueprint of how a cross-age tutoring system could be co-designed by children of different age groups and life circumstances. The CD2C Design Framework was derived as an abstraction of the second contribution of this research, the TitanTutor, an artifact designed using co-operative inquiry method and the Design Science Research approach. The third novelty of this research was contribution to Design Science Research theory, with the addition of new theory that states that cooperative design by children from different age groups and life circumstances is tempered by socio-environmental context and power relations between the co-design partners. This work provided important contributions to researchers in the areas of Cooperative Inquiry (CI), Human Computer Interaction (HCI), and Design Science Research (DSR). Future researchers could extend the CD2C Design Framework to make it even more abstract, thereby making it universally applicable to any co-design scenario.ComputingPh. D. (Information Systems

Artificial intelligence and robots in services : theory and management of (future) humanrobot service interactions

During the past decade, service robots have increasingly been deployed in a wide variety of services, where they co-produce service outcomes with and for the benefit of internal or external customers within humanrobot service interactions (HRSI). Although the introduction of different service robot types into the marketplace promises efficiency gains, it changes premises of service encounter theory and practice fundamentally. Moreover, introducing service robots without considering external or internal customers needs can lead to negative service outcomes. This thesis aims to generate knowledge on how the introduction of different service robot types (i.e., embodied and digital service robots) in internal and external service encounters changes fundamental premises of service encounter theory and impacts HRSI outcomes. In doing so, it leverages different scientific methods and focuses on external service encounters with digital and embodied service robots, as well as internal service encounters with digital service robots. Chapter 2 aims to advance service encounter theory in the context of HRSI in external service encounters by conceptually developing a service encounter theory evaluation scheme to assess a theorys fit to explain HRSI-related phenomena. The scheme includes individual and contextual factors that bound theoretical premises and, hence, supports scholars in assessing standing service encounter theories. The chapter also puts forth an exemplary assessment of role theory and provides detailed avenues for future research. Chapter 3 aims to synthesize the great wealth of knowledge on HRSI related to external service encounters with embodied service robots. By conducting a comprehensive systematic literature review, the chapter identifies 199 empirical research articles across scientific fields that can inform service research on how to successfully introduce service robots into the organizational frontline. To organize the plethora of research findings, this chapter develops a new structuring framework (D3: design, delegate, deploy). It utilizes this framework to provide a comprehensive overview of the empirical HRSI literature, delineates practical implications, and identifies gaps in literature to identify promising future research avenues. Chapter 4 also addresses HRSI in external service encounters but focuses specifically on the transformative potential of embodied service robots to enhance vulnerable consumers (i.e., children and older adults) well-being in social isolation. To identify how different robots can enhance well-being, this chapter follows a conceptual approach and integrates findings from service research, social robotics, social psychology, and medicine. The chapter develops a typology of robotic transformative service (i.e., entertainer, social enabler, mentor, and friend) as a function of consumers state of social isolation, well-being focus, and robot capabilities and a future research agenda for robotic transformative service research (RTSR). This work guides service consumers and providers, as well as robot developers, in identifying and developing the most appropriate robot type for advancing the well-being of vulnerable consumers in social isolation. Finally, Chapter 5 focuses on HRSI research in the context of interactions with digital service robots in internal service encounters. Based on a comprehensive literature review paired with a qualitative study, it conceptionally develops a new concept of a collaborative, digital service robot: a collaborative intelligence system (i.e., CI system) that co-produces service with employees. Drawing from service encounter needs theory, the chapter also empirically tests the effect of CI systems on employee need fulfillment (i.e., need for control, cognition, self-efficacy, and justice) and, in turn, on responsibility taking in two scenario-based experiments. The results uncover divergent mechanisms of how the fulfillment of service encounter needs drives the effect of CI systems on outcome responsibility for different employee groups. Service scholars and managers benefit from a blueprint for designing collaborative digital service robots and an understanding of their effects on employee outcomes in service co-production. In summary, this thesis contributes to literature by providing new insights into different types of HRSI by consolidating HRSI knowledge, developing and advancing HRSI concepts and theory, and empirically investigating HRSI-related phenomena. The new insights put forth in this thesis are discussed and implications for service theory and practice are delineated.Serviceroboter werden zunehmend für Dienstleistungen eingesetzt, wobei sie mit und zum Nutzen von internen oder externen Kunden im Rahmen von Mensch-Roboter-Service-Interaktionen (MRSI) Serviceergebnisse co-produzieren. Die Einführung verschiedener Arten von Servicerobotern (d. h. verkörperte und digitale) verspricht Effizienzgewinne, verändert jedoch grundlegende Prämissen der Theorie und Praxis von Dienstleistungsinteraktionen. Darüber hinaus kann die Einführung von Servicerobotern ohne die Berücksichtigung von Kundenbedürfnissen zu negativen Serviceergebnissen führen. Ziel dieser Dissertation ist es, Wissen darüber zu generieren, wie die Einführung verschiedener Robotertypen in internen und externen Dienstleistungsinteraktionen grundlegende theoretische Prämissen von Dienstleistungsinteraktionen verändert und sich auf die Ergebnisse von MRSI auswirkt. Um dieses Ziel zu erreichen werden unter Einsatz verschiedener wissenschaftlicher Methoden drei verschiedene Arten von MRSI untersucht. Kapitel 2 zielt darauf ab, die Theorie der Dienstleistungsinteraktion im Kontext von MRSI in externen Dienstleistungsinteraktionen weiterzuentwickeln und konzeptioniert ein Bewertungsschema für bestehende Theorien der Dienstleistungsbegegnung. Das Schema umfasst individuelle und kontextuelle Faktoren, die die ursprünglichen theoretischen Prämissen von Mensch-zu-Mensch Dienstleistungsinteraktionen beeinflussen und unterstützt somit Wissenschaftler bei der Bewertung von Theorien zur Verwendung im MRSI Kontext. Das Kapitel enthält eine beispielhafte Bewertung der Rollentheorie und zeigt detaillierte Wege für zukünftige Forschung auf. Kapitel 3 zielt darauf ab, die große Fülle an Wissen über MRSI im Kontext externer Dienstleistungsinteraktionen mit verkörperten Servicerobotern zu synthetisieren. Durch eine systematische, interdisziplinäre Literaturanalyse identifiziert das Kapitel 199 empirische Forschungsartikel, die Erkenntnisse liefern, wie Serviceroboter erfolgreich in den Dienstleistungsprozess eingebunden werden können. Um die Fülle an Forschungsergebnissen zu ordnen, entwickelt dieses Kapitel ein neues, strukturierendes Modell (D3 framework: design, delegate, deploy). Dieses wird im Rahmen des Kapitels genutzt, um einen umfassenden Überblick über die empirische MRSI-Forschung zu geben, praktische Implikationen abzuleiten und Forschungslücken aufzuzeigen. Kapitel 4 befasst sich ebenfalls mit MRSI in externen Dienstleistungsinteraktionen, konzentriert sich aber auf das transformative Potenzial von verkörperten Servicerobotern zur Steigerung des Wohlbefindens von sozial isolierten, vulnerablen Verbrauchern (d. h. Kinder und ältere Erwachsene). Um herauszufinden, wie verschiedene Roboter das Wohlbefinden steigern können, integriert das Kapitel Erkenntnisse aus der Dienstleistungsforschung, der sozialen Robotik, der Sozialpsychologie und der Medizin. Dabei wird eine Typologie von vier transformativen Robotern in Abhängigkeit vom Zustand der sozialen Isolation des Verbrauchers, der Art des Wohlbefindens und den Fähigkeiten des Roboters entwickelt. Weiterhin wird eine detaillierte Forschungsagenda im Kontext transformativer Dienstleistungserstellung durch Roboter erarbeitet. Diese Arbeit hilft Dienstleistungsnehmern und -anbietern sowie Roboterentwicklern bei der Identifizierung und Entwicklung des am besten geeigneten Robotertyps zur Förderung des Wohlbefindens von sozial isolierten, vulnerablen Verbrauchern. Abschließend fokussiert Kapitel 5 die MRSI-Forschung im Kontext interner Dienstleistungsinteraktionen mit digitalen Servicerobotern. Auf Basis einer Literaturanalyse gepaart mit einer qualitativen Studie wird ein neues Konzept eines kollaborativen, digitalen Serviceroboters entwickelt: ein Collaborative Intelligence System (CI System), das in Zusammenarbeit mit Mitarbeitern Dienstleistungsergebnisse co-produziert. Anhand szenariobasierter Experimente wird empirisch untersucht, ob CI Systeme psychosoziale Bedürfnisse von Mitarbeitern befriedigen können und damit die mitarbeiterseitige Übernahme von Verantwortung für gemeinsam produzierte Ergebnisse gefördert wird. Die Ergebnisse decken für verschiedene Mitarbeitergruppen unterschiedliche Mechanismen auf, wie die Erfüllung von Bedürfnissen in der Dienstleistungsinteraktionen die Wirkung von CI Systemen auf die mitarbeiterseitige Übernahme von Verantwortung beeinflusst. Dienstleistungsforscher und -manager profitieren von einer Blaupause für die Gestaltung kollaborativer Dienstleistungsroboter und einem Verständnis für deren Auswirkungen auf Mitarbeitende. Insgesamt leistet diese Dissertation einen Beitrag zur Dienstleistungsforschung, indem sie neue Erkenntnisse über verschiedene Arten von MRSI liefert, das bestehende MRSI-Wissen konsolidiert, neue MRSI-Konzepte und -Theorien entwickelt bzw. weiterentwickelt und MRSI-bezogene Phänomene empirisch untersucht. Die neuen Erkenntnisse werden diskutiert und Implikationen für die Dienstleistungstheorie und -praxis abgeleitet