Producing Acoustic-Prosodic Entrainment in a Robotic Learning Companion to Build Learner Rapport

abstract: With advances in automatic speech recognition, spoken dialogue systems are assuming increasingly social roles. There is a growing need for these systems to be socially responsive, capable of building rapport with users. In human-human interactions, rapport is critical to patient-doctor communication, conflict resolution, educational interactions, and social engagement. Rapport between people promotes successful collaboration, motivation, and task success. Dialogue systems which can build rapport with their user may produce similar effects, personalizing interactions to create better outcomes. This dissertation focuses on how dialogue systems can build rapport utilizing acoustic-prosodic entrainment. Acoustic-prosodic entrainment occurs when individuals adapt their acoustic-prosodic features of speech, such as tone of voice or loudness, to one another over the course of a conversation. Correlated with liking and task success, a dialogue system which entrains may enhance rapport. Entrainment, however, is very challenging to model. People entrain on different features in many ways and how to design entrainment to build rapport is unclear. The first goal of this dissertation is to explore how acoustic-prosodic entrainment can be modeled to build rapport. Towards this goal, this work presents a series of studies comparing, evaluating, and iterating on the design of entrainment, motivated and informed by human-human dialogue. These models of entrainment are implemented in the dialogue system of a robotic learning companion. Learning companions are educational agents that engage students socially to increase motivation and facilitate learning. As a learning companion’s ability to be socially responsive increases, so do vital learning outcomes. A second goal of this dissertation is to explore the effects of entrainment on concrete outcomes such as learning in interactions with robotic learning companions. This dissertation results in contributions both technical and theoretical. Technical contributions include a robust and modular dialogue system capable of producing prosodic entrainment and other socially-responsive behavior. One of the first systems of its kind, the results demonstrate that an entraining, social learning companion can positively build rapport and increase learning. This dissertation provides support for exploring phenomena like entrainment to enhance factors such as rapport and learning and provides a platform with which to explore these phenomena in future work.Dissertation/ThesisDoctoral Dissertation Computer Science 201

eXtended Reality for Education and Training

L'abstract è presente nell'allegato / the abstract is in the attachmen

Effective Tutoring with Empathic Embodied Conversational Agents

This thesis examines the prospect of using empathy in an Embodied Tutoring System (ETS) that guides students through an online quiz (by providing feedback on student answers and responding to self-reported student emotion). The ETS seeks to imitate human behaviours successfully used in one-to-one human tutorial interactions. The main hypothesis is that the interaction with an empathic ETS results in greater learning gains than a neutral ETS, primarily by encouraging positive and reducing negative student emotions using empathic feedback. In a preparatory study we investigated different strategies for expressing emotion by the ETS. We established that a multimodal strategy achieves the best results regarding how accurately human participants can recognise the emotions. This approach was used in developing the feedback strategy for our empathic ETS. The preparatory study was followed by two studies in which we compared a neutral with an empathic ETS. The ETS in the second of these studies was developed using results from the first of these studies. In both studies, we found no statistically significant difference in learning gains between the neutral and empathic ETS. However, we did discover a number of interactions between the ETS system, learning gains and, in particular 1) student scores on an empathic tendency test and 2) student ability. We also analysed the subjective responses and the relation between self-reported emotions during the quiz and student learning gains. Based on our studies in a formal class room setting, we assess the prospects of using empathic agents in a classroom setting and describe a number of requirements for their effective use

Affective Expressions in Conversational Agents for Learning Environments: Effects of curiosity, humour, and expressive auditory gestures

Conversational agents -- systems that imitate natural language discourse -- are becoming an increasingly prevalent human-computer interface, being employed in various domains including healthcare, customer service, and education. In education, conversational agents, also known as pedagogical agents, can be used to encourage interaction; which is considered crucial for the learning process. Though pedagogical agents have been designed for learners of diverse age groups and subject matter, they retain the overarching goal of eliciting learning outcomes, which can be broken down into cognitive, skill-based, and affective outcomes. Motivation is a particularly important affective outcome, as it can influence what, when, and how we learn. Understanding, supporting, and designing for motivation is therefore of great importance for the advancement of learning technologies. This thesis investigates how pedagogical agents can promote motivation in learners. Prior research has explored various features of the design of pedagogical agents and what effects they have on learning outcomes, and suggests that agents using social cues can adapt the learning environment to enhance both affective and cognitive outcomes. One social cue that is suggested to be of importance for enhancing learner motivation is the expression or simulation of affect in the agent. Informed by research and theory across multiple domains, three affective expressions are investigated: curiosity, humour, and expressive auditory gestures -- each aimed at enhancing motivation by adapting the learning environment in different ways, i.e., eliciting contagion effects, creating a positive learning experience, and strengthening the learner-agent relationship, respectively. Three studies are presented in which each expression was implemented in a separate type of agent: physically-embodied, text-based, and voice-based; with all agents taking on the role of a companion or less knowledgeable peer to the learner. The overall focus is on how each expression can be displayed, what the effects are on perception of the agent, and how it influences behaviour and learning outcomes. The studies result in theoretical contributions that add to our understanding of conversational agent design for learning environments. The findings provide support for: the simulation of curiosity, the use of certain humour styles, and the addition of expressive auditory gestures, in enhancing motivation in learners interacting with conversational agents; as well as indicating a need for further exploration of these strategies in future work

A study on the impact of a music looping technology intervention upon pre-service generalist teachers’ self-efficacy to teach music in primary schools

In Australia, in the current climate of economic rationalism in which there has been an increasing emphasis on literacy and numeracy, funding for specialised subjects like music has been reducing. As a result, generalist classroom teachers are being given more responsibility for delivering effective music education in primary schools. However, the time dedicated to training pre-service teachers in music education in tertiary institutions has diminished. Further, time constraints involved in building pre-service knowledge and skills in teaching music may impact many pre-service teachers’ beliefs about their ability to teach music. Within these constraints, digital technology may provide a key to improving pre-service teacher training in music education in universities, resulting in better quality delivery of music in schools. This study investigates the potential of digital looping technology to build generalist pre-service teachers’ knowledge of and efficacy for teaching music in primary schools. The study involved three stages of investigation: Stage One: an experimental and control intervention involving measuring the self-efficacy of pre-service teachers before and after they completed one unit of study incorporating looping technology; Stage Two: video analysis in a practicum setting; and Stage three: participant self-reflections following the practicum to investigate the transferability of pre-service teachers’ self-efficacy from university-based learning to classroom practice. Based upon the study, this thesis makes a number of recommendations for future practice in terms of generalist pre-service teacher training, as well as recommendations for future research

The Nature of Problem Solving

Solving non-routine problems is a key competence in a world full of changes, uncertainty and surprise where we strive to achieve so many ambitious goals. But the world is also full of solutions because of the extraordinary competences of humans who search for and find them. We must explore the world around us in a thoughtful way, acquire knowledge about unknown situations efficiently, and apply new and existing knowledge creatively. The Nature of Problem Solving presents the background and the main ideas behind the development of the PISA 2012 assessment of problem solving, as well as results from research collaborations that originated within the group of experts who guided the development of this assessment. It illustrates the past, present and future of problem-solving research and how this research is helping educators prepare students to navigate an increasingly uncertain, volatile and ambiguous world

Designing human-agent interaction for AI literacy of novice adult learners

In the coming few years, at least 50% of the workers would need to upskill for AI-based jobs such as data analysts, process automation specialists, and digital transformation specialists. Currently, existing AI courses are not suitable for novice adult learners learning online as these courses require technical knowledge, do not involve practical activities, or require teaching assistance. Previous user studies with practical AI literacy systems found students needed assistance in selecting input data, iteratively building AI, and measuring AI’s performance. A pedagogical agent (computer program to assist in multimedia teaching) could provide this assistance to novice adult learners. A limited number of AI literacy pedagogical agents for novice adult learners in an online context have been designed and evaluated. The goal of this thesis is to identify: (1) the user needs of novice adult learners, (2) user experience (UX) goals for an AI literacy system to be used by novice adult learners in an online self-study context, and (3) human-agent interaction that satisfies user needs and achieve UX goals. This thesis followed an Experience-Driven Design (EDD) approach; the user needs of novice adult learners in AI literacy in an online context were identified, and then human-agent interaction to aid novice adult learners in AI literacy in an online context was designed and evaluated. A user needs study with 8 participants helped to identify the UX goals of relatedness, engagement, and competence; and it revealed that adult learners need a social agent to provide emotional- and task-based feedback to guide them in the correct direction. Two human-agent interactions were iteratively designed based on feedback types: (1) task feedback, and (2) emotional and task-based feedback. An evaluation study with these agent feedback types was conducted with 8 participants using a between-subject design. The results showed that a pedagogical agent for AI literacy in an online context should (1) provide immediate emotions to communicate AI confidence in real-time, (2) use sounds to provide data input feedback, and (3) answer task-related questions

Teaching Statistics for Social Justice - An Autoethnographic Research Report

The following autoethnography was completed by two graduate students at University A learning to enact teaching for social justice while building content underpinnings in statistics at University B. The authors present a research base for teaching for social justice followed by a description of their lesson, observations during enactment, and reflection of change in beliefs about teaching for social justice afterward. Findings in this study are shared from the authors’ personal perspectives through the enactment of teaching a lesson for social justice in an undergraduate statistics course at University B. Implications provide encouragement that the inclusion of social justice topics in undergraduate and graduate level teacher educator coursework may improve teacher attention to equity in practice

Real-Time Affective Support to Promote Learner’s Engagement

abstract: Research has shown that the learning processes can be enriched and enhanced with the presence of affective interventions. The goal of this dissertation was to design, implement, and evaluate an affective agent that provides affective support in real-time in order to enrich the student’s learning experience and performance by inducing and/or maintaining a productive learning path. This work combined research and best practices from affective computing, intelligent tutoring systems, and educational technology to address the design and implementation of an affective agent and corresponding pedagogical interventions. It included the incorporation of the affective agent into an Exploratory Learning Environment (ELE) adapted for this research. A gendered, three-dimensional, animated, human-like character accompanied by text- and speech-based dialogue visually represented the proposed affective agent. The agent’s pedagogical interventions considered inputs from the ELE (interface, model building, and performance events) and from the user (emotional and cognitive events). The user’s emotional events captured by biometric sensors and processed by a decision-level fusion algorithm for a multimodal system in combination with the events from the ELE informed the production-rule-based behavior engine to define and trigger pedagogical interventions. The pedagogical interventions were focused on affective dimensions and occurred in the form of affective dialogue prompts and animations. An experiment was conducted to assess the impact of the affective agent, Hope, on the student’s learning experience and performance. In terms of the student’s learning experience, the effect of the agent was analyzed in four components: perception of the instructional material, perception of the usefulness of the agent, ELE usability, and the affective responses from the agent triggered by the student’s affective states. Additionally, in terms of the student’s performance, the effect of the agent was analyzed in five components: tasks completed, time spent solving a task, planning time while solving a task, usage of the provided help, and attempts to successfully complete a task. The findings from the experiment did not provide the anticipated results related to the effect of the agent; however, the results provided insights to improve diverse components in the design of affective agents as well as for the design of the behavior engines and algorithms to detect, represent, and handle affective information.Dissertation/ThesisDoctoral Dissertation Computer Science 201