Modelling speaker adaptation in second language learner dialogue

Understanding how tutors and students adapt to one another within Second Language (L2) learning is an important step in the development of better automated tutoring tools for L2 conversational practice. Such an understanding can not only inform conversational agent design, but can be useful for other pedagogic applications such as formative assessment, self reflection on tutoring practice, learning analytics, and conversation modelling for personalisation and adaptation. Dialogue is a challenging domain for natural language processing, understanding, and generation. It is necessary to understand how participants adapt to their interlocutor, changing what they express and how they express it as they update their beliefs about the knowledge, preferences, and goals of the other person. While this adaptation is natural to humans, it is an open problem for dialogue systems, where managing coherence across utterances is an active area of research, even without adaptation. This thesis extends our understanding of adaptation in human dialogue, to better implement this in agent-based conversational dialogue. This is achieved through comparison to fluent conversational dialogues and across student ability levels. Specifically, we are interested in how adaptation takes place in terms of the linguistic complexity, lexical alignment and the dialogue act usage demonstrated by the speakers within the dialogue. Finally, with the end goal of an automated tutor in mind, the student alignment levels are used to compare dialogues between student and human tutor with those where the tutor is an agent. We argue that the lexical complexity, alignment and dialogue style adaptation we model in L2 human dialogue are signs of tutoring strategies in action, and hypothesise that creating agents which adapt to these aspects of dialogue will result in better environments for learning. We hypothesise that with a more adaptive agent, student alignment may increase, potentially resulting in improved engagement and learning. We find that In L2 practice dialogues, both student and tutor adapt to each other, and this adaptation depends on student ability. Tutors adapt to push students of higher ability, and to encourage students of lower ability. Complexity, dialogue act usage and alignment are used differently by speakers in L2 dialogue than within other types of conversational dialogue, and changes depending on the learner proficiency. We also find different types of learner behaviours within automated L2 tutoring dialogues to those present in human ones, using alignment to measure this. This thesis contributes new findings on interlocutor adaptation within second language practice dialogue, with an emphasis on how these can be used to improve tutoring dialogue agents

Collaborative trails in e-learning environments

This deliverable focuses on collaboration within groups of learners, and hence collaborative trails. We begin by reviewing the theoretical background to collaborative learning and looking at the kinds of support that computers can give to groups of learners working collaboratively, and then look more deeply at some of the issues in designing environments to support collaborative learning trails and at tools and techniques, including collaborative filtering, that can be used for analysing collaborative trails. We then review the state-of-the-art in supporting collaborative learning in three different areas – experimental academic systems, systems using mobile technology (which are also generally academic), and commercially available systems. The final part of the deliverable presents three scenarios that show where technology that supports groups working collaboratively and producing collaborative trails may be heading in the near future

Persuasive and adaptive tutorial dialogues for a medical diagnosis tutoring system

The objective of this thesis is to address a key problem in the development of an intelligent tutoring system, that is, the implementation of the verbal exchange (a dialogue) that takes place between a student and the system. Here we consider TeachMed, a medical diagnosis tutoring system that teaches the students to diagnose clinical problems. However, approaches that are presented could also fit other tutoring systems. In such a system, a dialogue must be implemented that determines when and how pedagogic aid is provided to the student, that is, what to say to her, in what circumstances, and how to say it. Finite state machines and automated planning systems are so far the two most common approaches for implementing tutoring dialogues in intelligent tutoring systems. In the former approach, finite state machines of dialogues are manually designed and hard coded in intelligent tutoring systems. This is a straightforward but very time consuming approach. Furthermore, any change or extension to the hard coded finite state machines is very difficult as it requires reprogramming the system. On the other hand, automated planning has long been presented as a promising technique for automatic dialogue generating. However, in existing approaches, the requirement for the system to persuade the student is not formally acknowledged. Moreover, current dialogue planning approaches are not able to reason on uncertainties about the student's knowledge. This thesis presents two approaches for generating more effective tutorial dialogues.The first approach describes an argumentation framework for implementing persuasive tutoring dialogues. In this approach the entire interaction between the student and the tutoring system is seen as argumentation.The tutoring system and the student can settle conflicts arising during their argumentation by accepting, challenging, or questioning each other's arguments or withdrawing their own arguments. Pedagogic strategies guide the tutoring system by selecting arguments aimed at convincing the student.The second approach presents a non-deterministic planning technique which models the dialogue generation problem as one of planning with incomplete knowledge and sensing. This approach takes into account incomplete information about a particular fact of the student's knowledge by creating conditional branches in a dialogue plan such that each branch represents an adaptation of the dialogue plan with respect to a particular state of the student's knowledge or belief concerning the desired fact. In order to find out the real state of the student's knowledge and to choose the right branch at execution time, the planner includes some queries in the dialogue plan so that the tutoring system can ask the student to gather missing information. One contribution in this thesis is improving the quality of tutoring dialogues by engaging students in argumentative interactions and/or adapting the dialogues with respect to the student's knowledge. Another one is facilitating the design and implementation of tutoring by turning to automatically generated dialogues as opposed to manually generated ones

Comprehension based adaptive learning systems

Conversational Intelligent Tutoring Systems aim to mimic the adaptive behaviour of human tutors by delivering tutorial content as part of a dynamic exchange of information conducted using natural language. Deciding when it is beneficial to intervene in a student’s learning process is an important skill for tutoring. Human tutors use prior knowledge about the student, discourse content and learner non-verbal behaviour to choose when intervention will help learners overcome impasse. Experienced human tutors adapt discourse and pedagogy based on recognition of comprehension and non-comprehension indicative learner behaviour. In this research non-verbal behaviour is explored as a method of computationally analysing reading comprehension so as to equip an intelligent conversational agent with the human-like ability to estimate comprehension from non-verbal behaviour as a decision making trigger for feedback, prompts or hints. This thesis presents research that combines a conversational intelligent tutoring system (CITS) with near real-time comprehension classification based on modelling of e-learner non-verbal behaviour to estimate learner comprehension during on-screen conversational tutoring and to use comprehension classifications as a trigger for intervening with hints, prompts or feedback for the learner. To improve the effectiveness of tuition in e-learning, this research aims to design, develop and demonstrate novel computational methods for modelling e-learner comprehension of on-screen information in near real-time and for adapting CITS tutorial discourse and pedagogy in response to perception of comprehension indicative behaviour. The contribution of this research is to detail the motivation for, design of, and evaluation of a system which has the human-like ability to introduce micro-adaptive feedback into tutorial discourse in response to automatic perception of e-learner reading comprehension. This research evaluates empirically whether e-learner non-verbal behaviour can be modelled to classify comprehension in near real-time and presents a near real-time comprehension classification system which achieves normalised comprehension classification accuracy of 75%. Understanding e-learner comprehension creates exciting opportunities for advanced personalisation of materials, discourse, challenge and the digital environment itself. The research suggests a benefit is gained from comprehension based adaptation in conversational intelligent tutoring systems, with a controlled trial of a comprehension based adaptive CITS called Hendrix 2.0 showing increases in tutorial assessment scores of up to 17% when comprehension based discourse adaptation is deployed to scaffold the learning experience

Methodological Tools for Linguistic Description and Typology.

International audienc

Effective Tutoring with Empathic Embodied Conversational Agents

This thesis examines the prospect of using empathy in an Embodied Tutoring System (ETS) that guides students through an online quiz (by providing feedback on student answers and responding to self-reported student emotion). The ETS seeks to imitate human behaviours successfully used in one-to-one human tutorial interactions. The main hypothesis is that the interaction with an empathic ETS results in greater learning gains than a neutral ETS, primarily by encouraging positive and reducing negative student emotions using empathic feedback. In a preparatory study we investigated different strategies for expressing emotion by the ETS. We established that a multimodal strategy achieves the best results regarding how accurately human participants can recognise the emotions. This approach was used in developing the feedback strategy for our empathic ETS. The preparatory study was followed by two studies in which we compared a neutral with an empathic ETS. The ETS in the second of these studies was developed using results from the first of these studies. In both studies, we found no statistically significant difference in learning gains between the neutral and empathic ETS. However, we did discover a number of interactions between the ETS system, learning gains and, in particular 1) student scores on an empathic tendency test and 2) student ability. We also analysed the subjective responses and the relation between self-reported emotions during the quiz and student learning gains. Based on our studies in a formal class room setting, we assess the prospects of using empathic agents in a classroom setting and describe a number of requirements for their effective use

An Actor-Centric Approach to Facial Animation Control by Neural Networks For Non-Player Characters in Video Games

Game developers increasingly consider the degree to which character animation emulates facial expressions found in cinema. Employing animators and actors to produce cinematic facial animation by mixing motion capture and hand-crafted animation is labor intensive and therefore expensive. Emotion corpora and neural network controllers have shown promise toward developing autonomous animation that does not rely on motion capture. Previous research and practice in disciplines of Computer Science, Psychology and the Performing Arts have provided frameworks on which to build a workflow toward creating an emotion AI system that can animate the facial mesh of a 3d non-player character deploying a combination of related theories and methods. However, past investigations and their resulting production methods largely ignore the emotion generation systems that have evolved in the performing arts for more than a century. We find very little research that embraces the intellectual process of trained actors as complex collaborators from which to understand and model the training of a neural network for character animation. This investigation demonstrates a workflow design that integrates knowledge from the performing arts and the affective branches of the social and biological sciences. Our workflow begins at the stage of developing and annotating a fictional scenario with actors, to producing a video emotion corpus, to designing training and validating a neural network, to analyzing the emotion data annotation of the corpus and neural network, and finally to determining resemblant behavior of its autonomous animation control of a 3d character facial mesh. The resulting workflow includes a method for the development of a neural network architecture whose initial efficacy as a facial emotion expression simulator has been tested and validated as substantially resemblant to the character behavior developed by a human actor

The significance of silence. Long gaps attenuate the preference for ‘yes’ responses in conversation.

In conversation, negative responses to invitations, requests, offers and the like more often occur with a delay – conversation analysts talk of them as dispreferred. Here we examine the contrastive cognitive load ‘yes’ and ‘no’ responses make, either when given relatively fast (300 ms) or delayed (1000 ms). Participants heard minidialogues, with turns extracted from a spoken corpus, while having their EEG recorded. We find that a fast ‘no’ evokes an N400-effect relative to a fast ‘yes’, however this contrast is not present for delayed responses. This shows that an immediate response is expected to be positive – but this expectation disappears as the response time lengthens because now in ordinary conversation the probability of a ‘no’ has increased. Additionally, however, 'No' responses elicit a late frontal positivity both when they are fast and when they are delayed. Thus, regardless of the latency of response, a ‘no’ response is associated with a late positivity, since a negative response is always dispreferred and may require an account. Together these results show that negative responses to social actions exact a higher cognitive load, but especially when least expected, as an immediate response

Proceedings ICPW'07: 2nd International Conference on the Pragmatic Web, 22-23 Oct. 2007, Tilburg: NL

Proceedings ICPW'07: 2nd International Conference on the Pragmatic Web, 22-23 Oct. 2007, Tilburg: N