THE ROLE OF SIMULATION IN SUPPORTING LONGER-TERM LEARNING AND MENTORING WITH TECHNOLOGY

Mentoring is an important part of professional development and longer-term learning. The nature of longer-term mentoring contexts means that designing, developing, and testing adaptive learning sys-tems for use in this kind of context would be very costly as it would require substantial amounts of fi-nancial, human, and time resources. Simulation is a cheaper and quicker approach for evaluating the impact of various design and development decisions. Within the Artificial Intelligence in Education (AIED) research community, however, surprisingly little attention has been paid to how to design, de-velop, and use simulations in longer-term learning contexts. The central challenge is that adaptive learning system designers and educational practitioners have limited guidance on what steps to consider when designing simulations for supporting longer-term mentoring system design and development deci-sions. My research work takes as a starting point VanLehn et al.’s [1] introduction to applications of simulated students and Erickson et al.’s [2] suggested approach to creating simulated learning envi-ronments. My dissertation presents four research directions using a real-world longer-term mentoring context, a doctoral program, for illustrative purposes. The first direction outlines a framework for guid-ing system designers as to what factors to consider when building pedagogical simulations, fundamen-tally to answer the question: how can a system designer capture a representation of a target learning context in a pedagogical simulation model? To illustrate the feasibility of this framework, this disserta-tion describes how to build, the SimDoc model, a pedagogical model of a longer-term mentoring learn-ing environment – a doctoral program. The second direction builds on the first, and considers the issue of model fidelity, essentially to answer the question: how can a system designer determine a simulation model’s fidelity to the desired granularity level? This dissertation shows how data from a target learning environment, the research literature, and common sense are combined to achieve SimDoc’s medium fidelity model. The third research direction explores calibration and validation issues to answer the question: how many simulation runs does it take for a practitioner to have confidence in the simulation model’s output? This dissertation describes the steps taken to calibrate and validate the SimDoc model, so its output statistically matches data from the target doctoral program, the one at the university of Saskatchewan. The fourth direction is to demonstrate the applicability of the resulting pedagogical model. This dissertation presents two experiments using SimDoc to illustrate how to explore pedagogi-cal questions concerning personalization strategies and to determine the effectiveness of different men-toring strategies in a target learning context. Overall, this dissertation shows that simulation is an important tool in the AIED system design-ers’ toolkit as AIED moves towards designing, building, and evaluating AIED systems meant to support learners in longer-term learning and mentoring contexts. Simulation allows a system designer to exper-iment with various design and implementation decisions in a cost-effective and timely manner before committing to these decisions in the real world

"Teach AI How to Code": Using Large Language Models as Teachable Agents for Programming Education

This work investigates large language models (LLMs) as teachable agents for learning by teaching (LBT). LBT with teachable agents helps learners identify their knowledge gaps and discover new knowledge. However, teachable agents require expensive programming of subject-specific knowledge. While LLMs as teachable agents can reduce the cost, LLMs' over-competence as tutees discourages learners from teaching. We propose a prompting pipeline that restrains LLMs' competence and makes them initiate "why" and "how" questions for effective knowledge-building. We combined these techniques into TeachYou, an LBT environment for algorithm learning, and AlgoBo, an LLM-based tutee chatbot that can simulate misconceptions and unawareness prescribed in its knowledge state. Our technical evaluation confirmed that our prompting pipeline can effectively configure AlgoBo's problem-solving performance. Through a between-subject study with 40 algorithm novices, we also observed that AlgoBo's questions led to knowledge-dense conversations (effect size=0.73). Lastly, we discuss design implications, cost-efficiency, and personalization of LLM-based teachable agents

The ties that bind: Social Interaction in Conversational Agents

This document is the English version of: Justine Cassell, «« Tisser des liens »», Réseaux 2020/2 (No 220-221) , p. 21-45International audienceThe article argues for a genre of AI capable of building social bonds with humans. The argument’s starting point is the two competing origin stories of Artificial Intelligence. In one, the goal of AI was to create machines that could simulate every aspect of human intelligence. In the other, it was to build machines that adapt closely to natural human behaviour. While the first story is better known, it is argued that the second would have been more fruitful, as it places the human at the heart of the endeavour. Based on this historical perspective, the article provides several examples of conversational agents that engage in this kind of adaptive social behaviour. Results of experiments with these social agents find that they do in fact improve relations between people and the systems. Additionally, they improve performance on the task that the human and the conversational agent are conducting together

Interacting with educational chatbots: A systematic review

Chatbots hold the promise of revolutionizing education by engaging learners, personalizing learning activities, supporting educators, and developing deep insight into learners’ behavior. However, there is a lack of studies that analyze the recent evidence-based chatbot-learner interaction design techniques applied in education. This study presents a systematic review of 36 papers to understand, compare, and reflect on recent attempts to utilize chatbots in education using seven dimensions: educational field, platform, design principles, the role of chatbots, interaction styles, evidence, and limitations. The results show that the chatbots were mainly designed on a web platform to teach computer science, language, general education, and a few other fields such as engineering and mathematics. Further, more than half of the chatbots were used as teaching agents, while more than a third were peer agents. Most of the chatbots used a predetermined conversational path, and more than a quarter utilized a personalized learning approach that catered to students’ learning needs, while other chatbots used experiential and collaborative learning besides other design principles. Moreover, more than a third of the chatbots were evaluated with experiments, and the results primarily point to improved learning and subjective satisfaction. Challenges and limitations include inadequate or insufficient dataset training and a lack of reliance on usability heuristics. Future studies should explore the effect of chatbot personality and localization on subjective satisfaction and learning effectiveness

Rudeness and Rapport: Insults and Learning Gains in Peer Tutoring

Abstract. For 20 years, researchers have envisioned artificially intelligent learning companions that evolve with their students as they grow and learn. However, while communication theory suggests that positivity decreases over time in relationships, most tutoring systems designed to build rapport with a student remain adamantly polite, and may therefore inadvertently distance the learner from the agent over time. We present an analysis of high school friends interacting in a peer tutoring environment as a step towards designing agents that sustain long-term pedagogical relationships with learners. We find that tu-tees and tutors use different language behaviors: tutees express more playful-ness and face-threat, while tutors attend more to the task. This face-threat by the tutee is associated with increased learning gains for their tutor. Additionally, a small sample of partners who were strangers learned less than friends, and in these dyads increased face-threat was negatively correlated with learning. Our findings support the idea that learning companions should gradually move to-wards playful face-threat as they build relationships with their students

KARAKTERISASI KETERAMPILAN COLLABORATIVE PROBLEM SOLVING (CPS) PESERTA DIDIK SMK MELALUI PENILAIAN BERBASIS WEB PADA MATERI LISTRIK DINAMIS

Proses penilaian keterampilan Collaborative Problem Solving (CPS) khususnya di Indonesia masih jarang ditemukan. Studi ini menggunakan metode online untuk menggali karakteristik keterampilan CPS peserta didik melalui penilaian berbasis web interaktif pada laman webcps.site. Peserta didik bekerja secara berpasangan menyelesaikan dua task tentang materi listrik dinamis. Metode yang digunakan dalam penelitian ini adalah metode survey. Data aliran proses diambil dari 15 grup yang dikelompokkan secara acak dari 30 peserta didik (17 diantaranya peserta didik laki-laki dan 13 peserta didik perempuan) di salah satu Sekolah Menengah Kejuruan Pulau Bengkalis di provinsi Riau. Data ditransformasikan menjadi indikator keterampilan Collaborative Problem Solving berdasarkan kata kunci dengan bantuan software Nvivo 12 Plus, kemudian dikategorikan berdasarkan empat level yaitu Beginner (level 1), Emerging (level 2), intermediate (level 3), dan advance (level 4). Hasil penelitian menunjukkan bahwa siswa berada di berbagai level dengan karakteristik yang berbeda di kedua domain sosial dan domain kognitif dalam keterampilan CPS. Penilaian yang digunakan dalam penelitian ini dapat digunakan sebagai instrumen pengukuran keterampilan pemecahan masalah kolaboratif. The process of assessing Collaborative Problem Solving (CPS) skills, especially in Indonesia, is still rare. This study uses an online method to explore the characteristics of students' CPS skills through interactive web-based assessments on the webcps.site page. Students work in pairs to complete two tasks about dynamic electrical material. The method used in this research is a survey method. Process flow data were taken from 15 groups randomly grouped from 30 students (17 of whom were male students and 13 were female students) in one of the Bengkalis Island Vocational High Schools in Riau province. The data is transformed into indicators of Collaborative Problem Solving skills based on keywords with the help of Nvivo 12 Plus software, then categorized based on four levels, namely Beginner (level 1), Emerging (level 2), intermediate (level 3), and advanced (level 4). The results showed that students were at various levels with different characteristics in both the social domain and the cognitive domain in CPS skills. The assessment used in this study can be used as an instrument for measuring collaborative problem solving skills

A Systematic Review of Voice-based Intelligent Virtual Agents in EFL Education

Since its debut in the field of education nearly three decades ago, Artificial Intelligence (AI) has been considered as a powerful tool to facilitate new paradigms for instructional design and innovative educational practice in the form of intelligent tutoring systems, chatbots, teaching robots and adaptive learning systems among others. Recent technological advances in the adjacent areas of natural language processing, machine learning and computer graphics focusing primarily on design features that can improve their human-like qualities of naturalness and believability as human interlocutors have also amplified new application opportunities for Intelligent Virtual Agents (IVAs) or Animated Pedagogical Agents (APAs) within the area of Intelligent Computer-Assisted Language Learning (ICALL). Although AI-powered IVAs hold the potential to enhance the learning process in nearly any knowledge domain and personalize automation in teaching by embodying different roles in the learning environment, strikingly few studies have empirically attempted to assess IVAs impact on L2 learners’ academic achievement when learning English as a Foreign (EFL) so far. This study addresses this issue via a systematic review of relevant interventionist IVA studies that were conducted in EFL settings and published within the 2015-2020 timeframe examining IVAs key affordances, major barriers in their adoption for language learning purposes, and the CALL research trends currently prevalent on the topic. Pedagogical implications for the effective implementation of IVA technology in L2 contexts are discussed and future research avenues in the area are highlighted

Effective Tutoring with Empathic Embodied Conversational Agents

This thesis examines the prospect of using empathy in an Embodied Tutoring System (ETS) that guides students through an online quiz (by providing feedback on student answers and responding to self-reported student emotion). The ETS seeks to imitate human behaviours successfully used in one-to-one human tutorial interactions. The main hypothesis is that the interaction with an empathic ETS results in greater learning gains than a neutral ETS, primarily by encouraging positive and reducing negative student emotions using empathic feedback. In a preparatory study we investigated different strategies for expressing emotion by the ETS. We established that a multimodal strategy achieves the best results regarding how accurately human participants can recognise the emotions. This approach was used in developing the feedback strategy for our empathic ETS. The preparatory study was followed by two studies in which we compared a neutral with an empathic ETS. The ETS in the second of these studies was developed using results from the first of these studies. In both studies, we found no statistically significant difference in learning gains between the neutral and empathic ETS. However, we did discover a number of interactions between the ETS system, learning gains and, in particular 1) student scores on an empathic tendency test and 2) student ability. We also analysed the subjective responses and the relation between self-reported emotions during the quiz and student learning gains. Based on our studies in a formal class room setting, we assess the prospects of using empathic agents in a classroom setting and describe a number of requirements for their effective use

Astus, une plateforme pour créer et étudier les systèmes tutoriels intelligents « par traçage de modèle »

Cette thèse s’intéresse aux systèmes tutoriels intelligents (STI), un type d’environnement informatique pour l’apprentissage humain (EIAH) qui se distingue des autres (p. ex. les exerciseurs et les hypermédias éducatifs) en offrant un mécanisme d’évaluation plus sophistiqué. Parmi les différentes familles de STI, ce sont les STI « par traçage de modèle » (MTT) qui ont le plus fait leurs preuves. Les MTT sont critiqués, premièrement parce qu’ils évaluent l’apprenant de façon serrée (c.-à-d. qui positionne l’action de l’apprenant par rapport à une ou plusieurs méthodes pour effectuer la tâche), ce qui n’est possible que pour des tâches bien définies. Par conséquent, on leur reproche d’encourager un apprentissage superficiel. Deuxièmement, parce que les efforts de création qu’ils requièrent sont jugés prohibitifs, ce qui a mené à l’apparition d’autres familles de STI, comme les STI « par contraintes » et les STI « par traçage d’exemples » et ceux basés sur l’apprentissage automatique. Par cette thèse, nous voulons contribuer à renouveler l’intérêt pour les MTT en améliorant le rapport entre les efforts de création et l’efficacité potentielle des interventions, et en établissant plus clairement leur rôle pédagogique. Pour ce faire, nous proposons la plateforme Astus qui permet d’explorer l’espace qui existe entre les MTT créés avec les plateformes existantes, et des MTT dédiés ayant recours à des connaissances didactiques sophistiquées (p. ex. des dialogues) qui exigent des efforts de création encore plus importants. La plateforme Astus se distingue des plateformes existantes parce qu’elle génère des interventions plutôt que de recourir à des interventions prémâchées et qu’elle supporte les tâches s’effectuant dans des environnements qui ont une dimension physique. La génération des interventions dépend : d’un modèle de la tâche qui s’inscrit dans le paradigme du tuteur, c’est-à-dire qui représente une abstraction et une généralisation des instructions d’un tuteur humain; d’un modèle de l’UI qui permet des interventions riches comme une démonstration (c.-à-d. déplacements du pointeur et simulation des clics et des saisies); de langages dédiés et d’outils qui réduisent les efforts de création des auteurs; de mécanismes d’extension qui permettent d’adapter la génération en fonction d’une stratégie pédagogique particulière. Le paradigme du tuteur, parce qu’il favorise une communication transparente entre le système et l’apprenant, met en évidence les avantages et les désavantages de l’approche pédagogique des MTT, essentiellement une évaluation précise (c.-à-d. qui permet de produire des indices sur la prochaine étape et des rétroactions sur les erreurs), mais serrée. En s’inscrivant explicitement le paradigme du tuteur, entre autres en évitant de tirer profit de la nature de domaines particuliers ou de propriétés de tâches particulières pour assouplir l’évaluation, la plateforme Astus se démarque plus nettement des autres familles de STI que les autres MTT. Par conséquent, elle établit plus clairement le rôle pédagogique des MTT. Cinq expérimentations (menées par Luc Paquette) à petite échelle ont été réalisées auprès d’étudiants au baccalauréat au département d’informatique (un laboratoire pour la manipulation d’arbres binaires de recherche et un pour la conversion de nombres en virgule flottante). Ces expérimentations indiquent que les interventions générées sont efficaces. Au-delà de ces résultats, c’est le processus entourant ces expérimentations, parce qu’il est comparable au processus des chercheurs potentiellement intéressés par la plateforme Astus, qui montre que la version présentée dans cette thèse est plus qu’un prototype et qu’elle peut être utilisée à l’interne dans un contexte réel