THE ROLE OF SIMULATION IN SUPPORTING LONGER-TERM LEARNING AND MENTORING WITH TECHNOLOGY

Mentoring is an important part of professional development and longer-term learning. The nature of longer-term mentoring contexts means that designing, developing, and testing adaptive learning sys-tems for use in this kind of context would be very costly as it would require substantial amounts of fi-nancial, human, and time resources. Simulation is a cheaper and quicker approach for evaluating the impact of various design and development decisions. Within the Artificial Intelligence in Education (AIED) research community, however, surprisingly little attention has been paid to how to design, de-velop, and use simulations in longer-term learning contexts. The central challenge is that adaptive learning system designers and educational practitioners have limited guidance on what steps to consider when designing simulations for supporting longer-term mentoring system design and development deci-sions. My research work takes as a starting point VanLehn et al.’s [1] introduction to applications of simulated students and Erickson et al.’s [2] suggested approach to creating simulated learning envi-ronments. My dissertation presents four research directions using a real-world longer-term mentoring context, a doctoral program, for illustrative purposes. The first direction outlines a framework for guid-ing system designers as to what factors to consider when building pedagogical simulations, fundamen-tally to answer the question: how can a system designer capture a representation of a target learning context in a pedagogical simulation model? To illustrate the feasibility of this framework, this disserta-tion describes how to build, the SimDoc model, a pedagogical model of a longer-term mentoring learn-ing environment – a doctoral program. The second direction builds on the first, and considers the issue of model fidelity, essentially to answer the question: how can a system designer determine a simulation model’s fidelity to the desired granularity level? This dissertation shows how data from a target learning environment, the research literature, and common sense are combined to achieve SimDoc’s medium fidelity model. The third research direction explores calibration and validation issues to answer the question: how many simulation runs does it take for a practitioner to have confidence in the simulation model’s output? This dissertation describes the steps taken to calibrate and validate the SimDoc model, so its output statistically matches data from the target doctoral program, the one at the university of Saskatchewan. The fourth direction is to demonstrate the applicability of the resulting pedagogical model. This dissertation presents two experiments using SimDoc to illustrate how to explore pedagogi-cal questions concerning personalization strategies and to determine the effectiveness of different men-toring strategies in a target learning context. Overall, this dissertation shows that simulation is an important tool in the AIED system design-ers’ toolkit as AIED moves towards designing, building, and evaluating AIED systems meant to support learners in longer-term learning and mentoring contexts. Simulation allows a system designer to exper-iment with various design and implementation decisions in a cost-effective and timely manner before committing to these decisions in the real world

Mindful documentary

Thesis (Ph. D.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2005.Includes bibliographical references (leaves 86-92).In the practice of documentary creation, a videographer performs an elaborate balancing act between observing the world, deciding what to record, and understanding the implications of the recorded material, all with respect to her primary goal of story construction. This thesis presents mindful documentary, a model of a videographer's cyclical process of thinking and constructing during a documentary production. The purpose of this model is to better support documentary creation through systems that assist the documentary videographer in discovering new methods of observation, ways of thinking, and novel stories while recording the world. Based on the mindful documentary model, a reflective partnership is established between the videographer and a camera with commonsense reasoning abilities during capture and organization of documentary video collections. Knowledge is solicited from the videographer at the point of capture; it is used to generate narrative or contextual shot suggestions, which provide alternative recording path ideas for the videographer. Thus, the system encourages the videographer to reflect on the story possibilities of a documentary collection during real-time capture. Qualitative results of studies with a group of videographers - including novices and experts - showed a willingness to take suggestions during documentary production and, in some cases, to alter the recording path after reflection on shot possibilities presented by the system. Moreover, suggestions often had increased influence on the recording path if they were not taken as directives but as catalysts, i.e., prompts to expand thinking about the documentary subject rather than explicit shot instructions.(cont.) Critical lessons were learned about methodology and system design for documentary production. As a documentary is built, evidence of what the videographer has learned is represented in the documentary. The model, methodology, and system presented in this thesis provide a basis for understanding how videographers think during documentary construction and how machines with commonsense reasoning resources can serve as creative storytelling partners.by Barbara A. Barry.Ph.D

Evaluation of computer aided instruction: Assessing the value and effectiveness of operational systems

This thesis investigated a number of performance measures for computer-aided instruction (CAI) systems. These "evaluation metrics" are intended to assess the worth and value of teaching systems. An operational accounting tutor (which teaches marginal costing) was used to develop the metrics and a replication study was conducted on Application Program Tutor (a tutoring system designed to teach courses). Although, CAI is a mature technology which has evolved in a variety of fields and forms since the 1950s, its potential remains untapped. Factors attributed to this include resistance from teachers, lack of student involvement in design, and insufficient imagination in curriculum design. Inadequate system standards and a deficiency of good software tools, lack of documentation, maintenance and education value have also been key limiting factors. The overall picture seems akin to a cottage industry than a co-ordinated enterprise. Evaluation is significant, to developers and users in this field, because in the short-term it improves the usability and life-span of the numerous systems that have been developed and in the long-term it focuses attention (away from the impetus to deliver) towards issues of appropriateness and quality in system design. Different traditions of evaluation are explored, including the selection criteria used in educational technology and the impact of the quality philosophy on software engineering. This research was conducted using the Before-after Two-group design on forty-two accountancy students, where a conventionally taught group was compared with the accountancy tutor group. Performance on a number of marginal (or variable) costing problems was measured before and after both groups were taught. Moreover, the experimental group was given a questionnaire to complete (which was designed to capture their assessment of the system). The results derived from the well- crafted questionnaire were indicative of the systems strengths and weaknesses and supplied useful criteria for future research

Développement d'un système tutoriel intelligent pour l'apprentissage du raisonnement logique

Le raisonnement est un processus cognitif nous permettant de tirer à partir de règles, des conclusions sur des faits et situations de la vie. Il est considéré dans la littérature comme étant une partie intégrante de plusieurs autres processus humains comme la perception (résultat d'une combinaison inductive entre les senseurs et la mémoire), la catégorisation, la compréhension, la prise de décision, la résolution de problèmes. L'humain a tendance à effectuer des raisonnements erronés sans s'en rendre compte. Les erreurs de raisonnement seraient des inférences créatives dans un contexte où il aurait été approprié de faire une inférence déductive. L'acquisition de la compétence en raisonnement consiste alors à apprendre à être déductif et ainsi à acquérir des structures plus organisées pour mieux systématiser notre information et pour devenir cognitivement plus performants. L'avènement des technologies de l'information allié à l'évolution du domaine de l'Intelligence Artificielle, a permis le développement de systèmes tel que les systèmes tutoriels intelligents (STI). Ces systèmes sont caractérisés par le fait qu'ils permettent d'automatiser l'enseignement et de favoriser l'apprentissage sans l'intervention d'un tuteur humain. Ce projet vise le développement d'un STI générique appelé Muse-logique et dédié à l'apprentissage du raisonnement logique. Dans ce document, nous présentons en premier lieu la problématique et les objectifs de notre projet. Une étude détaillée des domaines en jeu est faite par la suite. Enfin, l'architecture, les éléments conceptuels et les résultats d'implémentation du système, sont exposés ainsi qu'une validation préliminaire du système. Le contenu et l'élaboration des différents composants du STI on fait l'objet d'un travail minutieux effectué avec la participation active des membres d'une équipe multidisciplinaire. Le composant expert est soutenu par un ensemble de règles du domaine. Le modèle cognitif de l'apprenant est soutenu par un réseau bayésien et enfin, le modèle pédagogique est soutenu par des règles tutorielles étudiées et validées théoriquement. La particularité et la robustesse de Muse-logique résident dans son cadre de développement composé d'experts en sciences cognitives, en systèmes tutoriels intelligent et en raisonnement logique. Des perspectives futures sont envisagées pour les prochaines versions.\ud ______________________________________________________________________________ \ud MOTS-CLÉS DE L’AUTEUR : raisonnement logique, systèmes tutoriels intelligents, logique, système adaptatif, réseau bayésien, feedbacks, représentation des connaissances

Augmented Conversation and Cognitive Apprenticeship Metamodel Based Intelligent Learning Activity Builder System

This research focused on a formal (theory based) approach to designing Intelligent Tutoring System (ITS) authoring tool involving two specific conventional pedagogical theories—Conversation Theory (CT) and Cognitive Apprenticeship (CA). The research conceptualised an Augmented Conversation and Cognitive Apprenticeship Metamodel (ACCAM) based on apriori theoretical knowledge and assumptions of its underlying theories. ACCAM was implemented in an Intelligent Learning Activity Builder System (ILABS)—an ITS authoring tool. ACCAM’s implementation aims to facilitate formally designed tutoring systems, hence, ILABS―the practical implementation of ACCAM― constructs metamodels for Intelligent Learning Activity Tools (ILATs) in a numerical problem-solving context (focusing on the construction of procedural knowledge in applied numerical disciplines). Also, an Intelligent Learning Activity Management System (ILAMS), although not the focus of this research, was developed as a launchpad for ILATs constructed and to administer learning activities. Hence, ACCAM and ILABS constitute the conceptual and practical contributions that respectively flow from this research. ACCAM’s implementation was tested through the evaluation of ILABS and ILATs within an applied numerical domain―the accounting domain. The evaluation focused on the key constructs of ACCAM―cognitive visibility and conversation, implemented through a tutoring strategy employing Process Monitoring (PM). PM augments conversation within a cognitive apprenticeship framework; it aims to improve the visibility of the cognitive process of a learner and infers intelligence in tutoring systems. PM was implemented via an interface that attempts to bring learner’s thought process to the surface. This approach contrasted with previous studies that adopted standard Artificial Intelligence (AI) based inference techniques. The interface-based PM extends the existing CT and CA work. The strategy (i.e. interface-based PM) makes available a new tutoring approach that aimed fine-grain (or step-wise) feedbacks, unlike the goal-oriented feedbacks of model-tracing. The impact of PM—as a preventive strategy (or intervention) and to aid diagnosis of learners’ cognitive process—was investigated in relation to other constructs from the literature (such as detection of misconception, feedback generation and perceived learning effectiveness). Thus, the conceptualisation and implementation of PM via an interface also contributes to knowledge and practice. The evaluation of the ACCAM-based design approach and investigation of the above mentioned constructs were undertaken through users’ reaction/perception to ILABS and ILAT. This involved, principally, quantitative approach. However, a qualitative approach was also utilised to gain deeper insight. Findings from the evaluation supports the formal (theory based) design approach—the design of ILABS through interaction with ACCAM. Empirical data revealed the presence of conversation and cognitive visibility constructs in ILATs, which were determined through its behaviour during the learning process. This research identified some other theoretical elements (e.g. motivation, reflection, remediation, evaluation, etc.) that possibly play out in a learning process. This clarifies key conceptual variables that should be considered when constructing tutoring systems for applied numerical disciplines (e.g. accounting, engineering). Also, the research revealed that PM enhances the detection of a learner’s misconception and feedback generation. Nevertheless, qualitative data revealed that frequent feedbacks due to the implementation of PM could be obstructive to thought process at advance stage of learning. Thus, PM implementations should also include delayed diagnosis, especially for advance learners who prefer to have it on request. Despite that, current implementation allows users to turn PM off, thereby using alternative learning route. Overall, the research revealed that the implementation of interface-based PM (i.e. conversation and cognitive visibility) improved the visibility of learner’s cognitive process, and this in turn enhanced learning—as perceived

Une approche propédagogique du diagnostic cognitif dans les STI : conception, formalisation et implémentation

Dans un STI, la question de la conception et de l'implémentation du diagnostic cognitif (DC) du comportement de l'apprenant est abordée de manière intuitive et au "cas par cas", selon les objectifs spécifiques de ce STI. D'un point de vue purement opérationnel, cette approche n'est pas vraiment néfaste puisque les STI qui en résultent "fonctionnent". Cependant lorsqu' il s'agit de rendre au DC dans un STI, la dimension pédagogique qui lui est naturellement sienne dans le tutorat humain, un regard en perspective sur des considérations plus fondamentales favorise la réflexion dans ce sens. Cette thèse formule, formalise et implémente explicitement une dimension «propédagogique» du DC dans les STI. Cette dimension pro-pédagogique s'exprime principalement à travers deux relations qui n'ont jamais été explicitement étudiées et mises en pratique dans la recherche sur le DC dans les STI : la relation entre le DC et les paradigmes de cognition; la relation entre le DC et son exploitation pédagogique, en l'occurrence la remédiation des difficultés de l'apprenant, ce à travers une boucle diagnostic-remédiation. Outre leur originalité conceptuelle, ces relations sont articulées dans un cadre de spécification pour le DC lorsqu'il s'agit de l'appliquer dans un STI. L'originalité de l'idée d'un tel cadre de référence est qu'il favorise une préservation de la fidélité pédagogique du DC au cours de son implémentation dans un système informatique. Sur le plan informatique, l'originalité de la contribution de cette thèse est qu'elle formalise cette perspective du processus de DC dans un STI, notamment en systématisant la dynamique de la boucle diagnostic-remédiation à travers: (1) un algorithme générique de type « générer et tester», (2) l'intégration à cet algorithme de mécanismes formels de raisonnement incertain par le biais des inférences bayésiennes (pour tenir compte des facteurs d'incertitude reliés au DC dans un STI) et (3) l'implémentation de cet algorithme dans une librairie de programmes génériques et donc réutilisables par tout membre de la communauté AIED désireux d'adopter cette philosophie de diagnostic cognitif dans un STI

A Classification of Evaluation Methods for Intelligent Tutoring Systems

Evaluation of intelligent tutoring systems (ITS) is an important area of research in current educational practices. There are many evaluation methods available but the literature does not suggest any clear guidelines for an evaluator – normally an educator – which methods to use in particular contexts. This paper proposes a classification of evaluation methods to simplify the selection task. The classification is based on two primary questions relating to the target of evaluation and learning environment in which the evaluation would be pursued. The classification is hoped to help in improving quality of computer based education by providing a practical and to the point way of selecting the appropriate evaluation methods for intelligent tutoring systems