Promoting Learning by Inducing and Scaffolding Cognitive Disequilibrium and Confusion through System Feedback

Learners frequently experience uncertainty about how to proceed during learning. These experiences cause learners to enter a state of cognitive disequilibrium and its affiliated affective state of confusion. Cognitive disequilibrium and confusion have been found to frequently occur during complex learning and provide opportunities for deeper learning. In the current thesis, a learning environment that induces confusion was investigated. In the environment, learners engaged in a dialogue on scientific reasoning with an animated pedagogical agent. Confusion was induced through false feedback provided by the tutor agent (e.g., when learners responded correctly and were told their response was incorrect). Self-reports of confusion during the training session indicated that false feedback was an effective method for inducing confusion. False feedback was also found to increase learners’ ability to apply this knowledge to new and novel situations, under certain conditions. Implications for the design of learning environments are also discussed

Understanding and Addressing Misconceptions in Introductory Programming: A Data-Driven Approach

With the expansion of computer science (CS) education, CS teachers in K-12 schools should be cognizant of student misconceptions and be prepared to help students establish accurate understanding of computer science and programming. This exploratory design-based research (DBR) study implemented a data-driven approach to identify secondary school students’ misconceptions using both their compilation and test errors and provide targeted feedback to promote students’ conceptual change in introductory programming. Research subjects were two groups of high school students enrolled in two sections of a Java-based programming course in a 2017 summer residential program for gifted and talented students. This study consisted of two stages. In the first stage, students of group 1 took the introductory programming class and used an automated learning system, Mulberry, which collected data on student problem-solving attempts. Data analysis was conducted to identify common programming errors students demonstrated in their programs and relevant misconceptions. In the second stage, targeted feedback to address these misconceptions was designed using principles from conceptual change and feedback theories and added to Mulberry. When students of group 2 took the same introductory programming class and solved programming problems in Mulberry, they received the targeted feedback to address their misconceptions. Data analysis was conducted to assess how the feedback affected the evolution of students’ (mis)conceptions. Using students’ erroneous solutions, 55 distinct compilation errors were identified, and 15 of them were categorized as common ones. The 15 common compilation errors accounted for 92% of all compilation errors. Based on the 15 common compilation errors, three underlying student misconceptions were identified, including deficient knowledge of fundamental Java program structure, misunderstandings of Java expressions, and confusion about Java variables. In addition, 10 common test errors were identified based on nine difficult problems. The results showed that 54% of all test errors were related to the difficult problems, and the 10 common test errors accounted for 39% of all test errors of the difficult problems. Four common student misconceptions were identified based on the 10 common test errors, including misunderstandings of Java input, misunderstandings of Java output, confusion about Java operators, and forgetting to consider special cases. Both quantitative and qualitative data analysis were conducted to see whether and how the targeted feedback affected students’ solutions. Quantitative analysis indicated that targeted feedback messages enhanced students’ rates of improving erroneous solutions. Group 2 students showed significantly higher improvement rates in all erroneous solutions and solutions with common errors compared to group 1 students. Within group 2, solutions with targeted feedback messages resulted in significantly higher improvement rates compared to solutions without targeted feedback messages. Results suggest that with targeted feedback messages students were more likely to correct errors in their code. Qualitative analysis of students’ solutions of four selected cases determined that students of group 2, when improving their code, made fewer intermediate incorrect solutions than students in group 1. The targeted feedback messages appear to have helped to promote conceptual change. The results of this study suggest that a data-driven approach to understanding and addressing student misconceptions, which is using student data in automated assessment systems, has the potential to improve students’ learning of programming and may help teachers build better understanding of their students’ common misconceptions and develop their pedagogical content knowledge (PCK). The use of automated assessment systems with misconception identification components may be helpful in pre-college introductory programming courses and so is encouraged as K-12 CS education expands. Researchers and developers of automated assessment systems should develop components that support identifying common student misconceptions using both compilation and non-compilation errors. Future research should continue to investigate the use of targeted feedback in automated assessment systems to address students’ misconceptions and promote conceptual change in computer science education

Paradigms for the design of multimedia learning environments in engineering

The starting point for this research was the belief that interactive multimedia learning environments represent a significant evolution in computer based learning and therefore their design requires a re-examination of the underlying principles of learning and knowledge representation. Current multimedia learning environments (MLEs) can be seen as descendants of the earlier technologies of computer-aided learning (CAL), intelligent tutoring systems (ITS) and videodisc-based learning systems. As such they can benefit from much of the wisdom which emerged from those technologies. However, multimedia can be distinguished from earlier technologies by its much greater facility in bringing to the learner high levels of interaction with and control over still and moving image, animation, sound and graphics. Our intuition tells us that this facility has the potential to create learning environments which are not merely substitutes for "live" teaching, but which are capable of elucidating complex conceptual knowledge in ways which have not previously been possible. If the potential of interactive multimedia for learning is to be properly exploited then it needs to be better understood. MLEs should not just be regarded as a slicker version of CAL, ITS or videodisc but a new technology requiring a reinterpretation of the existing theories of learning and knowledge representation. The work described in this thesis aims to contribute to a better understanding of the ways in which MLEs can aid learning. A knowledge engineering approach was taken to the design of a MLE for civil engineers. This involved analysing in detail the knowledge content of the learning domain in terms of different paradigms of human learning and knowledge representation. From this basis, a design strategy was developed which matched the nature of the domain knowledge to the most appropriate delivery techniques. The Cognitive Apprenticeship Model (CAM) was shown to be able to support the integration and presentation of the different categories of knowledge in a coherent instructional framework. It is concluded that this approach is helpful in enabling designers of multimedia systems both to capture and to present a rich picture of the domain. The focus of the thesis is concentrated on the domain of Civil Engineering and the learning of concepts and design skills within that domain. However, much of it could be extended to other highly visual domains such as mechanical engineering. Many of the points can also be seen to be much more widely relevant to the design of any MLE.Engineering and Physical Sciences Research Counci

Natural Language Tutoring and the Novice Programmer

For beginning programmers, inadequate problem solving and planning skills are among the most salient of their weaknesses. Novices, by definition, lack much of the tacit knowledge that underlies effective programming. This dissertation examines the efficacy of natural language tutoring (NLT) to foster acquisition of this tacit knowledge. Coached Program Planning (CPP) is proposed as a solution to the problem of teaching the tacit knowledge of programming. The general aim is to cultivate the development of such knowledge by eliciting and scaffolding the problem solving and planning activities that novices are known to underestimate or bypass altogether. ProPL (pro-PELL), a dialogue-based intelligent tutoring system based on CPP, is also described. In an evaluation, the primary findings were that students who received tutoring from ProPL seemed to exhibit an improved ability compose plans and displayed behaviors suggestive of thinking at greater levels of abstraction than students in a read-only control group. The major finding is that NLT appears to be effective in teaching program composition skills

Un modèle pour la génération d'indices par une plateforme de tuteurs par traçage de modèle

La présente thèse décrit des travaux de recherche effectués dans le domaine des systèmes tutoriels intelligents (STI). Plus particulièrement, elle s'intéresse aux tuteurs par traçage de modèle (MTT). Les MTTs ont montré leur efficacité pour le tutorat de la résolution de tâches bien définies. Par contre, les interventions pédagogiques qu'ils produisent doivent être incluses, par l'auteur du tuteur, dans le modèle de la tâche enseignée. La recherche effectuée répond à cette limite en proposant des méthodes et algorithmes permettant la génération automatique d'interventions pédagogiques. Une méthode a été développée afin de permettre à la plateforme Astus de générer des indices par rapport à la prochaine étape en examinant le contenu du modèle de la tâche enseignée. De plus, un algorithme a été conçu afin de diagnostiquer les erreurs des apprenants en fonction des actions hors trace qu'ils commettent. Ce diagnostic permet à Astus d'offrir une rétroaction par rapport aux erreurs sans que l'auteur du tuteur ait à explicitement modéliser les erreurs. Cinq expérimentations ont été effectuées lors de cours enseignés au département d'informatique de l'Université de Sherbrooke afin de valider de façon empirique les interventions générées par Astus. Le résultat de ces expérimentations montre que 1) il est possible de générer des indices par rapport à la prochaine étape qui sont aussi efficaces et aussi appréciés que ceux conçus par un enseignant et que 2) la plateforme Astus est en mesure de diagnostiquer un grand nombre d'actions hors trace des apprenants afin de fournir une rétroaction par rapport aux erreurs

Productive Failure in Virtual Language Learning for English

Vocabulary and syntax are challenges for English as Foreign Language (EFL) learners when they want to communicate in English. Task-based Language Teaching is commonly used in EFL teaching of vocabulary and syntax, which is a type of Direct Instruction (DI) that involves the initial use of explicit language instruction followed by a language learning activity. This study compared the efficacy for language learning of a different type of pedagogical approach, Productive Failure (PF), which delays instruction until after a language learning activity, to Direct Instruction (DI). There were three main language learning assessment areas: (a) students' declarative and procedural knowledge in the written production of the target language, (b) students' declarative and procedural knowledge in the spoken production of the target language, and (c) students' cognitive and metacognitive strategies in learning. English language education department freshmen in an Indonesian university (N=112) participated in the study by performing language learning activities in Second Life (SL), which is a 3-D virtual learning environment. They were randomly assigned to two language learning treatment groups. The PF group finished a communicative task on describing places prior to receiving explicit instruction. In contrast, the DI group watched an instructional video before completing a communicative task on describing places. This was followed by students in both groups finishing a similar communicative task in SL. Data from pre-and post-tests were analysed quantitatively, and video captures were transcribed and analysed qualitatively. The quantitative results found that PF group students performed significantly higher on the English syntax written assessment and both groups performed equally on the written vocabulary assessment. However, both groups performed equally on the spoken assessments of syntax and vocabulary. In the qualitative analysis, the PF students were found to use more self-regulated learning strategies and study tactics than DI students. The pattern of these findings is discussed in terms of previous research and theory. Overall, these findings suggest further research is warranted to investigate the use of PF language learning activities that involve the use of a virtual learning environment

Eficacia de la retroalimentación formativa para mejorar estrategias de competencia lectora en enseñanza secundaria

El objetivo general de la presente tesis doctoral es analizar la efectividad de la retroalimentación formativa para mejorar estrategias de competencia lectora en estudiantes de Enseñanza Secundaria. La investigación actual (Vidal-Abarca, Mañá y Gil, 2010) y los resultados de la evaluación del programa PISA (OECD, 2010), han mostrado que es necesario diseñar intervenciones para fomentar que los estudiantes españoles se conviertan en lectores competentes. La retroalimentación formativa constituiría un procedimiento instruccional apropiado para promover la adquisición de estrategias de auto-regulación dirigidas a mejorar la competencia lectora (Butler y Winne, 1995). Específicamente, el presente trabajo se centra en situaciones de lectura-orientada-a-tareas (responder preguntas de comprensión mientras el texto está disponible para buscar información) que permiten evaluar la competencia lectora de los estudiantes de acuerdo con las más actuales perspectivas (Snow, 2002; OECD, 2009). En estas situaciones de lectura, un lector competente debe implementar correctamente los procesos clásicos de comprensión (Kintsch, 1998), pero además, deben tomar una serie de decisiones estratégicas relacionadas con la búsqueda de información que requieren un alto nivel de auto-regulación (McNamara y Magliano, 2009; Rouet, 2006). La retroalimentación formativa proporcionaría una oportunidad para confirmar o reestructurar las estrategias de auto-regulación en situaciones de lectura-orientada-a-tareas, a través de la comparación del rendimiento obtenido en una pregunta (i.e., comprensión actual del estudiante) con un estándar deseado de rendimiento (i.e., contestar correctamente) (Butler y Winne, 1995). Abundante investigación ha demostrado que los estudiantes que auto-regularon correctamente cuándo volver al texto a buscar información y qué información es relevante para contestar la pregunta obtuvieron mejores resultados de comprensión en lectura-orientada-a-tareas (Cerdán y Vidal-Abarca, 2008; Cerdán, et al., 2011; Mañá, 2011; Vidal-Abarca et al., 2010; Vidal-Abarca, Salmerón y Mañá, 2010). Así, la retroalimentación formativa aplicada a situaciones de lectura-orientad-a-tareas tendría como objetivo principal fomentar decisiones adecuadas sobre cuándo y qué buscar. Asimismo, los sistemas de enseñanza asistida por ordenador serían una opción idónea para aplicar y analizar la eficacia de la retroalimentación formativa en situaciones de lectura-orientad-a-tareas. Por ejemplo, estos sistemas permitirían proporcionar retroalimentación inmediata a los estudiantes sobre la exactitud de sus respuestas a cada pregunta de comprensión para fomentar la adquisición efectiva de estrategias de auto-regulación más generales (Butler y Winne, 1995).No obstante, actualmente no existen estudios sobre cómo afectaría la retroalimentación formativa a las estrategias y rendimiento de los estudiantes en lectura-orientada-a-tareas, por tanto, multitud de cuestiones quedan abiertas. En el presente trabajo se ha tratado de dar respuesta a algunas de las cuestiones críticas sobre la aplicación de la retroalimentación en lectura-orientada-a-tareas a través de tres estudios experimentales. Los tres estudios que componen el presente trabajo examinan el impacto de la retroalimentación formativa en las estrategias de lectura y en la comprensión de los estudiantes en situaciones de lectura-orientada-a-tareas. En general, implican un énfasis progresivo hacia la intervención en la auto-regulación de los estudiantes. El Estudio 1 constituye un paso necesario para analizar si la retroalimentación formativa podía cambiar el comportamiento de búsqueda de los estudiantes cuando corregían respuestas incorrectas, pero el énfasis en las posibilidades de la retroalimentación para fomentar la auto-regulación fue casi nulo. Se planteó que la retroalimentación formativa podía presentar diferentes niveles de especificad según se centrara en las decisiones sobre cuándo buscar, o además, informara sobre qué información es relevante para contestar. Este último tipo resultaría ser más útil puesto que guiaría de forma más precisa el comportamiento de búsqueda. Los resultados apoyaron que la retroalimentación formativa más específica es efectiva para mejorar el comportamiento de búsqueda de los estudiantes y sus resultados de comprensión en lectura-orientada-a-tareas. En el Estudio 2, se analizó la efectividad de la retroalimentación formativa para guiar las decisiones estratégicas de búsqueda (cuándo y qué buscar) de los estudiantes al responder preguntas posteriores y así, los procesos de auto-regulación adquieren mayor relevancia aunque todavía quedan restringidos a la presencia de la retroalimentación. Se analizó si además de recibir la respuesta correcta para una pregunta los estudiantes necesitaban información más específica y adaptada sobre sus propias estrategias de búsqueda para mejorar su ejecución en preguntas posteriores (Whyte, et al, 1995; Butler, Godbole y Marsh, 2012). Los resultados del Estudio 2 mostraron que era necesario presentar información adaptada junto a la respuesta correcta para mejorar las decisiones estratégicas y la comprensión de los estudiantes en la fase final del estudio, tras un proceso de entrenamiento con la retroalimentación. Sin embargo, en este estudio no se incluyó una situación de transferencia sin el apoyo de la retroalimentación. Así, el Estudio 3 se centró en elaboración de un procedimiento de retroalimentación formativa para promover estrategias de búsqueda adecuadas cuando la retroalimentación desaparece, con un énfasis claro en la auto-regulación de los estudiantes. Se planteó que promover la transferencia de estrategias de auto-regulación en tareas complejas, requeriría procedimientos de retroalimentación que estimulen la auto-evaluación o reflexión sobre cómo las propias decisiones de búsqueda (cuándo y qué buscar) se relacionan con contestar correctamente las preguntas (Nicol y Macfarlane-Dick, 2006). Además, el procedimiento más efectivo de retroalimentación debía facilitar a los estudiantes la comprensión de sus errores cuando auto-evaluaban sus estrategias y rendimiento (Butler, et al., 2012). Los resultados obtenidos mostraron que el procedimiento que presentaba información más específica y fomentaba la mejor comprensión de las demandas de la tarea, fue más efectivo para garantizar la transferencia de estrategias de auto-regulación en lectura-orientada-a-tareas. Así, los tres estudios han mostrado que la retroalimentación formativa es una herramienta de instrucción efectiva para mejorar las estrategias de competencia lectora de los estudiantes de Secundaria. A lo largo de los tres estudios se ha mostrado que la retroalimentación formativa es efectiva para fomentar comportamientos de búsqueda apropiados; para guiar las decisiones de los estudiantes; y finalmente, para promover sus estrategias de auto-regulación. Por tanto, los elementos que se han ido modificando desde el diseño del Estudio 1 al diseño del Estudio 3 muestran un creciente énfasis hacia la intervención dirigida a mejorar la auto-regulación en lectura-orientada-a-tareas. En definitiva, los tres estudios del trabajo han permitido avanzar hacia la retroalimentación formativa más efectiva en lectura-orientada-a-tareas. Con respecto al contenido de la retroalimentación, los resultados muestran que informar sobre estrategias específicas de la lectura orientada-a-tareas facilita a los estudiantes captar las características de estas situaciones de lectura para mejorar su ejecución (Hattie y Timperley, 2007; Kluger y DeNisi, 1996; Narciss y Huth, 2004). Asimismo, los tres estudios coinciden en señalar los beneficios de la retroalimentación formativa más específica (Mason y Bruning, 2001; Mory, 2004; Narciss, 2004; Shute, 2008). Más allá, a lo largo de los tres estudios realizados se han obtenido resultados que contribuyen al estudio sistemático de las estrategias de auto-regulación necesarias en lectura-orientada-a-tareas. En general, el presente trabajo tiene importante implicaciones educativas para el diseño e implementación de procedimientos de retroalimentación formativa dirigidos a mejorar la competencia lectora en contextos avanzados de enseñanza asistida por ordenador, como por ejemplo el tutor inteligente TuinLEC, desarrollado por el grupo de investigación Psicotext de la Universidad de Valencia (Vidal-Abarca, et al, en prensa)

Designing Embodied Interactive Software Agents for E-Learning: Principles, Components, and Roles

Embodied interactive software agents are complex autonomous, adaptive, and social software systems with a digital embodiment that enables them to act on and react to other entities (users, objects, and other agents) in their environment through bodily actions, which include the use of verbal and non-verbal communicative behaviors in face-to-face interactions with the user. These agents have been developed for various roles in different application domains, in which they perform tasks that have been assigned to them by their developers or delegated to them by their users or by other agents. In computer-assisted learning, embodied interactive pedagogical software agents have the general task to promote human learning by working with students (and other agents) in computer-based learning environments, among them e-learning platforms based on Internet technologies, such as the Virtual Linguistics Campus (www.linguistics-online.com). In these environments, pedagogical agents provide contextualized, qualified, personalized, and timely assistance, cooperation, instruction, motivation, and services for both individual learners and groups of learners. This thesis develops a comprehensive, multidisciplinary, and user-oriented view of the design of embodied interactive pedagogical software agents, which integrates theoretical and practical insights from various academic and other fields. The research intends to contribute to the scientific understanding of issues, methods, theories, and technologies that are involved in the design, implementation, and evaluation of embodied interactive software agents for different roles in e-learning and other areas. For developers, the thesis provides sixteen basic principles (Added Value, Perceptible Qualities, Balanced Design, Coherence, Consistency, Completeness, Comprehensibility, Individuality, Variability, Communicative Ability, Modularity, Teamwork, Participatory Design, Role Awareness, Cultural Awareness, and Relationship Building) plus a large number of specific guidelines for the design of embodied interactive software agents and their components. Furthermore, it offers critical reviews of theories, concepts, approaches, and technologies from different areas and disciplines that are relevant to agent design. Finally, it discusses three pedagogical agent roles (virtual native speaker, coach, and peer) in the scenario of the linguistic fieldwork classes on the Virtual Linguistics Campus and presents detailed considerations for the design of an agent for one of these roles (the virtual native speaker)

Do Micro-Level Tutorial Decisions Matter: Applying Reinforcement Learning To Induce Pedagogical Tutorial Tactics

In this dissertation, I investigated applying a form of machine learning, reinforcement learning, to induce tutorial tactics from pre-existing data collected from real subjects. Tutorial tactics are policies as to how the tutor should select the next action when there are multiple ones available at each step. In order to investigate whether micro-level tutorial decisions would impact students' learning, we induced two sets of tutorial tactics: the ``Normalized Gain' tutorial tactics were derived with the goal of enhancing the tutorial decisions that contribute to the students' learning while the "Inverse Normalized Gain" ones were derived with the goal of enhancing those decisions that contribute less or even nothing to the students' learning. The two sets of tutorial tactics were compared on real human participants. Results showed that when the contents were controlled so as to be the same, different tutorial tactics would indeed make a difference in students' learning gains. The "Normalized Gain" students out-performed their ``Inverse Normalized Gain' peers. This dissertation sheds some light on how to apply reinforcement learning to induce tutorial tactics in natural language tutoring systems