Metacognition and Motivation: The Role of Time-Awareness in Preparation for Future Learning

In this work, we investigate how two factors, metacognitive skills and motivation, would impact student learning across domains. More specifically, our primary goal is to identify the critical, yet robust, interaction patterns of these two factors that would contribute to students' performance in learning logic first and then their performance on a subsequent new domain, probability. We are concerned with two types of metacognitive skills: strategy-awareness and time-awareness, that is, which problem-solving strategy to use and when to use it. Our data were collected from 495 participants across three consecutive semesters, and our results show that the only students who consistently outperform their peers across both domains are those who are not only highly motivated but also strategy-aware and time-aware

Stealth Assessment of Self-Regulative Behaviors within a Game-Based Environment

abstract: Students' ability to regulate and control their behaviors during learning has been shown to be a critical skill for academic success. However, researchers often struggle with ways to capture the nuances of this ability, often solely relying on self-report measures. This thesis proposal employs a novel approach to investigating variations in students' ability to self-regulate by using process data from the game-based Intelligent Tutoring System (ITS) iSTART-ME. This approach affords a nuanced examination of how students' regulate their interactions with game-based features at both a coarse-grained and fine-grain levels and the ultimate impact that those behaviors have on in-system performance and learning outcomes (i.e., self-explanation quality). This thesis is comprised of two submitted manuscripts that examined how a group of 40 high school students chose to engage with game-based features and how those interactions influenced their target skill performance. Findings suggest that in-system log data has the potential to provide stealth assessments of students' self-regulation while learning.Dissertation/ThesisM.A. Psychology 201

Retrieval-, Distributed-, and Interleaved Practice in the Classroom:A Systematic Review

Three of the most effective learning strategies identified are retrieval practice, distributed practice, and interleaved practice, also referred to as desirable difficulties. However, it is yet unknown to what extent these three practices foster learning in primary and secondary education classrooms (as opposed to the laboratory and/or tertiary education classrooms, where most research is conducted) and whether these strategies affect different students differently. To address these gaps, we conducted a systematic review. Initial and detailed screening of 869 documents found in a threefold search resulted in a pool of 29 journal articles published from 2006 through June 2020. Seventy-five effect sizes nested in 47 experiments nested in 29 documents were included in the review. Retrieval- and interleaved practice appeared to benefit students’ learning outcomes quite consistently; distributed practice less so. Furthermore, only cognitive Student*Task characteristics (i.e., features of the student’s cognition regarding the task, such as initial success) appeared to be significant moderators. We conclude that future research further conceptualising and operationalising initial effort is required, as is a differentiated approach to implementing desirable difficulties

A systematic review and meta-analysis of interventions based on metacognition and self-regulation in school-aged mathematics

Mathematics is an important gatekeeper for educational and professional opportunities and a useful tool for discovery and expression. Given previous research and theory demonstrating potential for metacognitive and self-regulated learning (MC/SRL) interventions to support mathematics achievement with diverse learners, a systematic review was conducted to evaluate their effectiveness within the years of general education, with pupils of ages three to 18. Appropriately-designed studies that were reported in English between 2005 and 2019 were included. Following a systematic search, with double-reviewing and expert consultation for consistency, 1,761 bibliographic items were screened, resulting in 60 included studies. Qualitative aspects of the designs, contexts, participants, and intervention activities were synthesised narratively. Posttest-only and adjusted, random effects meta-analyses were performed using a single mathematics achievement measure from each study. The results indicate a generally positive effect from the included interventions (combined Cohen’s d=0.46, SE=0.08, 95% CI=0.30 to 0.60). This represents a somewhat more modest effect compared with previous reviews in this area, possibly due to a greater range of included reports. No risk of publication bias was identified, reflecting the breadth and diversity of included studies, but efforts to mitigate heterogeneity were only partially successful. Interventions using structured problem-solving with metacognitive prompts were more effective than those not using it, while dissertations reported lower effects than journal articles. No differences were found based on participant age or intervention dose. Primary studies used a variety of assessments and differed on reporting of interventions and quality-related factors, and there remained substantial heterogeneity in the meta-analysis. Implications of this review for educational theory, research, and practice are discussed, with emphasis on reporting studies fully, using broad-scope, comparable assessments, and investing in comprehensive metacognitive and self-regulated learning interventions that can support lasting change in teaching and learning

Life-Worthy Learning Skills: A Curriculum Intervention to Promote Self-Regulated Learning

Taking the form of a portfolio of papers written across my six-year research journey, this professional doctoral thesis presents the findings of a discipline-independent curriculum intervention designed to improve students’ self-regulated learning skills. The study also examines the relationship between students’ self-regulated learning skills and their academic achievement. Research suggests that discipline-independent training interventions improve students’ self-regulated learning skills, also having a major impact on students’ academic achievement across childhood and adolescence. Founded on Zimmerman's (2000) cyclic model of self-regulated learning, the 10-week discipline-independent intervention underpinning this research was designed and implemented to support the development of Year 9 (13-14 year old) students’ self-regulated learning skills. Following a pre-test post-test non-equivalent group design, participants were divided into two groups by way of purposeful sampling, with each group receiving the training intervention consecutively during the academic year 2017/18. Data was collected at three timepoints (before the start of Phase 1, at the end of Phase 1, and at the end of Phase 2) using the Motivated Strategies for Learning Questionnaire (MSLQ) and an original and tailored instrument created specifically for this research; the Self-Regulated Learning Experimental Design Survey (SRLEDS). Forming a significant contribution to the field, this instrument was validated using the MSLQ. Results show that although students’ self-regulated learning skills and academic achievement improved across timepoint, there were not any significant differences between group and nor did students’ self-regulated learning skills predict their level of academic achievement. In light of this, the discussion focuses on providing context to these results, exploring local changes within the research setting that account for the findings before outlining the implications of this study for both research and practice. In addition to the development and validation of the SRLEDS, this study also contributes to the fields of research and practice by offering a critical reflection on the challenges of using control groups within a live school research setting. In response to this, an alternative research design is proposed as well as a range of future research directions, with implications for practice highlighted and discussed

Improving Mobile MOOC Learning via Implicit Physiological Signal Sensing

Massive Open Online Courses (MOOCs) are becoming a promising solution for delivering high- quality education on a large scale at low cost in recent years. Despite the great potential, today’s MOOCs also suffer from challenges such as low student engagement, lack of personalization, and most importantly, lack of direct, immediate feedback channels from students to instructors. This dissertation explores the use of physiological signals implicitly collected via a "sensorless" approach as a rich feedback channel to understand, model, and improve learning in mobile MOOC contexts. I first demonstrate AttentiveLearner, a mobile MOOC system which captures learners' physiological signals implicitly during learning on unmodified mobile phones. AttentiveLearner uses on-lens finger gestures for video control and monitors learners’ photoplethysmography (PPG) signals based on the fingertip transparency change captured by the back camera. Through series of usability studies and follow-up analyses, I show that the tangible video control interface of AttentiveLearner is intuitive to use and easy to operate, and the PPG signals implicitly captured by AttentiveLearner can be used to infer both learners’ cognitive states (boredom and confusion levels) and divided attention (multitasking and external auditory distractions). Building on top of AttentiveLearner, I design, implement, and evaluate a novel intervention technology, Context and Cognitive State triggered Feed-Forward (C2F2), which infers and responds to learners’ boredom and disengagement events in real time via a combination of PPG-based cognitive state inference and learning topic importance monitoring. C2F2 proactively reminds a student of important upcoming content (feed-forward interventions) when disengagement is detected. A 48-participant user study shows that C2F2 on average improves learning gains by 20.2% compared with a non-interactive baseline system and is especially effective for bottom performers (improving their learning gains by 41.6%). Finally, to gain a holistic understanding of the dynamics of MOOC learning, I investigate the temporal dynamics of affective states of MOOC learners in a 22 participant study. Through both a quantitative analysis of the temporal transitions of affective states and a qualitative analysis of subjective feedback, I investigate differences between mobile MOOC learning and complex learning activities in terms of affect dynamics, and discuss pedagogical implications in detail

Development and Evaluation of a Sustainable e-Learning Framework for Higher Education Institutions in Malaysia

This thesis focuses on developing and evaluating a sustainable e-learning framework (SeLF) for the higher education institutions in Malaysia. A mixed methods approach was used. Data was collected through surveys from Malaysian universities to assess the characteristics of SeLF. The framework was then evaluated through expert interviews. The thesis highlights SeLF elements and their contribution to the Triple Bottom Line, facilitating sustainable e-learning and its contribution to society, the environment, and the economy

Investigating the Prerequisites for a robust Neurotutor: The Detection of mixed User States containing Working Memory Load, Affective Valence and Affective Dominance

Intelligent tutoring systems are software environments that aim to simulate a human tutor. While current systems show effectiveness comparable to human tutors, they still suffer from the ‘assistance dilemma’. This drawback refers to the inability to infer the ongoing user state which can lead to situations where the system provides no or inadequate support. To alleviate this situation, user state detection has been implemented in some systems. However, at the current time, only behavioral indicators are used to infer the ongoing user state. Such overt behaviors are not specific enough to provide a detailed representation of the user state. This is the reason why I suggest to investigate the potential use of the electroencephalogram to infer the ongoing user state. This combination of an intelligent tutoring system and an EEG-based user state detection is called a neurotutor. EEG-based user state detection usually focusses on narrow user states which can be detected in controlled lab environments. I assume that real-life environments like a classroom evoke complex user states which consist of multiple different components. I therefore propose a three component framework that enables the tracking of different processes that are active during a complex user state. The first two studies focus on the separation of working memory load and affective valence in a highly controlled setting with the use of established measures from classical neuroscience. I found that measures used to infer working memory load can be used to track changes in working memory load under different affective valence. Furthermore, I found that said measures were also sensitive to changes in affective valence. Surprisingly, I found that measures used to infer affective valence were not sensitive to changes in affective valence under working memory load. Additional analyses revealed that working memory load and affective valence can be automatically detected with accuracies sufficient for the use in a neurotutor. The third study successfully replicated the findings from the first two studies in a more realistic, although less controlled setting. A simplified learning game was used to induce the complex user state of perceived loss of control that simultaneously evoked cognitive as well as affective processes. With the help of the framework I was able to integrate the findings from three different studies that all analyzed the same dataset. This would not have been possible without an adequate theoretical framework.Intelligente Tutorensysteme sind EDV-Programme, welche versuchen einen menschlichen Tutor zu simulieren. Obwohl derzeit erhältliche Systeme ähnlich effektiv sind wie menschliche Tutoren, leiden sie immer noch unter dem ‚Assistenzdilemma‘. Dies referenziert auf die Unfähigkeit den aktuellen Nutzerzustand zu erkennen, was dazu führen kann, dass das System keine oder inadäquate Unterstützung anbietet. Um diesen Umstand zu beseitigen wurde in manchen Systemen bereits eine Nutzerzustandserkennung implementiert. Derzeit werden jedoch nur beobachtbare Verhaltensweisen verwendet, um den aktuellen Nutzerzustand abzuleiten. Solch offenkundige Verhaltensweisen sind jedoch nicht spezifisch genug, um ein detailliertes Bild des Nutzers zu erfassen. Deshalb schlage ich vor, dass man das Elektroenzephalogramm verwenden soll, um den aktuellen Nutzerzustand zu erkennen. Diese Kombination eines intelligenten Tutorensystems mit einer EEG-basierten Zustandserkennung nennt man einen Neurotutor. Normalerweise fokussiert sich EEG-basierte Zustandserkennung jedoch nur auf begrenzte Zustände, welche innerhalb kontrollierter Laborumgebungen erkannt werden. Ich nehme an, dass realistische Settings wie ein Klassenzimmer komplexe Zustände hervorrufen, welche aus mehreren verschiedenen Komponenten bestehen. Daher stelle ich ein Bezugssystem mit drei Komponenten auf, welcher die Zuordnung der verschiedenen Prozesse, die während eines komplexen Nutzerzustandes aktiv sind, ermöglicht. Die ersten beiden Studien befassen sich mit der Separation von Arbeitsgedächtnisbelastungen und affektiver Valenz durch die Verwendung von etablierten EEG Maßen in einem hoch kontrollierten Setting. Dabei habe ich herausgefunden, dass Maße, welche zum Erkennen von Arbeitsgedächtnisbelastungen verwendet werden, auch geeignet sind, um Veränderungen in der Belastung des Arbeitsgedächtnisses unter gleichzeitiger emotioneller Stimulation zu erkennen. Außerdem habe ich entdeckt, dass die erwähnten Maße auch sensitiv gegenüber Veränderungen in der affektiven Valenz sind. Überraschender Weise hat sich herausgestellt, dass Maße, welche zum Erkennen von affektiver Valenz verwendet werden, nicht sensitiv gegenüber Veränderungen in der affektiven Valenz sind, wenn gleichzeitig das Arbeitsgedächtnis belastet wird. Zusätzliche Analysen haben aufgedeckt, dass Arbeitsgedächtnisbelastungen und affektive Valenz mit einer Genauigkeit erkannt werden kann, welche für den Gebrauch in einem Neurotutor ausreichend sind. In der dritten Studie konnten die Ergebnisse der ersten beiden Studien erfolgreich in einem relevanteren, aber weniger kontrollierten Kontext repliziert werden. Dabei wurde ein vereinfachtes Lernspiel verwendet, um einen wahrgenommenen Kontrollverlust zu induzieren. Durch Verwendung des Bezugsrahmens war es mir möglich die Ergebnisse von drei verschiedenen Studien zu integrieren, welche alle denselben Datensatz analysierten. Dies wäre nicht ohne adäquaten theoretischen Rahmen möglich gewesen

Effects of Timing and Level of Integration of Formative Feedback in Question-Answering for Concept Learning

De acuerdo con la literatura previa dentro del contexto de la retroalimentación formativa (Bangert-Downs et al., 2013; Butler & Roediger, 2008; Butler & Winne, 1995; Hattie & Timperley, 2007; Kapp et al., 2015; Karpicke, & Roediger, 2007, 2008; Llorens, et al., 2014; Máñez, Vidal-Abarca & Martínez, 2016; Máñez, Vidal-Abarca, Martínez & Kendeu, 2017; Newman, Williams & Hiller, 1974; Narciss, 2013; Narciss et al., 2014 ; van der Kleij, Eggen, Timmers, & Veldkamp, 2012; van der Kleij, et al., 2015; Vidal-Abarca et al., 2017; Vidal-Abarca, Martínez, Ferrer & García, 2017), el objetivo general de la presente tesis es la evaluación de la efectividad de un procedimiento formativo de lectura y contestación a preguntas que proporciona retroalimentación de manera inmediata frente a la retroalimentación demorada. En el presente trabajo, estudiantes universitarios y de secundaria leen un texto y responden preguntas mientras tienen el texto disponible. Además, el nivel de integración del mensaje de retroalimentación elaborada está específicamente diseñado para proporcionar a los estudiantes una relación de las ideas más relevantes del texto en un intento por favorecer así la comprensión del contenido de aprendizaje de una manera más coherente. El Estudio 1 analiza el impacto de un procedimiento de contestación a preguntas que proporciona retroalimentación de manera inmediata (pregunta por pregunta) o demorada (tras la compleción de cuatro preguntas), en comparación con una condición de no retroalimentación en la que los estudiantes se ven obligados a re-leer el texto. Las medidas dependientes incluyen la probabilidad del estudiante de corregir errores, estimaciones de certitud de respuesta, rendimiento y comportamiento en línea durante la compleción de la tarea (i.e., tiempos de lectura) en una muestra de estudiantes universitarios de pregrado. Además, el Estudio 2 se sumerge en la cuestión de qué tipo de nivel de integración del mensaje de retroalimentación elaborada es más beneficioso para el aprendizaje conceptual cuando se proporciona en forma inmediata, sumativa demorada o integrativa demorada. En este estudio, los efectos del nivel de tarea (bajo versus alto nivel) y las diferencias individuales basadas en el novel de conocimiento previo previo de los estudiantes se consideran y discuten a la luz de la literatura reciente en comprensión y aprendizaje del texto (Gilabert , Martínez, y Vidal-Abarca, 2005; Le Bigot y Rouet, 2007; McNamara y Kintsch, 1996; Priebe, Keenan y Miller, 2011; O´ Really y Sabatini, 2013; Ozuru et al., 2007; Salmerón, Kintsch & Cañas, 2006; Vidal-Abarca & SanJosé, 1998; Clemens et al., 2018). En el presente trabajo, las conclusiones y limitaciones abarcan algunos de los puntos más críticos sobre el papel de la retroalimentación formativa para la contestación de preguntas, comprensión del material de estudio y el aprendizaje de conceptos. De manera similar, las implicaciones educativas se discuten a fondo.Following previous literature within the context of feedback (Bangert-Downs et al., 2013; Butler & Roediger, 2008; Butler & Winne, 1995; Hattie & Timperley, 2007; Kapp et al., 2015; Karpicke, & Roediger, 2007, 2008; Llorens, et al., 2014; Máñez, Vidal-Abarca & Martínez, 2016; Máñez, Vidal-Abarca, Martínez & Kendeu, 2017; Newman, Williams & Hiller, 1974; Narciss, 2013; Narciss et al., 2014; van der Kleij, Eggen, Timmers, & Veldkamp, 2012; van der Kleij, et al., 2015; Vidal-Abarca et al., 2017; Vidal-Abarca, Martínez, Ferrer & García, 2017), the general objective of the present thesis is the evaluation of the effectiveness of a formative feedback question-answering procedure that provides feedback in an immediate and delayed manner. Students read a text and answer questions while having the text available. In addition, the level of integration of the elaborative feedback message is specifically designed to provide students with a relation of the most relevant ideas from the text in an attempt to help them comprehend the learning content in a more coherent manner. Study 1 analyzes the impact of a question-answering procedure that provides feedback in an immediate (question by question) or delayed manner (after the completion of four questions), compared to a no-feedback condition where students are forced to rereading a text. Dependent measures include the student´s probability to correct errors, response certitude estimates, question-answering performance and online behaviour on a sample of undergraduate university students. Additionally, Study 2 dives into the issue of what type of level of integration of the elaborative feedback message is more beneficial for conceptual learning when provided in an immediate, delayed summative or delayed integrative form. In this study, the effects of the task level (i.e., low versus high-level) and the individual differences in previous background knowledge of students are considered and further discussed under the light of the recent literature in reading comprehension and learning from text (Gilabert, Martínez, & Vidal-Abarca, 2005; Le Bigot & Rouet, 2007; McNamara & Kintsch, 1996; Priebe, Keenan, & Miller, 2011; O´ Really & Sabatini, 2013; Ozuru et al., 2007; Salmerón, Kintsch & Cañas, 2006; Vidal-Abarca & SanJosé, 1998; Clemens et al., 2018). Conclusions and limitations embrace some of the most critical points on the role of formative feedback for question-answering, reading comprehension and learning with an available text. Educational implications are further discussed