Integrating knowledge tracing and item response theory: A tale of two frameworks

Traditionally, the assessment and learning science commu-nities rely on different paradigms to model student performance. The assessment community uses Item Response Theory which allows modeling different student abilities and problem difficulties, while the learning science community uses Knowledge Tracing, which captures skill acquisition. These two paradigms are complementary - IRT cannot be used to model student learning, while Knowledge Tracing assumes all students and problems are the same. Recently, two highly related models based on a principled synthesis of IRT and Knowledge Tracing were introduced. However, these two models were evaluated on different data sets, using different evaluation metrics and with different ways of splitting the data into training and testing sets. In this paper we reconcile the models' results by presenting a unified view of the two models, and by evaluating the models under a common evaluation metric. We find that both models are equivalent and only differ in their training procedure. Our results show that the combined IRT and Knowledge Tracing models offer the best of assessment and learning sciences - high prediction accuracy like the IRT model, and the ability to model student learning like Knowledge Tracing

3. Toward a Cognitive Theory for the Measu rement of Achievement

INTRODUCTION Given the demands for higher levels of learning in our schools and the press for education in the skilled trades, the professions, and the sciences, we must develop more powerful and specific methods for assessing achievement. We need forms of assessment that educators can use to improve educational practice and to diagnose individual progress by monitoring the outcomes of learning and training. Compared to the well-developed technology for aptitude measurement and selection testing, however, the measurement of achievement and diagnosis of learning problems is underdeveloped. This is because the correlational models that support prediction are insufficient for the task of prescribing remediation or other instructional interventions. Tests can predict fa ilure without a theory of what causes success, but intervening to prevent failure and enhance competence requires deeper understanding. The study of the nature of learning is therefore integral to the assessment of achievement. We must use what we know about the cognitive properties of acquired proficiency and about the structures and processes that develop as a student becomes competent in a domain . We know that learning is not simply a matter of the accretion of subject-matter concepts and procedures; it consists rather of organizing and restructuring of this information to enable skillful procedures and processes of problem representation and solution. Somehow, tests must be sensitive to how well this structuring has proceeded in the student being tested. The usual forms of achievement tests are not effective diagnostic aids. In order for tests to become usefully prescriptive, they must identify performance components that facilitate or interfere with current proficiency and the attainment of eventual higher levels of achievement. Curriculum analysis of the content and skill to be learned in a subject matter does not automatically provide information about how students attain competence about the difficulties they meet in attaining it. An array of subject-matter subtests differing in difficulty is not enough for useful diagnosis. Rather, qualitative indicators of specific properties of performance that influence learning and characterize levels of competence need to be identified. In order to ascertain the critical differences between successful and unsuccessful student performance, we need to appraise the knowledge structures and cognitive processes that reveal degrees of competence in a field of study. We need a fuller understanding of what to test and how test items relate to target knowledge. In contrast, most of current testing technology is post hoc and has focused on what to do after test items are constructed. Analysis of item difficulty, development of discrimination indices, scaling and norming procedures, and analysis of test dimensions and factorial composition take place after the item is written. A theory of acquisition and performance is needed before and during item design

Effects of Wide Reading Vs. Repeated Readings on Struggling College Readers\u27 Comprehension Monitoring Skills

Fluency instruction has had limited effects on reading comprehension relative to reading rate and prosodic reading (Dowhower, 1987; Herman, 1985; National Institute of Child Health and Human Development, 2000a). More specific components (i.e., error detection) of comprehension may yield larger effects through exposure to a wider range of materials than repeated readings (Kuhn, 2005b). Thirty-three students reading below college level were randomly assigned to a Repeated Readings (RR), a Wide Reading (WR), or a Vocabulary Study (VS) condition and received training in 9 sessions of 30 minutes in a Southeast community college. RR students read an instructional-level text consecutively four times before answering comprehension questions about it; WR students read four instructional-level texts each once and answered questions while the VS group studied and took a quiz on academic vocabulary. An additional 13 students reading at college level provided comparison data. At pretest, all participants completed the Nelson Denny Reading Test, Test of Word Reading Efficiency, Error Detection task (Albrecht & O\u27Brien, 1993), working memory test, Metacognitive Awareness of Reading Strategies Inventory (MARSI; Mokhtari & Reichard, 2002), a maze test, Author Recognition Test (ART), and reading survey. All pretest measures except for the ART and reading surveys were re-administered at posttest to training groups. Paired-samples t-test analyses revealed (a) significant gains for the WR condition in vocabulary (p = .043), silent reading rate (p \u3c .05), maze (p \u3c .05) and working memory (p \u3c .05) (b) significant gains for the RR students in silent reading rate (p = .05) and maze (p = .006) and (c) significant increases on vocabulary (p \u3c .05), maze (p = .005), and MARSI (p \u3c .005) for the VS group at posttest. Unreliable patterns of error detection were observed for all groups at pretest and post-test. Results suggest that effects of fluency instruction be sought at the local level processes of reading using the maze test, which reliably detected reading improvements from fluency instruction (RR, WR) and vocabulary study (VS) in only 9 sessions. With significant gains on more reading measures, the WR condition appears superior to the RR condition as a fluency program for struggling college readers. Combining the WR condition with vocabulary study may augment students’ gains

Metacognitive Instruction in L2 French: An Analysis of Listening Performance and Automaticity

The goal of the present study was to investigate the instructional potential of metacognitive strategies to improve listening comprehension and the automaticity of listening processes. Metacognition can be described as focusing attention on the cognitive processes one is currently using. In the classroom, metacognitive listening instruction means guiding learners in applying metacognition to learning by monitoring the mental strategies they use while listening, evaluating the relative success of these strategies, and planning for future listening experiences. I argue that providing students with these tools to monitor and regulate the perceptual and cognitive processes involved in listening contributes significantly to building toward the automaticity of those processes, leading to improved performance and reduced reaction time on listening assessments. In order to test this hypothesis, a pretest—treatment—posttest design was adopted and seven intact sections of second-semester French were randomly assigned to the control or experimental condition. The experimental groups were instructed in using metacognitive strategies to regulate the listening process while the control groups were simply exposed to the listening passages and asked to verify comprehension. Results showed that treatment condition alone did not account for improvement in listening comprehension or automaticity. Differences were found, however, based on initial listening proficiency and metacognitive awareness: low proficiency learners in both conditions outperformed all high proficiency learners in listening gains over time. Initial level of metacognitive awareness had a significant impact on gains in listening performance, indicating that the learners in the experimental group who began the study with low metacognitive awareness achieved higher gains in listening. This suggests that increasing metacognitive awareness through instruction is effective in improving listening comprehension. A potential ceiling effect was indicated, however, since those who began the study with high metacognitive awareness in the control condition improved their listening more than those in the experimental condition. An analysis of reaction time gain scores suggested that automaticity was not affected by initial listening proficiency. On the other hand, it was the learners in the control condition with low initial metacognitive awareness who decreased their reaction time compared to other subgroups over the course of the study, suggesting that metacognitive abilities are indeed related to automaticity. On the whole, these results indicate that initial level of metacognitive awareness has a differential impact on listening comprehension gains as well as reaction time. This supports the idea that increasing metacognition can help learners to improve listening skills; however, further research must be done in order to clarify the nature of these interrelationships

"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers

Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.Comment: 26 pages, 2 figures, TOCH

Applying science of learning in education: Infusing psychological science into the curriculum

The field of specialization known as the science of learning is not, in fact, one field. Science of learning is a term that serves as an umbrella for many lines of research, theory, and application. A term with an even wider reach is Learning Sciences (Sawyer, 2006). The present book represents a sliver, albeit a substantial one, of the scholarship on the science of learning and its application in educational settings (Science of Instruction, Mayer 2011). Although much, but not all, of what is presented in this book is focused on learning in college and university settings, teachers of all academic levels may find the recommendations made by chapter authors of service. The overarching theme of this book is on the interplay between the science of learning, the science of instruction, and the science of assessment (Mayer, 2011). The science of learning is a systematic and empirical approach to understanding how people learn. More formally, Mayer (2011) defined the science of learning as the “scientific study of how people learn” (p. 3). The science of instruction (Mayer 2011), informed in part by the science of learning, is also on display throughout the book. Mayer defined the science of instruction as the “scientific study of how to help people learn” (p. 3). Finally, the assessment of student learning (e.g., learning, remembering, transferring knowledge) during and after instruction helps us determine the effectiveness of our instructional methods. Mayer defined the science of assessment as the “scientific study of how to determine what people know” (p.3). Most of the research and applications presented in this book are completed within a science of learning framework. Researchers first conducted research to understand how people learn in certain controlled contexts (i.e., in the laboratory) and then they, or others, began to consider how these understandings could be applied in educational settings. Work on the cognitive load theory of learning, which is discussed in depth in several chapters of this book (e.g., Chew; Lee and Kalyuga; Mayer; Renkl), provides an excellent example that documents how science of learning has led to valuable work on the science of instruction. Most of the work described in this book is based on theory and research in cognitive psychology. We might have selected other topics (and, thus, other authors) that have their research base in behavior analysis, computational modeling and computer science, neuroscience, etc. We made the selections we did because the work of our authors ties together nicely and seemed to us to have direct applicability in academic settings

Collaborative trails in e-learning environments

This deliverable focuses on collaboration within groups of learners, and hence collaborative trails. We begin by reviewing the theoretical background to collaborative learning and looking at the kinds of support that computers can give to groups of learners working collaboratively, and then look more deeply at some of the issues in designing environments to support collaborative learning trails and at tools and techniques, including collaborative filtering, that can be used for analysing collaborative trails. We then review the state-of-the-art in supporting collaborative learning in three different areas – experimental academic systems, systems using mobile technology (which are also generally academic), and commercially available systems. The final part of the deliverable presents three scenarios that show where technology that supports groups working collaboratively and producing collaborative trails may be heading in the near future

Metacognition and Decision-Making Style in Clinical Narratives

Clinical decision-making has high-stakes outcomes for both physicians and patients, yet little research has attempted to model and automatically annotate such decision-making. The dual process model (Evans, 2008) posits two types of decision-making, which may be ordered on a continuum from intuitive to analytical (Hammond, 1981). Training clinicians to recognize decision-making style and select the most appropriate mode of reasoning for a particular context may help reduce diagnostic error (Norman, 2009). This study makes preliminary steps towards detection of decision style, based on an annotated dataset of image-based clinical reasoning in which speech data were collected from physicians as they inspected images of dermatological cases and moved towards diagnosis (Hochberg et al., 2014a). A classifier was developed based on lexical, speech, disfluency, physician demographic, cognitive, and diagnostic difficulty features to categorize diagnostic narratives as intuitive vs. analytical; the model improved on the baseline by over 30%. The introduced computational model provides construct validity for the dual process theory. Eventually, such modeling may be incorporated into instructional systems that teach clinicians to become more effective decision makers. In addition, metacognition, or self-assessment and self-management of cognitive processes, has been shown beneficial to decision-making (Batha & Carroll, 2007; Ewell-Kumar, 1999). This study measured physicians\u27 metacognitive awareness, an online component of metacognition, based on the confidence-accuracy relationship, and also exploited the corpus annotation of decision style to derive decision metrics. These metrics were used to examine the relationships between decision style, metacognitive awareness, expertise, case difficulty, and diagnostic accuracy. Based on statistical analyses, intuitive reasoning was associated with greater diagnostic accuracy, with an advantage for expert physicians. Case difficulty was associated with greater user of analytical decision-making, while metacognitive awareness was linked to decreased diagnostic accuracy. These results offer a springboard for further research on the interactions between decision style, metacognitive awareness, physician and case characteristics, and diagnostic accuracy

Application of Voice Personal Assistants in the Context of Smart University

Los asistentes personales de voz basados en técnicas avanzadas de comprensión del lenguaje natural se muestran como un recurso prometedor frente al reto del diseño de plataformas virtuales de aprendizaje. Específicamente, estos recursos pueden servir de apoyo para la mejora del proceso de enseñanza-aprendizaje. El objetivo principal de este trabajo ha sido el de estudiar los desafíos actuales para la utilización de este tipo de asistentes en el ámbito de las universidades inteligentes. Asimismo, se ha analizado cómo esta nueva tecnología puede ayudar a los estudiantes en su proceso de aprendizaje y grado de satisfacción. Los resultados de este trabajo se presentan en tres artículos de investigación publicados en revistas científicas indexadas en Web of Science. También se aporta un Registro de la Propiedad Intelectual registrado en el Ministerio de Cultura de España, en la categoría de programa de ordenador, cuyos derechos fueron cedidos a la Universidad de Burgos.Personal voice assistants based on advanced natural language comprehension techniques are shown as a promising resource with regard to the challenge of designing virtual learning platforms. In particular, these resources can support the improvement of the teaching-learning process. The main objective of this work has been to study the current challenges for the use of this type of assistant in the field of smart universities. Likewise, it has been analyzed how this innovative technology can help students in their learning process and their degree of satisfaction. The results of this work are presented in three research articles published in scientific journals indexed on the Web of Science. Also, an Intellectual Property Registry registered with the Ministry of Culture of Spain in the category of computer programs is provided, whose rights were transferred to the University of Burgos

Rethinking Thinking About Thinking: Against a Pedagogical Imperative to Cultivate Metacognitive Skills

In summaries of “best practices” for pedagogy, one typically encounters enthusiastic advocacy for metacognition. Some researchers assert that the body of evidence supplied by decades of education studies indicates a clear pedagogical imperative: that if one wants their students to learn well, one must implement teaching practices that cultivate students’ metacognitive skills. In this dissertation, I counter that education research does not impose such a mandate upon instructors. We lack sufficient and reliable evidence from studies that use the appropriate research design to validate the efficacy of metacognitive skill-building interventions (not just evaluate their relationship to learning outcomes). I argue that improved academic outcomes following these interventions aren’t necessarily mediated by increased metacognitive skills; rather, enhanced student performance can be accounted for by other factors that accompany metacognitive training, particularly the explicit provision of domain-specific knowledge. On the way to this conclusion, I elaborate some complications and controversies surrounding “metacognition.” This is a sprawling and nebulous construct, which makes generalizations about its pedagogical value dubious from the outset. Moreover, it is unclear the extent to which the end goal of metacognitive skill-building (cognitive self-mastery, involving knowledge of and control over our own minds) is even possible, given our mental architecture; some cognitive scientists allege that we are subject to phenomenological illusions which make this seem more achievable than it actually is. I also attempt to provide an account of how metacognitive skill-building could receive glowing endorsements from educators and education researchers, even though its conceptual and empirical underpinnings are flimsy. The error theory I offer identifies two major factors that may foster belief in the efficacy of metacognitive training: goalpost-shifting around the objective of such efforts, and motivated reasoning in defense of the desirable conclusion that educators can significantly reshape students’ minds and unlock their intellectual potential through simple pedagogical interventions