It’s About Time: 4th International Workshop on Temporal Analyses of Learning Data

Interest in analyses that probe the temporal aspects of learning continues to grow. The study of common and consequential sequences of events (such as learners accessing resources, interacting with other learners and engaging in self-regulatory activities) and how these are associated with learning outcomes, as well as the ways in which knowledge and skills grow or evolve over time are both core areas of interest. Learning analytics datasets are replete with fine-grained temporal data: click streams; chat logs; document edit histories (e.g. wikis, etherpads); motion tracking (e.g. eye-tracking, Microsoft Kinect), and so on. However, the emerging area of temporal analysis presents both technical and theoretical challenges in appropriating suitable techniques and interpreting results in the context of learning. The learning analytics community offers a productive focal ground for exploring and furthering efforts to address these challenges as it is already positioned in the “‘middle space’ where learning and analytic concerns meet” (Suthers & Verbert, 2013, p 1). This workshop, the fourth in a series on temporal analysis of learning, provides a focal point for analytics researchers to consider issues around and approaches to temporality in learning analytics

Qualitative, quantitative, and data mining methods for analyzing log data to characterize students' learning strategies and behaviors [discussant]

This symposium addresses how different classes of research methods, all based upon the use of log data from educational software, can facilitate the analysis of students’ learning strategies and behaviors. To this end, four multi-method programs of research are discussed, including the use of qualitative, quantitative-statistical, quantitative-modeling, and educational data mining methods. The symposium presents evidence regarding the applicability of each type of method to research questions of different grain sizes, and provides several examples of how these methods can be used in concert to facilitate our understanding of learning processes, learning strategies, and behaviors related to motivation, meta-cognition, and engagement

The Search as Learning Spaceship: Toward a Comprehensive Model of Psychological and Technological Facets of Search as Learning

Using a Web search engine is one of today’s most frequent activities. Exploratory search activities which are carried out in order to gain knowledge are conceptualized and denoted as Search as Learning (SAL). In this paper, we introduce a novel framework model which incorporates the perspective of both psychology and computer science to describe the search as learning process by reviewing recent literature. The main entities of the model are the learner who is surrounded by a specific learning context, the interface that mediates between the learner and the information environment, the information retrieval (IR) backend which manages the processes between the interface and the set of Web resources, that is, the collective Web knowledge represented in resources of different modalities. At first, we provide an overview of the current state of the art with regard to the five main entities of our model, before we outline areas of future research to improve our understanding of search as learning processes. Copyright © 2022 von Hoyer, Hoppe, Kammerer, Otto, Pardi, Rokicki, Yu, Dietze, Ewerth and Holtz

The Search as Learning Spaceship: Toward a Comprehensive Model of Psychological and Technological Facets of Search as Learning

Using a Web search engine is one of today’s most frequent activities. Exploratory search activities which are carried out in order to gain knowledge are conceptualized and denoted as Search as Learning (SAL). In this paper, we introduce a novel framework model which incorporates the perspective of both psychology and computer science to describe the search as learning process by reviewing recent literature. The main entities of the model are the learner who is surrounded by a specific learning context, the interface that mediates between the learner and the information environment, the information retrieval (IR) backend which manages the processes between the interface and the set of Web resources, that is, the collective Web knowledge represented in resources of different modalities. At first, we provide an overview of the current state of the art with regard to the five main entities of our model, before we outline areas of future research to improve our understanding of search as learning processes

Effects of Metacognitive Monitoring on Academic Achievement in an Ill-Structured Problem-Solving Environment

Higher education courses are increasingly moving online while educational approaches are concurrently shifting their focus toward student-centered approaches to learning. These approaches promote critical thinking by asking students to solve a range of ill-structured problems that exist in the real world. Researchers have found that student-centered online learning environments require students to have self-regulated learning skills, including metacognitive skills to regulate their own learning processes. Much of the research suggests that externally supporting students while they are learning online, either directly or indirectly, helps them to succeed academically. However, few empirical studies have investigated what levels of support are most effective for promoting students\u27 self-regulated learning behaviors. Additionally, these studies reported conflicting results – some found maximum support to be most effective while others found no significant difference. The purpose of this study was to investigate the effectiveness of different levels of support for self-regulated learning during a complex learning activity to solve an ill-structured problem-solving situation in an online learning environment. In addition, the role of students\u27 self-efficacy on their academic achievement was examined. A total of 101 undergraduate students from three international studies courses offered at a large urban Southeastern public university in the United States participated in the study. The students were randomly assigned to treatment (minimum support, maximum support) and control groups. Students\u27 academic achievement scores were measured using a conceptual knowledge test created by the professor teaching the courses. O\u27Neil\u27s (1997) Trait Self-Regulation Questionnaire measured students\u27 self-efficacy. Analysis of Co-Variance (ANCOVA) was conducted to analyze the data. The ANCOVA results indicated significant improvement of the academic achievement of the minimum support group versus both the maximum support and control groups. Additionally, self-efficacy as a co-variable did not significantly impact students\u27 achievement scores in any of the groups. The overall results indicated that it is important to consider the level of self-regulated learning support when designing online learning environments promoting students\u27 critical thinking skills. Promoting students\u27 self-regulated learning skills is vital when designing online higher education courses

Towards investigating the validity of measurement of self-regulated learning based on trace data

Contains fulltext : 250033.pdf (Publisher’s version ) (Open Access)Contemporary research that looks at self-regulated learning (SRL) as processes of learning events derived from trace data has attracted increasing interest over the past decade. However, limited research has been conducted that looks into the validity of trace-based measurement protocols. In order to fill this gap in the literature, we propose a novel validation approach that combines theory-driven and data-driven perspectives to increase the validity of interpretations of SRL processes extracted from trace-data. The main contribution of this approach consists of three alignments between trace data and think aloud data to improve measurement validity. In addition, we define the match rate between SRL processes extracted from trace data and think aloud as a quantitative indicator together with other three indicators (sensitivity, specificity and trace coverage), to evaluate the "degree" of validity. We tested this validation approach in a laboratory study that involved 44 learners who learned individually about the topic of artificial intelligence in education with the use of a technology-enhanced learning environment for 45 minutes. Following this new validation approach, we achieved an improved match rate between SRL processes extracted from trace-data and think aloud data (training set: 54.24%; testing set: 55.09%) compared to the match rate before applying the validation approach (training set: 38.97%; test set: 34.54%). By considering think aloud data as "reference point", this improvement of the match rate quantified the extent to which validity can be improved by using our validation approach. In conclusion, the novel validation approach presented in this study used both empirical evidence from think aloud data and rationale from our theoretical framework of SRL, which now, allows testing and improvement of the validity of trace-based SRL measurements.39 p

Integrating knowledge tracing and item response theory: A tale of two frameworks

Traditionally, the assessment and learning science commu-nities rely on different paradigms to model student performance. The assessment community uses Item Response Theory which allows modeling different student abilities and problem difficulties, while the learning science community uses Knowledge Tracing, which captures skill acquisition. These two paradigms are complementary - IRT cannot be used to model student learning, while Knowledge Tracing assumes all students and problems are the same. Recently, two highly related models based on a principled synthesis of IRT and Knowledge Tracing were introduced. However, these two models were evaluated on different data sets, using different evaluation metrics and with different ways of splitting the data into training and testing sets. In this paper we reconcile the models' results by presenting a unified view of the two models, and by evaluating the models under a common evaluation metric. We find that both models are equivalent and only differ in their training procedure. Our results show that the combined IRT and Knowledge Tracing models offer the best of assessment and learning sciences - high prediction accuracy like the IRT model, and the ability to model student learning like Knowledge Tracing

Leveraging Multimodal Learning Analytics to Understand How Humans Learn with Emerging Technologies

Major education and training challenges are plaguing the United States in preparing the next generation of the future workforce to meet the demands of the 21st Century. Several calls have been released to improve education programs to ensure learners are acquiring 21st century knowledge, skills, and abilities (KSAs). As we embark on the digital and automation ages of the 21st century, it is essential that we move away from traditional education programs that define and measure KSAs as static constructs (e.g., standardized assessments) with little consideration of the actual real-time deployment of these processes, missing critical information on the degree to which learners are acquiring and applying 21st century KSAs. The objective of this dissertation is to use 1 book chapter and 2 journal articles to illustrate the value in leveraging emerging technologies and multimodal trace data to define and measure scientific thinking, reflection, and self-regulated learning--core 21st century skills, across contexts, domains, tasks, and populations (e.g., medical versus undergraduates versus middle-school students). Chapters 2-4 of this dissertation provide evidence of ways to leverage multimodal trace data guided by theoretical perspectives in cognitive and learning sciences, with a special focus in self-regulated learning, to assess the extent to which learners engaged in scientific thinking, reflection, and self-regulated learning during learning activities with emerging technologies. Overall, results from these chapters illustrate that it is necessary to utilize methods that capture learning processes as they unfold during learning activities that are guided by theoretical perspectives in self-regulated learning. Findings from this research hold significant broader impacts for addressing the education and training challenges in the United States by collecting multimodal trace data over the course of learning to not only detect and identify how learners are developing KSAs such as scientific thinking, reflection, and self-regulated learning, but where these data could be fed into an intelligent and adaptive system to repurpose it back to trainers, teachers, instructors, and learners for just-in-time interventions and individualized feedback. The intellectual merit of this dissertation focuses predominantly on the importance of utilizing rich streams of multimodal trace data that are mapped onto different theoretical perspectives on how humans self-regulate across tasks like clinical reasoning, scientific thinking, and reflection with emerging technologies such as a game-based learning environment called Crystal Island. Discussion is incorporated around ways to leverage multimodal trace data on undergraduate, middle-school, and medical student populations across a range of tasks including learning about microbiology to problem solving with a game-based learning environment called Crystal Island and clinically reasoning about diagnoses across emerging technologies