Stability and sensitivity of Learning Analytics based prediction models

Learning analytics seek to enhance the learning processes through systematic measurements of learning related data and to provide informative feedback to learners and educators. Track data from Learning Management Systems (LMS) constitute a main data source for learning analytics. This empirical contribution provides an application of Buckingham Shum and Deakin Crick’s theoretical framework of dispositional learning analytics: an infrastructure that combines learning dispositions data with data extracted from computer-assisted, formative assessments and LMSs. In two cohorts of a large introductory quantitative methods module, 2049 students were enrolled in a module based on principles of blended learning, combining face-to-face Problem-Based Learning sessions with e-tutorials. We investigated the predictive power of learning dispositions, outcomes of continuous formative assessments and other system generated data in modelling student performance and their potential to generate informative feedback. Using a dynamic, longitudinal perspective, computer-assisted formative assessments seem to be the best predictor for detecting underperforming students and academic performance, while basic LMS data did not substantially predict learning. If timely feedback is crucial, both use-intensity related track data from e-tutorial systems, and learning dispositions, are valuable sources for feedback generation

Individual differences in the preference for worked examples: lessons from an application of dispositional learning analytics

Worked-examples have been established as an effective instructional format in problem-solving practices. However, less is known about variations in the use of worked examples across individuals at different stages in their learning process in student-centred learning contexts. This study investigates different profiles of students’ learning behaviours based on clustering learning dispositions, prior knowledge, and the choice of feedback strategies in a naturalistic setting. The study was conducted on 1,072 students over an eight-week long introductory mathematics course in a blended instructional format. While practising exercises in a digital learning environment, students can opt for tutored problem-solving, untutored problem-solving, or call worked examples. The results indicated six distinct profiles of learners regarding their feedback preferences in different learning phases. Finally, we investigated antecedents and consequences of these profiles and investigated the adequacy of used feedback strategies concerning ‘help-abuse’. This research indicates that the use of instructional scaffolds as worked-examples or hints and the efficiency of that use differs from student to student, making the attempt to find patterns at an overall level a hazardous endeavour

The development of a questionnaire on metacognition for students in higher education

Background Interest in the role of metacognition has been steadily rising in most forms of education. This study focuses on the construction of a questionnaire for measuring metacognitive knowledge, metacognitive regulation and metacognitive responsiveness among students in higher education and the subsequent process of testing to determine its validity. Purpose The aim of the study was to construct an original instrument for measuring features of metacognition, henceforth referred to as the Awareness of Independent Learning Inventory (AILI), and further to establish the similarities and differences between this model and existing instruments for measuring metacognition. Sample The AILI questionnaire was distributed to 1058 students in various types of Teacher Training Institutes in the Netherlands and Belgium. The abridged English version of the questionnaire was administered to another sample of 729 students reading Economics and Business Administration at the University of Maastricht in the south of the Netherlands. Design and methods The AILI instrument was constructed on the basis of a facet design along two dimensions: components of metacognition and topics of concern to students in higher education. The data gathered with the instrument was analyzed by means of a generalisability study and a decision study, respectively. The validity of the instrument was investigated by using confirmatory factor analysis. Results The generalisability study showed that the reliability of the instrument was satisfactory. The decision study revealed that the number of items included in the questionnaire could be reduced substantially by leaving out two components of one of the dimensions in the facet design, without losing too much generalisability. The validity study showed that there was a considerable level of congruity between parts of the AILI questionnaire and the relevant parts of the Motivated Strategies for Learning Questionnaire (MSLQ). Conclusions The AILI questionnaire is a reliable and valid instrument for measuring metacognitive knowledge, regulation and responsiveness. It is suitable for use in the evaluation of the effects of interventions that purport to increase metacognitive knowledge, regulation and responsiveness of students in higher education

Dispositional Learning Analytics for Supporting Individualized Learning Feedback

An important goal of learning analytics (LA) is to improve learning by providing students with meaningful feedback. Feedback is often generated by prediction models of student success using data about students and their learning processes based on digital traces of learning activities. However, early in the learning process, when feedback is most fruitful, trace-data-based prediction models often have limited information about the initial ability of students, making it difficult to produce accurate prediction and personalized feedback to individual students. Furthermore, feedback generated from trace data without appropriate consideration of learners’ dispositions might hamper effective interventions. By providing an example of the role of learning dispositions in an LA application directed at predictive modeling in an introductory mathematics & statistics module, we make a plea for applying dispositional learning analytics (DLA) to make LA precise and actionable. DLA combines learning data with learners’ disposition data measured through for example self-report surveys. The advantage of DLA is twofold: first, to improve the accuracy of early predictions; and second, to link LA predictions with meaningful learning interventions that focus on addressing less developed learning dispositions. Dispositions in our DLA example include students’ mindsets, operationalized as entity and incremental theories of intelligence, and corresponding effort beliefs. These dispositions were inputs for a cluster analysis generating different learning profiles. These profiles were compared for other dispositions and module performance. The finding of profile differences suggests that the inclusion of disposition data and mindset data, in particular, adds predictive power to LA applications

Modeling temporality in person and variable centered approaches.

Learning analytics needs to pay more attention to the temporal aspect of learning processes, especially in self-regulated learning (SRL) research. In doing so, learning analytics models should incorporate both the duration and frequency of learning activities, the passage of time, and the temporal order of learning activities. However, where this exhortation is widely supported, there is less agreement on its consequences. Temporal aspects of learning processes could be presented as events, but does paying tribute to temporal aspects necessarily imply that event-based models are to replace variable-based models, and whether analytic discovery methods could or even should substitute traditional statistical methods? Our contribution will reason that we do not require such a paradigm shift to give temporal aspects the position it deserves. First, we argue that temporal aspects can be well integrated into variable-based models that apply statistical methods by carefully choosing appropriate time windows and granularity levels. Our second argument is that in addressing temporality in learning analytic models that describe authentic learning settings, heterogeneity is of crucial importance in both variable and event-based models. Variable-based person-centered modeling where a heterogeneous sample is split into homogeneous subsamples is suggested as a solution. Our conjecture is illustrated by an application of dispositional learning analytics, describing authentic learning processes over an eight-week full module of 2,360 students

Astro-WISE: Chaining to the Universe

The recent explosion of recorded digital data and its processed derivatives threatens to overwhelm researchers when analysing their experimental data or when looking up data items in archives and file systems. While current hardware developments allow to acquire, process and store 100s of terabytes of data at the cost of a modern sports car, the software systems to handle these data are lagging behind. This general problem is recognized and addressed by various scientific communities, e.g., DATAGRID/EGEE federates compute and storage power over the high-energy physical community, while the astronomical community is building an Internet geared Virtual Observatory, connecting archival data. These large projects either focus on a specific distribution aspect or aim to connect many sub-communities and have a relatively long trajectory for setting standards and a common layer. Here, we report "first light" of a very different solution to the problem initiated by a smaller astronomical IT community. It provides the abstract "scientific information layer" which integrates distributed scientific analysis with distributed processing and federated archiving and publishing. By designing new abstractions and mixing in old ones, a Science Information System with fully scalable cornerstones has been achieved, transforming data systems into knowledge systems. This break-through is facilitated by the full end-to-end linking of all dependent data items, which allows full backward chaining from the observer/researcher to the experiment. Key is the notion that information is intrinsic in nature and thus is the data acquired by a scientific experiment. The new abstraction is that software systems guide the user to that intrinsic information by forcing full backward and forward chaining in the data modelling.Comment: To be published in ADASS XVI ASP Conference Series, 2006, R. Shaw, F. Hill and D. Bell, ed

Overcoming cross-cultural group work tensions: mixed student perspectives on the role of social relationships

As universities worldwide rapidly internationalise, higher education classrooms have become unique spaces for collaboration between students from different countries. One common way to encourage collaboration between diverse peers is through group work. However, previous research has highlighted that cross-cultural group work can be challenging and has hinted at potential social tensions. To understand this notion better, we have used robust quantitative tools in this study to select 20 participants from a larger classroom of 860 students to take part in an in-depth qualitative interview about cross-cultural group work experiences. Participant views on social tensions in cross-cultural group work were elicited using a unique mediating artefact method to encourage reflection and in-depth discussion. In our analysis of emergent interview themes, we compared student perspectives on the role of social relationships in group work by their academic performance level. Our findings indicated that all students interviewed desired the opportunity to form social relationships with their group work members, but their motivations for doing so varied widely by academic performance level

Student engagement and perceptions of blended-learning of a clinical module in a veterinary degree program.

Blended learning has received much interest in higher education as a way to increase learning efficiency and effectiveness. By combining face-to-face teaching with technology-enhanced learning through online resources, students can manage their own learning. Blended methods are of particular interest in professional degree programs such as veterinary medicine in which students need the flexibility to undertake intra- and extramural activities to develop the range of competencies required to achieve professional qualification. Yet how veterinary students engage with blended learning activities and whether they perceive the approach as beneficial is unclear. We evaluated blended learning through review of student feedback on a 4-week clinical module in a veterinary degree program. The module combined face-to-face sessions with online resources. Feedback was collected by means of a structured online questionnaire at the end of the module and log data collected as part of a routine teaching audit. The features of blended learning that support and detract from students’ learning experience were explored using quantitative and qualitative methods. Students perceived a benefit from aspects of face-to-face teaching and technology-enhanced learning resources. Face-to-face teaching was appreciated for practical activities, whereas online resources were considered effective for facilitating module organization and allowing flexible access to learning materials. The blended approach was particularly appreciated for clinical skills in which students valued a combination of visual resources and practical activities. Although we identified several limitations with online resources that need to be addressed when constructing blended courses, blended learning shows potential to enhance student-led learning in clinical courses

The Toronto Adolescent and Youth Cohort Study:Study Design and Early Data Related to Psychosis Spectrum Symptoms, Functioning, and Suicidality

BACKGROUND: Psychosis spectrum symptoms (PSSs) occur in a sizable percentage of youth and are associated with poorer cognitive performance, poorer functioning, and suicidality (i.e., suicidal thoughts and behaviors). PSSs may occur more frequently in youths already experiencing another mental illness, but the antecedents are not well known. The Toronto Adolescent and Youth (TAY) Cohort Study aims to characterize developmental trajectories in youths with mental illness and understand associations with PSSs, functioning, and suicidality.METHODS: The TAY Cohort Study is a longitudinal cohort study that aims to assess 1500 youths (age 11-24 years) presenting to tertiary care. In this article, we describe the extensive diagnostic and clinical characterization of psychopathology, substance use, functioning, suicidality, and health service utilization in these youths, with follow-up every 6 months over 5 years, including early baseline data.RESULTS: A total of 417 participants were enrolled between May 4, 2021, and February 2, 2023. Participants met diagnostic criteria for an average of 3.5 psychiatric diagnoses, most frequently anxiety and depressive disorders. Forty-nine percent of participants met a pre-established threshold for PSSs and exhibited higher rates of functional impairment, internalizing and externalizing symptoms, and suicidality than participants without PSSs.CONCLUSIONS: Initial findings from the TAY Cohort Study demonstrate the feasibility of extensive clinical phenotyping in youths who are seeking help for mental health problems. PSS prevalence is much higher than in community-based studies. Our early data support the critical need to better understand longitudinal trajectories of clinical youth cohorts in relation to psychosis risk, functioning, and suicidality.</p