HCI expertise needed! Personalisation and feedback optimisation in online education

Two key challenges in education relate to how traditional educational providers can personalise online provisions to the students’ skill level, optimise the use of tools and increase both the generation and utilisation of feedback (in terms of timing, content, and subsequent use by students). The application of traditional programmes in the online setting is often complicated by the legacy of traditional universities infrastructures, knowledge bases (or lack thereof in the human-computer-interaction/HCI realm), and pedagogical priorities. It is here that HCI experts (designers and researchers) can have real-world impact in line with macro-HCI, while also being able to test new innovations in collaboration with educators (e.g., the practitioners in such education settings). In this note, we make a case that the HCI community is in a situation where it can make a significant contribution to traditional providers in two prospective areas: personalisation, feedback generation and increased feedback utilisation

Characterizing university students’ self-regulated learning behavior using dispositional learning analytics

[EN] Learning analytics can be used in conjunction with learner dispositions to identify at-risk students and provide personalized guidance on how to improve. Participants in the current study were students (n=192) studying a first year anatomy and physiology course. A two-step cluster analysis was performed using learning analytics data from the learning management system and self-regulated learning behavior from meta-learning assessment tasks. Three clusters of students were identified – high, medium and low self-regulated learners. High self-regulated learners were engaged with the meta-learning tasks, reported the most self-regulated learning strategies and used new strategies during semester. They also had the highest academic achievement. Compared to low self-regulated leaners, medium self-regulated learners were more engaged in the meta-learning tasks and used more learning strategies during semester, including new strategies; however, both medium and low self-regulated learners had similar levels of academic achievement. It is possible that the medium self-regulated learners represent students who were attempting to improve their learning, but had not yet found strategies that were right for them. Future evaluation of academic performance may determine whether the attempts to improve learning by medium self-regulated learners distinguishes them from low self-regulated learners in the later years of their study.Ainscough, L.; Leung, R.; Colthorpe, K.; Langfield, T. (2019). Characterizing university students’ self-regulated learning behavior using dispositional learning analytics. En HEAD'19. 5th International Conference on Higher Education Advances. Editorial Universitat Politècnica de València. 233-241. https://doi.org/10.4995/HEAD19.2019.9153OCS23324

Linking students' timing of engagement to learning design and academic performance

In recent years, the connection between Learning Design (LD) and Learning Analytics (LA) has been emphasized by many scholars as it could enhance our interpretation of LA findings and translate them to meaningful interventions. Together with numerous conceptual studies, a gradual accumulation of empirical evidence has indicated a strong connection between how instructors design for learning and student behaviour. Nonetheless, students' timing of engagement and its relation to LD and academic performance have received limited attention. Therefore, this study investigates to what extent students' timing of engagement aligned with instructor learning design, and how engagement varied across different levels of performance. The analysis was conducted over 28 weeks using trace data, on 387 students, and replicated over two semesters in 2015 and 2016. Our findings revealed a mismatch between how instructors designed for learning and how students studied in reality. In most weeks, students spent less time studying the assigned materials on the VLE compared to the number of hours recommended by instructors. The timing of engagement also varied, from in advance to catching up patterns. High-performing students spent more time studying in advance, while low-performing students spent a higher proportion of their time on catching-up activities. This study reinforced the importance of pedagogical context to transform analytics into actionable insights

Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context

The identification of students’ learning strategies by using multi-modal data that combine trace data with self-report data is the prime aim of this study. Our context is an application of dispositional learning analytics in a large introductory course mathematics and statistics, based on blended learning. Building on previous studies in which we found marked differences in how students use worked examples as a learning strategy, we compare different profiles of learning strategies on learning dispositions and learning outcome. Our results cast a new light on the issue of efficiency of learning by worked examples, tutored and untutored problem-solving: in contexts where students can apply their own preferred learning strategy, we find that learning strategies depend on learning dispositions. As a result, learning dispositions will have a confounding effect when studying the efficiency of worked examples as a learning strategy in an ecologically valid context

How do students engage with computer-based assessments: impact of study breaks on intertemporal engagement and pass rates

Study breaks and exam revision weeks are increasingly embedded in learning design under the assumption that students would make use of this time to catch up with their study or prepare for upcoming assessment tasks. However, there remains a paucity of empirical evidence to evaluate to what extent the implementation of study breaks, preparation and exam revision weeks impact students’ engagement and academic performance. By applying learning analytics in a Computer-Based Assessment (CBA) setting, this study investigates how study break weeks and assessment preparation weeks impacted the odds of passing a module using a mixed effect logistic regression on 123,916 undergraduate students in 205 modules over several semesters from 2015-2017 at the Open University. Furthermore, we investigated the intertemporal characteristics of student engagement during preparation weeks for a final assessment in an Introductory Business course over three semesters. A mixed-effect logistic regression was used to model behavioural engagement of 3,385 students on the VLE (i.e. click counts) over three semesters during the assessment preparation weeks. Our findings indicated a positive association between study breaks and the odds of passing a course, while there was no statistically significant effect in relation to the number of assessment preparation and revision weeks. Analysis of behavioural engagement on the VLE suggested that there was a higher proportion of passed students remained active during preparation and exam revision weeks compared to failed students. Compared to the pass group, the fail group also exhibited a stronger pattern of procrastination. This study offers new insights that could help institution management and course designer to evaluate the efficacy of using of study breaks and exam preparation weeks to improve student retention

Innovating Pedagogy 2017: Exploring new forms of teaching, learning and assessment, to guide educators and policy makers. Open University Innovation Report 6

This series of reports explores new forms of teaching, learning and assessment for an interactive world, to guide teachers and policy makers in productive innovation. This sixth report proposes ten innovations that are already in currency but have not yet had a profound influence on education. To produce it, a group of academics at the Institute of Educational Technology in The Open University collaborated with researchers from the Learning In a NetworKed Society (LINKS) Israeli Center of Research Excellence (I-CORE). Themes: • Big-data inquiry: thinking with data • Learners making science • Navigating post-truth societies • Immersive learning • Learning with internal values • Student-led analytics • Intergroup empathy • Humanistic knowledge-building communities • Open Textbooks • Spaced Learnin

Using Temporal Analytics to Detect Inconsistencies between Learning Design and Student Behaviours

Extensive research in learning science has established the importance of time management in online learning. Recently, learning analytics (LA) has shed further lights on the temporal characteristics of learning by allowing researchers to capture authentic digital footprints of student learning behaviours. Nonetheless, students’ timing of engagement and its relation to learning design (LD) and academic performance have received limited attention. This study investigates to what extent students’ timing of engagement aligned with instructor learning design, and how engagement varied across different levels of performance. Our findings revealed a mismatch between how instructors designed for learning and how students study. In most weeks, students spent less time studying the assigned materials on the virtual learning environment (VLE) compared to the number of hours recommended by instructors. The timing of engagement also varied, from in advance to catching up patterns. High-performing students spent more time studying in advance, while low-performing students spent a higher proportion of their time on catching-up activities. By incorporating the pedagogical context into learning analytics, not only we can understand what, why, and when students engage, but also how their behaviours are influenced by the way instructors design for learning

Affordances and limitations of learning analytics for computer-assisted language learning: a case study of the VITAL project

Learning analytics (LA) has emerged as a field that offers promising new ways to support failing or weaker students, prevent drop-out and aid retention. However, other research suggests that large datasets of learner activity can be used to understand online learning behaviour and improve pedagogy. While the use of LA in language learning has received little attention to date, available research suggests that understanding language learner behaviour could provide valuable insights into task design for instructors and materials designers, as well as help students with effective learning strategies and personalised learning pathways. This paper first discusses previous research in the field of language learning and teaching based on learner tracking and the specific affordances of LA for CALL, as well as its inherent limitations and challenges. The second part of the paper analyses data arising from the European Commission (EC) funded VITAL project that adopted a bottom-up pedagogical approach to LA and implemented learner activity tracking in different blended or distance learning settings. Referring to data arising from 285 undergraduate students on a Business French course at Hasselt University which used a flipped classroom design, statistical and process-mining techniques were applied to map and visualise actual uses of online learning resources over the course of one semester. Results suggested that most students planned their self-study sessions in accordance with the flipped classroom design, both in terms of their timing of online activity and selection of contents. Other metrics measuring active online engagement – a crucial component of successful flipped learning - indicated significant differences between successful and non-successful students. Meaningful learner patterns were revealed in the data, visualising students’ paths through the online learning environment and uses of the different activity types. The research implied that valuable insights for instructors, course designers and students can be acquired based on the tracking and analysis of language learner data and the use of visualisation and process-mining tools

Individual differences in the preference for worked examples: lessons from an application of dispositional learning analytics

Worked-examples have been established as an effective instructional format in problem-solving practices. However, less is known about variations in the use of worked examples across individuals at different stages in their learning process in student-centred learning contexts. This study investigates different profiles of students’ learning behaviours based on clustering learning dispositions, prior knowledge, and the choice of feedback strategies in a naturalistic setting. The study was conducted on 1,072 students over an eight-week long introductory mathematics course in a blended instructional format. While practising exercises in a digital learning environment, students can opt for tutored problem-solving, untutored problem-solving, or call worked examples. The results indicated six distinct profiles of learners regarding their feedback preferences in different learning phases. Finally, we investigated antecedents and consequences of these profiles and investigated the adequacy of used feedback strategies concerning ‘help-abuse’. This research indicates that the use of instructional scaffolds as worked-examples or hints and the efficiency of that use differs from student to student, making the attempt to find patterns at an overall level a hazardous endeavour

Unpacking the intertemporal impact of self-regulation in a blended mathematics environment

With the arrival of fine-grained log-data and the emergence of learning analytics, there may be new avenues to explore how Self-Regulated Learning (SRL) can provide a lens to how students learn in blended and online environments. In particular, recent research has found that the notion of time may be an essential but complex concept through which students make (un)conscious and self-regulated decisions as to when, what, and how to study. This study explored distinct clusters of behavioural engagement in an online e-tutorial called Sowiso at different time points (before tutorials, before quizzes, before exams), and their associations with self-regulated learning strategies, epistemic learning emotions, activity learning emotions, and academic performance. Using a cluster analysis on trace data of 1035 students practicing 429 online exercises in Sowiso, we identified four distinct cluster of students (e.g. early mastery, strategic, exam-driven, and inactive). Further analyses revealed significant differences between these four clusters in their academic performance, step-wise cognitive processing strategies, external self-regulation strategies, epistemic learning emotions and activity learning emotions. Our findings took a step forward towards personalised and actionable feedback in learning analytics by recognizing the complexity of how and when students engage in learning activities over time, and supporting educators to design early and theoretically informed interventions based on learning dispositions