License to evaluate: Preparing learning analytics dashboards for educational practice

Learning analytics can bridge the gap between learning sciences and data analytics, leveraging the expertise of both fields in exploring the vast amount of data generated in online learning environments. A typical learning analytics intervention is the learning dashboard, a visualisation tool built with the purpose of empowering teachers and learners to make informed decisions about the learning process. Related work has investigated learning dashboards, yet none have explored the theoretical foundation that should inform the design and evaluation of such interventions. In this systematic literature review, we analyse the extent to which theories and models from learning sciences have been integrated into the development of learning dashboards aimed at learners. Our analysis revealed that very few dashboard evaluations take into account the educational concepts that were used as a theoretical foundation for their design. Furthermore, we report findings suggesting that comparison with peers, a common reference frame for contextualising information on learning analytics dashboards, was not perceived positively by all learners. We summarise the insights gathered through our literature review in a set of recommendations for the design and evaluation of learning analytics dashboards for learners

Supporting students’ confidence judgement through visualising alignment in open learner models

Supporting students’ knowledge monitoring skills, a component of metacognition, can help students regulate their own learning. This thesis investigates the alignment of learners’ confidence in their knowledge with a computer’s assessment of their knowledge, visualised using an Open Learner Model (OLM). The research explored students’ preferred method for visualising inconsistent data (e.g. misalignment) in an OLM, and the ways in which visualising alignment can influence student interaction with the computer. The thesis demonstrates that visualising alignment in Open Learner Models signifi-cantly increases students’ confidence compared to a control condition. In particular, visualising alignment benefited low-achieving students, in terms of knowledge monitoring and this was associated with improvements in their performance. Students showed a preference towards the visualisations that provides an overview of the in-formation (i.e. opacity) rather than ones, which provide detailed information. Graph-ical representation is shown to be more beneficial in motivating students to interact with the system than text-based representation of the same information in the con-text of representing the alignment within OLMs

Learners' self-assessment and metacognition when using an open learner model with drill down

Metacognition is ‘thinking on thinking’. It is important to educational practices for learners/teachers, and in activities such as formative-assessment and self-directed learning. The ability to perform metacognition is not innate and requires fostering, and self-assessment contributes to this. Literature suggests proven practices for promoting metacognitive opportunities and ongoing enquiry about how technology best supports these. This thesis considers an open learner model (OLM) with a drill-down approach as a method to investigate support for metacognition and self-assessment. Measuring aspects of metacognition without unduly influencing it is challenging. Direct measures (e.g. learners ‘thinking-aloud’) could distort/disrupt/encourage/effect metacognition. The thesis develops methods to evaluate aspects of metacognition without directly affecting it, relevant to future learning-analytics research/OLM design. It proposes a technology specification/implementation for supporting metacognition research and highlights the relevance of using a drill-down approach. Using measures that correspond to post-hoc learner accounts, this thesis identifies a baseline of student activity that is consistent with important regulation of cognition tasks and students’ specific interest in problems. Whilst this does not always influence self-assessment accuracy, students indicating their self-assessment ability can be used as a proxy measure to identify those who will improve. Evidence supports claims that OLMs remain relevant in metacognition research