Supporting students’ confidence judgement through visualising alignment in open learner models

Supporting students’ knowledge monitoring skills, a component of metacognition, can help students regulate their own learning. This thesis investigates the alignment of learners’ confidence in their knowledge with a computer’s assessment of their knowledge, visualised using an Open Learner Model (OLM). The research explored students’ preferred method for visualising inconsistent data (e.g. misalignment) in an OLM, and the ways in which visualising alignment can influence student interaction with the computer. The thesis demonstrates that visualising alignment in Open Learner Models signifi-cantly increases students’ confidence compared to a control condition. In particular, visualising alignment benefited low-achieving students, in terms of knowledge monitoring and this was associated with improvements in their performance. Students showed a preference towards the visualisations that provides an overview of the in-formation (i.e. opacity) rather than ones, which provide detailed information. Graph-ical representation is shown to be more beneficial in motivating students to interact with the system than text-based representation of the same information in the con-text of representing the alignment within OLMs

Visualising alignment to support students’ judgment of confidence in open learner models

Knowledge monitoring is a component of metacognition which can help students regulate their own learning. In adaptive learning software, the system’s model of the student can be presented as an open learner model (OLM) which is intended to enable monitoring processes. We explore how presenting alignment, between students’ self-assessed confidence and the system’s model of the student, supports knowledge monitoring. When students can see their confidence and their performance (either combined within one skill meter or expanded as two separate skill meters), their knowledge monitoring and performance improves, particularly for low-achieving students. These results indicate the importance of communicating the alignment between the system’s evaluation of student performance and student confidence in the correctness of their answers as a means to support metacognitive skills

Individual and peer comparison open learner model visualisations to identify what to work on next

Open learner models (OLM) can support self-regulated learning, collaborative interaction, and navigation in adaptive educational systems. Previous research has found that learners have a range of preferences for learner model visualisation. However, research has focused mainly on visualisations that are available in a single system, meaning that not all visualisations have been compared to each other. We present a study using screen shots of OLM visualisations for individuals and for comparing one's own learner model to the models of other individuals or the group, to define visualisations that students would be able to use to identify their next steps, across a wider range of options

An Open Learner Model Dashboard for Adaptive Learning

The thesis describes the design process of the independent OLM dashboard, MittFagkart, that visualizes student activity data across digital math tools used in Norwegian classrooms for teachers.Masteroppgave i informasjonsvitenskapINFO390MASV-INF

Learners' self-assessment and metacognition when using an open learner model with drill down

Metacognition is ‘thinking on thinking’. It is important to educational practices for learners/teachers, and in activities such as formative-assessment and self-directed learning. The ability to perform metacognition is not innate and requires fostering, and self-assessment contributes to this. Literature suggests proven practices for promoting metacognitive opportunities and ongoing enquiry about how technology best supports these. This thesis considers an open learner model (OLM) with a drill-down approach as a method to investigate support for metacognition and self-assessment. Measuring aspects of metacognition without unduly influencing it is challenging. Direct measures (e.g. learners ‘thinking-aloud’) could distort/disrupt/encourage/effect metacognition. The thesis develops methods to evaluate aspects of metacognition without directly affecting it, relevant to future learning-analytics research/OLM design. It proposes a technology specification/implementation for supporting metacognition research and highlights the relevance of using a drill-down approach. Using measures that correspond to post-hoc learner accounts, this thesis identifies a baseline of student activity that is consistent with important regulation of cognition tasks and students’ specific interest in problems. Whilst this does not always influence self-assessment accuracy, students indicating their self-assessment ability can be used as a proxy measure to identify those who will improve. Evidence supports claims that OLMs remain relevant in metacognition research

Supporting students in the analysis of case studies for professional ethics education

Intelligent tutoring systems and computer-supported collaborative environments have been designed to enhance human learning in various domains. While a number of solid techniques have been developed in the Artificial Intelligence in Education (AIED) field to foster human learning in fundamental science domains, there is still a lack of evidence about how to support learning in so-called ill-defined domains that are characterized by the absence of formal domain theories, uncertainty about best solution strategies and teaching practices, and learners' answers represented through text and argumentation. This dissertation investigates how to support students' learning in the ill-defined domain of professional ethics through a computer-based learning system. More specifically, it examines how to support students in the analysis of case studies, which is a common pedagogical practice in the ethics domain. This dissertation describes our design considerations and a resulting system called Umka. In Umka learners analyze case studies individually and collaboratively that pose some ethical or professional dilemmas. Umka provides various types of support to learners in the analysis task. In the individual analysis it provides various kinds of feedback to arguments of learners based on predefined system knowledge. In the collaborative analysis Umka fosters learners' interactions and self-reflection through system suggestions and a specifically designed visualization. The system suggestions offer learners the chance to consider certain helpful arguments of their peers, or to interact with certain helpful peers. The visualization highlights similarities and differences between the learners' positions, and illustrates the learners' level of acceptance of each other's positions. This dissertation reports on a series of experiments in which we evaluated the effectiveness of Umka's support features, and suggests several research contributions. Through this work, it is shown that despite the ill-definedness of the ethics domain, and the consequent complications of text processing and domain modelling, it is possible to build effective tutoring systems for supporting students' learning in this domain. Moreover, the techniques developed through this research for the ethics domain can be readily expanded to other ill-defined domains, where argument, qualitative analysis, metacognition and interaction over case studies are key pedagogical practices

The Design and Evaluation of an Educational Software Development Process for First Year Computing Undergraduates

First year, undergraduate computing students experience a series of well-known challenges when learning how to design and develop software solutions. These challenges, which include a failure to engage effectively with planning solutions prior to implementation ultimately impact upon the students’ competency and their retention beyond the first year of their studies. In the software industry, software development processes systematically guide the development of software solutions through iterations of analysis, design, implementation and testing. Industry-standard processes are, however, unsuitable for novice programmers as they require prior programming knowledge. This study investigates how a researcher-designed educational software development process could be created for novice undergraduate learners, and the impact of this process on their competence in learning how to develop software solutions. Based on an Action Research methodology that ran over three cycles, this research demonstrates how an educational software development methodology (termed FRESH) and its operationalised process (termed CADET which is a concrete implementation of the FRESH methodology), was designed and implemented as an educational tool for enhancing student engagement and competency in software development. Through CADET, students were reframed as software developers who understand the value in planning and developing software solutions, and not as programmers who prematurely try to implement solutions. While there remain opportunities to further enhance the technical sophistication of the process as it is implemented in practice, CADET enabled the software development steps of analysis and design to be explicit elements of developing software solutions, rather than their more typically implicit inclusion in introductory CS courses. The research contributes to the field of computing education by exploring the possibilities of – and by concretely generating – an appropriate scaffolded methodology and process; by illustrating the use of computational thinking and threshold concepts in software development; and by providing a novel evaluation framework (termed AKM-SOLO) to aid in the continuous improvement of educational processes and courses by measuring student learning experiences and competencies

D3.1 Instructional Designs for Real-time Feedback

The main objective of METALOGUE is to produce a multimodal dialogue system that is able to implement an interactive behaviour that seems natural to users and is flexible enough to exploit the full potential of multimodal interaction. The METALOGUE system will be arranged in the context of educational use-case scenarios, i.e. for training active citizens (Youth Parliament) and call centre employees. This deliverable describes the intended real-time feedback and reflection in-action support to support the training. Real-time feedback informs learners how they perform key skills and enables them to monitor their progress and thus reflect in-action. This deliverable examines the theoretical considerations of reflection in-action, what type of data is available and should be used, the timing and type of real-time feedback and, finally, concludes with an instructional design blueprint giving a global outline of a set of tasks with stepwise increasing complexity and the feedback proposed.The underlying research project is partly funded by the METALOGUE project. METALOGUE is a Seventh Framework Programme collaborative project funded by the European Commission, grant agreement number: 611073 (http://www.metalogue.eu)

Technology enhanced assessment in complex collaborative settings

Building upon discussions by the Assessment Working Group at EDUsummIT 2013, this article reviews recent developments in technology enabled assessments of collaborative problem solving in order to point out where computerised assessments are particularly useful (and where non-computerised assessments need to be retained or developed) while assuring that the purposes and designs are transparent and empowering for teachers and learners. Technology enabled assessments of higher order critical thinking in a collaborative social context can provide data about the actions, communications and products created by a learner in a designed task space. Principled assessment design is required in order for such a space to provide trustworthy evidence of learning, and the design must incorporate and take account of the engagement of the audiences for the assessment as well as vary with the purposes and contexts of the assessment. Technology enhanced assessment enables in-depth unobtrusive documentation or ‘quiet assessment’ of the many layers and dynamics of authentic performance and allows greater flexibility and dynamic interactions in and among the design features. Most important for assessment FOR learning, are interactive features that allow the learner to turn up or down the intensity, amount and sharpness of the information needed for self-absorption and adoption of the feedback. Most important in assessment OF learning, are features that compare the learner with external standards of performance. Most important in assessment AS learning, are features that allow multiple performances and a wide array of affordances for authentic action, communication and the production of artefacts