Psychometrics in Practice at RCEC

A broad range of topics is dealt with in this volume: from combining the psychometric generalizability and item response theories to the ideas for an integrated formative use of data-driven decision making, assessment for learning and diagnostic testing. A number of chapters pay attention to computerized (adaptive) and classification testing. Other chapters treat the quality of testing in a general sense, but for topics like maintaining standards or the testing of writing ability, the quality of testing is dealt with more specifically.\ud All authors are connected to RCEC as researchers. They present one of their current research topics and provide some insight into the focus of RCEC. The selection of the topics and the editing intends that the book should be of special interest to educational researchers, psychometricians and practitioners in educational assessment

Informing Writing: The Benefits of Formative Assessment

Examines whether classroom-based formative writing assessment - designed to provide students with feedback and modified instruction as needed - improves student writing and how teachers can improve such assessment. Suggests best practices

The Learning Grid and E-Assessment using Latent Semantic Analysis

E-assessment is an important component of e-learning and e-qualification. Formative and summative assessment serve different purposes and both types of evaluation are critical to the pedagogicalprocess. While students are studying, practicing, working, or revising, formative assessment provides direction, focus, and guidance. Summative assessment provides the means to evaluate a learner's achievement and communicate that achievement to interested parties. Latent Semantic Analysis (LSA) is a statistical method for inferring meaning from a text. Applications based on LSA exist that provide both summative and formative assessment of a learner's work. However, the huge computational needs are a major problem with this promising technique. This paper explains how LSA works, describes the breadth of existing applications using LSA, explains how LSA is particularly suited to e-assessment, and proposes research to exploit the potential computational power of the Grid to overcome one of LSA's drawbacks

Using student experience as a model for designing an automatic feedback system for short essays

The SAFeSEA project (Supportive Automated Feedback for Short Essay Answers) aims to develop an automated feedback system to support university students as they write summative essays. Empirical studies carried out in the initial phase of the system’s development illuminated students’ approaches to and understandings of the essay-writing process. Findings from these studies suggested that, regardless of their experience of higher education, students consider essay-writing as: 1) a sequential set of activities, 2) a process that is enhanced through particular sources of support and 3) a skill that requires the development of personal strategies. Further data collected from tutors offered insight into the feedback and reflection stages of essay-writing. These perspectives offered a fundamental model of essay-writing and feedback to inform the ongoing, iterative development of this automated feedback system and indeed, for any institution developing tools to support students’ writing

Learning support for mature, part-time, evening students: providing feedback via frequent, computer-based assessments

A new module in our first year Biology curriculum was used as a vehicle to test strategies for improving learning support. To this end, we have administered frequent CBA, incorporating extensive feedback, both to pace the students’ study efforts and to pinpoint areas in which additional help from lecturers may be required. Three of the 7 CBA provided through the 15-week course were initially given as open-book summative tests, thus contributing to the overall mark for the module. Other CBA were formative: these included repeats of the summative CBA made available for revision purposes, as well as other CBA which focused mainly on aspects of the course that were summatively assessed by other means. A closedbook final exam, also computer-based, was given in the final week as a comprehensive assessment. We have evaluated the utility and effectiveness of our approach by surveying student opinion via questionnaires, examining patterns and extent of student use of formative assessments, and by analysing grades for the summative CBA. We have found the students’ perceptions of the approach to be largely positive and that the formative CBA were well-used, especially as revision aids for the final exam. Our analysis further indicates that the style of the assessments may have been especially helpful to students whose first language is not English

INQUIRE: a case study in evaluating the potential of online MCQ tests in a discursive subject

There has been a wealth of investigation into the use of online multiple-choice questions as a means of summative assessment, however the research into the use of formative MCQs by the same mode of delivery still remains patchy. Similarly, research and implementation has been largely concentrated within the Sciences and Medicine rather than the more discursive subjects within the Humanities and Social Sciences. The INQUIRE (Interactive Questions Reinforcing Education) Evaluation Project was jointly conducted by two groups at the University of Oxford-the Said Business School and the Academic Computing Development Team to evaluate the use of online MCQs as a mechanism to reinforce and extend student learning. This initial study used a small set of highly focused MCQ tests that were designed to complement an introductory series of first-year undergraduate management lectures. MCQ is a simple and well-established technology, and hence the emphasis was very much on situating the tests within the student experience. The paper will cover how the online MCQs are intended to fit into the Oxford Undergraduate study agenda, and how a simple evaluation was executed and planned to investigate their usage and impact. The chosen method of evaluation was to combine focus groups with automated online methods of tracking, and the paper discusses the findings of both of these

Implementation of computer assisted assessment: lessons from the literature

This paper draws attention to literature surrounding the subject of computer-assisted assessment (CAA). A brief overview of traditional methods of assessment is presented, highlighting areas of concern in existing techniques. CAA is then defined, and instances of its introduction in various educational spheres are identified, with the main focus of the paper concerning the implementation of CAA. Through referenced articles, evidence is offered to inform practitioners, and direct further research into CAA from a technological and pedagogical perspective. This includes issues relating to interoperability of questions, security, test construction and testing higher cognitive skills. The paper concludes by suggesting that an institutional strategy for CAA coupled with staff development in test construction for a CAA environment can increase the chances of successful implementation

Scoping a vision for formative e-assessment: a project report for JISC

Assessment is an integral part of teaching and learning. If the relationship between teaching and learning were causal, i. e. if students always mastered the intended learning outcomes of a particular sequence of instruction, assessment would be superfluous. Experience and research suggest this is not the case: what is learnt can often be quite different from what is taught. Formative assessment is motivated by a concern with the elicitation of relevant information about student understanding and / or achievement, its interpretation and an exploration of how it can lead to actions that result in better learning. In the context of a policy drive towards technology-enhanced approaches to teaching and learning, the question of the role of digital technologies is key and it is the latter on which this project particularly focuses. The project and its deliverables have been informed by recent and relevant literature, in particular recent work by Black andIn this work, they put forward a framework which suggests that assessment for learning their term for formative assessment can be conceptualised as consisting of a number of aspects and five keystrategies. The key aspects revolve around the where the learner is going, where the learner is right now and how she can get there and examines the role played by the teacher, peers and the learner. Language: English Keywords: assessments, case studies, design patterns, e-assessmen