Informing Writing: The Benefits of Formative Assessment

Examines whether classroom-based formative writing assessment - designed to provide students with feedback and modified instruction as needed - improves student writing and how teachers can improve such assessment. Suggests best practices

Measure for Measure: A Critical Consumers' Guide to Reading Comprehension Assessments for Adolescents

A companion report to Carnegie's Time to Act, analyzes and rates commonly used reading comprehension tests for various elements and purposes. Outlines trends in types of questions, stress on critical thinking, and screening or diagnostic functions

Development and Evaluation of the Nebraska Assessment of Computing Knowledge

One way to increase the quality of computing education research is to increase the quality of the measurement tools that are available to researchers, especially measures of students’ knowledge and skills. This paper represents a step toward increasing the number of available thoroughly-evaluated tests that can be used in computing education research by evaluating the psychometric properties of a multiple-choice test designed to differentiate undergraduate students in terms of their mastery of foundational computing concepts. Classical test theory and item response theory analyses are reported and indicate that the test is a reliable, psychometrically-sound instrument suitable for research with undergraduate students. Limitations and the importance of using standardized measures of learning in education research are discussed

Measuring beginner reading skills: An empirical evaluation of alternative instruments and their potential use for policymaking and accountability in Peru

Based on analysis of reading performance data from 475 third-graders in Peru, this study makes recommendations on improving reading tests, choice of reading standards, and how to present the results at the school and individual levels. The paper reviews the literature on using reading skills measurement in the early grades to guide policymaking, strengthen accountability, and improve education quality. It uses data generated from the same students using two common approaches to measuring reading skills: an individually-administered oral fluency test, and a group-administered written comprehension test designed by the Ministry of Education for the 2006 universal standard test of second grade reading comprehension. These two approaches have sometimes been presented as competing alternatives, but the paper shows that it is better if they are used together, as complements. Based on psychometric analysis, the paper shows that both the oral and written tests adequately measured studentsâÃÂàreading abilities. The results show that reading fluency and comprehension are correlated: fluent readers are more likely to understand what they read than non-fluent readers. The strength of the fluency-comprehension relationship depends on the level of fluency, the difficulty of the questions, and social characteristics of the school. The paper recommends using improved versions of both tests to evaluate early grade reading skills, as a central element of a system of accountability for results. It proposes a model for reporting test results desgned to highlight the importance of reading standards, mobilize the education community to reach them, track progress, and identify students in need of extra support.academic performance; access to education; access to preschool; access to preschool education; Achievement; achievements; addition; adults; Assessment of Literacy; average score; Basic Skills;

Mixed Methods: Using a Combination of Techniques to Assess Writing Ability

A productive ability such as writing can be assessed only through a candidate‘s performance on a task, giving rise to concerns about the reliability and validity of writing assessments. In this chapter, it is argued that a combination of different techniques can help improve the quality of an evaluation of writing ability. First, an indirect test can be applied to reliably assess specific components of the writing process (e.g., revision), adding to the validity of the assessment. Furthermore, an analytic rating procedure accompanied by anchor essays allows raters to reliably evaluate the content and overall structure of written pieces. And last, automated scoring techniques can be used to objectively score text features considered important to text quality. Combining these methods provides an evaluation that is solid and informativ

Estimating intra-rater reliability on an oral english proficiency test from a Bilingual Education Program

Este estudio tiene como objetivo presentar los resultados de una investigación la cual pretendía estimar el nivel de confiabilidad intra-evaluador en un examen de suficiencia oral en inglés, y determinar los diferentes factores internos y externos que afectan la consistencia del evaluador. Los participantes involucrados en el desarrollo de este estudio fueron dos profesores encargados de evaluar la sección de habla de un examen de suficiencia administrado en la Licenciatura en Bilingüismo con énfasis en inglés. Se calculó un coeficiente de correlación con el fin de establecer la consistencia de los evaluadores mientras que un protocolo verbal retrospectivo fue llevado a cabo para recopilar información acerca de los factores que influyen en la confiabilidad del evaluador. Los resultados sugieren que hay un alto nivel de confiabilidad intra-evaluador en el examen de suficiencia en cuanto el coeficiente de correlación arrojó valores superiores a .80. No obstante, aspectos relacionados con la falta de adhesión a los criterios de la rúbrica, la relación evaluador-estudiante, las condiciones físicas, y la presión y responsabilidad del evaluador para dar una nota precisa fueron identificados como factores que afectan la consistencia del evaluador. Finalmente, se proporcionaron algunas implicaciones procedentes de esta investigación

'It’s hard to define good writing, but i recognise it when i see it’: can consensus-based assessment evaluate the teaching of writing?

In a Higher Education environment where evidence-based practice and accountability are highly valued, most writing practitioners will be familiar with direct requests or less tangible pressures to demonstrate that their teaching has a positive impact on students’ writing skills. Although such evaluations are not devoid of risk and the need for them is contested, it can be argued that it is better to engage with them, as this can avoid the danger of overly simplistic forms of measurements being imposed. The current paper engages with this question by proposing the conceptual basis for a new measurement tool. Based on Amabile’s Consensual Assessment Technique (CAT), developed to assess creativity, the tool develops the idea of consensual assessment of writing as a methodology that can provide robust data through systematic measurement. At the same time, I argue consensual assessment reflects the evaluation of writing in real life situations more closely than many of the methodologies for writing assessment used in other contexts, primarily large scale tests. As such, it would allow writing practitioners to go beyond ethnographic methods, or self- reporting, in order to obtain greater insight into the ways in which their teaching helps change students’ actual writing, without sacrificing the complexity of writing as social interaction, which is fundamental to an academic literacies approach

Quantifying critical thinking: Development and validation of the Physics Lab Inventory of Critical thinking (PLIC)

Introductory physics lab instruction is undergoing a transformation, with increasing emphasis on developing experimentation and critical thinking skills. These changes present a need for standardized assessment instruments to determine the degree to which students develop these skills through instructional labs. In this article, we present the development and validation of the Physics Lab Inventory of Critical thinking (PLIC). We define critical thinking as the ability to use data and evidence to decide what to trust and what to do. The PLIC is a 10-question, closed-response assessment that probes student critical thinking skills in the context of physics experimentation. Using interviews and data from 5584 students at 29 institutions, we demonstrate, through qualitative and quantitative means, the validity and reliability of the instrument at measuring student critical thinking skills. This establishes a valuable new assessment instrument for instructional labs.Comment: 16 pages, 4 figure