Measuring Youth Program Quality: A Guide to Assessment Tools

Thanks to growing interest in the subject of youth program quality, many tools are now available to help organizations and systems assess and improve quality. Given the size and diversity of the youth-serving sector, it is unrealistic to expect that any one tool or process will fit all programs or circumstances. This report compares the purpose, history, structure, methodology, content and technical properties of nine different program observation tools

The problems of assessing software reliability ...When you really need to depend on it

This paper looks at the ways in which the reliability of software can be assessed and predicted. It shows that the levels of reliability that can be claimed with scientific justification are relatively modest

Using Wireless Pedometers to Measure Children’s Physical Activity: How Reliable is the Fitbit Zip?

The purpose of this study is to examine the reliability of wireless pedometers in measuring elementary school children’s physical activity. Activity measurement using a wireless pedometer Fitbit ZipTM was compared to activity measurement using Yamax Digi-WalkerTM SW701 for a group of randomly selected 25 children in Grades 3, 4, and 5. Fitbit ZipTM wireless pedometers were found to have an appropriate degree (Nunnally & Bernstein, 1994) of accuracy and reliability compared to the Yamax Digi-WalkerTM SW701 pedometer. The Fitbit ZipTM wireless pedometer collected more step counts than the Yamax Digi-WalkerTM SW701 pedometer; however, the difference was not statistically significant. Participants reported that they preferred wearing the Fitbit ZipTM to the Yamax Digi-WalkerTM SW701 because the Fitbit ZipTM was more comfortable to wear and less likely to fall off. Participants also reported being more motivated to move while wearing the Fitbit ZipTM

Ensuring Fair and Reliable Measures of Effective Teaching: Culminating Findings from the MET Project's Three-Year Study

States and districts have launched unprecedented efforts in recent years to build new feedback and evaluation systems that support teacher growth and development. The goal is to improve practice so that teachers can better help their students graduate from high school ready to succeed in college and beyond. These systems depend on trustworthy information about teaching effectiveness -- information that recognizes th complexity of teaching and is trusted by both teachers and administrators. To that end, the Measures of Effective Teaching (MET) project set out three years ago to investigate how a set of measures could identify effective teaching fairly and reliably. With the help of 3,000 teacher volunteers who opened up their classrooms to us -- along with scores of academic and organizational partners -- we have studied, among other measures:Classroom observation instruments, including both subject-specific and cross-subject tools, that define discrete teaching competencies and describe different levels of performance for each;Student perception surveys that assess key characteristics of the classroom environment, including supportiveness, challenge, and order; andStudent achievement gains on state tests and on more cognitively challenging assessments. We have reported findings as we learned them in order to provide states and districts with evidence-based guidance to inform their ongoing work. In our initial report in 2010 (Learning about Teaching), we found that a well-designed student perception survey can provide reliable feedback on aspects of teaching practice that are predictive of student learning.In 2012 (Gathering Feedback for Teaching), we presented similar results for classroom observations. We also found that an accurate observation rating requires two or more lessons, each scored by a different certified observer. With each analysis we have better understood the particular contribution that each measure makes to a complete picture of effective teaching and how those measures should be implemented to provide teachers with accurate and meaningful feedback.This final brief from the MET project's three-year study highlights new analyses that extend and deepen the insights from our previous work. These studies address three fundamental questions that face practitioners and policymakers engaged in creating teacher support and evaluation systems

A Large-Scale Comparison of Main Concept Production Between Persons with Aphasia and Persons Without Brain Injury

Purpose: The purposes of this study are to provide clinicians and researchers with introductory psychometric data for the main concept analysis (MCA), a measure of discourse informativeness, and specifically, to provide descriptive and comparative statistical information about the performance of a large sample of persons not brain injured (PNBIs) and persons with aphasia (PWAs) on AphasiaBank discourse tasks. Method: Transcripts of 5 semi-spontaneous discourse tasks were retrieved from the AphasiaBank database and scored according to detailed checklists and scoring procedures. Transcripts from 145 PNBIs and 238 PWAs were scored; descriptive statistics, median tests, and effect sizes are reported. Results: PWAs demonstrated overall lower informativeness scores and more frequent production of statements that were inaccurate and/or incomplete. Differences between PNBIs and PWAs were observed for all main concept measures and stories. Comparisons of PNBIs and aphasia subtypes revealed significant differences for all groups, although the pattern of differences and strength of effect sizes varied by group and discourse task. Conclusions: These results may improve the investigative and clinical utility of the MCA by providing descriptive and comparative information for PNBIs and PWAs for standardized discourse tasks that can be reliably scored. The results indicate that the MCA is sensitive to differences in discourse as a result of aphasia

Measuring is more than assigning numbers

'Measurement is fundamental to research-related activities in social science (hence this Handbook). In my own field of education research, perhaps the most discussed element of education lies in test scores. Examination results are measurements, the number of students attaining a particular standard in a test is a measurement; indeed the standard of a test is a measurement. The allocation of places at school, college or university, student:teacher ratios, funding plans, school timetables, staff workloads, adult participation rates, and the stratification of educational outcomes by sex, social class, ethnicity or geography for example, are all based on measurements. Good and careful work has been done in all of these areas (Nuttall 1987). However, the concept of measurement itself remains under-examined, and is often treated in an uncritical way. In saying this I mean more than the usual lament about qualitative:quantitative schism or the supposed reluctance of social scientists to engage with numeric analysis (Gorard et al. 2004a). I mean that even where numeric analysis is being conducted, the emphasis is on collecting, collating, analysing, and reporting the kinds of data generated by measurement, with the process of measurement and the rigor of the measurement instrument being somewhat taken for granted by many commentators. Issues that are traditionally considered by social scientists include levels of measurement, reliability, validity, and the creation of complex indices (as illustrated in some of the chapters contained in this volume). But these matters are too often dealt with primarily as technical matters – such as how to assess reliability or which statistical test to use with which combination of levels of measurement. The process of quantification itself is just assumed'

“It’s hard to tell”. The challenges of scoring patients on standardised outcome measures by multidisciplinary teams: a case study of Neurorehabilitation

Background Interest is increasing in the application of standardised outcome measures in clinical practice. Measures designed for use in research may not be sufficiently precise to be used in monitoring individual patients. However, little is known about how clinicians and in particular, multidisciplinary teams, score patients using these measures. This paper explores the challenges faced by multidisciplinary teams in allocating scores on standardised outcome measures in clinical practice. Methods Qualitative case study of an inpatient neurorehabilitation team who routinely collected standardised outcome measures on their patients. Data were collected using non participant observation, fieldnotes and tape recordings of 16 multidisciplinary team meetings during which the measures were recited and scored. Eleven clinicians from a range of different professions were also interviewed. Data were analysed used grounded theory techniques. Results We identified a number of instances where scoring the patient was 'problematic'. In 'problematic' scoring, the scores were uncertain and subject to revision and adjustment. They sometimes required negotiation to agree on a shared understanding of concepts to be measured and the guidelines for scoring. Several factors gave rise to this problematic scoring. Team members' knowledge about patients' problems changed over time so that initial scores had to be revised or dismissed, creating an impression of deterioration when none had occurred. Patients had complex problems which could not easily be distinguished from each other and patients themselves varied in their ability to perform tasks over time and across different settings. Team members from different professions worked with patients in different ways and had different perspectives on patients' problems. This was particularly an issue in the scoring of concepts such as anxiety, depression, orientation, social integration and cognitive problems. Conclusion From a psychometric perspective these problems would raise questions about the validity, reliability and responsiveness of the scores. However, from a clinical perspective, such characteristics are an inherent part of clinical judgement and reasoning. It is important to highlight the challenges faced by multidisciplinary teams in scoring patients on standardised outcome measures but it would be unwarranted to conclude that such challenges imply that these measures should not be used in clinical practice for decision making about individual patients. However, our findings do raise some concerns about the use of such measures for performance management

Evaluation of Intelligent Intrusion Detection Models

This paper discusses an evaluation methodology that can be used to assess the performance of intelligent techniques at detecting, as well as predicting, unauthorised activities in networks. The effectiveness and the performance of any developed intrusion detection model will be determined by means of evaluation and validation. The evaluation and the learning prediction performance for this task will be discussed, together with a description of validation procedures. The performance of developed detection models that incorporate intelligent elements can be evaluated using well known standard methods, such as matrix confusion, ROC curves and Lift charts. In this paper these methods, as well as other useful evaluation approaches, are discussed.Peer reviewe