Ensuring Fair and Reliable Measures of Effective Teaching: Culminating Findings from the MET Project's Three-Year Study

States and districts have launched unprecedented efforts in recent years to build new feedback and evaluation systems that support teacher growth and development. The goal is to improve practice so that teachers can better help their students graduate from high school ready to succeed in college and beyond. These systems depend on trustworthy information about teaching effectiveness -- information that recognizes th complexity of teaching and is trusted by both teachers and administrators. To that end, the Measures of Effective Teaching (MET) project set out three years ago to investigate how a set of measures could identify effective teaching fairly and reliably. With the help of 3,000 teacher volunteers who opened up their classrooms to us -- along with scores of academic and organizational partners -- we have studied, among other measures:Classroom observation instruments, including both subject-specific and cross-subject tools, that define discrete teaching competencies and describe different levels of performance for each;Student perception surveys that assess key characteristics of the classroom environment, including supportiveness, challenge, and order; andStudent achievement gains on state tests and on more cognitively challenging assessments. We have reported findings as we learned them in order to provide states and districts with evidence-based guidance to inform their ongoing work. In our initial report in 2010 (Learning about Teaching), we found that a well-designed student perception survey can provide reliable feedback on aspects of teaching practice that are predictive of student learning.In 2012 (Gathering Feedback for Teaching), we presented similar results for classroom observations. We also found that an accurate observation rating requires two or more lessons, each scored by a different certified observer. With each analysis we have better understood the particular contribution that each measure makes to a complete picture of effective teaching and how those measures should be implemented to provide teachers with accurate and meaningful feedback.This final brief from the MET project's three-year study highlights new analyses that extend and deepen the insights from our previous work. These studies address three fundamental questions that face practitioners and policymakers engaged in creating teacher support and evaluation systems

Peer assessment of language proficiency

2004-2005 > Academic research: refereed > Publication in refereed journalAccepted ManuscriptPublishe

Youth Development Approaches in Adolescent Family Life Demonstration Projects

Youth development (YD) strategies in conjunction with appropriate age-graded sexuality and family life education programs/curricula may have an important role to play in formulating convincing answers to these questions. Youth development approaches help youth enhance their assets rather than concentrating on their difficulties. They focus on where youth are going, helping them develop a belief in a viable future and in their ability to take actions that will bring that future about. The commitment to a future that would be disrupted by a pregnancy during adolescence is about the only thing that Zabin and her colleagues (1986) found to differentiate among Baltimore adolescents using teen clinics who did and did not get pregnant. Teens without a strong reason to avoid pregnancy got pregnant at the same rate as those who wanted to get pregnant; the only teens who were successful at avoiding pregnancy were those who had a future goal that a pregnancy would disrupt. Thus, incorporating youth development principles along with some specific techniques into the work of the Office of Adolescent Pregnancy Programs' (OAPP) abstinence-oriented programs would seem to be an important program enhancement with potentially valuable impacts

The efficacy of learning analytics interventions in higher education: A systematic review

This is the author accepted manuscript. The final version is available from Wiley via the DOI in this record.Educational institutions are increasingly turning to learning analytics to identify and intervene with students at risk of underperformance or discontinuation. However, the extent to which the current evidence base supports this investment is currently unclear, and particularly so in relation to the effectiveness of interventions based on predictive models. The aim of the present paper was to conduct a systematic review and quality assessment of studies on the use of learning analytics in higher education, focusing specifically on intervention studies. Search terms identified 689 articles, but only 11 studies evaluated the effectiveness of interventions based on learning analytics. These studies highlighted the potential of such interventions, but the general quality of the research was moderate, and left several important questions unanswered. The key recommendation based on this review is that more research into the implementation and evaluation of scientifically driven learning analytics is needed to build a solid evidence base for the feasibility, effectiveness, and generalizability of such interventions. This is particularly relevant when considering the increasing tendency of educational institutions around the world to implement learning analytics interventions with only little evidence of their effectiveness.The research reported in this paper was supported by the University of Exeter’s Effective Learning Analytics project

Evaluating service supply in conditional cash transfers

textConditional cash transfers are poverty reduction mechanisms that seek to increase demand of social services by combining an income effect with a health or education requirement. This demand-side strategy relies on a tacit assumption about the quality of and access to those services as a path to improve human capital outcomes. Some conditional cash transfers have included supply-side complementary incentives to ensure that services are suitable to deliver a good education and better health. This study reviews the existing evidence on the impact of supply-side incentives in the context of conditional cash transfers. The review finds that a limited number of studies estimate effects of supply in human capital outcomes and only a few impact evaluations consider the role of schools or health centers in enabling access. The evaluations revised find no evidence that supply side interventions coupled with conditional cash transfers directly improve program outcomes. Nonetheless, several studies highlight the relevance of school organization, in terms of school modalities and student/teacher ratios in school enrollment and attendance. Impact estimations as well as the implementation of the supply-side programs also signal the need for a more nuanced understanding of how school management influences a variety of schooling outcomes. In general, the small number of impact estimations and the restricted set of variables used limits the generalizability of the results. For this reason, a principal conclusion of the review is the need for further research on the topic, as well as consistency across impact measures and a more in-depth analysis of school supply and their influence on learning outcomes.Global Policy Studie

Generalization Programming and the Instructional Hierarchy: A Performance Feedback Intervention in Writing

National assessments have indicated that a large number of students in the United States are underperforming in writing (National Center for Education Statistics, 2012; Persky, Daane, &;; Jin, 2003). Accordingly, there is a clear need for research to identify appropriate interventions targeting writing. One intervention that has received empirical support is performance feedback (Van Houten et al., 1974, 1975, 1979). However, few performance feedback studies have explicitly targeted generalization. The primary purpose of the current study was to examine the extent to which 116 third-grade students, randomly assigned to a generalization programming (n = 39), performance feedback (n = 38), or practice-only condition (n = 39), demonstrated gains in writing fluency and were able to demonstrate stimulus and response generalization. It was hypothesized that (a) students in both intervention conditions would significantly outperform the practice-only condition in writing fluency growth and (b) students in the generalization programming condition would demonstrate stronger performance on measures of stimulus and response generalization as a result of explicit programming tactics targeting generalized skills along the sequence of the Instructional Hierarchy (Haring &;; Eaton, 1978). There was mixed support for the hypotheses, as students in the performance feedback condition demonstrated statistically significant greater writing fluency growth in comparison to both the practice-only and generalization programming conditions. In comparison to students assigned to practice-only or performance feedback conditions, students assigned to the generalization programming condition demonstrated significantly greater performance on a measure of response generalization; however, there were no differences between the conditions on a measure of stimulus generalization

A Review of Behavior Analysis in Education

Most education research in applied behavior analysis (ABA) is specific to early learners, which presents significant opportunity to better determine the effectiveness of various instruction methods for college-aged learners (CALs). Within the context of pedagogy, or method of teaching, ABA is the scientific study of the effect of instructional techniques on student behaviors. The available research for college-aged learners is scattered, non-definitive, and has gaps over time. Despite declension over the past half-century, research examining evidence-based practices in education has identified multiple approaches to help instructors manage and improve individual student behaviors and academic performance for CALs. In this review, I evaluated the available research in education generally, identified methods with the most empirically supported evidence as a best practice for teachers, and suggested topics for future research to help fill in some of the current knowledge gaps in ABA within the confines of education. Additionally, I reviewed the available behavior-analytic research with CALs. These studies were evaluated by determining whether the instructional methods demonstrated clear results with significant improvements in student behaviors and student satisfaction. Analysis of the research identified personalized system of instruction (PSI) as the most effective teaching method for improving CAL outcomes. Given this finding, it is recommended that researchers in the field of ABA who are focused on CALs, develop more methods specific to PSI, encourage teachers apply them in a real-world setting, and determine how best to make this information widely available

Dermatology residency selection criteria with an emphasis on program characteristics: a national program director survey.

Background. Dermatology residency programs are relatively diverse in their resident selection process. The authors investigated the importance of 25 dermatology residency selection criteria focusing on differences in program directors' (PDs') perception based on specific program demographics. Methods. This cross-sectional nationwide observational survey utilized a 41-item questionnaire that was developed by literature search, brainstorming sessions, and online expert reviews. The data were analyzed utilizing the reliability test, two-step clustering, and K-means methods as well as other methods. The main purpose of this study was to investigate the differences in PDs' perception regarding the importance of the selection criteria based on program demographics. Results. Ninety-five out of 114 PDs (83.3%) responded to the survey. The top five criteria for dermatology residency selection were interview, letters of recommendation, United States Medical Licensing Examination Step I scores, medical school transcripts, and clinical rotations. The following criteria were preferentially ranked based on different program characteristics: "advanced degrees," "interest in academics," "reputation of undergraduate and medical school," "prior unsuccessful attempts to match," and "number of publications." Conclusions. Our survey provides up-to-date factual data on dermatology PDs' perception in this regard. Dermatology residency programs may find the reported data useful in further optimizing their residency selection process

Integrating Item Accuracy and Reaction Time to Improve the Measurement of Inhibitory Control Abilities in Early Childhood

Efforts to improve children’s executive function are often hampered by the lack of measures that are optimized for use during the transition from preschool to elementary school. Whereas preschool-based measures often emphasize response accuracy, elementary school-based measures emphasize reaction time (RT)—especially for measures inhibitory control (IC) tasks that typically have a speeded component. The primary objective of this study was to test in a preschool-aged sample whether the joint use of item-level accuracy and RT data resulted in improved scoring for three IC tasks relative to scores derived from accuracy data alone. Generally, the joint use of item-level accuracy and RT data resulted in modest improvements in the measurement precision of IC abilities. Moreover, the joint use of item-level accuracy and RT helped eliminate floor and ceiling effects that occurred when accuracy data were considered alone. Results are discussed with respect to the importance of scoring IC tasks in ways that are maximally informative for program evaluation and longitudinal modeling

An examination of collaborative strategic reading-high school (CSR-HS) intervention in students with ASD

textThis study investigates the effects of implementing Collaborative Strategic Reading–High School (CSR–HS) on reading comprehension and challenging behavior outcomes for three high school students with Autism Spectrum Disorder (ASD). Using a combined single subject research design consisting of a delayed, concurrent multiple-baseline and an alternating treatments with reversal, three high school students with ASD were paired with neurotypical reading partners to learn and use reading strategies with informational text two to three times per week. The alternating treatment conditions were CSR-HS with choice of text (i.e., CSR-HS-C) and CSR-HS without the opportunity to choose the reading text (i.e., CSR-HS-NC). Daily comprehension checks were collected and visually inspected along with data on occurrences of various challenging behaviors exhibited by each participant during intervention. Fidelity of implementation was also measured. Increased reading comprehension scores and decreased incidences of challenges behaviors were detected for the three participants upon implementation of intervention conditions. As for the influence of the choice component on the measured outcomes, no clear differentiation between conditions was observed in terms of reading comprehension gains and reduction in challenging behavior across the three participants, suggesting that the addition of choice did not show an added value to CSR-HS intervention.Special Educatio