Does content knowledge matter in scoring teacher work samples?: A study of rater differences

The study examines the role of rater characteristics in the assessment of teacher practice as presented in Renaissance Teacher Work Samples (RTWS). The study analyzed ratings of 10 teacher work samples submitted by teacher candidates at the University of Northern Iowa between Fall 2000 and Spring 2004. These teacher work samples, created in the area of Spanish language learning at 7-11 grade levels, were analyzed to determine the impact of content expertise, amount of teaching experience, and previous RTWS rating experience on reviewer\u27s ratings. Three study questions form the foundation for the investigation: (1) Is there a significant difference between ratings assigned by raters with foreign language content experience and raters without foreign language content experience? Does this differ by sections of the teacher work sample? (2) What is the relationship between rater\u27s overall teaching experience and his/her scoring of foreign language teacher work samples? (3) What is the impact of a rater\u27s work sample scoring experience on his/her scoring of foreign language work samples? In order to address these questions, the study used a causal-comparative research design. Dependent variables in this study were scores of work samples reported by 30 raters; independent variables were presence or absence of foreign language content expertise, as well as other demographic characteristics, such as (a) the amount of foreign language teaching experience, (b) amount of teaching experience, (c) experience with scoring work samples, (d) gender, and (e) level of education. The study employed two instruments: (a) a demographic questionnaire and (b) a RTWS scoring rubric. The investigator recruited 30 participants from various middle and high schools in Iowa. Sixteen of these participants were foreign language teachers, while the remaining fourteen were educators teaching in various content areas other than foreign language or ESL. Participants of the study were asked to participate in a Renaissance Teacher Work Sample training and scoring session, rating 10 Spanish work samples submitted by UNI teacher candidates. The analysis of the demographic data revealed that participants varied greatly in almost all the areas of the questionnaire. The TWS data analysis contributed to the further understanding of the participants\u27 rating process and outcomes, their scoring speed, and allowed to answer the questions of the study. The findings of the study indicate that there is no statistically significant difference between the ratings of the Spanish teacher work samples reported by the participants in the study. Thus, the study did not find any statistically significant impact of the rater characteristics on scorers\u27 perception of teacher practice as presented in the Spanish teacher work samples. The findings of the study support other validity and reliability studies of the RTWS methodology and instrumentation. Additionally, the study outlines several areas for further research

The surgical safety checklist and patient outcomes after surgery: a prospective observational cohort study, systematic review and meta-analysis

© 2017 British Journal of Anaesthesia Background: The surgical safety checklist is widely used to improve the quality of perioperative care. However, clinicians continue to debate the clinical effectiveness of this tool. Methods: Prospective analysis of data from the International Surgical Outcomes Study (ISOS), an international observational study of elective in-patient surgery, accompanied by a systematic review and meta-analysis of published literature. The exposure was surgical safety checklist use. The primary outcome was in-hospital mortality and the secondary outcome was postoperative complications. In the ISOS cohort, a multivariable multi-level generalized linear model was used to test associations. To further contextualise these findings, we included the results from the ISOS cohort in a meta-analysis. Results are reported as odds ratios (OR) with 95% confidence intervals. Results: We included 44 814 patients from 497 hospitals in 27 countries in the ISOS analysis. There were 40 245 (89.8%) patients exposed to the checklist, whilst 7508 (16.8%) sustained ≥1 postoperative complications and 207 (0.5%) died before hospital discharge. Checklist exposure was associated with reduced mortality [odds ratio (OR) 0.49 (0.32–0.77); P\u3c0.01], but no difference in complication rates [OR 1.02 (0.88–1.19); P=0.75]. In a systematic review, we screened 3732 records and identified 11 eligible studies of 453 292 patients including the ISOS cohort. Checklist exposure was associated with both reduced postoperative mortality [OR 0.75 (0.62–0.92); P\u3c0.01; I2=87%] and reduced complication rates [OR 0.73 (0.61–0.88); P\u3c0.01; I2=89%). Conclusions: Patients exposed to a surgical safety checklist experience better postoperative outcomes, but this could simply reflect wider quality of care in hospitals where checklist use is routine

Prospective observational cohort study on grading the severity of postoperative complications in global surgery research

Background The Clavien–Dindo classification is perhaps the most widely used approach for reporting postoperative complications in clinical trials. This system classifies complication severity by the treatment provided. However, it is unclear whether the Clavien–Dindo system can be used internationally in studies across differing healthcare systems in high- (HICs) and low- and middle-income countries (LMICs). Methods This was a secondary analysis of the International Surgical Outcomes Study (ISOS), a prospective observational cohort study of elective surgery in adults. Data collection occurred over a 7-day period. Severity of complications was graded using Clavien–Dindo and the simpler ISOS grading (mild, moderate or severe, based on guided investigator judgement). Severity grading was compared using the intraclass correlation coefficient (ICC). Data are presented as frequencies and ICC values (with 95 per cent c.i.). The analysis was stratified by income status of the country, comparing HICs with LMICs. Results A total of 44 814 patients were recruited from 474 hospitals in 27 countries (19 HICs and 8 LMICs). Some 7508 patients (16·8 per cent) experienced at least one postoperative complication, equivalent to 11 664 complications in total. Using the ISOS classification, 5504 of 11 664 complications (47·2 per cent) were graded as mild, 4244 (36·4 per cent) as moderate and 1916 (16·4 per cent) as severe. Using Clavien–Dindo, 6781 of 11 664 complications (58·1 per cent) were graded as I or II, 1740 (14·9 per cent) as III, 2408 (20·6 per cent) as IV and 735 (6·3 per cent) as V. Agreement between classification systems was poor overall (ICC 0·41, 95 per cent c.i. 0·20 to 0·55), and in LMICs (ICC 0·23, 0·05 to 0·38) and HICs (ICC 0·46, 0·25 to 0·59). Conclusion Caution is recommended when using a treatment approach to grade complications in global surgery studies, as this may introduce bias unintentionally

Critical care admission following elective surgery was not associated with survival benefit:prospective analysis of data from 27 countries

Purpose: As global initiatives increase patient access to surgical treatments, there is a need to define optimal levels of perioperative care. Our aim was to describe the relationship between the provision and use of critical care resources and postoperative mortality. Methods: Planned analysis of data collected during an international 7-day cohort study of adults undergoing elective in-patient surgery. We used risk-adjusted mixed-effects logistic regression models to evaluate the association between admission to critical care immediately after surgery and in-hospital mortality. We evaluated hospital-level associations between mortality and critical care admission immediately after surgery, critical care admission to treat life-threatening complications, and hospital provision of critical care beds. We evaluated the effect of national income using interaction tests. Results: 44,814 patients from 474 hospitals in 27 countries were available for analysis. Death was more frequent amongst patients admitted directly to critical care after surgery (critical care: 103/4317 patients [2%], standard ward: 99/39,566 patients [0.3%]; adjusted OR 3.01 [2.10–5.21]; p < 0.001). This association may differ with national income (high income countries OR 2.50 vs. low and middle income countries OR 4.68; p = 0.07). At hospital level, there was no association between mortality and critical care admission directly after surgery (p = 0.26), critical care admission to treat complications (p = 0.33), or provision of critical care beds (p = 0.70). Findings of the hospital-level analyses were not affected by national income status. A sensitivity analysis including only high-risk patients yielded similar findings. Conclusions: We did not identify any survival benefit from critical care admission following surgery