Reviewer agreement trends from four years of electronic submissions of conference abstract

BACKGROUND: The purpose of this study was to determine the inter-rater agreement between reviewers on the quality of abstract submissions to an annual national scientific meeting (Canadian Association of Emergency Physicians; CAEP) to identify factors associated with low agreement. METHODS: All abstracts were submitted using an on-line system and assessed by three volunteer CAEP reviewers blinded to the abstracts' source. Reviewers used an on-line form specific for each type of study design to score abstracts based on nine criteria, each contributing from two to six points toward the total (maximum 24). The final score was determined to be the mean of the three reviewers' scores using Intraclass Correlation Coefficient (ICC). RESULTS: 495 Abstracts were received electronically during the four-year period, 2001 – 2004, increasing from 94 abstracts in 2001 to 165 in 2004. The mean score for submitted abstracts over the four years was 14.4 (95% CI: 14.1–14.6). While there was no significant difference between mean total scores over the four years (p = 0.23), the ICC increased from fair (0.36; 95% CI: 0.24–0.49) to moderate (0.59; 95% CI: 0.50–0.68). Reviewers agreed less on individual criteria than on the total score in general, and less on subjective than objective criteria. CONCLUSION: The correlation between reviewers' total scores suggests general recognition of "high quality" and "low quality" abstracts. Criteria based on the presence/absence of objective methodological parameters (i.e., blinding in a controlled clinical trial) resulted in higher inter-rater agreement than the more subjective and opinion-based criteria. In future abstract competitions, defining criteria more objectively so that reviewers can base their responses on empirical evidence may lead to increased consistency of scoring and, presumably, increased fairness to submitters

Guidelines on uncomplicated urinary tract infections are difficult to follow: perceived barriers and suggested interventions

Contains fulltext : 88451.pdf (publisher's version ) (Open Access)BACKGROUND: Urinary tract infections (UTI) are among the most common health problems seen in general practice. Evidence-based guidelines on UTI are available, but adherence to these guidelines varies widely among practitioners for reasons not well understood. The aim of this study was to identify the barriers to the implementation of a guideline on UTI perceived by Dutch general practitioners (GPs) and to explore interventions to overcome these barriers. METHODS: A focus group study, including 13 GPs working in general practices in the Netherlands, was conducted. Key recommendations on diagnosis and treatment of uncomplicated UTI were selected from the guideline. Barriers to guideline adherence and possible interventions to address these barriers were discussed. The focus group session was audio-taped and transcribed verbatim. Barriers were classified according to an existing framework. RESULTS: Lack of agreement with the recommendations, unavailable and inconvenient materials (i.e. dipslides), and organisational constraints were perceived as barriers for the diagnostic recommendations. Barriers to implementing the treatment recommendations were lack of applicability and organisational constraints related to the availability of drugs in pharmacies. Suggested interventions were to provide small group education to GPs and practice staff members, to improve organisation and coordination of care in out of hour services, to improve the availability of preferred dosages of drugs, and to pilot-test guidelines regionally. CONCLUSIONS: Despite sufficient knowledge of the recommendations on UTI, attitudinal and external barriers made it difficult to follow them in practice. The care concerning UTI could be optimized if these barriers are adequately addressed in implementation strategies. The feasibility and success of these strategies could be improved by involving the target group of the guideline in selecting useful interventions to address the barriers to implementation