The reliability of assessing the appropriateness of requested diagnostic tests

Despite a poor reliability, peer assessment is the traditional method to assess the appropriateness of health care activities. This article describes the reliability of the human assessment of the appropriateness of diagnostic tests requests. The authors used a random selection of 1217 tests from 253 request forms submitted by general practitioners in the Maastricht region of the Netherlands. Three reviewers independently assessed the appropriateness of each requested test. Interrater kappa values ranged from 0.33 to 0.42, and kappa values of intrarater agreement ranged from 0.48 to 0.68. The joint reliability coefficient of the 3 reviewers was 0.66. This reliability is sufficient to review test ordering over a series of cases but is not sufficient to make case-by-case assessments. Sixteen reviewers are needed to obtain a joint reliability of 0.95. The authors conclude that there is substantial variation in assessment concerning what is an appropriately requested diagnostic test and that this feedback method is not reliable enough to make a case-by-case assessment. Computer support may be beneficial to support and make the process of peer review more uniform

Comparing assessment of appropriateness of diagnostic tests between a human expert and an automated reminder system

This paper describes the validation of the GRIF automated reminder system. The reminder system has been developed to influence diagnostic test ordering of General Practitioners (GPs). It generates critical comments on the basis of accepted guidelines. A retrospective random selection of 253 request forms has been taken. We compared the comments of a human expert to the comments of the reminder system. A panel of two independent reviewers judged the requested tests based on the strict interpretation of the guidelines. The sensitivity, specificity and 'predictive values' of the comments of the reminder system and the human expert were calculated using the judgement of the two reviewers as 'gold standard'