Influencing clinicians and healthcare managers: can ROC be more persuasive?

Receiver Operating Characteristic analysis provides a reliable and cost effective performance measurement tool, without using full clinical trials. However, when ROC analysis shows that performance is statistically superior in one condition than another it is difficult to relate this result to effects in practice, or even to determine whether it is clinically significant. In this paper we present two concurrent analyses: using ROC methods alongside single threshold recall rate data, and suggest that reporting both provides complimentary data. Four mammographers read 160 difficult cases (41% malignant) twice, with and without prior mammograms. Lesion location and probability of malignancy was reported for each case and analyzed using JAFROC. Concurrently each participant chose recall or return to screen for each case. JAFROC analysis showed that the presence of prior mammograms improved performance (p<.05). Single threshold data showed a trend towards a 26% increase in the number of false positive recalls without prior mammograms (p=.056). If this trend were present throughout the NHS Breast Screening Programme then discarding prior mammograms would correspond to an increase in recall rate from 4.6% to 5.3%, and 12,414 extra women recalled annually for assessment. Whilst ROC methods account for all possible thresholds of recall and have higher power, providing a single threshold example of false positive, false negative, and recall rates when reporting results could be more influential for clinicians. This paper discusses whether this is a useful additional method of presenting data, or whether it is misleading and inaccurat

The time course of cancer detection performance

The purpose of this study was to measure how mammography readers' performance varies with time of day and time spent reading. This was investigated in screening practice and when reading an enriched case set. In screening practice records of time and date that each case was read, along with outcome (whether the woman was recalled for further tests, and biopsy results where performed) was extracted from records from one breast screening centre in UK (4 readers). Patterns of performance with time spent reading was also measured using an enriched test set (160 cases, 41% malignant, read three times by eight radiologists). Recall rates varied with time of day, with different patterns for each reader. Recall rates decreased as the reading session progressed both when reading the enriched test set and in screening practice. Further work is needed to expand this work to a greater number of breast screening centres, and to determine whether these patterns of performance over time can be used to optimize overall performance

Relationship between the donor population density (i.e. density of the population previously contained by the culture medium) and the proportion of dispersing cells (experiment 1).

<p>In this experiment, we manipulated the nature of the medium but not the cell concentration. Populations were exposed to medium from donor populations of higher or lower density. The non-significant relationship does not support the chemical mediation hypothesis. Each point represents an independent replicate.</p

Statistical analysis of the proportion of paramecium cells dispersing in experiments 1 and 2.

<p>The values provided for significant terms are those in the final model; for the others, the values are those when added one by one to the final model.</p>1<p>because clone and initial density are partly confounded, these two terms were not significant when added simultaneously to the model (SAS-type 3 fitting), but each was significant when added first in sequential (type 1) fitting procedure.</p

Duncan_et_al_single_inoculations

Duncan_et_al_single_inoculation

Duncan_et_al_double_inoculations

Duncan_et_al_double_inoculation

Serratia Abundance

Raw data from the main experiment, pertaining to Serratia abundance over time. Observations were input by row. Column titles specify the following: "day" -- day of the experiment on which the observation was made; "repID" -- replicate label of the microcosm; "trtmt" -- treatment label of the microcosm; "prey" -- presence (yes) or absence (no) of Paramecium in the microcosm; "predator -- presence (yes) or absence (no) of Didinium in the microcosm; "parasite" -- presence (yes) or absence (no) of Holospora in the microcosm; "species" -- the species to which the observation pertains ("SERRA" -- Serratia); "Platevol" -- volume of sample (in microliters) plated on agar gel for incubation; "Dilution" -- the factor by which the original sample from the microcosm had been diluted before plating; "Colonies" -- the number of colony-forming units counted on the plate

Paramecium Morphology & Trajectory

Raw data obtained from video capture and image analysis of swimming Paramecium ("un" -- uninfected; "inf" -- infected by Holospora)

Paramecium & Didinium Abundance

Raw data from the main experiment, pertaining to ciliate abundances over time. Observations were input by row. Column titles specify the following: "day" -- day of the experiment on which the observation was made; "repID" -- replicate label of the microcosm; "trtmt" -- treatment label of the microcosm; "prey" -- presence (yes) or absence (no) of Paramecium in the microcosm; "predator" -- presence (yes) or absence (no) of Didinium in the microcosm; "parasite" -- presence (yes) or absence (no) of Holospora in the microcosm; "species" -- the species to which the observation pertains ("PARA" -- Paramecium; "DIDI" -- Didinium); "vol" -- the volume sampled for the observation; "count" -- the number of individual present within the volume sampled

Standard electroretinograms (ERGs) recorded from normal and affected Swedish vallhund dogs.

<p>Compared to the ERG recordings of a normal dog (top row) both rod- and cone-mediated retinal functions are clearly reduced at <i>Stage 2</i>. ERG amplitudes are further decreased at <i>Stage 3</i> (third row), especially rod-mediated responses, and may not even be recordable in some affected dogs (flat lines in fourth row).</p