AFROC analysis of reporting radiographer’s performance in CT head interpretation

Aim: A preliminary small scale study to assess the diagnostic performance of a limited group of reporting radiographers and consultant radiologists in clinical practice undertaking computer tomography (CT) head interpretation . Method: A multiple reader multiple case (MRMC) alternative free response receiver operating characteristic (AFROC) methodology was applied. Utilising an image bank of 30 CT head examinations, with a 1:1 ratio of normal to abnormal cases. A reference standard was established by double reporting the original reports using two additional independent consultant radiologists with arbitration of discordance by the researcher. Twelve observers from six southern National Health Service (NHS) trusts were invited to participate. The results were compared for accuracy, agreement, sensitivity, specificity. Data analysis used AFROC and area under the curve (AUC) with standard error. Results: The reporting radiographers results demonstrated a mean sensitivity rate of 88.7% (95% CI 82.3 to 95.1%), specificity 95.6% (96% CI 90.1 to 100%) and accuracy of 92.2% (95% CI 89.3 to 95%). The consultant radiologists mean sensitivity rate was 83.35% (95% CI 80 to 86.7%), specificity 90% (95% CI 86.7 to 93.3%) and accuracy of 86.65% (95% CI 83.3 to 90%). Observer performance between the two groups was compared with AFROC, AUC, and standard error analysis (p=0.94, SE 0.202). Conclusion: The findings of this research indicate that within a limited study, a small group of reporting radiographers demonstrated high levels of diagnostic accuracy in the interpretation of CT head examinations that was equivalent to a small selection of consultant radiologists

CT head reporting by radiographers: results of an accredited postgraduate programme

Aim: To evaluate the results of the summative objective structured examination (OSE) for the first four cohorts of radiographers (n ¼ 24) undertaking an accredited postgraduate course in reporting computer tomography (CT) head examinations. Method: The construction of a summative OSE contained twenty five CT head examinations that incorporated 1:1 normal to abnormal pathological examples. All cases were blind reported by three consultant radiologists to produce a valid reference standard report for comparison with the radiographer's interpretation. The radiographers (n ¼ 24) final reports (n ¼ 600) were analysed to determine the sensitivity, specificity and agreement values and concordance for the four cohorts. Results: The four cohorts (2007e2013) of postgraduate radiography students' collective OSE results established a mean sensitivity rate of 99%, specificity 95% and agreement concordance rates of 90%. The final grades indicate that within an academic environment, trained radiographers possess high levels of diagnostic performance accuracy in the interpretation of CT head examinations

Risk analysis, ideal observers, and receiver operating characteristic curves for tasks that combine detection and estimation

Previously published work on joint estimation/detection tasks has focused on the area under the estimation receiver operating characteristic (EROC) curve as a figure of merit (FOM) for these tasks in imaging. Another FOM for these joint tasks is the Bayesian risk, where a cost is assigned to all detection outcomes and to the estimation errors, and then averaged over all sources of randomness in the object ensemble and the imaging system. Important elements of the cost function, which are not included in standard EROC analysis, are that the cost for a false positive depends on the estimate produced for the parameter vector, and the cost for a false negative depends on the true value of the parameter vector. The ideal observer in this setting, which minimizes the risk, is derived for two applications. In the first application, a parameter vector is estimated only in the case of a signal present classification. For the second application, parameter vectors are estimated for either classification, and these vectors may have different dimensions. In both applications, a risk-based estimation receiver operating characteristic curve is defined and an expression for the area under this curve is given. It is also shown that, for some observers, this area may be estimated from a two alternative forced choice test. Finally, if the classifier is optimized for a given estimator, then it is shown that the slope of the risk-based estimation receiver operating characteristic curve at each point is the negative of the ratio of the prior probabilities for the two classes. (C) 2019 Society of Photo-Optical Instrumentation Engineers (SPIE)National Institutes of Health [R01-EB000803, P41-EB002035]; Department of Homeland Security [HSHQDC-14-C-BOOIO]This item from the UA Faculty Publications collection is made available by the University of Arizona with support from the University of Arizona Libraries. If you have questions, please contact us at [email protected]

Receiver Operating Characteristic (ROC) Analysis

Visual expertise covers a broad range of types of studies and methodologies. Many studies incorporate some measure(s) of observer performance or how well participants perform on a given task. Receiver Operating Characteristic (ROC) analysis is a method commonly used in signal detection tasks (i.e., those in which the observer must decide whether or not a target is present or absent; or must classify a given target as belonging to one category or another), especially those in the medical imaging literature. This frontline paper will review some of the core theoretical underpinnings of ROC analysis, provide an overview of how to conduct an ROC study, and discuss some of the key variants of ROC analysis and their applications

Interpretation of clinical imagingexaminations by radiographers:a programme of research

Background Studies which have investigated the interpretation of plain skeletal examinations by radiographers have demonstrated encouraging findings, however, the studies have not extended beyond this area of practice and radiographers' diagnostic performance for other more complex investigations has not been established. Comparisons of performance between groups of healthcare practitioners to date, has also been limited. Aim This research programme aimed to investigate the interpretation of clinical imaging examinations by radiographers, and other healthcare practitioners, in the provision of initial interpretations and/or definitive reports of plain imaging ( skeletal and chest) and crosssectional (magnetic resonance imaging [MRI] – lumbar/thoracic spine, knees and internal auditory meati [IAM]) investigations. Methods The eight studies utilised a variety of methodological approaches and included quasiexperimental and observational studies. One quasi-experimental study compared the performance of radiographers, nurses and junior doctors in initial image interpretation and another similar study included a training intervention; both utilised alternate free-response receiver operating characteristic curve (AFROC) methodology. Three of the observational studies investigated the ability of radiographers to provide definitive reports on a wide range of clinical examinations, including chest and MRI investigations, in a controlled environment. One large multi-centre observational study investigated the performance of radiographers, in clinical practice (A/E: skeletal examinations) during the implementation of a radiographic reporting service. The agreement between consultant radiologists' MRI reports of lumbar/thoracic spine, knee and IAM examinations was investigated in another observational study. The final study compared the reports of trained radiographers and consultant radiologists, with those of an index radiologist, when reporting on MRI examinations of the knee and lumbar spine, as part of a prospective pre-implementation agreement study. Results The first AFROC study demonstrated statistically significant improvements after training, for radiographers (A1=0.55 - 0.72) and nurses (A1=0.65 - 0.63), although the radiographers maintained a better overall performance post training (p=0.004) in providing an initial image interpretation of trauma radiographs of the appendicular skeleton. Radiographers also achieved statistically higher (p<0.01) AUC values (A1=0.75) than nurses (A1=0.58) and junior doctors (A1=0.54) in the second AFROC study. Three studies, which examined 11155 reports, were conducted under controlled conditions in an academic setting and provided evidence of radiographers’ high levels of accuracy in reporting of skeletal A/E (93.9%); skeletal non A/E (92.5%); chest (89.0%); MRI lumbar/thoracic spine (87.2%), knees (86.3%) and IAM (98.4%) examinations. In the multi-centre clinical study, the mean accuracy, sensitivity and specificity rates of the radiographers reports (n=7179) of plain examinations of the skeletal system in the trauma setting was found to be 99%, 98% and 99%, respectively. The considerable range of values for agreement, between consultant radiologists reports of MRI examinations of the thoracic/lumbar spine (k=0 – 0.8), knee (k=0.3 – 0.8) and IAM (k=1.0) was similar to other studies and resulted in a reasonable estimation of the performance, in the UK, of an average non specialist consultant radiologist in MRI reporting. In the final study, radiographers reported in clinical practice conditions, on a prospective random sample of knee and lumbar spine MRI examinations, to a level of agreement comparable with non-musculoskeletal consultant radiologists (Mean difference in observer agreement <1%, p=0.86). Less than 10% of observers' reports (radiographers and consultant radiologists) were found to be sufficiently discordant to be clinically important. Conclusion The outcomes of this research programme demonstrate that radiographers can provide initial interpretations of radiographic examinations of the appendicular skeleton, in the trauma setting, to a higher level of accuracy than A/E practitioners. The findings also provide evidence that selected radiographers with appropriate education and training can provide definitive reports on plain clinical examinations (A/E and non A/E referral sources) of the skeletal system and the chest; and MRI examinations of the knee, lumbar/thoracic spine and IAM to a level of performance comparable to the average non specialist consultant radiologist. Wider implementation of radiographer reporting is therefore indicated and future multi-centre research, including economic evaluations, to further inform practice at a national level, is recommended

Breast Tomosynthesis: Aspects on detection and perception of simulated lesions

The aim of this thesis was to investigate aspects on detectability of simulated lesions (microcalcifications and masses) in digital mammography (DM) and breast tomosynthesis (BT). Perception in BT image volumes were also investigated by evaluating certain reading conditions. The first study concerned the effect of system noise on the detection of masses and microcalcification clusters in DM images using a free-response task. System noise has an impact on image quality and is related to the dose level. It was found to have a substantial impact on the detection of microcalcification clusters, whereas masses were relatively unaffected. The effect of superimposed tissue in DM is the major limitation hampering the detection of masses. BT is a three-dimensional technique that reduces the effect of superimposed tissue. In the following two studies visibility was quantified for both imaging modalities in terms of the required contrast at a fixed detection performance (92% correct decisions). Contrast detail plots for lesions with sizes 0.2, 1, 3, 8 and 25 mm were generated. The first study involved only an in-plane BT slice, where the lesion centre appeared. The second study repeated the same procedure in BT image volumes for 3D distributed microcalcification clusters and 8 mm masses at two dose levels. Both studies showed that BT needs substantially less contrast than DM for lesions above 1 mm. Furthermore, the contrast threshold increased as the lesion size increased for both modalities. This is in accordance with the reduced effect of superimposed tissue in BT. For 0.2 mm lesions, substantially more contrast was needed. At equal dose, DM was better than BT for 0.2 mm lesions and microcalcification clusters. Doubling the dose substantially improved the detection in BT. Thus, system noise has a substantial impact on detection. The final study evaluated reading conditions for BT image volumes. Four viewing procedures were assessed: free scroll browsing only or combined with initial cine loops at frame rates of 9, 14 and 25 fps. They were viewed on a wide screen monitor placed in vertical or horizontal positions. A free-response task and eye tracking were utilized to record the detection performance, analysis time, visual attention and search strategies. Improved reading conditions were found for horizontally aligned BT image volumes when using free scroll browsing only or combined with a cine loop at the fastest frame rate

Quantitative Techniques for PET/CT: A Clinical Assessment of the Impact of PSF and TOF

Tomographic reconstruction has been a challenge for many imaging applications, and it is particularly problematic for count-limited modalities such as Positron Emission Tomography (PET). Recent advances in PET, including the incorporation of time-of-flight (TOF) information and modeling the variation of the point response across the imaging field (PSF), have resulted in significant improvements in image quality. While the effects of these techniques have been characterized with simulations and mathematical modeling, there has been relatively little work investigating the potential impact of such methods in the clinical setting. The objective of this work is to quantify these techniques in the context of realistic lesion detection and localization tasks for a medical environment. Mathematical observers are used to first identify optimal reconstruction parameters and then later to evaluate the performance of the reconstructions. The effect on the reconstruction algorithms is then evaluated for various patient sizes and imaging conditions. The findings for the mathematical observers are compared to, and validated by, the performance of three experienced nuclear medicine physicians completing the same task