Area of ischemia assessed by physicians and software packages from myocardial perfusion scintigrams.

The European Society of Cardiology recommends that patients with >10% area of ischemia should receive revascularization. We investigated inter-observer variability for the extent of ischemic defects reported by different physicians and by different software tools, and if inter-observer variability was reduced when the physicians were provided with a computerized suggestion of the defects

On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation

Uncertainty estimation methods are expected to improve the understanding and quality of computer-assisted methods used in medical applications (e.g., neurosurgical interventions, radiotherapy planning), where automated medical image segmentation is crucial. In supervised machine learning, a common practice to generate ground truth label data is to merge observer annotations. However, as many medical image tasks show a high inter-observer variability resulting from factors such as image quality, different levels of user expertise and domain knowledge, little is known as to how inter-observer variability and commonly used fusion methods affect the estimation of uncertainty of automated image segmentation. In this paper we analyze the effect of common image label fusion techniques on uncertainty estimation, and propose to learn the uncertainty among observers. The results highlight the negative effect of fusion methods applied in deep learning, to obtain reliable estimates of segmentation uncertainty. Additionally, we show that the learned observers' uncertainty can be combined with current standard Monte Carlo dropout Bayesian neural networks to characterize uncertainty of model's parameters.Comment: Appears in Medical Image Computing and Computer Assisted Interventions (MICCAI), 201

Characteristics of conventional high-risk coronary plaques and a novel CT defined thin-cap fibroatheroma in patients undergoing CCTA with stable chest pain

Background: Coronary computed tomography angiography (CCTA) can identify high-risk coronary plaque types. However, the inter-observer variability for high-risk plaque features, including low attenuation plaque (LAP), positive remodelling (PR), and the Napkin-Ring sign (NRS), may reduce their utility, especially amongst less experienced readers.Methodology: In a prospective study, we compared the prevalence, location and inter-observer variability of both conventional CT-defined high-risk plaques with a novel index based on quantifying the ratio of necrotic core to fibrous plaque using individualised X-ray attenuation cut-offs (the CT-defined thin-cap fibroatheroma - CT-TCFA) in 100 patients followed-up for 7 years.Results: In total, 346 plaques were identified in all patients. Seventy-two (21%) of all plaques were classified by conventional CT parameters as high-risk (either NRS or PR and LAP combined), and 43 (12%) of plaques were considered high-risk using the novel CT-TCFA definition of (Necrotic Core/fibrous plaque ratio of >0.9). The majority (80%) of the high-risk plaques (LAP&PR, NRS and CT-TCFA) were located in the proximal and mid-LAD and RCA. The kappa co-efficient of inter-observer variability (k) for NRS was 0.4 and for PR and LAP combined 0.4. While the kappa co-efficient of inter-observer variability (k) for the new CT-TCFA definition was 0.7. During follow-up, patients with either conventional high-risk plaques or CT-TCFAs were significantly more likely to have MACE (Major adverse cardiovascular events) compared to patients without coronary plaques (p value 0.03 & 0.03, respectively).Conclusion: The novel CT-TCFA is associated with MACE and has improved inter-observer variability compared with current CT-defined high-risk plaques

The impact of a radiologist-led workshop on MRI target volume delineation for radiotherapy

Introduction: Magnetic resonance imaging (MRI) is increasingly used for target volume delineation in radiotherapy due to its superior soft tissue visualisation compared to computed tomography (CT). The aim of this study was to assess the impact of a radiologist-led workshop on inter-observer variability in volume delineation on MRI. Methods: Data from three separate studies evaluating the impact of MRI in lung, breast and cervix were collated. At pre-workshop evaluation, observers involved in each clinical site were instructed to delineate specified volumes. Radiologists specialising in each cancer site conducted an interactive workshop on interpretation of images and anatomy for each clinical site. At post-workshop evaluation, observers repeated delineation a minimum of 2 weeks after the workshops. Inter-observer variability was evaluated using dice similarity coefficient (DSC) and volume similarity (VOLSIM) index comparing reference and observer volumes. Results: Post-workshop primary gross tumour volumes (GTV) were smaller than pre-workshop volumes for lung with a mean percentage reduction of 10.4%. Breast clinical target volumes (CTV) were similar but seroma volumes were smaller post-workshop on both supine (65% reduction) and prone MRI (73% reduction). Based on DSC scores, improvement in inter-observer variability was seen for the seroma cavity volume on prone MRI with a reduction in DSC score range from 0.4-0.8 to 0.7-0.9. Breast CTV demonstrated good inter-observer variability scores (mean DSC 0.9) for both pre- and post-workshop. Post-workshop observer delineated cervix GTV was smaller than pre-workshop by 26.9%. Conclusion: A radiologist-led workshop did not significantly reduce inter-observer variability in volume delineation for the three clinical sites. However, some improvement was noted in delineation of breast CTV, seroma volumes and cervix GTV

Grading of carotid artery stenosis with multidetector-row CT angiography: visual estimation or caliper measurements?

To assess the optimal method for grading carotid artery stenosis with computed tomographic angiography (CTA), we compared visual estimation to caliper measurements, and determined inter-observer variability and agreement relative to digital subtraction angiography (DSA). We included 46 patients with symptomatic carotid stenosis for whom CTA and DSA of 55 carotids was available. Stenosis quantification by CTA using visual estimation (CTAVE) (method 1) was compared with caliper measurements using subjectively optimized wide window settings (method 2) or predefined contrast-dependent narrow window settings (method 3). Measurements were independently performed by two radiologists and two residents. To determine accuracy and inter-observer variability, we calculated linear weighted kappa, performed a Bland-Altman analysis and calculated mean difference (bias) and standard deviation of differences (SDD). For inter-observer variability, kappa analysis was “very good” (0.85) for expert observers using CTAVE compared with “good” (0.61) for experts using DSA. Compared with DSA, method 1 led to overestimation (bias 5.8–8.0%, SDD 10.6–14.4), method 3 led to underestimation (bias −6.3 to −3.0%, SDD 13.0–18.1). Measurement variability between DSA and visual estimation on CTA (SDD 11.5) is close to the inter-observer variability of repeated measurements on DSA that we found in this study (SDD 11.6). For CTA of carotids, stenosis grading based on visual estimation provides better agreement to grading by DSA compared with stenosis grading based on caliper measurements

Chest computed tomography in severe bronchopulmonary dysplasia:Comparing quantitative scoring methods

Purpose: Bronchopulmonary dysplasia (BPD) is the most common complication of extreme preterm birth and structural lung abnormalities are frequently found in children with BPD. To quantify lung damage in BPD, three new Hounsfield units (HU) based chest-CT scoring methods were evaluated in terms of 1) intra- and inter-observer variability, 2) correlation with the validated Perth-Rotterdam-Annotated-Grid-Morphometric-Analysis (PRAGMA)-BPD score, and 3) correlation with clinical data. Methods: Chest CT scans of children with severe BPD were performed at a median of 7 months corrected age. Hyper- and hypo-attenuated regions were quantified using PRAGMA-BPD and three new HU based scoring methods (automated, semi-automated, and manual). Intra- and inter-observer variability was measured using intraclass correlation coefficients (ICC) and Bland-Altman plots. The correlation between the 4 scoring methods and clinical data was assessed using Spearman rank correlation. Results: Thirty-five patients (median gestational age 26.1 weeks) were included. Intra- and inter-observer variability was excellent for hyper- and hypo-attenuation regions for the manual HU method and PRAGMA-BPD (ICCs range 0.80–0.97). ICC values for the semi-automated HU method were poorer, in particular for the inter-observer variability of hypo- (0.22–0.71) and hyper-attenuation (-0.06–0.89). The manual HU method was highly correlated with PRAGMA-BPD score for both hyper- (ρs0.92, p &lt; 0.001) and hypo-attenuation (ρs0.79, p &lt; 0.001), while automated and semi-automated HU methods showed poor correlation for hypo- (ρs &lt; 0.22) and good correlation for hyper-attenuation (ρs0.72–0.74, p &lt; 0.001). Several scores of hyperattenuation correlated with the use of inhaled bronchodilators in the first year of life; two hypoattenuation scores correlated with birth weight. Conclusions: PRAGMA-BPD and the manual HU method have the best reproducibility for quantification of CT abnormalities in BPD.</p

Intra- and inter-operator reproducibility of automated cloud-based carotid lumen diameter ultrasound measurement

Background: Common carotid artery lumen diameter (LD) ultrasound measurement systems are either manual or semi-automated and lack reproducibility and variability studies. This pilot study presents an automated and cloud-based LD measurements software system (AtheroCloud) and evaluates its: (i) intra/inter-operator reproducibility and (ii) intra/inter-observer variability. Methods: 100 patients (83 M, mean age: 68 ± 11 years), IRB approved, consisted of L/R CCA artery (200 ultrasound images), acquired using a 7.5-MHz linear transducer. The intra/inter-operator reproducibility was verified using three operator's readings. Near-wall and far carotid wall borders were manually traced by two observers for intra/inter-observer variability analysis. Results: The mean coefficient of correlation (CC) for intra- and inter-operator reproducibility between all the three automated reading pairs were: 0.99 (P &lt; 0.0001) and 0.97 (P &lt; 0.0001), respectively. The mean CC for intra- and inter-observer variability between both the manual reading pairs were 0.98 (P &lt; 0.0001) and 0.98 (P &lt; 0.0001), respectively. The Figure-of-Merit between the mean of the three automated readings against the four manuals were 98.32%, 99.50%, 98.94% and 98.49%, respectively. Conclusions: The AtheroCloud LD measurement system showed high intra/inter-operator reproducibility hence can be adapted for vascular screening mode or pharmaceutical clinical trial mode

Implications of inter observer variability in cervical smear reporting

Background: In spite of the Bethesda system 2001 (TBS 2001), formulating strict guidelines for reporting cervical smears, intra observer and inter observer variations are unavoidable and can be considered an inherent part of the reporting system. The implications of this variation are in the quality of performance of the reporting laboratory and in the patient management. Rescreening is a tool to reduce the variations and improve the quality of both the laboratory staff and laboratory as such. Rescreening by two or more experienced observers has helped in identifying new cases better. The present study aims to rescreen cervical smears by two independent observers, to compare the results of the two independent observers and to understand the implications of this variability on the quality of cervical smear reporting.Methods: 1000 consecutive cervical smears were rescreened by two experienced cyto-pathologists independently. Their findings were charted out and analyzed statistically for kappa value.Results: Initial reporting had identified 20 cases of neoplastic nature. First observer identified, in addition, 6 new cases and second observer identified 12 new cases. The inter observer variability of 6 cases showed a kappa value of 0.89.Conclusions: Rescreening is a safe way of picking up missed cases. Rescreening by two or more observers is better in identifying new cases. This helps in improving the quality of reporting personnel and the laboratory as well as in improving patient care