Erratum to: Methods for evaluating medical tests and biomarkers

[This corrects the article DOI: 10.1186/s41512-016-0001-y.]

Evidence synthesis to inform model-based cost-effectiveness evaluations of diagnostic tests: a methodological systematic review of health technology assessments

Background: Evaluations of diagnostic tests are challenging because of the indirect nature of their impact on patient outcomes. Model-based health economic evaluations of tests allow different types of evidence from various sources to be incorporated and enable cost-effectiveness estimates to be made beyond the duration of available study data. To parameterize a health-economic model fully, all the ways a test impacts on patient health must be quantified, including but not limited to diagnostic test accuracy. Methods: We assessed all UK NIHR HTA reports published May 2009-July 2015. Reports were included if they evaluated a diagnostic test, included a model-based health economic evaluation and included a systematic review and meta-analysis of test accuracy. From each eligible report we extracted information on the following topics: 1) what evidence aside from test accuracy was searched for and synthesised, 2) which methods were used to synthesise test accuracy evidence and how did the results inform the economic model, 3) how/whether threshold effects were explored, 4) how the potential dependency between multiple tests in a pathway was accounted for, and 5) for evaluations of tests targeted at the primary care setting, how evidence from differing healthcare settings was incorporated. Results: The bivariate or HSROC model was implemented in 20/22 reports that met all inclusion criteria. Test accuracy data for health economic modelling was obtained from meta-analyses completely in four reports, partially in fourteen reports and not at all in four reports. Only 2/7 reports that used a quantitative test gave clear threshold recommendations. All 22 reports explored the effect of uncertainty in accuracy parameters but most of those that used multiple tests did not allow for dependence between test results. 7/22 tests were potentially suitable for primary care but the majority found limited evidence on test accuracy in primary care settings. Conclusions: The uptake of appropriate meta-analysis methods for synthesising evidence on diagnostic test accuracy in UK NIHR HTAs has improved in recent years. Future research should focus on other evidence requirements for cost-effectiveness assessment, threshold effects for quantitative tests and the impact of multiple diagnostic tests

Evaluating the potential negative effects of school-based prevention programs aiming to reduce alcohol and drug misuse in adolescents: A systematic review of research articles prior to 2013

Issues: Reviews of alcohol and drug prevention programs commonly focus on positive effects, whilst disregarding possible iatrogenic effects. Our objective was to summarize evidence on iatrogenic effects of alcohol and drug prevention programs targeting adolescents. We systematically investigated the nature of these effects, the number of iatrogenic effects reported on, the sort of impact they have (e.g. in terms of severity) and the type of programs associated with iatrogenic effects. Approach: On January 2013, we searched the Cochrane Central Register of Controlled Trials, Medline, PsycINFO, Web of Science, Eric, Scirus and we actively searched for grey literature via Google Scholar and OpenGrey. Included were (quasi) RCTs that evaluated the effectiveness of school-based alcohol and drug prevention programs for adolescents. Ninety-three articles were eligible for inclusion and were screened for potential iatrogenic effects. Key findings: Ten articles reported on iatrogenic effects, which were predominantly found on substance use outcomes. The quality of these articles was assessed and a meta-analysis was not possible due to the heterogeneity in interventions and outcome measures. An increase in substance use was found in 5 out of 7 RCTs and in 1 quasi RCT. The magnitude of the iatrogenic effects found was not always clear due to the outcome measures used. Implications: Iatrogenic effects were assessed in a limited number of studies. In future studies on prevention programs the number, nature and impact of iatrogenic effects should standardly be assessed and reported. It allows us to detect potential problem areas in the conceptualization of prevention programs.nrpages: 33status: publishe

Regression shrinkage methods for clinical prediction models do not guarantee improved performance: Simulation study

When developing risk prediction models on datasets with limited sample size, shrinkage methods are recommended. Earlier studies showed that shrinkage results in better predictive performance on average. This simulation study aimed to investigate the variability of regression shrinkage on predictive performance for a binary outcome. We compared standard maximum likelihood with the following shrinkage methods: uniform shrinkage (likelihood-based and bootstrap-based), penalized maximum likelihood (ridge) methods, LASSO logistic regression, adaptive LASSO, and Firth's correction. In the simulation study, we varied the number of predictors and their strength, the correlation between predictors, the event rate of the outcome, and the events per variable. In terms of results, we focused on the calibration slope. The slope indicates whether risk predictions are too extreme (slope  1). The results can be summarized into three main findings. First, shrinkage improved calibration slopes on average. Second, the between-sample variability of calibration slopes was often increased relative to maximum likelihood. In contrast to other shrinkage approaches, Firth's correction had a small shrinkage effect but showed low variability. Third, the correlation between the estimated shrinkage and the optimal shrinkage to remove overfitting was typically negative, with Firth's correction as the exception. We conclude that, despite improved performance on average, shrinkage often worked poorly in individual datasets, in particular when it was most needed. The results imply that shrinkage methods do not solve problems associated with small sample size or low number of events per variable.status: publishe

The use of IETA terminology and other ultrasound features in the diagnosis of intracavitary fibroids and polyps

ObjectivesThe primary aim is to evaluate interobserver agreement on International Endometrial Tumour Analysis (IETA) terminology and other ultrasound (US) features (“bridge sign”, lesion shape and "fan‐shaped shadowing") of polyps and fibroids. The secondary aim is to assess the relevance of these US features in the diagnosis of polyps and fibroids.MethodsFrom Leuven University Hospital Bleeding Clinic we selected stored 3D‐volumes of 113 non‐consecutive patients with a final pathologic diagnosis of fibroid or polyp. 3D‐volumes of unenhanced transvaginal US examinations on grayscale (GS) and with Doppler signal (DS) as well as fluid instillation sonography (FIS) in GS and adding DS (FIS&DS) were converted in video clips. Two experienced sonographers, blinded to pathology results, scored the clips for the presence of the US features and proposed a diagnosis. Interobserver agreement was determined using Cohen's kappa coefficient (k).ResultsThere were 71 polyps and 42 fibroids. Interobserver agreement was good for “circular flow”, both on DS (k = 0.66) and on FIS&DS (k = 0.62), moderate for the “bridge sign” on GS (k = 0.50), lesion shape on FIS (k = 0.44), “single dominant vessel” on FIS&DS (k = 0.57), “fan‐shaped shadowing” on FIS (k = 0.57) and for “multiple vessels with focal origin” on FIS&DS (k = 0.60). In polyps, sonographers reported an oval shape on FIS in 76‐77% (percentage for sonographer 1 and 2), a “single dominant vessel” on FIS&DS in 32‐37% and “multiple vessels with focal origin” on FIS&DS in 15‐20%. In fibroids, the sonographers reported the “bridge sign” on GS in 36‐45%, “fan‐shaped shadowing” on FIS in 38‐52% and “circular flow” on DS in 29‐52% and on FIS&DS in 38‐57%. Examiners agreed on the diagnosis in 82% of the cases (k = 0.69); 63% were polyps.ConclusionsInterobserver agreement for some of the US features is good to moderate. These may be selected in the differential diagnosis between polyps and intracavitary fibroids. Overall there was a good agreement on the diagnosis between both examiners.info:eu-repo/semantics/publishe

ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models

OBJECTIVES: Receiver operating characteristic (ROC) curves show how well a risk prediction model discriminates between patients with and without a condition. We aim to investigate how ROC curves are presented in the literature and discuss and illustrate their potential limitations. STUDY DESIGN AND SETTING: We conducted a pragmatic literature review of contemporary publications that externally validated clinical prediction models. We illustrated limitations of ROC curves using a testicular cancer case study and simulated data. RESULTS: Of 86 identified prediction modeling studies, 52 (60%) presented ROC curves without thresholds and one (1%) presented an ROC curve with only a few thresholds. We illustrate that ROC curves in their standard form withhold threshold information have an unstable shape even for the same area under the curve (AUC) and are problematic for comparing model performance conditional on threshold. We compare ROC curves with classification plots, which show sensitivity and specificity conditional on risk thresholds. CONCLUSION: ROC curves do not offer more information than the AUC to indicate discriminative ability. To assess the model's performance for decision-making, results should be provided conditional on risk thresholds. Therefore, if discriminatory ability must be visualized, classification plots are attractive.status: publishe

Doctors' experiences and their perception of the most stressful aspects of complaints processes in the UK: an analysis of qualitative survey data

Objectives: To examine doctors’ experiences of complaints, including which aspects are most stressful. We also investigated how doctors felt complaints processes could be improved. Design and methods: A qualitative study based on a cross-sectional survey of members of the British Medical Association (BMA). We asked the following: (1) Try to summarise as best as you can your experience of the complaints process and how it made you feel. (2) What were the most stressful aspects of the complaint? (3) What would you improve in the complaints system? Participants: We sent the survey to 95 636 doctors, and received 10 930 (11.4%) responses. Of these, 6146 had a previous, recent or current complaint and 3417 (31.3%) of these respondents answered questions 1 and 2. We randomly selected 1000 answers for analysis, and included 100 using the saturation principle. Of this cohort, 93 responses for question 3 were available. Main results: Doctors frequently reported feeling powerless, emotionally distressed, and experiencing negative feelings towards both those managing complaints and the complainants themselves. Many felt unsupported, fearful of the consequences and that the complaint was unfair. The most stressful aspects were the prolonged duration and unpredictability of procedures, managerial incompetence, poor communication and perceiving that processes are biased in favour of complainants. Many reported practising defensively or considering changing career after a complaint, and few found any positive outcomes from complaints investigations. Physicians suggested procedures should be more transparent, competently managed, time limited, and that there should be an open dialogue with complainants and policies for dealing with vexatious complaints. Some felt more support for doctors was needed. Conclusions: Complaints seriously impact on doctors’ psychological wellbeing, and are associated with defensive practise. This is not beneficial to patient care. To improve procedures, doctors propose they are simplified, time limited and more transparent.status: publishe

Age-related differences in the sonographic characteristics of endometriomas

Study Question Do sonographic characteristics of ovarian endometriomas vary with age in premenopausal women? Summary Answer With increasing age, multilocular cysts and cysts with papillations and other solid components become more common whereas ground glass echogenicity of cyst fluid becomes less common. What is Known Already Expectant or medical management of women with endometriomas is now accepted. Therefore, the accuracy of non-invasive diagnosis of these cysts is pivotal. A clinically relevant question is whether the sonographic characteristics of ovarian endometriomas are the same irrespective of the age of the woman. Study Design, Size, Duration This is a secondary analysis of cross-sectional data in the International Ovarian Tumor Analysis (IOTA) database. The database contains clinical and ultrasound information collected pre-operatively between 1999 and 2012 from 5914 patients with adnexal masses in 24 ultrasound centres in 10 countries. Participants/Materials, Setting, Methods There were 1005 histologically confirmed endometriomas in adult premenopausal patients found in the database and these were used in our analysis. The following ultrasound variables (defined using IOTA terminology) were used to describe the ultrasound appearance of the endometriomas: tender mass at ultrasound, largest diameter of lesion, tumour type (unilocular, unilocular-solid, multilocular, multilocular-solid, solid), echogenicity of cyst content, presence of papillations, number of papillations, height (mm) of largest papillation, presence and proportion of solid tissue and number of cyst locules, as well as vascularity in papillations and colour content of the tumour scan (colour score) on colour or power Doppler ultrasounds. Results are reported as median difference or odds ratio (OR) per 10 years increase in age. Main Results and the Role of Chance Maximal lesion diameter did not vary substantially with age (+1.3 mm difference per 10 years increase in age, 95% confidence interval (CI)-1.4 to 4.0). Tender mass at scan was less common in the older the woman (OR 0.75, 95% CI 0.63-0.89), as were unilocular cysts relative to multilocular cysts (OR 0.70, 95% CI 0.57-0.85) and to lesions with solid components (OR 0.61, 95% CI 0.48-0.77), and ground glass echogenicity relative to homogeneous low-level echogenicity (OR 0.74, 95% CI 0.58-0.94) and other types of echogenicity of cyst contents (OR 0.64, 95% CI 0.50-0.81). Papillations were more common the older the woman (OR 1.65, 95% CI 1.24-2.21), but their height and vascularization showed no clear relation to age. LIMITATIONS, REASONS FOR CAUTION It is a limitation that we have little clinical information on the women included, e.g. previous surgery or medical treatment for endometriosis. It is important to emphasize that we do not know the age of the endometrioma itself and that our study is not longitudinal and so does not describe changes in endometriomas over time. The differences in the ultrasound appearance of endometriomas between women of different ages might be explained by previous surgery or medical treatment and might not be an effect of age per se. Wider Implications of the Findings Awareness of physicians that the ultrasound appearance of endometriomas differs between women of different ages may facilitate a correct diagnosis of endometrioma. STUDY FUNDING/COMPETING INTEREST(S) This study was supported in part by the Regione Autonoma della Sardegna (project code CPR-24750). B.V.C., A.C. and D.T. are supported by the Fund for Scientific Research Flanders, Belgium (FWO). The authors declare that there is no conflict of interest

The Risk of Endometrial Malignancy and Other Endometrial Pathology in Women with Abnormal Uterine Bleeding:An Ultrasound-Based Model Development Study by the IETA Group

Objectives: The aim of this study was to develop a model that can discriminate between different etiologies of abnormal uterine bleeding. Design: The International Endometrial Tumor Analysis 1 study is a multicenter observational diagnostic study in 18 bleeding clinics in 9 countries. Consecutive women with abnormal vaginal bleeding presenting for ultrasound examination (n = 2,417) were recruited. The histology was obtained from endometrial sampling, D&C, hysteroscopic resection, hysterectomy, or ultrasound follow-up for >1 year. Methods: A model was developed using multinomial regression based on age, body mass index, and ultrasound predictors to distinguish between: (1) endometrial atrophy, (2) endometrial polyp or intracavitary myoma, (3) endometrial malignancy or atypical hyperplasia, (4) proliferative/secretory changes, endometritis, or hyperplasia without atypia and validated using leave-center-out cross-validation and bootstrapping. The main outcomes are the model's ability to discriminate between the four outcomes and the calibration of risk estimates. Results: The median age in 2,417 women was 50 (interquartile range 43-57). 414 (17%) women had endometrial atrophy; 996 (41%) had a polyp or myoma; 155 (6%) had an endometrial malignancy or atypical hyperplasia; and 852 (35%) had proliferative/secretory changes, endometritis, or hyperplasia without atypia. The model distinguished well between malignant and benign histology (c-statistic 0.88 95% CI: 0.85-0.91) and between all benign histologies. The probabilities for each of the four outcomes were over- or underestimated depending on the centers. Limitations: Not all patients had a diagnosis based on histology. The model over- or underestimated the risk for certain outcomes in some centers, indicating local recalibration is advisable. Conclusions: The proposed model reliably distinguishes between four histological outcomes. This is the first model to discriminate between several outcomes and is the only model applicable when menopausal status is uncertain. The model could be useful for patient management and counseling, and aid in the interpretation of ultrasound findings. Future research is needed to externally validate and locally recalibrate the model