19 research outputs found
Erratum to: Methods for evaluating medical tests and biomarkers
[This corrects the article DOI: 10.1186/s41512-016-0001-y.]
Erratum to: Methods for evaluating medical tests and biomarkers
[This corrects the article DOI: 10.1186/s41512-016-0001-y.]
Evidence synthesis to inform model-based cost-effectiveness evaluations of diagnostic tests: a methodological systematic review of health technology assessments
Background: Evaluations of diagnostic tests are challenging because of the indirect nature of their impact on patient outcomes. Model-based health economic evaluations of tests allow different types of evidence from various sources to be incorporated and enable cost-effectiveness estimates to be made beyond the duration of available study data. To parameterize a health-economic model fully, all the ways a test impacts on patient health must be quantified, including but not limited to diagnostic test accuracy. Methods: We assessed all UK NIHR HTA reports published May 2009-July 2015. Reports were included if they evaluated a diagnostic test, included a model-based health economic evaluation and included a systematic review and meta-analysis of test accuracy. From each eligible report we extracted information on the following topics: 1) what evidence aside from test accuracy was searched for and synthesised, 2) which methods were used to synthesise test accuracy evidence and how did the results inform the economic model, 3) how/whether threshold effects were explored, 4) how the potential dependency between multiple tests in a pathway was accounted for, and 5) for evaluations of tests targeted at the primary care setting, how evidence from differing healthcare settings was incorporated. Results: The bivariate or HSROC model was implemented in 20/22 reports that met all inclusion criteria. Test accuracy data for health economic modelling was obtained from meta-analyses completely in four reports, partially in fourteen reports and not at all in four reports. Only 2/7 reports that used a quantitative test gave clear threshold recommendations. All 22 reports explored the effect of uncertainty in accuracy parameters but most of those that used multiple tests did not allow for dependence between test results. 7/22 tests were potentially suitable for primary care but the majority found limited evidence on test accuracy in primary care settings. Conclusions: The uptake of appropriate meta-analysis methods for synthesising evidence on diagnostic test accuracy in UK NIHR HTAs has improved in recent years. Future research should focus on other evidence requirements for cost-effectiveness assessment, threshold effects for quantitative tests and the impact of multiple diagnostic tests
Evaluating the potential negative effects of school-based prevention programs aiming to reduce alcohol and drug misuse in adolescents: A systematic review of research articles prior to 2013
Issues: Reviews of alcohol and drug prevention programs commonly focus on positive effects, whilst disregarding possible iatrogenic effects. Our objective was to summarize evidence on iatrogenic effects of alcohol and drug prevention programs targeting adolescents. We systematically investigated the nature of these effects, the number of iatrogenic effects reported on, the sort of impact they have (e.g. in terms of severity) and the type of programs associated with iatrogenic effects.
Approach: On January 2013, we searched the Cochrane Central Register of Controlled Trials, Medline, PsycINFO, Web of Science, Eric, Scirus and we actively searched for grey literature via Google Scholar and OpenGrey. Included were (quasi) RCTs that evaluated the effectiveness of school-based alcohol and drug prevention programs for adolescents. Ninety-three articles were eligible for inclusion and were screened for potential iatrogenic effects.
Key findings: Ten articles reported on iatrogenic effects, which were predominantly found on substance use outcomes. The quality of these articles was assessed and a meta-analysis was not possible due to the heterogeneity in interventions and outcome measures. An increase in substance use was found in 5 out of 7 RCTs and in 1 quasi RCT. The magnitude of the iatrogenic effects found was not always clear due to the outcome measures used.
Implications: Iatrogenic effects were assessed in a limited number of studies. In future studies on prevention programs the number, nature and impact of iatrogenic effects should standardly be assessed and reported. It allows us to detect potential problem areas in the conceptualization of prevention programs.nrpages: 33status: publishe
Regression shrinkage methods for clinical prediction models do not guarantee improved performance: Simulation study
When developing risk prediction models on datasets with limited sample size, shrinkage methods are recommended. Earlier studies showed that shrinkage results in better predictive performance on average. This simulation study aimed to investigate the variability of regression shrinkage on predictive performance for a binary outcome. We compared standard maximum likelihood with the following shrinkage methods: uniform shrinkage (likelihood-based and bootstrap-based), penalized maximum likelihood (ridge) methods, LASSO logistic regression, adaptive LASSO, and Firth's correction. In the simulation study, we varied the number of predictors and their strength, the correlation between predictors, the event rate of the outcome, and the events per variable. In terms of results, we focused on the calibration slope. The slope indicates whether risk predictions are too extreme (slope 1). The results can be summarized into three main findings. First, shrinkage improved calibration slopes on average. Second, the between-sample variability of calibration slopes was often increased relative to maximum likelihood. In contrast to other shrinkage approaches, Firth's correction had a small shrinkage effect but showed low variability. Third, the correlation between the estimated shrinkage and the optimal shrinkage to remove overfitting was typically negative, with Firth's correction as the exception. We conclude that, despite improved performance on average, shrinkage often worked poorly in individual datasets, in particular when it was most needed. The results imply that shrinkage methods do not solve problems associated with small sample size or low number of events per variable.status: publishe
The use of IETA terminology and other ultrasound features in the diagnosis of intracavitary fibroids and polyps
ObjectivesThe primary aim is to evaluate interobserver agreement on International Endometrial Tumour Analysis (IETA) terminology and other ultrasound (US) features (“bridge sign”, lesion shape and "fan‐shaped shadowing") of polyps and fibroids. The secondary aim is to assess the relevance of these US features in the diagnosis of polyps and fibroids.MethodsFrom Leuven University Hospital Bleeding Clinic we selected stored 3D‐volumes of 113 non‐consecutive patients with a final pathologic diagnosis of fibroid or polyp. 3D‐volumes of unenhanced transvaginal US examinations on grayscale (GS) and with Doppler signal (DS) as well as fluid instillation sonography (FIS) in GS and adding DS (FIS&DS) were converted in video clips. Two experienced sonographers, blinded to pathology results, scored the clips for the presence of the US features and proposed a diagnosis. Interobserver agreement was determined using Cohen's kappa coefficient (k).ResultsThere were 71 polyps and 42 fibroids. Interobserver agreement was good for “circular flow”, both on DS (k = 0.66) and on FIS&DS (k = 0.62), moderate for the “bridge sign” on GS (k = 0.50), lesion shape on FIS (k = 0.44), “single dominant vessel” on FIS&DS (k = 0.57), “fan‐shaped shadowing” on FIS (k = 0.57) and for “multiple vessels with focal origin” on FIS&DS (k = 0.60). In polyps, sonographers reported an oval shape on FIS in 76‐77% (percentage for sonographer 1 and 2), a “single dominant vessel” on FIS&DS in 32‐37% and “multiple vessels with focal origin” on FIS&DS in 15‐20%. In fibroids, the sonographers reported the “bridge sign” on GS in 36‐45%, “fan‐shaped shadowing” on FIS in 38‐52% and “circular flow” on DS in 29‐52% and on FIS&DS in 38‐57%. Examiners agreed on the diagnosis in 82% of the cases (k = 0.69); 63% were polyps.ConclusionsInterobserver agreement for some of the US features is good to moderate. These may be selected in the differential diagnosis between polyps and intracavitary fibroids. Overall there was a good agreement on the diagnosis between both examiners.info:eu-repo/semantics/publishe
A calibration hierarchy for risk models was defined: from utopia to empirical data
OBJECTIVE: Calibrated risk models are vital for valid decision support. We define four levels of calibration and describe implications for model development and external validation of predictions. STUDY DESIGN AND SETTING: We present results based on simulated data sets. RESULTS: A common definition of calibration is "having an event rate of R% among patients with a predicted risk of R%," which we refer to as "moderate calibration." Weaker forms of calibration only require the average predicted risk (mean calibration) or the average prediction effects (weak calibration) to be correct. "Strong calibration" requires that the event rate equals the predicted risk for every covariate pattern. This implies that the model is fully correct for the validation setting. We argue that this is unrealistic: the model type may be incorrect, the linear predictor is only asymptotically unbiased, and all nonlinear and interaction effects should be correctly modeled. In addition, we prove that moderate calibration guarantees nonharmful decision making. Finally, results indicate that a flexible assessment of calibration in small validation data sets is problematic. CONCLUSION: Strong calibration is desirable for individualized decision support but unrealistic and counter productive by stimulating the development of overly complex models. Model development and external validation should focus on moderate calibration.publisher: Elsevier
articletitle: A calibration hierarchy for risk models was defined: from utopia to empirical data
journaltitle: Journal of Clinical Epidemiology
articlelink: http://dx.doi.org/10.1016/j.jclinepi.2015.12.005
content_type: article
copyright: © 2016 Elsevier Inc. All rights reserved.status: publishe
ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models
OBJECTIVES: Receiver operating characteristic (ROC) curves show how well a risk prediction model discriminates between patients with and without a condition. We aim to investigate how ROC curves are presented in the literature and discuss and illustrate their potential limitations. STUDY DESIGN AND SETTING: We conducted a pragmatic literature review of contemporary publications that externally validated clinical prediction models. We illustrated limitations of ROC curves using a testicular cancer case study and simulated data. RESULTS: Of 86 identified prediction modeling studies, 52 (60%) presented ROC curves without thresholds and one (1%) presented an ROC curve with only a few thresholds. We illustrate that ROC curves in their standard form withhold threshold information have an unstable shape even for the same area under the curve (AUC) and are problematic for comparing model performance conditional on threshold. We compare ROC curves with classification plots, which show sensitivity and specificity conditional on risk thresholds. CONCLUSION: ROC curves do not offer more information than the AUC to indicate discriminative ability. To assess the model's performance for decision-making, results should be provided conditional on risk thresholds. Therefore, if discriminatory ability must be visualized, classification plots are attractive.status: publishe
Doctors' experiences and their perception of the most stressful aspects of complaints processes in the UK: an analysis of qualitative survey data
Objectives: To examine doctors’ experiences of complaints, including which aspects are most stressful. We also investigated how doctors felt complaints processes could be improved.
Design and methods: A qualitative study based on a cross-sectional survey of members of the British Medical Association (BMA). We asked the following:
(1) Try to summarise as best as you can your experience of the complaints process and how it made you feel. (2) What were the most stressful aspects of the complaint? (3) What would you improve in the complaints system?
Participants: We sent the survey to 95 636 doctors, and received 10 930 (11.4%) responses. Of these, 6146 had a previous, recent or current complaint and 3417 (31.3%) of these respondents answered questions 1 and 2. We randomly selected 1000 answers for analysis, and included 100 using the saturation principle. Of this cohort, 93 responses for
question 3 were available.
Main results: Doctors frequently reported feeling powerless, emotionally distressed, and experiencing negative feelings towards both those managing complaints and the complainants themselves. Many felt unsupported, fearful of the consequences and that the complaint was unfair. The most stressful aspects were the prolonged duration and unpredictability of procedures, managerial incompetence, poor communication and perceiving that processes are biased in favour of complainants. Many reported practising defensively or considering changing career after a complaint, and few found any positive
outcomes from complaints investigations. Physicians suggested procedures should be more transparent, competently managed, time limited, and that there should be an open dialogue with complainants and policies for dealing with vexatious complaints. Some felt more support for doctors was needed.
Conclusions: Complaints seriously impact on doctors’ psychological wellbeing, and are associated with defensive practise. This is not beneficial to patient care. To improve procedures, doctors propose they are simplified, time limited and more transparent.status: publishe
Age-related differences in the sonographic characteristics of endometriomas
Study Question Do sonographic characteristics of ovarian endometriomas vary with age in premenopausal women? Summary Answer With increasing age, multilocular cysts and cysts with papillations and other solid components become more common whereas ground glass echogenicity of cyst fluid becomes less common. What is Known Already Expectant or medical management of women with endometriomas is now accepted. Therefore, the accuracy of non-invasive diagnosis of these cysts is pivotal. A clinically relevant question is whether the sonographic characteristics of ovarian endometriomas are the same irrespective of the age of the woman. Study Design, Size, Duration This is a secondary analysis of cross-sectional data in the International Ovarian Tumor Analysis (IOTA) database. The database contains clinical and ultrasound information collected pre-operatively between 1999 and 2012 from 5914 patients with adnexal masses in 24 ultrasound centres in 10 countries. Participants/Materials, Setting, Methods There were 1005 histologically confirmed endometriomas in adult premenopausal patients found in the database and these were used in our analysis. The following ultrasound variables (defined using IOTA terminology) were used to describe the ultrasound appearance of the endometriomas: tender mass at ultrasound, largest diameter of lesion, tumour type (unilocular, unilocular-solid, multilocular, multilocular-solid, solid), echogenicity of cyst content, presence of papillations, number of papillations, height (mm) of largest papillation, presence and proportion of solid tissue and number of cyst locules, as well as vascularity in papillations and colour content of the tumour scan (colour score) on colour or power Doppler ultrasounds. Results are reported as median difference or odds ratio (OR) per 10 years increase in age. Main Results and the Role of Chance Maximal lesion diameter did not vary substantially with age (+1.3 mm difference per 10 years increase in age, 95% confidence interval (CI)-1.4 to 4.0). Tender mass at scan was less common in the older the woman (OR 0.75, 95% CI 0.63-0.89), as were unilocular cysts relative to multilocular cysts (OR 0.70, 95% CI 0.57-0.85) and to lesions with solid components (OR 0.61, 95% CI 0.48-0.77), and ground glass echogenicity relative to homogeneous low-level echogenicity (OR 0.74, 95% CI 0.58-0.94) and other types of echogenicity of cyst contents (OR 0.64, 95% CI 0.50-0.81). Papillations were more common the older the woman (OR 1.65, 95% CI 1.24-2.21), but their height and vascularization showed no clear relation to age. LIMITATIONS, REASONS FOR CAUTION It is a limitation that we have little clinical information on the women included, e.g. previous surgery or medical treatment for endometriosis. It is important to emphasize that we do not know the age of the endometrioma itself and that our study is not longitudinal and so does not describe changes in endometriomas over time. The differences in the ultrasound appearance of endometriomas between women of different ages might be explained by previous surgery or medical treatment and might not be an effect of age per se. Wider Implications of the Findings Awareness of physicians that the ultrasound appearance of endometriomas differs between women of different ages may facilitate a correct diagnosis of endometrioma. STUDY FUNDING/COMPETING INTEREST(S) This study was supported in part by the Regione Autonoma della Sardegna (project code CPR-24750). B.V.C., A.C. and D.T. are supported by the Fund for Scientific Research Flanders, Belgium (FWO). The authors declare that there is no conflict of interest