No role for quality scores in systematic reviews of diagnostic accuracy studies

BACKGROUND: There is a lack of consensus regarding the use of quality scores in diagnostic systematic reviews. The objective of this study was to use different methods of weighting items included in a quality assessment tool for diagnostic accuracy studies (QUADAS) to produce an overall quality score, and to examine the effects of incorporating these into a systematic review. METHODS: We developed five schemes for weighting QUADAS to produce quality scores. We used three methods to investigate the effects of quality scores on test performance. We used a set of 28 studies that assessed the accuracy of ultrasound for the diagnosis of vesico-ureteral reflux in children. RESULTS: The different methods of weighting individual items from the same quality assessment tool produced different quality scores. The different scoring schemes ranked different studies in different orders; this was especially evident for the intermediate quality studies. Comparing the results of studies stratified as "high" and "low" quality based on quality scores resulted in different conclusions regarding the effects of quality on estimates of diagnostic accuracy depending on the method used to produce the quality score. A similar effect was observed when quality scores were included in meta-regression analysis as continuous variables, although the differences were less apparent. CONCLUSION: Quality scores should not be incorporated into diagnostic systematic reviews. Incorporation of the results of the quality assessment into the systematic review should involve investigation of the association of individual quality items with estimates of diagnostic accuracy, rather than using a combined quality score

Application of GRADE: Making evidence-based recommendations about diagnostic tests in clinical practice guidelines

<p>Abstract</p> <p>Background</p> <p>Accurate diagnosis is a fundamental aspect of appropriate healthcare. However, clinicians need guidance when implementing diagnostic tests given the number of tests available and resource constraints in healthcare. Practitioners of health often feel compelled to implement recommendations in guidelines, including recommendations about the use of diagnostic tests. However, the understanding about diagnostic tests by guideline panels and the methodology for developing recommendations is far from completely explored. Therefore, we evaluated the factors that guideline developers and users need to consider for the development of implementable recommendations about diagnostic tests.</p> <p>Methods</p> <p>Using a critical analysis of the process, we present the results of a case study using the Grading of Recommendations Applicability, Development and Evaluation (GRADE) approach to develop a clinical practice guideline for the diagnosis of Cow Milk Allergy with the World Allergy Organization.</p> <p>Results</p> <p>To ensure that guideline panels can develop informed recommendations about diagnostic tests, it appears that more emphasis needs to be placed on group processes, including question formulation, defining patient-important outcomes for diagnostic tests, and summarizing evidence. Explicit consideration of concepts of diagnosis from evidence-based medicine, such as pre-test probability and treatment threshold, is required to facilitate the work of a guideline panel and to formulate implementable recommendations.</p> <p>Discussion</p> <p>This case study provides useful guidance for guideline developers and clinicians about what they ought to demand from clinical practice guidelines to facilitate implementation and strengthen confidence in recommendations about diagnostic tests. Applying a structured framework like the GRADE approach with its requirement for transparency in the description of the evidence and factors that influence recommendations facilitates laying out the process and decision factors that are required for the development, interpretation, and implementation of recommendations about diagnostic tests.</p

Quality and Reporting of Diagnostic Accuracy Studies in TB, HIV and Malaria: Evaluation Using QUADAS and STARD Standards

BackgroundPoor methodological quality and reporting are known concerns with diagnostic accuracy studies. In 2003, the QUADAS tool and the STARD standards were published for evaluating the quality and improving the reporting of diagnostic studies, respectively. However, it is unclear whether these tools have been applied to diagnostic studies of infectious diseases. We performed a systematic review on the methodological and reporting quality of diagnostic studies in TB, malaria and HIV.MethodsWe identified diagnostic accuracy studies of commercial tests for TB, malaria and HIV through a systematic search of the literature using PubMed and EMBASE (2004–2006). Original studies that reported sensitivity and specificity data were included. Two reviewers independently extracted data on study characteristics and diagnostic accuracy, and used QUADAS and STARD to evaluate the quality of methods and reporting, respectively.FindingsNinety (38%) of 238 articles met inclusion criteria. All studies had design deficiencies. Study quality indicators that were met in less than 25% of the studies included adequate description of[...] and description of the team executing the test and management of indeterminate/outlier results (both 17%). The use of STARD was not explicitly mentioned in any study. Only 22% of 46 journals that published the studies included in this review required authors to use STARD

Further investigation of confirmed urinary tract infection (UTI) in children under five years: a systematic review.

Background: Further investigation of confirmed UTI in children aims to prevent renal scarring and future complications. Methods: We conducted a systematic review to determine the most effective approach to the further investigation of confirmed urinary tract infection (UTI) in children under five years of age. Results: 73 studies were included. Many studies had methodological limitations or were poorly reported. Effectiveness of further investigations: One study found that routine imaging did not lead to a reduction in recurrent UTIs or renal scarring. Diagnostic accuracy: The studies do not support the use of less invasive tests such as ultrasound as an alternative to renal scintigraphy, either to rule out infection of the upper urinary tract (LR- = 0.57, 95%CI: 0.47, 0.68) and thus to exclude patients from further investigation or to detect renal scarring (LR+ = 3.5, 95% CI: 2.5, 4.8). None of the tests investigated can accurately predict the development of renal scarring. The available evidence supports the consideration of contrast-enhanced ultrasound techniques for detecting vesico-ureteric reflux (VUR), as an alternative to micturating cystourethrography (MCUG) (LR+ = 14.1, 95% CI: 9.5, 20.8; LR- = 0.20, 95%CI: 0.13, 0.29); these techniques have the advantage of not requiring exposure to ionising radiation. Conclusion: There is no evidence to support the clinical effectiveness of routine investigation of children with confirmed UTI. Primary research on the effectiveness, in terms of improved patient outcome, of testing at all stages in the investigation of confirmed urinary tract infection is urgently required

How does study quality affect the results of a diagnostic meta-analysis?

Background: The use of systematic literature review to inform evidence based practice in diagnostics is rapidly expanding. Although the primary diagnostic literature is extensive, studies are often of low methodological quality or poorly reported. There has been no rigorously evaluated, evidence based tool to assess the methodological quality of diagnostic studies. The primary objective of this study was to determine the extent to which variations in the quality of primary studies impact the results of a diagnostic meta-analysis and whether this differs with diagnostic test type. A secondary objective was to contribute to the evaluation of QUADAS, an evidence-based tool for the assessment of quality in diagnostic accuracy studies. Methods: This study was conducted as part of large systematic review of tests used in the diagnosis and further investigation of urinary tract infection (UTI) in children. All studies included in this review were assessed using QUADAS, an evidence-based tool for the assessment of quality in systematic reviews of diagnostic accuracy studies. The impact of individual components of QUADAS on a summary measure of diagnostic accuracy was investigated using regression analysis. The review divided the diagnosis and further investigation of UTI into the following three clinical stages: diagnosis of UTI, localisation of infection, and further investigation of the UTI. Each stage used different types of diagnostic test, which were considered to involve different quality concerns. Results: Many of the studies included in our review were poorly reported. The proportion of QUADAS items fulfilled was similar for studies in different sections of the review. However, as might be expected, the individual items fulfilled differed between the three clinical stages. Regression analysis found that different items showed a strong association with test performance for the different tests evaluated. These differences were observed both within and between the three clinical stages assessed by the review. The results of regression analyses were also affected by whether or not a weighting (by sample size) was applied. Our analysis was severely limited by the completeness of reporting and the differences between the index tests evaluated and the reference standards used to confirm diagnoses in the primary studies. Few tests were evaluated by sufficient studies to allow meaningful use of meta-analytic pooling and investigation of heterogeneity. This meant that further analysis to investigate heterogeneity could only be undertaken using a subset of studies, and that the findings are open to various interpretations. Conclusion: Further work is needed to investigate the influence of methodological quality on the results of diagnostic meta-analyses. Large data sets of well-reported primary studies are needed to address this question. Without significant improvements in the completeness of reporting of primary studies, progress in this area will be limited

Quasi-Normal Modes of Stars and Black Holes

Perturbations of stars and black holes have been one of the main topics of relativistic astrophysics for the last few decades. They are of particular importance today, because of their relevance to gravitational wave astronomy. In this review we present the theory of quasi-normal modes of compact objects from both the mathematical and astrophysical points of view. The discussion includes perturbations of black holes (Schwarzschild, Reissner-Nordstr\"om, Kerr and Kerr-Newman) and relativistic stars (non-rotating and slowly-rotating). The properties of the various families of quasi-normal modes are described, and numerical techniques for calculating quasi-normal modes reviewed. The successes, as well as the limits, of perturbation theory are presented, and its role in the emerging era of numerical relativity and supercomputers is discussed.Comment: 74 pages, 7 figures, Review article for "Living Reviews in Relativity

A cost-effectiveness analysis evaluating endoscopic surveillance for gastric cancer for populations with low to intermediate risk

10.1371/journal.pone.0083959PLoS ONE812-POLN

Diagnostic value of fine-needle aspiration biopsy for breast mass: a systematic review and meta-analysis

<p>Abstract</p> <p>Background</p> <p>Fine-needle aspiration biopsy (FNAB) of the breast is a minimally invasive yet maximally diagnostic method. However, the clinical use of FNAB has been questioned. The purpose of our study was to establish the overall value of FNAC in the diagnosis of breast lesions.</p> <p>Methods</p> <p>After a review and quality assessment of 46 studies, sensitivity, specificity and other measures of accuracy of FNAB for evaluating breast lesions were pooled using random-effects models. Summary receiver operating characteristic curves were used to summarize overall accuracy. The sensitivity and specificity for the studies data (included unsatisfactory samples) and underestimation rate of unsatisfactory samples were also calculated.</p> <p>Results</p> <p>The summary estimates for FNAB in diagnosis of breast carcinoma were as follows (unsatisfactory samples was temporarily exluded): sensitivity, 0.927 (95% confidence interval [CI], 0.921 to 0.933); specificity, 0.948 (95% CI, 0.943 to 0.952); positive likelihood ratio, 25.72 (95% CI, 17.35 to 28.13); negative likelihood ratio, 0.08 (95% CI, 0.06 to 0.11); diagnostic odds ratio, 429.73 (95% CI, 241.75 to 763.87); The pooled sensitivity and specificity for 11 studies, which reported unsatisfactory samples (unsatisfactory samples was considered to be positive in this classification) were 0.920 (95% CI, 0.906 to 0.933) and 0.768 (95% CI, 0.751 to 0.784) respectively. The pooled proportion of unsatisfactory samples that were subsequently upgraded to various grade cancers was 27.5% (95% CI, 0.221 to 0.296).</p> <p>Conclusions</p> <p>FNAB is an accurate biopsy for evaluating breast malignancy if rigorous criteria are used. With regard to unsatisfactory samples, futher invasive procedures are required in order to minimize the chance of a missed diagnosis of breast cancer.</p

Extending an evidence hierarchy to include topics other than treatment: revising the Australian 'levels of evidence'

Background: In 1999 a four-level hierarchy of evidence was promoted by the National Health and Medical Research Council in Australia. The primary purpose of this hierarchy was to assist with clinical practice guideline development, although it was co-opted for use in systematic literature reviews and health technology assessments. In this hierarchy interventional study designs were ranked according to the likelihood that bias had been eliminated and thus it was not ideal to assess studies that addressed other types of clinical questions. This paper reports on the revision and extension of this evidence hierarchy to enable broader use within existing evidence assessment systems. Methods: A working party identified and assessed empirical evidence, and used a commissioned review of existing evidence assessment schema, to support decision-making regarding revision of the hierarchy. The aim was to retain the existing evidence levels I-IV but increase their relevance for assessing the quality of individual diagnostic accuracy, prognostic, aetiologic and screening studies. Comprehensive public consultation was undertaken and the revised hierarchy was piloted by individual health technology assessment agencies and clinical practice guideline developers. After two and a half years, the hierarchy was again revised and commenced a further 18 month pilot period. Results: A suitable framework was identified upon which to model the revision. Consistency was maintained in the hierarchy of "levels of evidence" across all types of clinical questions; empirical evidence was used to support the relationship between study design and ranking in the hierarchy wherever possible; and systematic reviews of lower level studies were themselves ascribed a ranking. The impact of ethics on the hierarchy of study designs was acknowledged in the framework, along with a consideration of how harms should be assessed. Conclusion: The revised evidence hierarchy is now widely used and provides a common standard against which to initially judge the likelihood of bias in individual studies evaluating interventional, diagnostic accuracy, prognostic, aetiologic or screening topics. Detailed quality appraisal of these individual studies, as well as grading of the body of evidence to answer each clinical, research or policy question, can then be undertaken as required.Tracy Merlin, Adele Weston and Rebecca Toohe