Comparison of tools for assessing the methodological quality of primary and secondary studies in health technology assessment reports in Germany

Health care policy background: Findings from scientific studies form the basis for evidence-based health policy decisions. Scientific background: Quality assessments to evaluate the credibility of study results are an essential part of health technology assessment reports and systematic reviews. Quality assessment tools (QAT) for assessing the study quality examine to what extent study results are systematically distorted by confounding or bias (internal validity). The tools can be divided into checklists, scales and component ratings. Research questions: What QAT are available to assess the quality of interventional studies or studies in the field of health economics, how do they differ from each other and what conclusions can be drawn from these results for quality assessments? Methods: A systematic search of relevant databases from 1988 onwards is done, supplemented by screening of the references, of the HTA reports of the German Agency for Health Technology Assessment (DAHTA) and an internet search. The selection of relevant literature, the data extraction and the quality assessment are carried out by two independent reviewers. The substantive elements of the QAT are extracted using a modified criteria list consisting of items and domains specific to randomized trials, observational studies, diagnostic studies, systematic reviews and health economic studies. Based on the number of covered items and domains, more and less comprehensive QAT are distinguished. In order to exchange experiences regarding problems in the practical application of tools, a workshop is hosted. Results: A total of eight systematic methodological reviews is identified as well as 147 QAT: 15 for systematic reviews, 80 for randomized trials, 30 for observational studies, 17 for diagnostic studies and 22 for health economic studies. The tools vary considerably with regard to the content, the performance and quality of operationalisation. Some tools do not only include the items of internal validity but also the items of quality of reporting and external validity. No tool covers all elements or domains. Design-specific generic tools are presented, which cover most of the content criteria. Discussion: The evaluation of QAT by using content criteria is difficult, because there is no scientific consensus on the necessary elements of internal validity, and not all of the generally accepted elements are based on empirical evidence. Comparing QAT with regard to contents neglects the operationalisation of the respective parameters, for which the quality and precision are important for transparency, replicability, the correct assessment and interrater reliability. QAT, which mix items on the quality of reporting and internal validity, should be avoided. Conclusions: There are different, design-specific tools available which can be preferred for quality assessment, because of its wider coverage of substantive elements of internal validity. To minimise the subjectivity of the assessment, tools with a detailed and precise operationalisation of the individual elements should be applied. For health economic studies, tools should be developed and complemented with instructions, which define the appropriateness of the criteria. Further research is needed to identify study characteristics that influence the internal validity of studies

Systematic evaluation of written health information on PSA based screening in Germany.

BackgroundProstate-specific antigen (PSA) based screening for early detection of prostate cancer is common although it is associated with both benefits and potential harms (e.g., the risk of overdiagnosis). Evidence-based health information could help individuals make informed decisions about whether to undergo PSA testing or not. This evaluation aimed to determine whether the written health information materials available in Germany provide appropriate information for informed decision-making on PSA based screening.MethodsA list of criteria was developed and used to systematically assess the quality of information on the benefits and harms of prostate cancer screening included in written health information materials. Fourteen information materials identified by information requests and online searches were evaluated independently by two of three reviewers. Consensus was achieved with a third reviewer.ResultsOf the 14 information materials evaluated, 10 (71%) list the ability to reduce the absolute risk of death from prostate cancer as a benefit of PSA testing, 9 (64%) point out the risks of follow-up diagnostics, 13 (93%) describe the risks of the available prostate cancer treatments, and all 14 specify the risk of overdiagnosis. The minority provide numerical data on benefits and risks. Partially mismatched framing was identified in four cases: two information materials report only the relative frequencies of benefits, and two report only the absolute frequencies of harms. Half of the materials encouraged participation using downplaying or frightening language.ConclusionsThe majority of health information materials in Germany describe the benefits and harms of PSA based screening, including overdiagnosis, but often lack adequate balance, neutrality and numbers

Reported harms of screening colonoscopy due to colonoscopy preparation, sedation and/or the procedure itself in the identified leaflets (n = 28) and booklets (n = 13).

<p>Reported harms of screening colonoscopy due to colonoscopy preparation, sedation and/or the procedure itself in the identified leaflets (n = 28) and booklets (n = 13).</p

Information on baseline risks.

<p>Information on baseline risks.</p

Reported benefits of CRC screening in general and for colonoscopy and the FOBT in particular in the identified leaflets (n = 28) and booklets (n = 13).

<p>FOBT: faecal occult blood test, guaiac-based.</p><p>CRC: colorectal cancer.</p><p>Reported benefits of CRC screening in general and for colonoscopy and the FOBT in particular in the identified leaflets (n = 28) and booklets (n = 13).</p

Reported accuracy of CRC screening tests in leaflets (n = 28) and booklets (n = 13).

<p>Reported accuracy of CRC screening tests in leaflets (n = 28) and booklets (n = 13).</p

Aggregated results for reported benefits and harms, stratified by leaflets and booklets.

<p>The figure indicates whether a leaflet contains <u>any</u> information on the benefits of CRC screening a) in general or specifically for b) colonoscopy or c) the FOBT, and whether it contains any information on the d) general and e) specific harms of colonoscopy. To be rated positive for harms, it was not sufficient if the only information about possible harms referred to pain, stating incorrectly that there is no pain involved.</p