Health technology assessment of diagnostic tests: a state of the art review of methods guidance from international organizations

This is the final version. Available on open access from Cambridge University Press via the DOI in this recordOBJECTIVES: To identify which international health technology assessment (HTA) agencies are undertaking evaluations of medical tests, summarize commonalities and differences in methodological approach, and highlight examples of good practice. METHODS: A methodological review incorporating: systematic identification of HTA guidance documents mentioning evaluation of tests; identification of key contributing organizations and abstraction of approaches to all essential HTA steps; summary of similarities and differences between organizations; and identification of important emergent themes which define the current state of the art and frontiers where further development is needed. RESULTS: Seven key organizations were identified from 216 screened. The main themes were: elucidation of claims of test benefits; attitude to direct and indirect evidence of clinical effectiveness (including evidence linkage); searching; quality assessment; and health economic evaluation. With the exception of dealing with test accuracy data, approaches were largely based on general approaches to HTA with few test-specific modifications. Elucidation of test claims and attitude to direct and indirect evidence are where we identified the biggest dissimilarities in approach. CONCLUSIONS: There is consensus on some aspects of HTA of tests, such as dealing with test accuracy, and examples of good practice which HTA organizations new to test evaluation can emulate. The focus on test accuracy contrasts with universal acknowledgment that it is not a sufficient evidence base for test evaluation. There are frontiers where methodological development is urgently required, notably integrating direct and indirect evidence and standardizing approaches to evidence linkage.Medical Research Council (MRC

Ultrasound, CT, MRI, or PET-CT for staging and re-staging of adults with cutaneous melanoma.

BACKGROUND: Melanoma is one of the most aggressive forms of skin cancer, with the potential to metastasise to other parts of the body via the lymphatic system and the bloodstream. Melanoma accounts for a small percentage of skin cancer cases but is responsible for the majority of skin cancer deaths. Various imaging tests can be used with the aim of detecting metastatic spread of disease following a primary diagnosis of melanoma (primary staging) or on clinical suspicion of disease recurrence (re-staging). Accurate staging is crucial to ensuring that patients are directed to the most appropriate and effective treatment at different points on the clinical pathway. Establishing the comparative accuracy of ultrasound, computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET)-CT imaging for detection of nodal or distant metastases, or both, is critical to understanding if, how, and where on the pathway these tests might be used. OBJECTIVES: Primary objectivesWe estimated accuracy separately according to the point in the clinical pathway at which imaging tests were used. Our objectives were:• to determine the diagnostic accuracy of ultrasound or PET-CT for detection of nodal metastases before sentinel lymph node biopsy in adults with confirmed cutaneous invasive melanoma; and• to determine the diagnostic accuracy of ultrasound, CT, MRI, or PET-CT for whole body imaging in adults with cutaneous invasive melanoma:○ for detection of any metastasis in adults with a primary diagnosis of melanoma (i.e. primary staging at presentation); and○ for detection of any metastasis in adults undergoing staging of recurrence of melanoma (i.e. re-staging prompted by findings on routine follow-up).We undertook separate analyses according to whether accuracy data were reported per patient or per lesion.Secondary objectivesWe sought to determine the diagnostic accuracy of ultrasound, CT, MRI, or PET-CT for whole body imaging (detection of any metastasis) in mixed or not clearly described populations of adults with cutaneous invasive melanoma.For study participants undergoing primary staging or re-staging (for possible recurrence), and for mixed or unclear populations, our objectives were:• to determine the diagnostic accuracy of ultrasound, CT, MRI, or PET-CT for detection of nodal metastases;• to determine the diagnostic accuracy of ultrasound, CT, MRI, or PET-CT for detection of distant metastases; and• to determine the diagnostic accuracy of ultrasound, CT, MRI, or PET-CT for detection of distant metastases according to metastatic site. SEARCH METHODS: We undertook a comprehensive search of the following databases from inception up to August 2016: Cochrane Central Register of Controlled Trials; MEDLINE; Embase; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We studied reference lists as well as published systematic review articles. SELECTION CRITERIA: We included studies of any design that evaluated ultrasound (with or without the use of fine needle aspiration cytology (FNAC)), CT, MRI, or PET-CT for staging of cutaneous melanoma in adults, compared with a reference standard of histological confirmation or imaging with clinical follow-up of at least three months' duration. We excluded studies reporting multiple applications of the same test in more than 10% of study participants. DATA COLLECTION AND ANALYSIS: Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2)). We estimated accuracy using the bivariate hierarchical method to produce summary sensitivities and specificities with 95% confidence and prediction regions. We undertook analysis of studies allowing direct and indirect comparison between tests. We examined heterogeneity between studies by visually inspecting the forest plots of sensitivity and specificity and summary receiver operating characteristic (ROC) plots. Numbers of identified studies were insufficient to allow formal investigation of potential sources of heterogeneity. MAIN RESULTS: We included a total of 39 publications reporting on 5204 study participants; 34 studies reporting data per patient included 4980 study participants with 1265 cases of metastatic disease, and seven studies reporting data per lesion included 417 study participants with 1846 potentially metastatic lesions, 1061 of which were confirmed metastases. The risk of bias was low or unclear for all domains apart from participant flow. Concerns regarding applicability of the evidence were high or unclear for almost all domains. Participant selection from mixed or not clearly defined populations and poorly described application and interpretation of index tests were particularly problematic.The accuracy of imaging for detection of regional nodal metastases before sentinel lymph node biopsy (SLNB) was evaluated in 18 studies. In 11 studies (2614 participants; 542 cases), the summary sensitivity of ultrasound alone was 35.4% (95% confidence interval (CI) 17.0% to 59.4%) and specificity was 93.9% (95% CI 86.1% to 97.5%). Combining pre-SLNB ultrasound with FNAC revealed summary sensitivity of 18.0% (95% CI 3.58% to 56.5%) and specificity of 99.8% (95% CI 99.1% to 99.9%) (1164 participants; 259 cases). Four studies demonstrated lower sensitivity (10.2%, 95% CI 4.31% to 22.3%) and specificity (96.5%,95% CI 87.1% to 99.1%) for PET-CT before SLNB (170 participants, 49 cases). When these data are translated to a hypothetical cohort of 1000 people eligible for SLNB, 237 of whom have nodal metastases (median prevalence), the combination of ultrasound with FNAC potentially allows 43 people with nodal metastases to be triaged directly to adjuvant therapy rather than having SLNB first, at a cost of two people with false positive results (who are incorrectly managed). Those with a false negative ultrasound will be identified on subsequent SLNB.Limited test accuracy data were available for whole body imaging via PET-CT for primary staging or re-staging for disease recurrence, and none evaluated MRI. Twenty-four studies evaluated whole body imaging. Six of these studies explored primary staging following a confirmed diagnosis of melanoma (492 participants), three evaluated re-staging of disease following some clinical indication of recurrence (589 participants), and 15 included mixed or not clearly described population groups comprising participants at a number of different points on the clinical pathway and at varying stages of disease (1265 participants). Results for whole body imaging could not be translated to a hypothetical cohort of people due to paucity of data.Most of the studies (6/9) of primary disease or re-staging of disease considered PET-CT, two in comparison to CT alone, and three studies examined the use of ultrasound. No eligible evaluations of MRI in these groups were identified. All studies used histological reference standards combined with follow-up, and two included FNAC for some participants. Observed accuracy for detection of any metastases for PET-CT was higher for re-staging of disease (summary sensitivity from two studies: 92.6%, 95% CI 85.3% to 96.4%; specificity: 89.7%, 95% CI 78.8% to 95.3%; 153 participants; 95 cases) compared to primary staging (sensitivities from individual studies ranged from 30% to 47% and specificities from 73% to 88%), and was more sensitive than CT alone in both population groups, but participant numbers were very small.No conclusions can be drawn regarding routine imaging of the brain via MRI or CT. AUTHORS' CONCLUSIONS: Review authors found a disappointing lack of evidence on the accuracy of imaging in people with a diagnosis of melanoma at different points on the clinical pathway. Studies were small and often reported data according to the number of lesions rather than the number of study participants. Imaging with ultrasound combined with FNAC before SLNB may identify around one-fifth of those with nodal disease, but confidence intervals are wide and further work is needed to establish cost-effectiveness. Much of the evidence for whole body imaging for primary staging or re-staging of disease is focused on PET-CT, and comparative data with CT or MRI are lacking. Future studies should go beyond diagnostic accuracy and consider the effects of different imaging tests on disease management. The increasing availability of adjuvant therapies for people with melanoma at high risk of disease spread at presentation will have a considerable impact on imaging services, yet evidence for the relative diagnostic accuracy of available tests is limited

Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines

The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77–94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines

At what times during infection is SARS-CoV-2 detectable and no longer detectable using RT-PCR based tests?: A systematic review of individual participant data

Background Tests for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) viral ribonucleic acid (RNA) using reverse transcription polymerase chain reaction (RT-PCR) are pivotal to detecting current coronavirus disease (COVID-19) and duration of detectable virus indicating potential for infectivity. Methods We conducted an individual participant data (IPD) systematic review of longitudinal studies of RT-PCR test results in symptomatic SARS-CoV-2. We searched PubMed, LitCOVID, medRxiv, and COVID-19 Living Evidence databases. We assessed risk of bias using a QUADAS-2 adaptation. Outcomes were the percentage of positive test results by time and the duration of detectable virus, by anatomical sampling sites. Results Of 5078 studies screened, we included 32 studies with 1023 SARS-CoV-2 infected participants and 1619 test results, from − 6 to 66 days post-symptom onset and hospitalisation. The highest percentage virus detection was from nasopharyngeal sampling between 0 and 4 days post-symptom onset at 89% (95% confidence interval (CI) 83 to 93) dropping to 54% (95% CI 47 to 61) after 10 to 14 days. On average, duration of detectable virus was longer with lower respiratory tract (LRT) sampling than upper respiratory tract (URT). Duration of faecal and respiratory tract virus detection varied greatly within individual participants. In some participants, virus was still detectable at 46 days post-symptom onset. Conclusions RT-PCR misses detection of people with SARS-CoV-2 infection; early sampling minimises false negative diagnoses. Beyond 10 days post-symptom onset, lower RT or faecal testing may be preferred sampling sites. The included studies are open to substantial risk of bias, so the positivity rates are probably overestimated

The science of clinical practice: disease diagnosis or patient prognosis? Evidence about "what is likely to happen" should shape clinical practice.

BACKGROUND: Diagnosis is the traditional basis for decision-making in clinical practice. Evidence is often lacking about future benefits and harms of these decisions for patients diagnosed with and without disease. We propose that a model of clinical practice focused on patient prognosis and predicting the likelihood of future outcomes may be more useful. DISCUSSION: Disease diagnosis can provide crucial information for clinical decisions that influence outcome in serious acute illness. However, the central role of diagnosis in clinical practice is challenged by evidence that it does not always benefit patients and that factors other than disease are important in determining patient outcome. The concept of disease as a dichotomous 'yes' or 'no' is challenged by the frequent use of diagnostic indicators with continuous distributions, such as blood sugar, which are better understood as contributing information about the probability of a patient's future outcome. Moreover, many illnesses, such as chronic fatigue, cannot usefully be labelled from a disease-diagnosis perspective. In such cases, a prognostic model provides an alternative framework for clinical practice that extends beyond disease and diagnosis and incorporates a wide range of information to predict future patient outcomes and to guide decisions to improve them. Such information embraces non-disease factors and genetic and other biomarkers which influence outcome. SUMMARY: Patient prognosis can provide the framework for modern clinical practice to integrate information from the expanding biological, social, and clinical database for more effective and efficient care

Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension

The SPIRIT 2013 statement aims to improve the completeness of clinical trial protocol reporting by providing evidence-based recommendations for the minimum set of items to be addressed. This guidance has been instrumental in promoting transparent evaluation of new interventions. More recently, there has been a growing recognition that interventions involving artificial intelligence (AI) need to undergo rigorous, prospective evaluation to demonstrate their impact on health outcomes. The SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials-Artificial Intelligence) extension is a new reporting guideline for clinical trial protocols evaluating interventions with an AI component. It was developed in parallel with its companion statement for trial reports: CONSORT-AI (Consolidated Standards of Reporting Trials-Artificial Intelligence). Both guidelines were developed through a staged consensus process involving literature review and expert consultation to generate 26 candidate items, which were consulted upon by an international multi-stakeholder group in a two-stage Delphi survey (103 stakeholders), agreed upon in a consensus meeting (31 stakeholders) and refined through a checklist pilot (34 participants). The SPIRIT-AI extension includes 15 new items that were considered sufficiently important for clinical trial protocols of AI interventions. These new items should be routinely reported in addition to the core SPIRIT 2013 items. SPIRIT-AI recommends that investigators provide clear descriptions of the AI intervention, including instructions and skills required for use, the setting in which the AI intervention will be integrated, considerations for the handling of input and output data, the human–AI interaction and analysis of error cases. SPIRIT-AI will help promote transparency and completeness for clinical trial protocols for AI interventions. Its use will assist editors and peer reviewers, as well as the general readership, to understand, interpret, and critically appraise the design and risk of bias for a planned clinical trial

Erratum to: Methods for evaluating medical tests and biomarkers

[This corrects the article DOI: 10.1186/s41512-016-0001-y.]

Evidence synthesis to inform model-based cost-effectiveness evaluations of diagnostic tests: a methodological systematic review of health technology assessments

Background: Evaluations of diagnostic tests are challenging because of the indirect nature of their impact on patient outcomes. Model-based health economic evaluations of tests allow different types of evidence from various sources to be incorporated and enable cost-effectiveness estimates to be made beyond the duration of available study data. To parameterize a health-economic model fully, all the ways a test impacts on patient health must be quantified, including but not limited to diagnostic test accuracy. Methods: We assessed all UK NIHR HTA reports published May 2009-July 2015. Reports were included if they evaluated a diagnostic test, included a model-based health economic evaluation and included a systematic review and meta-analysis of test accuracy. From each eligible report we extracted information on the following topics: 1) what evidence aside from test accuracy was searched for and synthesised, 2) which methods were used to synthesise test accuracy evidence and how did the results inform the economic model, 3) how/whether threshold effects were explored, 4) how the potential dependency between multiple tests in a pathway was accounted for, and 5) for evaluations of tests targeted at the primary care setting, how evidence from differing healthcare settings was incorporated. Results: The bivariate or HSROC model was implemented in 20/22 reports that met all inclusion criteria. Test accuracy data for health economic modelling was obtained from meta-analyses completely in four reports, partially in fourteen reports and not at all in four reports. Only 2/7 reports that used a quantitative test gave clear threshold recommendations. All 22 reports explored the effect of uncertainty in accuracy parameters but most of those that used multiple tests did not allow for dependence between test results. 7/22 tests were potentially suitable for primary care but the majority found limited evidence on test accuracy in primary care settings. Conclusions: The uptake of appropriate meta-analysis methods for synthesising evidence on diagnostic test accuracy in UK NIHR HTAs has improved in recent years. Future research should focus on other evidence requirements for cost-effectiveness assessment, threshold effects for quantitative tests and the impact of multiple diagnostic tests

Erratum to: Methods for evaluating medical tests and biomarkers

[This corrects the article DOI: 10.1186/s41512-016-0001-y.]