Using randomised controlled trials to evaluate the clinical effectiveness of diagnostic tests: how useful are test-treatment RCTs?

Background: Decisions on which tests to use should be informed by evidence that they do more good than harm. Test-treatment RCTs are recommended as the ‘gold–standard’ approach, but have attracted criticism that question whether they are fit for purpose. Confronting this question, the thesis investigates four key challenges by finding and analysing all identifiable test-treatment RCTs (2004–2007). Methods: Capture–recapture analysis estimated the total population of trials; descriptive analysis characterised the diagnostic questions evaluated by RCT; reviews of reporting and methodological quality investigated how informative and valid trials are; analytic induction was used to develop a theoretical framework linking tests to health outcomes, from which a tool was designed. Results: Published trials were poor quality, and found to be highly complex studies that will be challenging to evaluate reliably: interventions are difficult to capture and translate into protocols; several methods traditionally used to eliminate bias are more difficult to implement; test-treatment strategies impact on patient health in numerous and highly complicated ways. Conclusion: Test-treatment trials have the potential to be very useful instruments, and though highly challenging they could be both reliable and informative. However, it must be acknowledged that trials will not be suited to all comparisons

Test-treatment RCTs are susceptible to bias:a review of the methodological quality of randomized trials that evaluate diagnostic tests

Abstract Background There is a growing recognition for the need to expand our evidence base for the clinical effectiveness of diagnostic tests. Many international bodies are calling for diagnostic randomized controlled trials to provide the most rigorous evidence of impact to patient health. Although these so-called test-treatment RCTs are very challenging to undertake due to their methodological complexity, they have not been subjected to a systematic appraisal of their methodological quality. The extent to which these trials may be producing biased results therefore remains unknown. We set out to address this issue by conducting a methodological review of published test-treatment trials to determine how often they implement adequate methods to limit bias and safeguard the validity of results. Methods We ascertained all test-treatment RCTs published 2004–2007, indexed in CENTRAL, including RCTs which randomized patients to diagnostic tests and measured patient outcomes after treatment. Tests used for screening, monitoring or prognosis were excluded. We assessed adequacy of sequence generation, allocation concealment and intention-to-treat, appropriateness of primary analyses, blinding and reporting of power calculations, and extracted study characteristics including the primary outcome. Results One hundred three trials compared 105 control with 119 experimental interventions, and reported 150 primary outcomes. Randomization and allocation concealment were adequate in 57 and 37% of trials. Blinding was uncommon (patients 5%, clinicians 4%, outcome assessors 21%), as was an adequate intention-to-treat analysis (29%). Overall 101 of 103 trials (98%) were at risk of bias, as judged using standard Cochrane criteria. Conclusion Test-treatment trials are particularly susceptible to attrition and inadequate primary analyses, lack of blinding and under-powering. These weaknesses pose much greater methodological and practical challenges to conducting reliable RCT evaluations of test-treatment strategies than standard treatment interventions. We suggest a cautious approach that first examines whether a test-treatment intervention can accommodate the methodological safeguards necessary to minimize bias, and highlight that test-treatment RCTs require different methods to ensure reliability than standard treatment trials. Please see the companion paper to this article: http://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-016-0286-0

Exfoliative cytology for diagnosing basal cell carcinoma and other skin cancers in adults

Background: Early accurate detection of all skin cancer types is essential to guide appropriate management and to reduce morbidity and improve survival. Basal cell carcinoma (BCC) is usually localised to the skin with potential to infiltrate and damage surrounding tissue, while cutaneous squamous cell carcinoma (cSCC) and melanoma have a much higher potential to metastasise and ultimately lead to death. Exfoliative cytology is a non–invasive test that uses the Tzanck smear technique to identify disease by examining the structure of cells obtained from scraped samples. This simple procedure is a less invasive diagnostic test than a skin biopsy, and for BCC has the potential to provide an immediate diagnosis that avoids an additional visit to clinic to receive skin biopsy results. This may benefit patients scheduled for either Mohs micrographic surgery or non–surgical treatments such as radiotherapy. A cytology scrape can never give the same information as a skin biopsy, however, so it is important to know more about which skin cancer situations it may be helpful.Objectives: The primary objective was to determine the diagnostic accuracy of exfoliative cytology for the detection of basal cell carcinoma (BCC) in adults. Secondary objectives were to determine diagnostic accuracy for the detection of i) cutaneous squamous cell carcinoma, ii) invasive melanoma and atypical intraepidermal melanocytic variants, and iii) any skin cancer, including keratinocyte skin cancer, invasive melanoma and atypical intraepidermal melanocytic variants, or any other skin cancer.Search methods: We undertook a comprehensive search of the following databases from inception up to August 2016: Cochrane Central Register of Controlled Trials; MEDLINE; EMBASE; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We also studied the reference lists of published systematic review articles.Selection criteria: Studies evaluating exfoliative cytology in adults with lesions suspicious for BCC, cSCC or melanoma, compared with a reference standard of histological confirmation.Data collection and analysis: Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS-2). Where possible we estimated summary sensitivities and specificities using the bivariate hierarchical model.Main results: This review reports on nine studies with a total of 1655 lesions including 1120 BCCs (14 datasets), 401 lesions with 44 cSCCs (two datasets), and 200 lesions with 10 melanomas (one dataset). Three of these datasets (one each for BCC, melanoma, and any malignant condition) were derived from one study which also performed a direct comparison with dermoscopy. Studies were of moderate to poor quality providing inadequate descriptions of participant selection, thresholds used to make cytological and histological diagnoses, and blinding. Reporting of patients’ prior referral pathways was particularly poor, as were descriptions of the cytodiagnostic criteria used to make diagnoses. No studies evaluated the use of exfoliative cytology as a primary diagnostic test for detecting BCC or other skin cancers in lesions suspicious for skin cancer. Pooled data from seven studies using standard cytomorphological criteria (but various stain methods) to detect BCC in patients with a high clinical suspicion of BCC estimated the sensitivity and specificity of exfoliative cytology as 97.5% (95% CI: 94.5 to 98.9%) and 90.1% (95% CI: 81.1 to 95.1%) respectively. When applied to a hypothetical population of 1000 clinically suspected BCC lesions with a median observed BCC prevalence of 86%, exfoliative cytology would miss 21 BCCs and would lead to 14 false positive diagnoses of BCC. No false positive cases were histologically confirmed to be melanoma. Insufficient data are available to make summary statements regarding the accuracy of exfoliative cytology to detect melanoma or cSCC, or its accuracy compared to dermoscopy.Authors' conclusions: The utility of exfoliative cytology for the primary diagnosis of skin cancer is unknown, as all included studies focused on the use of this technique for confirming strongly suspected clinical diagnoses. For the confirmation of BCC in lesions with a high clinical suspicion, there is evidence of high sensitivity and specificity for exfoliative cytology. Since decisions to treat low riskBCCs are unlikely in practice to require diagnostic confirmation given that clinical suspicion is already high, exfoliative cytology might be most useful for cases of BCC where the treatments being contemplated require a tissue diagnosis (e.g. radiotherapy). The small number of included studies, poor reporting and varying methodological quality means that no strong conclusions can currently be drawn to guide clinical practice. Despite insufficient data on the use of cytology for cSCC or melanoma, it is unlikely that cytology would be useful in these scenarios since preservation of the architecture of the whole lesion that would be available from a biopsy provides crucial diagnostic information. Given the paucity of good quality data, appropriately designed prospective comparative studies may be required to evaluate both the diagnostic value of exfoliative cytology by comparison to dermoscopy, and its confirmatory value in adequately reported populations with a high probability of BCC scheduled for further treatment requiring a tissue diagnosis

Management of endocrine disease:Imaging for the diagnosis of malignancy in incidentally discovered adrenal masses: a systematic review and meta-analysis

Objective: Adrenal masses are incidentally discovered in 5% of CT scans. In 2013/2014, 81 million CT examinations were undertaken in the USA and 5 million in the UK. However, uncertainty remains around the optimal imaging approach for diagnosing malignancy. We aimed to review the evidence on the accuracy of imaging tests for differentiating malignant from benign adrenal masses. Design: A systematic review and meta-analysis was conducted. Methods: We searched MEDLINE, EMBASE, Cochrane CENTRAL Register of Controlled Trials, Science Citation Index, Conference Proceedings Citation Index, and ZETOC (January 1990 to August 2015). We included studies evaluating the accuracy of CT, MRI, or F-18-fluoro-deoxyglucose (FDG)-PET compared with an adequate histological or imaging-based follow-up reference standard. Results: We identified 37 studies suitable for inclusion, after screening 5469 references and 525 full-text articles. Studies evaluated the accuracy of CT (n = 16), MRI (n = 15), and FDG-PET (n = 9) and were generally small and at high or unclear risk of bias. Only 19 studies were eligible for meta-analysis. Limited data suggest that CT density >10 HU has high sensitivity for detection of adrenal malignancy in participants with no prior indication for adrenal imaging, that is, masses with <= 10 HU are unlikely to be malignant. All other estimates of test performance are based on too small numbers. Conclusions: Despite their widespread use in routine assessment, there is insufficient evidence for the diagnostic value of individual imaging tests in distinguishing benign from malignant adrenal masses. Future research is urgently needed and should include prospective test validation studies for imaging and novel diagnostic approaches alongside detailed health economics analysis

Exfoliative cytology for the diagnosis of basal cell carcinoma and other skin cancers in adults

Background: Early accurate detection of all skin cancer types is essential to guide appropriate management and to reduce morbidity and improve survival. Basal cell carcinoma (BCC) is usually localised to the skin with potential to infiltrate and damage surrounding tissue, while cutaneous squamous cell carcinoma (cSCC) and melanoma have a much higher potential to metastasise and ultimately lead to death. Exfoliative cytology is a non–invasive test that uses the Tzanck smear technique to identify disease by examining the structure of cells obtained from scraped samples. This simple procedure is a less invasive diagnostic test than a skin biopsy, and for BCC has the potential to provide an immediate diagnosis that avoids an additional visit to clinic to receive skin biopsy results. This may benefit patients scheduled for either Mohs micrographic surgery or non–surgical treatments such as radiotherapy. A cytology scrape can never give the same information as a skin biopsy, however, so it is important to know more about which skin cancer situations it may be helpful. Objectives: The primary objective was to determine the diagnostic accuracy of exfoliative cytology for the detection of basal cell carcinoma (BCC) in adults. Secondary objectives were to determine diagnostic accuracy for the detection of i) cutaneous squamous cell carcinoma, ii) invasive melanoma and atypical intraepidermal melanocytic variants, and iii) any skin cancer, including keratinocyte skin cancer, invasive melanoma and atypical intraepidermal melanocytic variants, or any other skin cancer. Search methods: We undertook a comprehensive search of the following databases from inception up to August 2016: Cochrane Central Register of Controlled Trials; MEDLINE; EMBASE; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We also studied the reference lists of published systematic review articles. Selection criteria: Studies evaluating exfoliative cytology in adults with lesions suspicious for BCC, cSCC or melanoma, compared with a reference standard of histological confirmation. Data collection and analysis: Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS-2). Where possible we estimated summary sensitivities and specificities using the bivariate hierarchical model. Main results: This review reports on nine studies with a total of 1655 lesions including 1120 BCCs (14 datasets), 401 lesions with 44 cSCCs (two datasets), and 200 lesions with 10 melanomas (one dataset). Three of these datasets (one each for BCC, melanoma, and any malignant condition) were derived from one study which also performed a direct comparison with dermoscopy. Studies were of moderate to poor quality providing inadequate descriptions of participant selection, thresholds used to make cytological and histological diagnoses, and blinding. Reporting of patients’ prior referral pathways was particularly poor, as were descriptions of the cytodiagnostic criteria used to make diagnoses. No studies evaluated the use of exfoliative cytology as a primary diagnostic test for detecting BCC or other skin cancers in lesions suspicious for skin cancer. Pooled data from seven studies using standard cytomorphological criteria (but various stain methods) to detect BCC in patients with a high clinical suspicion of BCC estimated the sensitivity and specificity of exfoliative cytology as 97.5% (95% CI: 94.5 to 98.9%) and 90.1% (95% CI: 81.1 to 95.1%) respectively. When applied to a hypothetical population of 1000 clinically suspected BCC lesions with a median observed BCC prevalence of 86%, exfoliative cytology would miss 21 BCCs and would lead to 14 false positive diagnoses of BCC. No false positive cases were histologically confirmed to be melanoma. Insufficient data are available to make summary statements regarding the accuracy of exfoliative cytology to detect melanoma or cSCC, or its accuracy compared to dermoscopy. Authors' conclusions: The utility of exfoliative cytology for the primary diagnosis of skin cancer is unknown, as all included studies focused on the use of this technique for confirming strongly suspected clinical diagnoses. For the confirmation of BCC in lesions with a high clinical suspicion, there is evidence of high sensitivity and specificity for exfoliative cytology. Since decisions to treat low risk BCCs are unlikely in practice to require diagnostic confirmation given that clinical suspicion is already high, exfoliative cytology might be most useful for cases of BCC where the treatments being contemplated require a tissue diagnosis (e.g. radiotherapy). The small number of included studies, poor reporting and varying methodological quality means that no strong conclusions can currently be drawn to guide clinical practice. Despite insufficient data on the use of cytology for cSCC or melanoma, it is unlikely that cytology would be useful in these scenarios since preservation of the architecture of the whole lesion that would be available from a biopsy provides crucial diagnostic information. Given the paucity of good quality data, appropriately designed prospective comparative studies may be required to evaluate both the diagnostic value of exfoliative cytology by comparison to dermoscopy, and its confirmatory value in adequately reported populations with a high probability of BCC scheduled for further treatment requiring a tissue diagnosis

Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines

The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77–94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines

Dermoscopy, with and without visual inspection, for the diagnosis of melanoma in adults

Background: Melanoma has one of the fastest rising incidence rates of any cancer. It accounts for a small percentage of skin cancer cases but is responsible for the majority of skin cancer deaths. Although history-taking and visual inspection of a suspicious lesion by a clinician are usually the first in a series of ‘tests’ to diagnose skin cancer, dermoscopy has become an important tool to assist diagnosis by specialist clinicians and is increasingly used in primary care settings. Dermoscopy is a magnification technique using visible light that allows more detailed examination of the skin compared to examination by the naked eye alone. Establishing the additive value of dermoscopy over and above visual inspection alone across a range of observers and settings is critical to understanding its contribution for the diagnosis of melanoma and to future understanding of the potential role of the growing number of other highresolution image analysis techniques. Objectives: To determine the diagnostic accuracy of dermoscopy for the detection of cutaneous invasive melanoma and atypical intraepidermal melanocytic variants in adults, and to compare its accuracy with that of visual inspection alone. Studies were separated according to whether the diagnosis was recorded face-to-face (in-person) or based on remote (image-based) assessment. Search methods: We undertook a comprehensive search of the following databases from inception up to August 2016: Cochrane Central Register of Controlled Trials; MEDLINE; Embase; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We studied reference lists and published systematic review articles. Selection criteria: Studies of any design that evaluated dermoscopy in adults with lesions suspicious for melanoma, compared with a reference standard of either histological confirmation or clinical follow-up. Data on the accuracy of visual inspection, to allow comparisons of tests, was included only if reported in the included studies of dermoscopy. Data collection and analysis: Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS-2). We contacted authors of included studies where information related to the target condition or diagnostic threshold were missing. We estimated accuracy using hierarchical summary ROC methods. Analysis of studies allowing direct comparison between tests was undertaken. To facilitate interpretation of results, we computed values of sensitivity at the point on the SROC curve with 80% fixed specificity and values of specificity with 80% fixed sensitivity. We investigated the impact of in-person test interpretation; use of a purposely developed algorithm to assist diagnosis; observer expertise; and dermoscopy training. Main results: A total of 104 study publications reporting on 103 study cohorts with 42,788 lesions (including 5700 cases) were included, providing 354 datasets for dermoscopy. The risk of bias was mainly low for the index test and reference standard domains and mainly high or unclear for participant selection and participant flow. Concerns regarding the applicability of study findings were largely scored as ‘High’ concern in three of four domains assessed. Selective participant recruitment, lack of reproducibility of diagnostic thresholds and lack of detail on observer expertise were particularly problematic. The accuracy of dermoscopy for the detection of invasive melanoma or atypical intraepidermal melanocytic variants was reported in 86 datasets; 26 for evaluations conducted in-person (dermoscopy added to visual inspection) and 60 for image-based evaluations (diagnosis based on interpretation of dermoscopic images). Analyses of studies by prior testing revealed no obvious effect on accuracy; analyses were hampered by the lack of studies in primary care, lack of relevant information and the restricted inclusion of lesions selected for biopsy or excision. Accuracy was higher for in-person diagnosis compared to image-based evaluations (relative diagnostic odds ratio (RDOR) of 4.6; 95% CI 2.4, 9.0, P<0.001). Accuracy was compared for (a) in-person evaluations of dermoscopy (26 evaluations; 23,169 lesions and 1664 melanomas) versus visual inspection alone (13 evaluations; 6740 lesions and 459 melanomas) and for (b) image-based evaluations of dermoscopy (60 evaluations; 13,475 lesions and 2851 melanomas) versus image-based visual inspection (11 evaluations; 1740 lesions and 305 melanomas). For both comparisons, meta-analysis found dermoscopy to be more accurate than visual inspection alone, with RDORs of (a) 4.7 (95% CI: 3.0 to 7.5; P < 0.001) and (b) 5.6 (95% CI: 3.7 to 8.5; P < 0.001). These effects correspond to predicted differences in sensitivity of (a) 16% (95% CI: 8%, 23%) (92% for dermoscopy+visual inspection vs 76% for visual inspection) and (b) 35% (95% CI 24% to 46%) (81% for dermoscopy vs 47% for visual inspection) at a fixed specificity of 80%; and topredicted differences in specificity of (a) 20% (95% CI 7%, 33) (95% for dermoscopy plus visual inspection vs 75% for visual inspection) and (b) 40% (95% CI 27, 57) (82% for dermoscopy vs 42% for visual inspection) at a fixed sensitivity of 80%. Using the median prevalence of disease in each set of studies ((a) 12% for in-person and (b) 24% for image-based) for a hypothetical population of 1000 lesions, an increase in sensitivity of (a) 16% (in-person) and (b) 35% (image-based) from using dermoscopy at a fixed specificity of 80% equates to a reduction in the number of melanomas missed of (a) 19 and (b) 81 with (a) 176 and (b) 152 false positive results. An increase in specificity of (a) 20% (in-person) and (b) 40% (image-based) at a fixed sensitivity of 80% equates to a reduction in the number of unnecessary excisions from using dermoscopy of (a) 176 and (b) 304 with (a) 24 and (b) 48 melanomas missed. The use of a named or published algorithm to assist dermoscopy interpretation (as opposed to no reported algorithm or reported use of pattern analysis) had no significant impact on accuracy either for in-person (RDOR 1.4, 95% CI 0.34, 5.6; P=0.17) or image-based (RDOR 1.4, 95% CI 0.60, 3.3; P=0.22) evaluations. This result was supported by subgroup analysis according to algorithm used. Higher accuracy for observers reported as having high experience and for those classed as ‘expert consultants’ in comparison to those considered to have less experience in dermoscopy was observed, particularly for image-based evaluations. Evidence for the effect of dermoscopy training on test accuracy was very limited but suggested associated improvements in sensitivity. Authors' conclusions: Despite the observed limitations in the evidence base, dermoscopy is a valuable tool to support the visual inspection of a suspicious skin lesion for the detection of melanoma and atypical intraepidermal melanocytic variants, particularly in referred populations and in the hands of experienced users. Data to support its use in primary care is limited however it may assist in triaging suspicious lesions for urgent referral when employed by suitably trained clinicians. Formal algorithms may be of most use for dermoscopy training purposes and for less expert observers, however reliable data comparing approaches using dermoscopy in-person are lacking