Quality assessment standards in artificial intelligence diagnostic accuracy systematic reviews: a meta-research study

Artificial intelligence (AI) centred diagnostic systems are increasingly recognized as robust solutions in healthcare delivery pathways. In turn, there has been a concurrent rise in secondary research studies regarding these technologies in order to influence key clinical and policymaking decisions. It is therefore essential that these studies accurately appraise methodological quality and risk of bias within shortlisted trials and reports. In order to assess whether this critical step is performed, we undertook a meta-research study evaluating adherence to the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) tool within AI diagnostic accuracy systematic reviews. A literature search was conducted on all studies published from 2000 to December 2020. Of 50 included reviews, 36 performed quality assessment, of which 27 utilised the QUADAS-2 tool. Bias was reported across all four domains of QUADAS-2. 243 of 423 studies (57.5%) across all systematic reviews utilising QUADAS-2 reported a high or unclear risk of bias in the patient selection domain, 110 (26%) reported a high or unclear risk of bias in the index test domain, 121 (28.6%) in the reference standard domain and 157 (37.1%) in the flow and timing domain. This study demonstrates incomplete uptake of quality assessment tools in reviews of AI-based diagnostic accuracy studies and highlights inconsistent reporting across all domains of quality assessment. Poor standards of reporting act as barriers to clinical implementation. The creation of an AI specific extension for quality assessment tools of diagnostic accuracy AI studies may facilitate the safe translation of AI tools into clinical practice

A systematic review of interventions to improve breast cancer screening health behaviours

Whilst breast cancer screening has been implemented in many countries, uptake is often suboptimal. Consequently, several interventions targeting non-attendance behaviour have been developed. This systematic review aims to appraise the successes of interventions, identifying and comparing the specific techniques they use to modify health behaviours. A literature search (PROSPERO CRD42020212090) between January 2005 and December 2020 using PubMed, Medline, PsycInfo, EMBASE and Google Scholar was conducted. Studies which investigated patient-facing interventions to increase attendance at breast cancer screening appointments were included. Details regarding the intervention delivery, theoretical background, and contents were extracted, as was quantitative data on the impact on attendance rates, compared to control measures. Interventions were also coded using the Behavioural Change Techniques (BCT) Taxonomy. In total fifty-four studies, detailing eighty interventions, met the inclusion criteria. Only 50% of interventions reported a significant impact on screening attendance. Thirty-two different BCTs were used, with 'prompts/cues' the most commonly incorporated (77.5%), however techniques from the group 'covert learning' had the greatest pooled effect size 0.12 (95% CI 0.05-0.19, P < 0·01, I2 = 91.5%). 'Problem solving' was used in the highest proportion of interventions that significantly increased screening attendance (69.0%). 70% of the interventions were developed using behavioural theories. These results show interventions aimed at increasing screening uptake are often unsuccessful. Commonly used approaches which focus upon explaining the consequences of not attending mammograms were often ineffective. Problem solving, however, has shown promise. These techniques should be investigated further, as should emerging technologies which can enable interventions to be feasibly translated at a population-level

Simulated wound assessment using digital planimetry versus three-dimensional cameras: implications for clinical assessment.

BACKGROUND: Clinical management of wounds can benefit from objective measures of response to treatment. Wound surface area and volume are objective measures of wound healing. Using a synthetic wound model, we compare the accuracy and reproducibility of 2 commercially available 3-dimensional (3D) cameras against planimetry and water displacement. METHODS: Twelve ulcers of various sizes and colors were reproduced in modeling clay and cured. Five naive observers used digital planimetry, water displacement, Eykona camera (Fuel 3D, UK), and Silhouette camera (ARANZ, New Zealand) to measure the wounds. RESULTS: When compared with traditional planimetry, wound surface area measurement with Eykona and Silhouette tended to underestimate wounds by 1.7% and 3.7%, respectively. Spearman correlation coefficients were 0.94 (Eykona) and 0.92 (Silhouette). Intraclass correlations for planimetry and the 2 cameras were all 1. Eykona and Silhouette tended to underestimate wound volumes when compared with water displacement by 58% and 23%, respectively. Spearman correlation coefficients were 0.92 (Eykona) and 0.72 (Silhouette). Intraclass correlations for water displacement and the two cameras were all 1. DISCUSSION: Serial accurate objective area measurements are feasible as part of ongoing clinical assessment of wounds. 3D cameras are reliable but have not shown superior accuracy to manual planimetry, and financial concerns and IT integration may limit general clinical usage. Volume measurements of wounds are practicable as part of clinical care

Avoidable 30‐day readmissions in patients undergoing vascular surgery

Background: Vascular surgery has one of the highest unplanned 30-day readmission rates of all surgical specialities. The degree to which these may be avoidable and the optimal strategies to reduce their occurrence is unknown. The aim of this study was to identify and classify avoidable 30-day readmissions in patients undergoing vascular surgery in order to plan targeted interventions to reduce their occurrence, improve outcomes and reduce cost. Methods: A retrospective analysis of discharges over a 12-month period from a single tertiary vascular unit was performed. A multidisciplinary panel conducted a manual case note review to identify and classify those 30-day unplanned emergency readmissions deemed avoidable. Results: An unplanned 30-day readmission occurred in 72/885 (8.1%) admissions. These unplanned readmissions were deemed avoidable in 50.0% (36/72) and were most frequently due to unresolved medical issues (19/36, 52.8%) and inappropriate admission with the potential for outpatient management (7/36, 19.4%). A smaller number were due to inadequate social care provision (4/36, 11.1%) and the occurrence of other avoidable adverse events (4/36, 11.1%). Conclusion: Half of all 30-day readmissions in vascular patients are potentially avoidable. Multidisciplinary coordination of inpatient care and the transition from hospital to community care following discharge need to be improved

A national survey assessing public readiness for digital health strategies against COVID-19 within the United Kingdom

There is concern that digital public health initiatives used in the management of COVID-19 may marginalise certain population groups. There is an overlap between the demographics of groups at risk of digital exclusion (older, lower social grade, low educational attainment and ethnic minorities) and those who are vulnerable to poorer health outcomes from SARS-CoV-2. In this national survey study (n=2040), we assessed how the UK population; particularly these overlapping groups, reported their preparedness for digital health strategies. We report, with respect to using digital information to make health decisions, that those over 60 are less comfortable (net comfort: 57%) than those between 18-39 (net comfort: 78%) and lower social grades are less comfortable (net comfort: 63%) than higher social grades (net comfort: 75%). With respect to a preference for digital over non-digital sources in seeking COVID-19 health information, those over 60 (net preference: 21%) are less inclined than those between 18-39 (net preference: 60%) and those of low educational attainment (net preference: 30%) are less inclined than those of high educational attainment (net preference: 52%). Lastly, with respect to distinguishing reliable digital COVID-19 information, lower social grades (net confidence: 55%) are less confident than higher social grades (net confidence: 68%) and those of low educational attainment (net confidence: 51%) are less confident than those of high educational attainment (net confidence: 71%). All reported differences are statistically significant (p<0.01) following multivariate regression modelling. This study suggests that digital public health approaches to COVID-19 have the potential to marginalise groups who are concurrently at risk of digital exclusion and poor health outcomes from SARS-CoV-2

Investigating the implementation of SMS and mobile messaging In Population Screening (The SIPS Study): Protocol for a Delphi Study

Background The use of mobile messaging including Short Message Service (SMS) and Web-based messaging in healthcare has grown significantly. Using messaging to facilitate patient communication has been advocated in several circumstances including population screening. These programmes, however, pose unique challenges to mobile communication, as messaging is often sent from a central hub to a diverse population with differing needs. Despite this, there is a paucity of robust frameworks to guide implementation. Objective This protocol describes the methods that will be used to develop a guide for the principles of use of mobile messaging for population screening programmes in England. Methods This modified Delphi study will be conducted in two parts: evidence synthesis and consensus generation. The former will incorporate a literature review of publications from 1st January 2000 to the present. This will elicit key themes to inform an online scoping questionnaire posed to a group of experts from academia, clinical medicine, industry and public health. Thematic analysis of free-text responses by two independent authors will elicit items to be used in the consensus generation. Patient and Public Involvement groups will be convened to ensure that a comprehensive item list is generated, which represents the public’s perspective. Each item will then be anonymously voted upon by experts as to its importance and feasibility of implementation in screening, during three rounds of a Delphi process. Consensus will be defined a priori at 70%, with items considered important and feasible eligible for inclusion into the final recommendation. A list of desirable items (important, but not currently feasible) will be developed to guide future work. Results The Institutional Review Board at Imperial College London has granted ethical approval (20IC6088). Results are expected to involve a list of recommendations to screening services with findings made available to screening services through Public Health England. This study will thus provide a formal guideline for the use of mobile messaging in screening services and provide future directions in this field. Discussion The use of mobile messaging has grown significantly across healthcare services, especially given the COVID-19 pandemic, but its implementation in screening programmes remains challenging. this modified Delphi approach with leading experts, will provide invaluable insights to facilitate incorporating messaging in these programmes, and create awareness of future developments in this area

Defining the Enablers and Barriers to the Implementation of Large-scale, Health Care-Related Mobile Technology: Qualitative Case Study in a Tertiary Hospital Setting

BACKGROUND: The successful implementation of clinical smartphone apps in hospital settings requires close collaboration with industry partners. A large-scale, hospital-wide implementation of a clinical mobile app for health care professionals developed in partnership with Google Health and academic partners was deployed on a bring-your-own-device basis using mobile device management at our UK academic hospital. As this was the first large-scale implementation of this type of innovation in the UK health system, important insights and lessons learned from the deployment may be useful to other organizations considering implementing similar technology in partnership with commercial companies. OBJECTIVE: The aims of this study are to define the key enablers and barriers and to propose a road map for the implementation of a hospital-wide clinical mobile app developed in collaboration with an industry partner as a data processor and an academic partner for independent evaluation. METHODS: Semistructured interviews were conducted with high-level stakeholders from industry, academia, and health care providers who had instrumental roles in the implementation of the app at our hospital. The interviews explored the participants' views on the enablers and barriers to the implementation process. The interviews were analyzed using a broadly deductive approach to thematic analysis. RESULTS: In total, 14 participants were interviewed. Key enablers identified were the establishment of a steering committee with high-level clinical involvement, well-defined roles and responsibilities between partners, effective communication strategies with end users, safe information governance precautions, and increased patient engagement and transparency. Barriers identified were the lack of dedicated resources for mobile change at our hospital, risk aversion, unclear strategy and regulation, and the implications of bring-your-own-device and mobile device management policies. The key lessons learned from the deployment process were highlighted, and a road map for the implementation of large-scale clinical mobile apps in hospital settings was proposed. CONCLUSIONS: Despite partnering with one of the world's biggest technology companies, the cultural and technological change required for mobile working and implementation in health care was found to be a significant challenge. With an increasing requirement for health care organizations to partner with industry for advanced mobile technologies, the lessons learned from our implementation can influence how other health care organizations undertake a similar mobile change and improve the chances of successful widespread mobile transformation

The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review

ABSTRACT Objective To evaluate the accuracy of digital and online symptom checkers in providing diagnoses and appropriate triage advice. Design Systematic review. Data sources Medline and Web of Science were searched up to 15 February 2021. Eligibility criteria for study selection Prospective and retrospective cohort, vignette, or audit studies that utilised an online or application-based service designed to input symptoms and biodata in order to generate diagnoses, health advice and direct patients to appropriate services were included. Main outcome measures The primary outcomes were (1) the accuracy of symptom checkers for providing the correct diagnosis and (2) the accuracy of subsequent triage advice given. Data extraction and synthesis Data extraction and quality assessment (using the QUADAS-2 tool) were performed by two independent reviewers. Owing to heterogeneity of the studies, meta-analysis was not possible. A narrative synthesis of the included studies and pre-specified outcomes was completed. Results Of the 177 studies retrieved, nine cohort studies and one cross-sectional study met the inclusion criteria. Symptom checkers evaluated a variety of medical conditions including ophthalmological conditions, inflammatory arthritides and HIV. 50% of the studies recruited real patients, while the remainder used simulated cases. The diagnostic accuracy of the primary diagnosis was low (range: 19% to 36%) and varied between individual symptom checkers, despite consistent symptom data input. Triage accuracy (range: 48.8% to 90.1%) was typically higher than diagnostic accuracy. Of note, one study found that 78.6% of emergency ophthalmic cases were under-triaged. Conclusions The diagnostic and triage accuracy of symptom checkers are variable and of low accuracy. Given the increasing push towards population-wide digital health technology adoption, reliance upon symptom checkers in lieu of traditional assessment models, poses the potential for clinical risk. Further primary studies, utilising improved study reporting, core outcome sets and subgroup analyses, are warranted to demonstrate equitable and non-inferior performance of these technologies to that of current best practice. PROSPERO registration number CRD42021271022. SUMMARY BOXES What is already known on this topic Chambers et al. (2019) have previously examined the evidence underpinning digital and online symptom checkers, including the accuracy of the diagnostic and triage information, for urgent health problems and found that diagnostic accuracy was generally low and varied depending on the symptom checker used. Given the increased reliance upon digital health technologies by health systems in light of the ongoing COVID-19 pandemic, in addition to the marked increase in availability of similarly themed digital health products since the last systematic review, a contemporary and comprehensive reassessment of this class of technologies to ascertain their diagnostic and triage accuracy is warranted. What this study adds Our systematic review demonstrates that the diagnostic accuracy of symptom checkers remains low and varies significantly depending on the pathology or symptom checker used. The findings of this systematic review suggests that this class of technologies, in their current state, poses significant risk for patient safety, particularly if utilised in isolation

The Role of Wearable Technologies and Telemonitoring in Managing Vascular Disease

Wearable devices and telemonitoring are becoming increasingly widespread in the clinical environment and have many applications in the tracking and maintenance of patient wellbeing. Interventions incorporating these technologies have been used with some success in patients with vascular disorders. Wearable fitness monitors and telemonitoring have been used in the community to mobilise patients with peripheral vascular disease with good results. Additionally, wearable monitors and telemonitoring have been studied for blood pressure monitoring in patients with hypertension. Telemonitoring interventions incorporating electronic medication trays and ingestible sensors have also been found to increase drug adherence in hypertensive patients and ultimately improve health outcomes. However, wearable and telemonitoring interventions often face problems with patient adherence, digital literacy and infrastructure. Further work needs to address these challenges and validate the technology before widespread implementation can occur

Evaluating the diagnostic and triage performance of digital and online symptom checkers for the presentation of myocardial infarction:A retrospective cross-sectional study

Online symptom checkers are increasingly popular health technologies that enable patients to input their symptoms to produce diagnoses and triage advice. However, there is concern regarding the performance and safety of symptom checkers in diagnosing and triaging patients with life-threatening conditions. This retrospective cross-sectional study aimed to evaluate and compare commercially available symptom checkers for performance in diagnosing and triaging myocardial infarctions (MI). Symptoms and biodata of MI patients were inputted into 8 symptom checkers identified through a systematic search. Anonymised clinical data of 100 consecutive MI patients were collected from a tertiary coronary intervention centre between 1st January 2020 to 31st December 2020. Outcomes included (1) diagnostic sensitivity as defined by symptom checkers outputting MI as the primary diagnosis (D1), or one of the top three (D3), or top five diagnoses (D5); and (2) triage sensitivity as defined by symptom checkers outputting urgent treatment recommendations. Overall D1 sensitivity was 48±31% and varied between symptom checkers (range: 6–85%). Overall D3 and D5 sensitivity were 73±20% (34–92%) and 79±14% (63–94%), respectively. Overall triage sensitivity was 83±13% (55–91%). 24±16% of atypical cases had a correct D1 though for female atypical cases D1 sensitivity was only 10%. Atypical MI D3 and D5 sensitivity were 44±21% and 48±24% respectively and were significantly lower than typical MI cases (p&lt;0.01). Atypical MI triage sensitivity was significantly lower than typical cases (53±20% versus 84±15%, p&lt;0.01). Female atypical cases had significantly lower diagnostic and triage sensitivity than typical female MI cases (p&lt;0.01).Given the severity of the pathology, the diagnostic performance of symptom checkers for correctly diagnosing an MI is concerningly low. Moreover, there is considerable inter-symptom checker performance variation. Patients presenting with atypical symptoms were under-diagnosed and under-triaged, especially if female. This study highlights the need for improved clinical performance, equity and transparency associated with these technologies