An Evaluation Service for Digital Public Health Interventions: User-Centered Design Approach

BACKGROUND: Digital health interventions (DHIs) have the potential to improve public health by combining effective interventions and population reach. However, what biomedical researchers and digital developers consider an effective intervention differs, thereby creating an ongoing challenge to integrating their respective approaches when evaluating DHIs. OBJECTIVE: This study aims to report on the Public Health England (PHE) initiative set out to operationalize an evaluation framework that combines biomedical and digital approaches and demonstrates the impact, cost-effectiveness, and benefit of DHIs on public health. METHODS: We comprised a multidisciplinary project team including service designers, academics, and public health professionals and used user-centered design methods, such as qualitative research, engagement with end users and stakeholders, and iterative learning. The iterative approach enabled the team to sequentially define the problem, understand user needs, identify opportunity areas, develop concepts, test prototypes, and plan service implementation. Stakeholders, senior leaders from PHE, and a working group critiqued the outputs. RESULTS: We identified 26 themes and 82 user needs from semistructured interviews (N=15), expressed as 46 Jobs To Be Done, which were then validated across the journey of evaluation design for a DHI. We identified seven essential concepts for evaluating DHIs: evaluation thinking, evaluation canvas, contract assistant, testing toolkit, development history, data hub, and publish health outcomes. Of these, three concepts were prioritized for further testing and development, and subsequently refined into the proposed PHE Evaluation Service for public health DHIs. Testing with PHE’s Couch-to-5K app digital team confirmed the viability, desirability, and feasibility of both the evaluation approach and the Evaluation Service. CONCLUSIONS: An iterative, user-centered design approach enabled PHE to combine the strengths of academic and biomedical disciplines with the expertise of nonacademic and digital developers for evaluating DHIs. Design-led methodologies can add value to public health settings. The subsequent service, now known as Evaluating Digital Health Products, is currently in use by health bodies in the United Kingdom and is available to others for tackling the problem of evaluating DHIs pragmatically and responsively

Proteomics identifies neddylation as a potential therapy target in small intestinal neuroendocrine tumors.

Patients with small intestinal neuroendocrine tumors (SI-NETs) frequently develop spread disease; however, the underlying molecular mechanisms of disease progression are not known and effective preventive treatment strategies are lacking. Here, protein expression profiling was performed by HiRIEF-LC-MS in 14 primary SI-NETs from patients with and without liver metastases detected at the time of surgery and initial treatment. Among differentially expressed proteins, overexpression of the ubiquitin-like protein NEDD8 was identified in samples from patients with liver metastasis. Further, NEDD8 correlation analysis indicated co-expression with RBX1, a key component in cullin-RING ubiquitin ligases (CRLs). In vitro inhibition of neddylation with the therapeutic agent pevonedistat (MLN4924) resulted in a dramatic decrease of proliferation in SI-NET cell lines. Subsequent mass spectrometry-based proteomics analysis of pevonedistat effects and effects of the proteasome inhibitor bortezomib revealed stabilization of multiple targets of CRLs including p27, an established tumor suppressor in SI-NET. Silencing of NEDD8 and RBX1 using siRNA resulted in a stabilization of p27, suggesting that the cellular levels of NEDD8 and RBX1 affect CRL activity. Inhibition of CRL activity, by either NEDD8/RBX1 silencing or pevonedistat treatment of cells resulted in induction of apoptosis that could be partially rescued by siRNA-based silencing of p27. Differential expression of both p27 and NEDD8 was confirmed in a second cohort of SI-NET using immunohistochemistry. Collectively, these findings suggest a role for CRLs and the ubiquitin proteasome system in suppression of p27 in SI-NET, and inhibition of neddylation as a putative therapeutic strategy in SI-NET

Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

A growing number of artificial intelligence (AI)-based clinical decision support systems are showing promising performance in preclinical, in silico, evaluation, but few have yet demonstrated real benefit to patient care. Early stage clinical evaluation is important to assess an AI system’s actual clinical performance at small scale, ensure its safety, evaluate the human factors surrounding its use, and pave the way to further large scale trials. However, the reporting of these early studies remains inadequate. The present statement provides a multistakeholder, consensus-based reporting guideline for the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence (DECIDE-AI). We conducted a two round, modified Delphi process to collect and analyse expert opinion on the reporting of early clinical evaluation of AI systems. Experts were recruited from 20 predefined stakeholder categories. The final composition and wording of the guideline was determined at a virtual consensus meeting. The checklist and the Explanation & Elaboration (E&E) sections were refined based on feedback from a qualitative evaluation process. 123 experts participated in the first round of Delphi, 138 in the second, 16 in the consensus meeting, and 16 in the qualitative evaluation. The DECIDE-AI reporting guideline comprises 17 AI specific reporting items (made of 28 subitems) and 10 generic reporting items, with an E&E paragraph provided for each. Through consultation and consensus with a range of stakeholders, we have developed a guideline comprising key items that should be reported in early stage clinical studies of AI-based decision support systems in healthcare. By providing an actionable checklist of minimal reporting items, the DECIDE-AI guideline will facilitate the appraisal of these studies and replicability of their findings

An evaluation service for digital public health interventions: concepts, development and ways of working

Background: Digital health interventions have potential to improve public health by combining effective intervention and population reach. However, what biomedical researchers and digital developers consider an effective intervention differs, thereby creating an ongoing challenge to integrate their respective approaches when evaluating digital health interventions. / Objective: Public Health England set out to operationalise an evaluation framework that combines biomedical and digital approaches, and demonstrates the impact, cost-effectiveness and benefit of digital health interventions to public health. / Methods: A multidisciplinary project team, composed of service designers, academics and public health professionals, employed user-centred design methods such as qualitative research, engagement with end-users and stakeholders, and iterative learning. An iterative approach enabled the team to sequentially define the problem, understand user needs, identify opportunity areas, develop concepts, test prototypes, and plan service implementation. Outputs were critiqued by stakeholders, system leaders and a Working Group. / Results: Semi-structured interviews (N=15) identified 26 themes and 82 user needs, expressed as 46 Jobs To Be Done, which were then validated across the journey of evaluation design for a digital health intervention. Seven essential concepts for evaluating digital health interventions were identified: (i) Evaluation Thinking; (ii) Evaluation Canvas; (iii) Contract Assistant; (iv) Testing Toolkit; (v) Development History; (vi) Data Hub, and (vii) Publish Health Outcomes. Three concepts were prioritised for further testing and development and subsequently refined into the proposed Public Health England Evaluation Service for public health digital health interventions. Testing with Public Health England's Couch-to-5K App digital team confirmed the viability, desirability and feasibility of both the evaluation approach and the Evaluation Service. / Conclusions: An iterative, user-centred design approach enabled Public Health England to combine the strengths of academic and biomedical disciplines alongside the expertise of non-academic and/or digital developers for evaluating digital health interventions. Design-led methodologies can add value in a public health setting. The subsequent service, now known as Evaluating Digital Health Products, is currently in use by health bodies in the United Kingdom and available to others tackling the problem of evaluating digital health interventions pragmatically and responsively