Methodological challenges in the evidence synthesis of health outcomes of digital health technologies [védés előtt]

Medical devices and pharmaceuticals are worlds apart, but healthcare would be impossible without them. Digital biomarkers are the subject of this thesis defined as objective, measurable, physiological, and behavioural parameters collected using wearable, portable, implantable, or digestible digital devices. Since the 1970s, systematic reviews and meta-analyses have dominated medical evidence synthesis. They provide medical decision-making evidence. To avoid biases and maintain methodological quality, the Cochrane Handbook recommends systematic reviews follow certain procedures during study stages. This thesis comprises six hypotheses related to digital biomarkers. The first hypothesis aimed to evaluate the suitability of using tools provided by the World Health Organization (WHO), including ICD-11 (International Classification of Diseases, 11th Revision), ICHI (International Classification of Health Interventions), and ICF (International Classification of Functioning, Disability and Health), for categorizing populations, interventions, outcomes, and behavioral/physiological data in studies involving digital biomarkers. The results indicated that these tools were not applicable for categorizing digital biomarker studies as a whole. However, further analysis revealed that these tools were suitable for categorizing digital biomarker studies involving non-general populations or populations with specific diseases. The second hypothesis focused on comparing the statistical power of direct and indirect digital biomarkers. The results indicated that there was no significant difference in power between these two types of digital biomarkers (p-value > 0.05). The next three hypotheses compared the characteristics of systematic reviews and meta-analyses of digital biomarker-based interventions with those of non-digital biomarkers or pharmaceuticals. The comparisons were made in terms of methodological quality, quality of evidence, and publication bias. Although all these hypotheses revealed non-significant differences between the two groups (p-values > 0.05), the results showed that both digital biomarkers and non-digital biomarkers or pharmaceuticals systematic reviews did not exhibit high methodological quality or quality of evidence. The Medical Device Regulation (MDR) has significantly improved European medical device regulatory standards, addressing the above concerns and improving clinical evidence. Despite MDR implementation delays, digital health technology evidence requirements are rising. Companies that achieve these higher clinical requirements will survive and obtain access to large interconnected markets, while those that fail may lose their market authorisation. Thus, medical technology enterprises may gain a competitive edge by strategically planning and executing extensive clinical investigations to provide high-quality clinical data. Developing these essential skills needs immediate attention and effort. Digital health investors should actively monitor industry players' evidence quality and clinical trial competence, since these characteristics may significantly increase company risk

The Features of Cardiovascular Papers and Impact on Citations

Introduction: The number of citations is a factor in evaluating the quality of scientific articles. The present study aims to examine the factors affecting the citation rate of cardiovascular articles. Methods: In this scientometrics study, the research population is all cardiovascular articles in 2014 in Web of Science (WoS), including a sample of 381 articles studied. Pearson correlation coefficient, Mann–Whitney, Kruskal–Wallis, and Bonferroni tests were used to examine the impact of article features on citations. Results: The results indicated that all quantitative variables (title length, number of authors, author's H-index, journal IF, number of pages, number of author's keywords, number of keywords-plus, number of references)had a significant relationship with the number of citations (P-value<0.001), except for the number of article keywords.  All of the qualitative variables (title length, number of authors, author's H-index, journal IF, number of pages, number of author's keywords, number of keywords-plus, number of references) also affect the number of citations (P-value<0.001). Open access articles, articles with the first author from Australia and North America, articles with international collaboration, and meta-analysis articles have received a more citation rate. Conclusion: Paying attention to the factors affecting the citation rate of cardiovascular articles can be of help to cardiovascular centers for policy-making and researchers in determining the research approach. In this way, they can improve the citation of their works

Methodological and Evidence Synthesis Quality Evaluation of MetaAnalyses Assessing the Effect of Antibacterial Envelopes to Reduce CIEDRelated Infections

Purpose: Antibacterial envelopes have been demonstrated to be therapeutically helpful in patients with Cardiac Implantable Electronic Devices (CIEDs). We examined the methodological and evidence synthesis quality of meta-analyses evaluating the effect of envelopes to reduce CIED infections. Methods: Full-text English systematic reviews published in peer-reviewed journals that described meta-analyses of the therapeutic efficacy of envelope on CIED-related infection were explored. A complete literature search was conducted from conception to September 27, 2021, using the electronic databases PubMed, Scopus, and Web of Science. On the 2nd of January 2022, the search was updated. Two reviewers independently screened the titles/abstracts and full-texts and extracted the data. The methodological quality of the included studies was assessed using the AMSTAR-2 tool. The GRADE technique was used to evaluate the quality of evidence synthesis. Results: Six reviews with a total of 15 outcomes were included. All of the reviews had a critically low methodological quality. Nine (60%) and six (40%) outcomes had low and moderate-quality evidence synthesis, respectively. Regarding the GRADE criteria, all outcomes were at risk of bias (n=15, 100%), followed by inconsistency (n=12, 80%), and publication bias (n=10, 67%). Researchers in the field should use the AMSTAR-2 scale and GRADE to perform high-quality studies in the future. Conclusion: To our knowledge, the current study is the first to analyze the methodological and evidence quality of systematic reviews providing meta-analyses on the effect of antibacterial envelopes on CIED-related infections. This is to help physicians, policymakers, and researchers to make better therapeutic decisions by revealing the methodological and evidence synthesis quality of systematic reviews

Health outcomes of Fitbit, Garmin or Apple Watch-based interventions

Introduction: A rising number of systematic reviews of smartwatch clinical effectiveness have been published; however, the findings are conflicting and need further investigation. Our purpose was to assess the methodological and evidence synthesis quality of meta-analyses of the therapeutic efficacy of Fitbit, Garmin, or Apple Watch-based interventions. Material and Methods: Systematic searches were carried out with three electronic databases from inception to October 2021. Full text systematic reviews originally published in English in peer-reviewed journals describing meta-analyses of all clinical outcomes of Fitbit-, Garmin-, or Apple Watch -based interventions were eligible for this study. The Amstar-2 scale and GRADE were used to assess the methodological and evidence synthesis quality, respectively. Results: One study with low methodological quality (overall score of 13.5) was identified, which contained five outcomes. None of the outcomes met the GRADE criteria for high-quality evidence (0%). Two outcomes (40%) had moderate-quality evidence synthesis, while three outcomes (60%) had low-quality evidence synthesis. The most prevalent downgrading GRADE factor was a risk of bias (100%), followed by imprecision (80%), and publication bias (60%). Conclusion: Despite the widespread adoption of smartwatches, there is a large gap in the literature as there is limited evidence. This review suggests that further clinical trials and meta- analyses are needed and it formulates research recommendations

Outcomes of Digital Biomarker–Based Interventions: Protocol for a Systematic Review of Systematic Reviews

BackgroundDigital biomarkers are defined as objective, quantifiable, physiological, and behavioral data that are collected and measured using digital devices such as portables, wearables, implantables, or digestibles. For their widespread adoption in publicly financed health care systems, it is important to understand how their benefits translate into improved patient outcomes, which is essential for demonstrating their value. ObjectiveThe paper presents the protocol for a systematic review that aims to assess the quality and strength of the evidence reported in systematic reviews regarding the impact of digital biomarkers on clinical outcomes compared to interventions without digital biomarkers. MethodsA comprehensive search for reviews from 2019 to 2020 will be conducted in PubMed and the Cochrane Library using keywords related to digital biomarkers and a filter for systematic reviews. Original full-text English publications of systematic reviews comparing clinical outcomes of interventions with and without digital biomarkers via meta-analysis will be included. The AMSTAR-2 tool will be used to assess the methodological quality of these reviews. To assess the quality of evidence, we will evaluate the systematic reviews using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) tool. To detect the possible presence of reporting bias, we will determine whether a protocol was published prior to the start of the studies. A qualitative summary of the results by digital biomarker technology and outcomes will be provided. ResultsThis protocol was submitted before data collection. Search, screening, and data extraction will commence in December 2021 in accordance with the published protocol. ConclusionsOur study will provide a comprehensive summary of the highest level of evidence available on digital biomarker interventions, providing practical guidance for health care providers. Our results will help identify clinical areas in which the use of digital biomarkers has led to favorable clinical outcomes. In addition, our findings will highlight areas of evidence gaps where the clinical benefits of digital biomarkers have not yet been demonstrated. International Registered Report Identifier (IRRID)PRR1-10.2196/2820

Digital Biomarker–Based Studies: Scoping Review of Systematic Reviews

BackgroundSensors and digital devices have revolutionized the measurement, collection, and storage of behavioral and physiological data, leading to the new term digital biomarkers. ObjectiveThis study aimed to investigate the scope of clinical evidence covered by systematic reviews (SRs) of randomized controlled trials involving digital biomarkers. MethodsThis scoping review was organized using the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. With the search limited to English publications, full-text SRs of digital biomarkers included randomized controlled trials that involved a human population and reported changes in participants’ health status. PubMed and the Cochrane Library were searched with time frames limited to 2019 and 2020. The World Health Organization’s classification systems for diseases (International Classification of Diseases, Eleventh Revision), health interventions (International Classification of Health Interventions), and bodily functions (International Classification of Functioning, Disability, and Health [ICF]) were used to classify populations, interventions, and outcomes, respectively. ResultsA total of 31 SRs met the inclusion criteria. The majority of SRs studied patients with circulatory system diseases (19/31, 61%) and respiratory system diseases (9/31, 29%). Most of the prevalent interventions focused on physical activity behavior (16/31, 52%) and conversion of cardiac rhythm (4/31, 13%). Looking after one’s health (physical activity; 15/31, 48%), walking (12/31, 39%), heart rhythm functions (8/31, 26%), and mortality (7/31, 23%) were the most commonly reported outcomes. In total, 16 physiological and behavioral data groups were identified using the ICF tool, such as looking after one’s health (physical activity; 14/31, 45%), walking (11/31, 36%), heart rhythm (7/31, 23%), and weight maintenance functions (7/31, 23%). Various digital devices were also studied to collect these data in the included reviews, such as smart glasses, smartwatches, smart bracelets, smart shoes, and smart socks for measuring heart functions, gait pattern functions, and temperature. A substantial number (24/31, 77%) of digital biomarkers were used as interventions. Moreover, wearables (22/31, 71%) were the most common types of digital devices. Position sensors (21/31, 68%) and heart rate sensors and pulse rate sensors (12/31, 39%) were the most prevalent types of sensors used to acquire behavioral and physiological data in the SRs. ConclusionsIn recent years, the clinical evidence concerning digital biomarkers has been systematically reviewed in a wide range of study populations, interventions, digital devices, and sensor technologies, with the dominance of physical activity and cardiac monitors. We used the World Health Organization’s ICF tool for classifying behavioral and physiological data, which seemed to be an applicable tool to categorize the broad scope of digital biomarkers identified in this review. To understand the clinical value of digital biomarkers, the strength and quality of the evidence on their health consequences need to be systematically evaluated

Digital Biomarker–Based Interventions: Systematic Review of Systematic Reviews

BackgroundThe introduction of new medical technologies such as sensors has accelerated the process of collecting patient data for relevant clinical decisions, which has led to the introduction of a new technology known as digital biomarkers. ObjectiveThis study aims to assess the methodological quality and quality of evidence from meta-analyses of digital biomarker–based interventions. MethodsThis study follows the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline for reporting systematic reviews, including original English publications of systematic reviews reporting meta-analyses of clinical outcomes (efficacy and safety endpoints) of digital biomarker–based interventions compared with alternative interventions without digital biomarkers. Imaging or other technologies that do not measure objective physiological or behavioral data were excluded from this study. A literature search of PubMed and the Cochrane Library was conducted, limited to 2019-2020. The quality of the methodology and evidence synthesis of the meta-analyses were assessed using AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews 2) and GRADE (Grading of Recommendations, Assessment, Development, and Evaluations), respectively. This study was funded by the National Research, Development and Innovation Fund of Hungary. ResultsA total of 25 studies with 91 reported outcomes were included in the final analysis; 1 (4%), 1 (4%), and 23 (92%) studies had high, low, and critically low methodologic quality, respectively. As many as 6 clinical outcomes (7%) had high-quality evidence and 80 outcomes (88%) had moderate-quality evidence; 5 outcomes (5%) were rated with a low level of certainty, mainly due to risk of bias (85/91, 93%), inconsistency (27/91, 30%), and imprecision (27/91, 30%). There is high-quality evidence of improvements in mortality, transplant risk, cardiac arrhythmia detection, and stroke incidence with cardiac devices, albeit with low reporting quality. High-quality reviews of pedometers reported moderate-quality evidence, including effects on physical activity and BMI. No reports with high-quality evidence and high methodological quality were found. ConclusionsResearchers in this field should consider the AMSTAR-2 criteria and GRADE to produce high-quality studies in the future. In addition, patients, clinicians, and policymakers are advised to consider the results of this study before making clinical decisions regarding digital biomarkers to be informed of the degree of certainty of the various interventions investigated in this study. The results of this study should be considered with its limitations, such as the narrow time frame. International Registered Report Identifier (IRRID)RR2-10.2196/2820

The Reporting Quality of Machine Learning Studies on Pediatric Diabetes Mellitus: Systematic Review

BackgroundDiabetes mellitus (DM) is a major health concern among children with the widespread adoption of advanced technologies. However, concerns are growing about the transparency, replicability, biasedness, and overall validity of artificial intelligence studies in medicine. ObjectiveWe aimed to systematically review the reporting quality of machine learning (ML) studies of pediatric DM using the Minimum Information About Clinical Artificial Intelligence Modelling (MI-CLAIM) checklist, a general reporting guideline for medical artificial intelligence studies. MethodsWe searched the PubMed and Web of Science databases from 2016 to 2020. Studies were included if the use of ML was reported in children with DM aged 2 to 18 years, including studies on complications, screening studies, and in silico samples. In studies following the ML workflow of training, validation, and testing of results, reporting quality was assessed via MI-CLAIM by consensus judgments of independent reviewer pairs. Positive answers to the 17 binary items regarding sufficient reporting were qualitatively summarized and counted as a proxy measure of reporting quality. The synthesis of results included testing the association of reporting quality with publication and data type, participants (human or in silico), research goals, level of code sharing, and the scientific field of publication (medical or engineering), as well as with expert judgments of clinical impact and reproducibility. ResultsAfter screening 1043 records, 28 studies were included. The sample size of the training cohort ranged from 5 to 561. Six studies featured only in silico patients. The reporting quality was low, with great variation among the 21 studies assessed using MI-CLAIM. The number of items with sufficient reporting ranged from 4 to 12 (mean 7.43, SD 2.62). The items on research questions and data characterization were reported adequately most often, whereas items on patient characteristics and model examination were reported adequately least often. The representativeness of the training and test cohorts to real-world settings and the adequacy of model performance evaluation were the most difficult to judge. Reporting quality improved over time (r=0.50; P=.02); it was higher than average in prognostic biomarker and risk factor studies (P=.04) and lower in noninvasive hypoglycemia detection studies (P=.006), higher in studies published in medical versus engineering journals (P=.004), and higher in studies sharing any code of the ML pipeline versus not sharing (P=.003). The association between expert judgments and MI-CLAIM ratings was not significant. ConclusionsThe reporting quality of ML studies in the pediatric population with DM was generally low. Important details for clinicians, such as patient characteristics; comparison with the state-of-the-art solution; and model examination for valid, unbiased, and robust results, were often the weak points of reporting. To assess their clinical utility, the reporting standards of ML studies must evolve, and algorithms for this challenging population must become more transparent and replicable