How to conduct a systematic review and meta-analysis of prognostic model studies

Background: Prognostic models are typically developed to estimate the risk that an individual in a particular health state will develop a particular health outcome, to support (shared) decision making. Systematic reviews of prognostic model studies can help identify prognostic models that need to further be validated or are ready to be implemented in healthcare. Objectives: To provide a step-by-step guidance on how to conduct and read a systematic review of prognostic model studies and to provide an overview of methodology and guidance available for every step of the review progress. Sources: Published, peer-reviewed guidance articles. Content: We describe the following steps for conducting a systematic review of prognosis studies: 1) Developing the review question using the Population, Index model, Comparator model, Outcome(s), Timing, Setting format, 2) Searching and selection of articles, 3) Data extraction using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist, 4) Quality and risk of bias assessment using the Prediction model Risk Of Bias ASsessment (PROBAST) tool, 5) Analysing data and undertaking quantitative meta-analysis, and 6) Presenting summary of findings, interpreting results, and drawing conclusions. Guidance for each step is described and illustrated using a case study on prognostic models for patients with COVID-19. Implications: Guidance for conducting a systematic review of prognosis studies is available, but the implications of these reviews for clinical practice and further research highly depend on complete reporting of primary studies

Prognostic models for radiation-induced complications after radiotherapy in head and neck cancer patients

Objectives: This is a protocol for a Cochrane Review (prognosis). The objectives are as follows:. Primary objective The review question is “Which prognostic models are available to predict the risk of radiation-induced side effects after radiation exposure to patients with head and neck cancer, what is their quality, and what is their predictive performance?”. Investigation of sources of heterogeneity between studies We will assess sources of heterogeneity among the prognostic models developed in the eligible studies. The potential sources are study population (e.g. site/stage of cancer, the use of other treatment [surgery and chemotherapy]), predictors, definition and incidence of the predicted outcomes, and prediction horizons. If there are multiple validation studies for the same model, the same sources of between-study heterogeneity will be investigated

The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement

Many randomized controlled trials (RCTs) are biased and difficult to reproduce due to methodological flaws and poor reporting. There is increasing attention for responsible research practices and implementation of reporting guidelines, but whether these efforts have improved the methodological quality of RCTs (e.g., lower risk of bias) is unknown. We, therefore, mapped risk-of-bias trends over time in RCT publications in relation to journal and author characteristics. Meta-information of 176,620 RCTs published between 1966 and 2018 was extracted. The risk-of-bias probability (random sequence generation, allocation concealment, blinding of patients/personnel, and blinding of outcome assessment) was assessed using a risk-of-bias machine learning tool. This tool was simultaneously validated using 63,327 human risk-of-bias assessments obtained from 17,394 RCTs evaluated in the Cochrane Database of Systematic Reviews (CDSR). Moreover, RCT registration and CONSORT Statement reporting were assessed using automated searches. Publication characteristics included the number of authors, journal impact factor (JIF), and medical discipline. The annual number of published RCTs substantially increased over 4 decades, accompanied by increases in authors (5.2 to 7.8) and institutions (2.9 to 4.8). The risk of bias remained present in most RCTs but decreased over time for allocation concealment (63% to 51%), random sequence generation (57% to 36%), and blinding of outcome assessment (58% to 52%). Trial registration (37% to 47%) and the use of the CONSORT Statement (1% to 20%) also rapidly increased. In journals with a higher impact factor (>10), the risk of bias was consistently lower with higher levels of RCT registration and the use of the CONSORT Statement. Automated risk-of-bias predictions had accuracies above 70% for allocation concealment (70.7%), random sequence generation (72.1%), and blinding of patients/personnel (79.8%), but not for blinding of outcome assessment (62.7%). In conclusion, the likelihood of bias in RCTs has generally decreased over the last decades. This optimistic trend may be driven by increased knowledge augmented by mandatory trial registration and more stringent reporting guidelines and journal requirements. Nevertheless, relatively high probabilities of bias remain, particularly in journals with lower impact factors. This emphasizes that further improvement of RCT registration, conduct, and reporting is still urgently needed

Outcomes of Minimally Invasive Thyroid Surgery - A Systematic Review and Meta-Analysis

Purpose: Conventional thyroidectomy has been standard of care for surgical thyroid nodules. For cosmetic purposes different minimally invasive and remote-access surgical approaches have been developed. At present, the most used robotic and endoscopic thyroidectomy approaches are minimally invasive video assisted thyroidectomy (MIVAT), bilateral axillo-breast approach endoscopic thyroidectomy (BABA-ET), bilateral axillo-breast approach robotic thyroidectomy (BABA-RT), transoral endoscopic thyroidectomy via vestibular approach (TOETVA), retro-auricular endoscopic thyroidectomy (RA-ET), retro-auricular robotic thyroidectomy (RA-RT), gasless transaxillary endoscopic thyroidectomy (GTET) and robot assisted transaxillary surgery (RATS). The purpose of this systematic review was to evaluate whether minimally invasive techniques are not inferior to conventional thyroidectomy. Methods: A systematic search was conducted in Medline, Embase and Web of Science to identify original articles investigating operating time, length of hospital stay and complication rates regarding recurrent laryngeal nerve injury and hypocalcemia, of the different minimally invasive techniques. Results: Out of 569 identified manuscripts, 98 studies met the inclusion criteria. Most studies were retrospective in nature. The results of the systematic review varied. Thirty-one articles were included in the meta-analysis. Compared to the standard of care, the meta-analysis showed no significant difference in length of hospital stay, except a longer stay after BABA-ET. No significant difference in incidence of recurrent laryngeal nerve injury and hypocalcemia was seen. As expected, operating time was significantly longer for most minimally invasive techniques. Conclusions: This is the first comprehensive systematic review and meta-analysis comparing the eight most commonly used minimally invasive thyroid surgeries individually with standard of care. It can be concluded that minimally invasive techniques do not lead to more complications or longer hospital stay and are, therefore, not inferior to conventional thyroidectomy

Risk of bias assessments in individual participant data meta-analyses of test accuracy and prediction models:a review shows improvements are needed

OBJECTIVES: Risk of bias assessments are important in meta-analyses of both aggregate and individual participant data (IPD). There is limited evidence on whether and how risk of bias of included studies or datasets in IPD meta-analyses (IPDMAs) is assessed. We review how risk of bias is currently assessed, reported, and incorporated in IPDMAs of test accuracy and clinical prediction model studies and provide recommendations for improvement.STUDY DESIGN AND SETTING: We searched PubMed (January 2018-May 2020) to identify IPDMAs of test accuracy and prediction models, then elicited whether each IPDMA assessed risk of bias of included studies and, if so, how assessments were reported and subsequently incorporated into the IPDMAs.RESULTS: Forty-nine IPDMAs were included. Nineteen of 27 (70%) test accuracy IPDMAs assessed risk of bias, compared to 5 of 22 (23%) prediction model IPDMAs. Seventeen of 19 (89%) test accuracy IPDMAs used Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2), but no tool was used consistently among prediction model IPDMAs. Of IPDMAs assessing risk of bias, 7 (37%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided details on the information sources (e.g., the original manuscript, IPD, primary investigators) used to inform judgments, and 4 (21%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided information or whether assessments were done before or after obtaining the IPD of the included studies or datasets. Of all included IPDMAs, only seven test accuracy IPDMAs (26%) and one prediction model IPDMA (5%) incorporated risk of bias assessments into their meta-analyses. For future IPDMA projects, we provide guidance on how to adapt tools such as Prediction model Risk Of Bias ASsessment Tool (for prediction models) and QUADAS-2 (for test accuracy) to assess risk of bias of included primary studies and their IPD.CONCLUSION: Risk of bias assessments and their reporting need to be improved in IPDMAs of test accuracy and, especially, prediction model studies. Using recommended tools, both before and after IPD are obtained, will address this.</p

Editor's Choice – Prognostic Factors and Models to Predict Mortality Outcomes in Patients with Peripheral Arterial Disease: A Systematic Review

Objective: Predicting adverse outcomes in patients with peripheral arterial disease (PAD) is a complex task owing to the heterogeneity in patient and disease characteristics. This systematic review aimed to identify prognostic factors and prognostic models to predict mortality outcomes in patients with PAD Fontaine stage I – III or Rutherford category 0 – 4. Data Sources: PubMed, Embase, and Cochrane Database of Systematic Reviews were searched to identify studies examining individual prognostic factors or studies aiming to develop or validate a prognostic model for mortality outcomes in patients with PAD. Review Methods: Information on study design, patient population, prognostic factors, and prognostic model characteristics was extracted, and risk of bias was evaluated. Results: Sixty nine studies investigated prognostic factors for mortality outcomes in PAD. Over 80 single prognostic factors were identified, with age as a predictor of death in most of the studies. Other common factors included sex, diabetes, and smoking status. Six studies had low risk of bias in all domains, and the remainder had an unclear or high risk of bias in at least one domain. Eight studies developed or validated a prognostic model. All models included age in their primary model, but not sex. All studies had similar discrimination levels of > 70%. Five of the studies on prognostic models had an overall high risk of bias, whereas two studies had an overall unclear risk of bias. Conclusion: This systematic review shows that a large number of prognostic studies have been published, with heterogeneity in patient populations, outcomes, and risk of bias. Factors such as sex, age, diabetes, hypertension, and smoking are significant in predicting mortality risk among patients with PAD Fontaine stage I – III or Rutherford category 0 – 4

Diagnostic yield and safety of navigation bronchoscopy: A systematic review and meta-analysis

Background: Navigation bronchoscopy has seen rapid development in the past decade in terms of new navigation techniques and multi-modality approaches utilizing different techniques and tools. This systematic review analyses the diagnostic yield and safety of navigation bronchoscopy for the diagnosis of peripheral pulmonary nodules suspected of lung cancer. Methods: An extensive search was performed in Embase, Medline and Cochrane CENTRAL in May 2022. Eligible studies used cone-beam CT-guided navigation (CBCT), electromagnetic navigation (EMN), robotic navigation (RB) or virtual bronchoscopy (VB) as the primary navigation technique. Primary outcomes were diagnostic yield and adverse events. Quality of studies was assessed using QUADAS-2. Random effects meta-analysis was performed, with subgroup analyses for different navigation techniques, newer versus older techniques, nodule size, publication year, and strictness of diagnostic yield definition. Explorative analyses of subgroups reported by studies was performed for nodule size and bronchus sign. Results: A total of 95 studies (n = 10,381 patients; n = 10,682 nodules) were included. The majority (n = 63; 66.3%) had high risk of bias or applicability concerns in at least one QUADAS-2 domain. Summary diagnostic yield was 70.9% (95%-CI 68.4%-73.2%). Overall pneumothorax rate was 2.5%. Newer navigation techniques using advanced imaging and/or robotics (CBCT, RB, tomosynthesis guided EMN; n = 24 studies) had a statistically significant higher diagnostic yield compared to longer established techniques (EMN, VB; n = 82 studies): 77.5% (95%-CI 74.7%-80.1%) vs 68.8% (95%-CI 65.9%-71.6%) (p < 0.001). Explorative subgroup analyses showed that larger nodule size and bronchus sign presence were associated with a statistically significant higher diagnostic yield. Other subgroup analyses showed no significant differences. Conclusion: Navigation bronchoscopy is a safe procedure, with the potential for high diagnostic yield, in particular using newer techniques such as RB, CBCT and tomosynthesis-guided EMN. Studies showed a large amount of heterogeneity, making comparisons difficult. Standardized definitions for outcomes with relevant clinical context will improve future comparability

Effectiveness of FeNO-guided treatment in adult asthma patients: A systematic review and meta-analysis

Objective: Asthma control is generally monitored by assessing symptoms and lung function. However, optimal treatment is also dependent on the type and extent of airway inflammation. Fraction of exhaled Nitric Oxide (FeNO) is a noninvasive biomarker of type 2 airway inflammation, but its effectiveness in guiding asthma treatment remains disputed. We performed a systematic review and meta-analysis to obtain summary estimates of the effectiveness of FeNO-guided asthma treatment. Design: We updated a Cochrane systematic review from 2016. Cochrane Risk of Bias tool was used to assess risk of bias. Inverse-variance random-effects meta-analysis was performed. Certainty of evidence was assessed using GRADE. Subgroup analyses were performed based on asthma severity, asthma control, allergy/atopy, pregnancy and obesity. Data Sources: The Cochrane Airways Group Trials Register was searched on 9 May 2023. Eligibility Criteria: We included randomized controlled trials (RCTs) comparing the effectiveness of a FeNO-guided treatment versus usual (symptom-guided) treatment in adult asthma patients. Results: We included 12 RCTs (2,116 patients), all showing high or unclear risk of bias in at least one domain. Five RCTs reported support from a FeNO manufacturer. FeNO-guided treatment probably reduces the number of patients having ≥1 exacerbation (OR = 0.61; 95%CI 0.44 to 0.83; six RCTs; GRADE moderate certainty) and exacerbation rate (RR = 0.67; 95%CI 0.54 to 0.82; six RCTs; moderate certainty), and may slightly improve Asthma Control Questionnaire score (MD = −0.10; 95%CI −0.18 to −0.02, six RCTs; low certainty), however, this change is unlikely to be clinically important. An effect on severe exacerbations, quality of life, FEV1, treatment dosage and FeNO values could not be demonstrated. There were no indications that effectiveness is different in subgroups of patients, although evidence for subgroup analysis was limited. Conclusions: FeNO-guided asthma treatment probably results in fewer exacerbations but may not have clinically important effects on other asthma outcomes

Completeness of reporting of clinical prediction models developed using supervised machine learning: A systematic review

ABSTRACTObjectiveWhile many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement.Study design and settingWe included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields (PROSPERO, CRD42019161764). We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies (www.TRIPOD-statement.org). We measured the overall adherence per article and per TRIPOD item.ResultsOur search identified 24 814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0-46.4) of TRIPOD items. No articles fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model’s predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3).ConclusionSimilar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste.What is new?Key findings: Similar to prediction model studies developed using regression techniques, machine learning (ML)-based prediction model studies adhered poorly to the TRIPOD statement, the current standard reporting guideline.What this adds to what is known? In addition to efforts to improve the completeness of reporting in ML-based prediction model studies, an extension of TRIPOD for these type of studies is needed.What is the implication, what should change now? While TRIPOD-AI is under development, we urge authors to follow the recommendations of the TRIPOD statement to improve the completeness of reporting and reduce potential research waste of ML-based prediction model studies.</jats:sec

Pediatric Achalasia: Diagnosis, Management and Follow-Up in the Netherlands

Su206