The judgement of biases included in the category “other bias” in Cochrane systematic reviews of interventions: a systematic survey

Abstract Background Clinical decisions are made based on Cochrane reviews, but the implementation of results of evidence syntheses such as Cochrane reviews is problematic if the evidence is not prepared consistently. All systematic reviews should assess the risk of bias (RoB) in included studies, and in Cochrane reviews, this is done by using Cochrane RoB tool. However, the tool is not necessarily applied according to the instructions. In this study, we aimed to determine the types of bias and their corresponding judgements noted in the ‘other bias’ domain of Cochrane RoB tool. Methods We analyzed Cochrane reviews that included randomized controlled trials (RCTs) and extracted data regarding ‘other bias’ from the RoB table and accompanying support for the judgment. We categorized different types of other bias. Results We analyzed 768 Cochrane reviews that included 11,369 RCTs. There were 602 (78%) Cochrane reviews that had ‘other bias’ domain in the RoB tool, and they included a total of 7811 RCTs. In the RoB table of 337 Cochrane reviews for at least one of the included trials it was indicated that no other bias was found and supporting explanations were inconsistently judged as low, unclear or high RoB. In the 524 Cochrane reviews that described various sources of other bias, there were 5762 individual types of explanations which we categorized into 31 groups. The judgments of the same supporting explanations were highly inconsistent. We found numerous other inconsistencies in reporting of sources of other bias in Cochrane reviews. Conclusion Cochrane authors mention a wide range of sources of other bias in the RoB tool and they inconsistently judge the same supporting explanations. Inconsistency in appraising risk of other bias hinders reliability and comparability of Cochrane systematic reviews. Discrepant and erroneous judgments of bias in evidence synthesis may hinder implementation of evidence in routine clinical practice and reduce confidence in otherwise trustworthy sources of information. These results can help authors of Cochrane and non-Cochrane reviews to gain insight into various sources of other bias that can be found in trials, and also to help them avoid mistakes that were recognized in published Cochrane reviews

APACHE - Review of current KPIs and proposal for new ones

This Report1 is part of a project that has received funding from the SESAR Joint Undertaking under grant agreement No 699338 under European Union’s Horizon 2020 research and innovation programme.The main objective of this report is to review the current KPIs and PIs used by the SESAR, Performance Review Body (PRB) and other relevant institutions and to propose new PIs which could be measured using the new framework proposed by the APACHE project. For this purpose, past reports and guidance material is reviewed in order to determine which KPAs are covered and specific KPIs/PIs used in Europe. Apart from that, relevant ICAO and CANSO documents are also reviewed, among others. Special attention is given to SESAR Performance Framework which is quite specific in its purpose and perspective as it aims to estimate the performance benefits of SESAR solutions before the execution phase of operations, which is in line with the APACHE project as it focuses mainly on Pre-OPS ATM performance assessment. Based on current KPIs/PIs review and objectives of ATM performance assessment framework from WP2, a set of novel PIs which could be measured using new framework introduced by the APACHE project are proposed in collaboration with the SJU and the PRB considering their valuable feedback. From this assessment, the APACHE System will implement a total of 42 new (or enhanced) performance indicators (25 main indicators and 17 variants)

APACHE - Review of current KPIs and proposal for new ones

This Report1 is part of a project that has received funding from the SESAR Joint Undertaking under grant agreement No 699338 under European Union’s Horizon 2020 research and innovation programme.The main objective of this report is to review the current KPIs and PIs used by the SESAR, Performance Review Body (PRB) and other relevant institutions and to propose new PIs which could be measured using the new framework proposed by the APACHE project. For this purpose, past reports and guidance material is reviewed in order to determine which KPAs are covered and specific KPIs/PIs used in Europe. Apart from that, relevant ICAO and CANSO documents are also reviewed, among others. Special attention is given to SESAR Performance Framework which is quite specific in its purpose and perspective as it aims to estimate the performance benefits of SESAR solutions before the execution phase of operations, which is in line with the APACHE project as it focuses mainly on Pre-OPS ATM performance assessment. Based on current KPIs/PIs review and objectives of ATM performance assessment framework from WP2, a set of novel PIs which could be measured using new framework introduced by the APACHE project are proposed in collaboration with the SJU and the PRB considering their valuable feedback. From this assessment, the APACHE System will implement a total of 42 new (or enhanced) performance indicators (25 main indicators and 17 variants).Postprint (published version

Assessments of attrition bias in Cochrane systematic reviews are highly inconsistent and thus hindering trial comparability

Abstract Background An important part of the systematic review methodology is appraisal of the risk of bias in included studies. Cochrane systematic reviews are considered golden standard regarding systematic review methodology, but Cochrane’s instructions for assessing risk of attrition bias are vague, which may lead to inconsistencies in authors’ assessments. The aim of this study was to analyze consistency of judgments and support for judgments of attrition bias in Cochrane reviews of interventions published in the Cochrane Database of Systematic Reviews (CDSR). Methods We analyzed Cochrane reviews published from July 2015 to June 2016 in the CDSR. We extracted data on number of included trials, judgment of attrition risk of bias for each included trial (low, unclear or high) and accompanying support for the judgment (supporting explanation). We also assessed how many Cochrane reviews had different judgments for the same supporting explanations. Results In the main analysis we included 10,292 judgments and supporting explanations for attrition bias from 729 Cochrane reviews. We categorized supporting explanations for those judgments into four categories and we found that most of the supporting explanations were unclear. Numerical indicators for percent of attrition, as well as statistics related to attrition were judged very differently. One third of Cochrane review authors had more than one category of supporting explanation; some had up to four different categories. Inconsistencies were found even with the number of judgments, names of risk of bias domains and different judgments for the same supporting explanations in the same Cochrane review. Conclusion We found very high inconsistency in methods of appraising risk of attrition bias in recent Cochrane reviews. Systematic review authors need clear guidance about different categories they should assess and judgments for those explanations. Clear instructions about appraising risk of attrition bias will improve reliability of the Cochrane’s risk of bias tool, help authors in making decisions about risk of bias and help in making reliable decisions in healthcare

APACHE - Scope and definition of the concept of operations for the project

The APACHE project proposes a new approach to assess European ATM performance based on simulation, optimization and performance assessment tools that will be able to capture the complex interdependencies between KPAs at different modelling scales. This document is the baseline for the Project and defines the operational context which encompasses the evaluation studies that will be carried out in the Project. The baseline and SESAR 2020 target operations definition within the context of APACHE will permit to settle the scope of the project and trace it within the context of the SESAR programme. This traceability is carried out as per SESAR solutions to be assessed, that could be assessed or that enable other solutions to be assessed within the Project.Postprint (published version

APACHE - Scope and definition of the concept of operations for the project

The APACHE project proposes a new approach to assess European ATM performance based on simulation, optimization and performance assessment tools that will be able to capture the complex interdependencies between KPAs at different modelling scales. This document is the baseline for the Project and defines the operational context which encompasses the evaluation studies that will be carried out in the Project. The baseline and SESAR 2020 target operations definition within the context of APACHE will permit to settle the scope of the project and trace it within the context of the SESAR programme. This traceability is carried out as per SESAR solutions to be assessed, that could be assessed or that enable other solutions to be assessed within the Project

APACHE - Scope and definition of the concept of operations for the project

The APACHE project proposes a new approach to assess European ATM performance based on simulation, optimization and performance assessment tools that will be able to capture the complex interdependencies between KPAs at different modelling scales. This document is the baseline for the Project and defines the operational context which encompasses the evaluation studies that will be carried out in the Project. The baseline and SESAR 2020 target operations definition within the context of APACHE will permit to settle the scope of the project and trace it within the context of the SESAR programme. This traceability is carried out as per SESAR solutions to be assessed, that could be assessed or that enable other solutions to be assessed within the Project