Public availability and adherence to prespecified statistical analysis approaches was low in published randomized trials

BACKGROUND AND OBJECTIVE: Prespecification of statistical methods in clinical trial protocols and statistical analysis plans can help to deter bias from p-hacking but is only effective if the prespecified approach is made available. STUDY DESIGN AND SETTING: For 100 randomized trials published in 2018 and indexed in PubMed, we evaluated how often a prespecified statistical analysis approach for the trial's primary outcome was publicly available. For each trial with an available prespecified analysis, we compared this with the trial publication to identify whether there were unexplained discrepancies. RESULTS: Only 12 of 100 trials (12%) had a publicly available prespecified analysis approach for their primary outcome; this document was dated before recruitment began for only two trials. Of the 12 trials with an available prespecified analysis approach, 11 (92%) had one or more unexplained discrepancies. Only 4 of 100 trials (4%) stated that the statistician was blinded until the SAP was signed off, and only 10 of 100 (10%) stated the statistician was blinded until the database was locked. CONCLUSION: For most published trials, there is insufficient information available to determine whether the results may be subject to p-hacking. Where information was available, there were often unexplained discrepancies between the prespecified and final analysis methods

Relevant Accessible Sensitivity Analysis for Clinical Trials with Missing Data

The statistical analysis of longitudinal randomised controlled trials is frequently complicated by the occurrence of protocol deviations which result in incomplete datasets for analysis. However analysis is approached, an unverifiable assumption about the distribution of the unobserved post-deviation data must be made. In such circumstances it is consequently important to assess the robustness of the primary analysis of the trial to different credible assumptions about the distribution of the missing data. Reference based multiple imputation procedures have been proposed for contextually relevant sensitivity analysis of longitudinal trials. Differences between the mean and variance of observed and missing data are specified with qualitative reference to trial arms and multiple imputation is used for estimation and inference. The primary analysis model is retained in the sensitivity analysis to assess the impact of alternative sampling behaviour on the original planned analysis. Rubin's rules are used to combine the treatment effect and variance estimates across imputed datasets, however it is unclear precisely what an appropriate measure of variance is in this setting and how Rubin's variance formula relates to this. We begin by defining a lower bound for variance estimation in the reference based settings as the variance estimate we would obtain were we able to observe the deviation data under the postulated post-deviation data assumption. We show Rubin's variance estimate always exceeds this and moreover it approximately preserves the loss of information in the primary analysis. We also explore Rubin's variance estimate in the δ-adjusted sensitivity analysis setting and show that Rubin's variance formula preserves the loss of information in this context. Alongside, we develop a new Stata command “mimix" for implementation of reference based sensitivity analyses. We illustrate the relevance and accessibility of the proposed methods of sensitivity analysis using data from a chronic asthma trial and a study of peer review

Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomized trials

Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trial

An evaluation of inverse probability weighting using the propensity score for baseline covariate adjustment in smaller population randomised controlled trials with a continuous outcome

Background It is important to estimate the treatment effect of interest accurately and precisely within the analysis of randomised controlled trials. One way to increase precision in the estimate and thus improve the power for randomised trials with continuous outcomes is through adjustment for pre-specified prognostic baseline covariates. Typically covariate adjustment is conducted using regression analysis, however recently, Inverse Probability of Treatment Weighting (IPTW) using the propensity score has been proposed as an alternative method. For a continuous outcome it has been shown that the IPTW estimator has the same large sample statistical properties as that obtained via analysis of covariance. However the performance of IPTW has not been explored for smaller population trials (< 100 participants), where precise estimation of the treatment effect has potential for greater impact than in larger samples. Methods In this paper we explore the performance of the baseline adjusted treatment effect estimated using IPTW in smaller population trial settings. To do so we present a simulation study including a number of different trial scenarios with sample sizes ranging from 40 to 200 and adjustment for up to 6 covariates. We also re-analyse a paediatric eczema trial that includes 60 children. Results In the simulation study the performance of the IPTW variance estimator was sub-optimal with smaller sample sizes. The coverage of 95% CI’s was marginally below 95% for sample sizes < 150 and ≥ 100. For sample sizes < 100 the coverage of 95% CI’s was always significantly below 95% for all covariate settings. The minimum coverage obtained with IPTW was 89% with n = 40. In comparison, regression adjustment always resulted in 95% coverage. The analysis of the eczema trial confirmed discrepancies between the IPTW and regression estimators in a real life small population setting. Conclusions The IPTW variance estimator does not perform so well with small samples. Thus we caution against the use of IPTW in small sample settings when the sample size is less than 150 and particularly when sample size < 100

Barriers and facilitators to the recruitment of disabled people to clinical trials: a scoping review

Introduction Underrepresentation of disabled groups in clinical trials results in an inadequate evidence base for their clinical care, which drives health inequalities. This study aims to review and map the potential barriers and facilitators to the recruitment of disabled people in clinical trials to identify knowledge gaps and areas for further extensive research. The review addresses the question: ‘What are the barriers and facilitators to recruitment of disabled people to clinical trials?’. Methods The Joanna Briggs Institute (JBI) Scoping review guidelines were followed to complete the current scoping review. MEDLINE and EMBASE databases were searched via Ovid. The literature search was guided by a combination of four key concepts from the research question: (1) disabled populations, (2) patient recruitment, (3) barriers and facilitators, and (4) clinical trials. Papers discussing barriers and facilitators of all types were included. Papers that did not have at least one disabled group as their population were excluded. Data on study characteristics and identified barriers and facilitators were extracted. Identified barriers and facilitators were then synthesised according to common themes. Results The review included 56 eligible papers. The evidence on barriers and facilitators was largely sourced from Short Communications from Researcher Perspectives (N = 22) and Primary Quantitative Research (N = 17). Carer perspectives were rarely represented in articles. The most common disability types for the population of interest in the literature were neurological and psychiatric disabilities. A total of five emergent themes were determined across the barriers and facilitators. These were as follows: risk vs benefit assessment, design and management of recruitment protocol, balancing internal and external validity considerations, consent and ethics, and systemic factors. Conclusions Both barriers and facilitators were often highly specific to disability type and context. Assumptions should be minimised, and study design should prioritise principles of co-design and be informed by a data-driven assessment of needs for the study population. Person-centred approaches to consent that empower disabled people to exercise their right to choose should be adopted in inclusive practice. Implementing these recommendations stands to improve inclusive practices in clinical trial research, serving to produce a well-rounded and comprehensive evidence base

Eliminating ambiguous treatment effects using estimands

Most reported treatment effects in medical research studies are ambiguously defined, which can lead to misinterpretation of study results. This is because most studies do not attempt to describe what the treatment effect represents, and instead require readers to deduce this based on the reported statistical methods. However, this approach is fraught, as many methods provide counterintuitive results. For example, some methods include data from all patients, yet the resulting treatment effect applies only to a subset of patients, whereas other methods will exclude certain patients while results will apply to everyone. Additionally, some analyses provide estimates pertaining to hypothetical settings where patients never die or discontinue treatment. Herein we introduce estimands as a solution to the aforementioned problem. An estimand is a clear description of what the treatment effect represents, thus saving readers the necessity of trying to infer this from study methods and potentially getting it wrong. We provide examples of how estimands can remove ambiguity from reported treatment effects and describe their current use in practice. The crux of our argument is that readers should not have to infer what investigators are estimating; they should be told explicitly

Access to unpublished protocols and statistical analysis plans of randomised trials

Background: Access to protocols and statistical analysis plans (SAPs) increases the transparency of randomised trial by allowing readers to identify and interpret unplanned changes to study methods, however they are often not made publicly available. We sought to determine how often study investigators would share unavailable documents upon request. Methods: We used trials from two previously identified cohorts (cohort 1: 101 trials published in high impact factor journals between January and April of 2018; cohort 2: 100 trials published in June 2018 in journals indexed in PubMed) to determine whether study investigators would share unavailable protocols/SAPs upon request. We emailed corresponding authors of trials with no publicly available protocol or SAP up to four times. Results: Overall, 96 of 201 trials (48%) across the two cohorts had no publicly available protocol or SAP (11/101 high-impact cohort, 85/100 PubMed cohort). In total, 8/96 authors (8%) shared some trial documentation (protocol only [n = 5]; protocol and SAP [n = 1]; excerpt from protocol [n = 1]; research ethics application form [n = 1]). We received protocols for 6/96 trials (6%), and a SAP for 1/96 trial (1%). Seventy-three authors (76%) did not respond, 7 authors responded (7%) but declined to share a protocol or SAP, and eight email addresses were invalid (8%). A total of 329 emails were sent (an average of 41 emails for every trial which sent documentation). After emailing authors, the total number of trials with an available protocol increased by only 3%, from 52% in to 55%. Conclusions: Most study investigators did not share their unpublished protocols or SAPs upon direct request. Alternative strategies are needed to increase transparency of randomised trials and ensure access to protocols and SAPs

Treatment of pustular psoriasis with anakinra: a statistical analysis plan for stage 1 of an adaptive two-staged randomised placebo-controlled trial

Background Palmoplantar pustulosis (PPP) is a rare, chronic inflammatory skin disease. It is known to affect quality of life at a level comparable to that from major medical and psychiatric illness, yet current treatment options are remarkably limited. Recent evidence however suggests that interleukin-1 (IL-1) blockade with anakinra will deliver therapeutic benefit in PPP. Methods Anakinra for Pustular psoriasis: Response in a Controlled Trial (APRICOT) is a two-staged, adaptive, double-blind, randomised placebo-controlled trial which aims to test the hypothesis that IL-1 blockade with anakinra will deliver therapeutic benefit in PPP. During stage 1 a total of 24 patients will be randomised (1:1) to receive either placebo or anakinra. The two candidate primary outcomes are fresh pustule count (across palms and soles) and the Palmoplantar Pustulosis Area and Severity Index (PPPASI) score, recorded at baseline and at weeks 1, 4 and 8. Analysis at the end of stage 1 will compare treatment arms to ensure sufficient efficacy and safety in order to progress to stage 2. The primary outcome for stage 2 will also be identified following an assessment of the reliability and discriminative ability of fresh pustule count and PPPASI. The trial is powered to detect efficacy and will recruit an additional 40 patients in stage 2 (n = 64 in total). Analysis will follow the intention-to-treat principle and analyse patients as randomised. Discussion This manuscript describes the important features of the small population trial design for APRICOT and the pre-specified statistical analysis plan for stage 1. The statistical analysis plan has been developed prior to data extraction and in compliance with international guidelines. It will increase the transparency of the data analysis for the APRICOT trial. The findings of the trial will help to clarify the role of anakinra in the treatment of PPP

Knee surgery and its evidence base

Introduction Evidence driven orthopaedics is gaining prominence. It enables better management decisions and therefore better patient care. The aim of our study was to review a selection of the leading publications pertaining to knee surgery to assess changes in levels of evidence over a decade. Methods Articles from the years 2000 and 2010 in The Knee, the Journal of Arthroplasty, Knee Surgery, Sports Traumatology, Arthroscopy, the Journal of Bone and Joint Surgery (American Volume) and the Bone and Joint Journal were analysed and ranked according to guidelines from the Centre for Evidence-Based Medicine. The intervening years (2003, 2005 and 2007) were also analysed to further define the trend. Results The percentage of high level evidence (level I and II) studies increased albeit without reaching statistical significance. Following a significant downward trend, the latter part of the decade saw a major rise in levels of published evidence. The most frequent type of study was therapeutic. Conclusions Although the rise in levels of evidence across the decade was not statistically significant, there was a significant drop and then rise in these levels in the interim. It is therefore important that a further study is performed to assess longer-term trends. Recent developments have made clear that high quality evidence will be having an ever increasing influence on future orthopaedic practice. We suggest that journals implement compulsory declaration of a published study's level of evidence and that authors consider their study designs carefully to enhance the quality of available evidence

Reference based sensitivity analysis for longitudinal trials with protocol deviation via multiple imputation

Randomised controlled trials provide essential evidence for the evaluation of new and existing medical treatments. Unfortunately the statistical analysis is often complicated by the occurrence of protocol deviations, which mean we cannot always measure the intended outcomes for individuals who deviate, resulting in a missing data problem. In such settings, however one approaches the analysis, an untestable assumption about the distribution of the unobserved data must be made. To understand how far the results depend on these assumptions, the primary analysis should be supplemented by a range of sensitivity analyses, which explore how the conclusions vary over a range of different credible assumptions for the missing data. In this article we describe a new command, mimix, that can be used to perform reference based sensitivity analyses for randomised controlled trials with longitudinal quantitative outcome data, using the approach proposed by Carpenter, Roger, and Kenward (2013). Under this approach, we make qualitative assumptions about how individuals' missing outcomes relate to those observed in relevant groups in the trial, based on plausible clinical scenarios. Statistical analysis then proceeds using the method of multiple imputation