What is a pilot or feasibility study? A review of current practice and editorial policy

Background: In 2004, a review of pilot studies published in seven major medical journals during 2000-01 recommended that the statistical analysis of such studies should be either mainly descriptive or focus on sample size estimation, while results from hypothesis testing must be interpreted with caution. We revisited these journals to see whether the subsequent recommendations have changed the practice of reporting pilot studies. We also conducted a survey to identify the methodological components in registered research studies which are described as 'pilot' or 'feasibility' studies. We extended this survey to grant-awarding bodies and editors of medical journals to discover their policies regarding the function and reporting of pilot studies. Methods: Papers from 2007-08 in seven medical journals were screened to retrieve published pilot studies. Reports of registered and completed studies on the UK Clinical Research Network (UKCRN) Portfolio database were retrieved and scrutinized. Guidance on the conduct and reporting of pilot studies was retrieved from the websites of three grant giving bodies and seven journal editors were canvassed. Results: 54 pilot or feasibility studies published in 2007-8 were found, of which 26 (48%) were pilot studies of interventions and the remainder feasibility studies. The majority incorporated hypothesis-testing (81%), a control arm (69%) and a randomization procedure (62%). Most (81%) pointed towards the need for further research. Only 8 out of 90 pilot studies identified by the earlier review led to subsequent main studies. Twelve studies which were interventional pilot/feasibility studies and which included testing of some component of the research process were identified through the UKCRN Portfolio database. There was no clear distinction in use of the terms 'pilot' and 'feasibility'. Five journal editors replied to our entreaty. In general they were loathe to publish studies described as 'pilot'. Conclusion: Pilot studies are still poorly reported, with inappropriate emphasis on hypothesis-testing. Authors should be aware of the different requirements of pilot studies, feasibility studies and main studies and report them appropriately. Authors should be explicit as to the purpose of a pilot study. The definitions of feasibility and pilot studies vary and we make proposals here to clarify terminology

Recurrence rates for SIDS - the importance of risk stratification

Objective: To investigate the importance of stratification by risk factors in computing the probability of a second SIDS in a family. Design: Simulation Study Background: The fact that a baby dies suddenly and unexpectedly means that there is a raised probability that the baby’s family have risk factors associated with Sudden Infant Death Syndrome (SIDS). Thus one cannot consider the risk of a subsequent death to be that of the general population. The Confidential Enquiry into Stillbirths and Deaths in Infancy (CESDI)6 identified three major social risk factors: smoking, age1, and unemployed/unwaged as major risk factors. It gave estimates of risk for families with different numbers of these risk factors. We investigate whether it is reasonable to assume that, conditional on these risk factors, the risk of a second event is independent of the risk of the first and as a consequence one can square the risks to get the risk of two SIDS in a family. We have used CESDI data to estimate the probability of a second SID in a family under different plausible scenarios of the prevalence of the risk factors. We have applied the model to make predictions in the Care of Next Infant (CONI) study7. Results: The model gave plausible predictions. The CONI study observed 18 second SIDS. Our model predicted 14 (95% prediction interval 7 to 21). Conclusion: When considering the risk of a subsequent SIDS in a family one should always take into account the known risk factors. If all risks have been identified, then conditional on these risks, the risk of two events is the product of the individual risks However for a given family we cannot quantify the magnitude of the increased risk because of other possible risk factors not accounted for in the model

Understanding FEV1 for the purpose of cystic fibrosis registry comparisons: Does bias in annual review FEV1 affect between-centre comparison within the UK? An analysis of registry data

Rationale, aims and objective: We previously demonstrated that annual review %FEV1 under-estimates lung health of adults with CF compared to %FEV1 captured during periods of clinical stability. This has implications in the comparisons against registries with encounter-based FEV1, such as the US. It is uncertain whether this bias affects between-centre comparison within the UK. Previous funnel plot analyses have identified variation in annual review %FEV1 according to centre size, hence we investigated whether paired differences between annual review and best %FEV1 also vary according to centre size. Methods: This registry analysis included 18 adult CF centres in the UK with ≥80% completeness for best FEV1 data in 2014. Mean discrepancy between annual review and best %FEV1 is a surrogate for the extent by which annual review %FEV1 underestimates lung health; and was plotted against centre size. A Local Polynomial Regression (LOESS) curve was used to explore the relationship between the two variables. An appropriate model is fitted based on the LOESS curve to determine the strength of relationship between discrepancies in %FEV1 and centre size. Results: There is an inverted U-shaped relationship between mean discrepancies in %FEV1 and centre size. A regression of the paired mean difference in %FEV1 against centre size showed a significant improvement in the goodness of fit for a quadratic model (R2 = 23.8% for a quadratic model compared with 0.4% for a linear one; p = 0.048 for the quadratic term). Conclusions: Annual review %FEV1 under-estimated lung health of adults from small and large centres in the UK to a greater extent compared to medium-sized centres. A plot of %FEV1 against centre size (e.g. funnel plot comparison) would be affected by systematic bias in annual review %FEV1. Therefore, annual review %FEV1 is an unreliable metric to compare health outcomes of adult CF centres within the UK

Rescue therapy within the UK Cystic Fibrosis registry: an exploration of the predictors of intravenous antibiotic use amongst adults with CF

Background and objective: Intravenous (i.v.) antibiotics are needed for rescue when preventative therapy fails to achieve stability among adults with cystic fibrosis (CF). Understanding the distribution of i.v. days can provide insight into the care that adults with CF need. We aim to determine the baseline characteristics that are associated with higher i.v. use, in particular to test the hypothesis that prior-year i.v. use is associated with future-year i.v. use. Methods: This is a cross-sectional analysis of the 2013–2014 UK CF registry data. Stepwise logistic regression was performed using current-year i.v. days as the dependent variable, and demographic variables including prior-year i.v. days as the covariates. Based on these results, study sample was divided into clinically meaningful subgroups using analysis similar to tree-based method. Results: Data were available for 4269 adults in 2013 and 4644 adults in 2014. Prior-year i.v. use was the strongest predictor for current-year i.v. use followed by forced expiratory volume in 1 s (FEV1). Adults with high prioryear i.v. use (>14 days) continued to require high levels of i.v., regardless of FEV1. Those with high prior-year i. v. use and FEV1 ≥70% had higher current-year i.v. days compared to adults with low prior-year i.v. use and FEV1 <40% (28 days, interquartile range (IQR): 11–41 days vs 14 days, IQR: 0–28 days; Mann–Whitney P-value <0.001 in 2013). Conclusion: CF people with prior high levels of rescue often continue to need high levels of rescue even if they have good FEV1. The reasons for this require further investigations

Acute effects of aircraft noise on cardiovascular admissions - an interrupted time-series analysis of a six-day closure of London Heathrow Airport caused by volcanic ash

Acute noise exposure may acutely increase blood pressure but the hypothesis that acute exposure to aircraft noise may trigger cardiovascular events has not been investigated. This study took advantage of a six-day closure of a major airport in April 2010 caused by volcanic ash to examine if there was a decrease in emergency cardiovascular hospital admissions during or immediately after the closure period, using an interrupted daily time-series study design. The population living within the 55 dB(A) noise contour was substantial at 0.7 million. The average daily admission count was 13.9 (SD 4.4). After adjustment for covariates, there was no evidence of a decreased risk of hospital admission from cardiovascular disease during the closure period (relative risk 0.97 (95% CI 0.75–1.26)). Using lags of 1–7 days gave similar results. Further studies are needed to investigate if transient aircraft noise exposure can trigger acute cardiovascular events

Exploring the implications of different approaches to estimate centre-level adherence using objective adherence data in an adult cystic fibrosis centre - a retrospective observational study

Background Accurate centre-level medication adherence measurement allows identification of highly performing CF centres, drives shared learning and informs quality improvement. Self-reported adherence is unreliable but data-logging nebulisers can capture objective data. However, adherence levels in current literature are limited by the use of agreed prescriptions and convenience sampling. In this single-centre retrospective study, we quantified the differences in centre-level adherence with different methods of calculating adherence (unadjusted vs normative adherence) and different data sampling frames (convenience sampling vs including difficult to obtain data). Methods Adherence data were objectively captured using I-neb® from 2013-2016 in Sheffield Adult CF Centre. Adults on non data-logging devices, on ivacaftor or with previous lung transplantation were excluded. Adherence was calculated based on agreed regimen (‘unadjusted adherence’) or minimum required regimen (‘normative adherence’). I-nebs® not brought to clinic were downloaded during home visits. Adults not on any inhaled therapy but with chronic Pseudomonas aeruginosa infection were included by counting their adherence as “0”. Results Of the 131 included adults, 126 provided I-neb® data. Calculating unadjusted adherence from I-nebs® brought to clinics resulted in the highest centre-level adherence (median 41.8% in 2013). Median adherence reduced after sequentially accounting for minimum required regimen (40.0% in 2013), I-nebs® not brought to clinics (32.9% in 2013) and adults not on any inhaled therapy (31.0% in 2013). Conclusions Different approaches of calculating adherence produced different adherence levels. Adherence levels based only on agreed regimen among adults who readily brought their nebulisers to clinics can over-estimate the effective adherence of CF centres

Understanding basic numbers : examples from Brexit, Covid and common medical conditions

The UK Brexit debate and the current Covid pandemic have been fertile grounds for people seeking poor use of statistics, and demonstrate a need to reiterate some basic principles of data presentation. Communicating basic numbers to convey the correct message is a vital skill for a public health professional but even basic numbers can be difficult to understand, and are susceptible to misuse. The first issue is how to understand ‘orphan’ numbers; numbers quoted without comparison or context. This leads on to the problems of understanding numbers as proportions and how to make comparisons using proportions. Percentages, and in particular percentage changes, are also a major source of misunderstanding and the baseline percentage should always be given. The use of relative risk can also convey the wrong message and should always be accompanied by a measure of absolute risk. Similarly, numbers needed to treat should also refer to baseline risks. Communicating numbers is often more effective using natural counts or frequencies rather than fractions or proportions, and using pictorial representations of proportions can also be effective. The paper will also examine the problems of using simple ratios to try and adjust one continuous variable by another in particular the use of the BMI and for standardising death rates by institution. The misuse of reporting occurs in primary sources such as academic papers, but even more so in secondary reporting sources such as general media reports. It is natural to try and convey complex messages using a single summary number, but there are assumptions behind these summaries that should be questioned. It is usually better to give the individual numbers rather than a ratio of them