The two study protocols are published in this issue Colorectal Disease: FALCON, a

multicentre randomised controlled trial of strategies to reduce surgical site infection, and

AFAR, a predictive model of atrial fibrillation after colonic resection. Both are exemplars of excellent research design that surgeon researchers should seek to emulate. Trial statisticians

were involved at an early stage and the protocols have been through several rounds of peer

review by trial methodologists, prior to being funded by the National Institute for Health

Research (NIHR). In this article we address the important question of sample size

calculations and how they should be approached for these very different forms of study

Fiorentino, F

Treasure, T

English

UCL Discovery

Sample size calculations for randomised controlled trials and for prediction models 1  2 Francesca Fiorentino 3 Department of Surgery and Cancer, Imperial College London, London UK 4  5 Tom Treasure* 6 Clinical Operational Research Unit, University College London 7  8  9 Corresponding author 10 Tom Treasure: tom.treasure@gmail.com 11  12  13  14  15  16  17 There was no funding for this work. 18 The authors have no conflicts of interest to declare. 19   20 Abstract 21  22 The two study protocols are published in this issue Colorectal Disease: FALCON, a 23 multicentre randomised controlled trial of strategies to reduce surgical site infection, and 24 AFAR, a predictive model of atrial fibrillation after colonic resection. Both are exemplars of 25 excellent research design that surgeon researchers should seek to emulate. Trial statisticians 26 were involved at an early stage and the protocols have been through several rounds of peer 27 review by trial methodologists, prior to being funded by the National Institute for Health 28 Research (NIHR). In this article we address the important question of sample size 29 calculations and how they should be approached for these very different forms of study. 30  31  32  33 Main Text 34  35 Most surgical procedures came into practice without randomised trials because, against a 36 well-known experience of clinical outcomes over many years, an appropriate and well 37 conducted operation was seen to make a dramatic and lasting difference. For example, 38 Thomas’s splint only had to be seen in use for injured farmers in north Wales, and then 39 soldiers in the 1914-18 war, to become universally adopted. The relief of pain in the hours 40 and days after injury was evident, followed by recovery to walk on legs of matching length, 41 with both feet pointing forward. To generalise that process of deduction, the features that 42 indicate that an RCT is not needed are a close temporal and mechanistic relationship between 43 the intervention and the effect, resulting in a large and sustained benefit.(1) The Thomas’s 44 splint became the standard initial treatment, applicable to the large majority of patients with 45 femoral fracture.  46  47 In contrast, lung metastasectomy is carried out in fewer than one in thirty of the patients who 48 have lung metastases.(2) The outcome of importance is survival. For lung metastasectomy, 49 results are usually given as survival rate, usually at an interval of five-years, but there are too 50 many factors and uncertainties to conclude that metastasectomy has a survival benefit by 51 observation alone.(3) 52  53 Calculating the sample size for a randomised controlled trial 54 It is wasteful of time and effort to embark on a study that is not large enough to provide a 55 conclusive answer, or so large as to be wasteful of effort and resources.(4)  To calculate a 56 sample size, the statistician needs to know what is (a) the outcome of importance, (b) the 57 outcome measure and (c) the clinically meaningful effect size.  58   59 For lung metastasectomy, survival beyond five years was the only outcome reported in the 51 60 follow-up studies found in a systematic review (5) so for our first illustration (a) survival is 61 the outcome of importance. Survival of ~40% at five years has been consistently reported and 62 was confirmed in a meta-analysis including 2925 patients. (6) For the illustration we will 63 identify the survival rate at 5-years to be the outcome measure (b). The effect size depends on 64 what would be the survival without metastasectomy. The US Society of Thoracic Surgeons 65 based its recommendations on a consensus assumption of zero survival, but for this 66 illustration will use the more cautious “worse than 5%” suggested by the authors of the meta-67 analysis. Then (c) is the absolute difference between 40% and 5%, the effect size of 35%.  68  69 The surgeons need to agree with statistician the value of alpha—the probability of a false 70 positive—usually set at 5% and hence the familiar P<0.05. The value of beta—the probability 71 of a false negative—is usually set at 20% or more cautiously 10%. Power is 1-beta so in 72 percentage terms these are expressed as 80% or 90%, that is the power to avoid a false 73 negative. Given these estimates a statistician can generate Table 1. This is for 1:1 74 randomisation and shows that 44 patients (22 in each arm) would provide 80% power for a 75 two-sample proportion test. There are likely to be patients lost to follow up, so the target 76 recruitment might be set at 50.  77  78 In cancer trials it is usual to use time to death (overall survival) or cancer progression 79 (progression free survival) for (b) the outcome measure. The statistical test used for the 80 sample size calculation is the two-sample comparison of survivor functions (log-rank test). 81 The same assumptions can be used to do the calculation, but the statistical method takes into 82 account the time of the event, death. It captures more information than a simple count of 5-83 year survivors, so it requires commensurately fewer patients. Using the log rank test, the 84 statistician can produce Table 2. Randomisation is still 1:1 and shows that 36 patients (18 in 85 each arm) would provide 80% power with a two-sample survivor function test. A total of42 86 patients would allow for loss to follow up.  87  88 In the discussion between the investigators and the statistician, all should be alert to the 89 possibility of “back calculation”. The surgeons know the number of patients available for 90 recruitment and can tweak the effect size to give an achievable number of randomised 91 patients. In the case of lung metastasectomy the consensus assumption of zero survival(7) had 92 for years ruled out the possibility of randomisation at all; there was no prospect of equipoise. 93 Also, it conveniently attributed all the credit for survival to the effect of the operation and 94 trumps any likely effect from chemotherapy. 95  96 Tables 1 and 2 illustrate the principle, but it is not how the conversation with the statistician 97 went in the case of the PulMiCC trial. The investigators had reason to believe that patients 98 eligible for metastasectomy had better survival than was widely assumed. This came from a 99 comparative study in 1980(8) and a modelling study on cancer registry data in 2006.(9) Both 100 suggested the possibility that metastasectomy makes a much smaller difference to survival 101 than assumed. Knowing that, the statistician asked what was the smallest clinically 102 meaningful difference in the five-year survival that might justify lung metastasectomy. A 103 difference from 40% survival in the treated down to 30% survival in the control was 104 suggested (10% difference). Table 3 shows the calculation using a two-sample survivor 105 function test.  106  107 As we said, the actual sample size calculation may be much more complicated. In fact, the 108 PulMiCC trial was powered for non-inferiority of leaving the metastases unresected using 109 time to event analysis.(10) With this smaller difference (40%  and 30%)  the numbers needed 110 to power the study were commensurately higher, and in the event not achievable due to the 111 tenacity with which cancer teams held on to the near zero assumption and its implications. 112 (11, 12) It is also important to remember that for the sample size calculation it was important 113 to be realistic at the planning stage, The assumptions are replaced by findings once the data 114 are in, and the prior power calculation plays no part in the analysis or interpretation of the 115 results.(13) 116  117 It may be important to not rely on randomisation, but to ensure that there is a balance in 118 prognostic factors between the randomized groups, particularly if these factors might create 119 differences of a magnitude that compete with the treatment effect (confounding factors). For 120 example, obesity in studies of surgical site infection(14) which might be relevant in 121 FALCON.(15) In the case of the PulMiCC trial the unfavourable features were more than one 122 metastasis, liver involvement, carcinoembryonic antigen elevation and shorter interval since 123 the primary resection. In large drug trials this process is done by stratification but in trials of 124 limited size an alternative is minimisation which adjusts the probability of a patient being 125 assigned to one or other arm in order to achieve balance between the groups in the known 126 factors, relying on randomisation to balance the unknown confounders.(16, 17) It is essential 127 that this is done by a strict algorithm out of sight of anyone involved in the trial. 128  129  130 Prediction models 131  132 Prediction models are used for investigating patient outcomes in relation to patient and 133 disease characteristics. They may be of use in surgical practice and we give three examples. 134  135 1. In the AFAR study(18) the adverse outcome to be “predicted” is the onset of new 136 atrial fibrillation during the recovery period. Patients in the stratum more likely to 137 have this problem can then have further planned screening or prophylactic 138 approaches. The model is intended to target more costly and labour intensive methods 139 to where they will achieve the greatest benefit for patients. 140 2. A predictive model has been developed to risk adjust postoperative mortality among 141 patients having of colorectal cancer.(19) It allows fair comparisons to be made 142 between hospitals, clinical teams and individual surgeons. Implementation of public 143 reporting in 2013 was followed by a fall in the observed surgical mortality. The model 144 allowed this to be interpreted as a real reduction in mortality without risk avoidance. 145 (20) 146 3. A third example is to select patients for surgery by gaining insights into their 147 likelihood of death or survival after surgery. We will return to an unsatisfactory 148 example in the development of a model with this purpose as a cautionary tale.(21) 149  150 A standard approach is to use a “training” dataset and the model is then tested with a separate 151 “validation” dataset which has been held back for the purpose. Following the same principle 152 as the sample size calculation the statistician must be provide with the best available data, 153 informed estimates of as yet unquantified factors and what outcome would be useful. The 154 outcome can be a continuous scale, categorical or estimated survival (time to event). 155  156 The model developed by Walker, Finan and van der Meulen(22) used internal validation and 157 is more sophisticated than can be described here but it illustrates the power of a collaborative 158 effort with data available on 62,314 patients in the National Bowel Cancer Audit and 159 collaboration with very highly skilled data analysts. Eight risk factors were included and 160 mortality was counted up to 90 days. This captures 50% more deaths, virtually all having a 161 relationship to treatment. The methods of “imputation” for missing data (and missingness is 162 inevitable) and validation were at a high level of expertise.  163  164 The tables are set up to illustrate the fewest of counts that might allow for a valid model. In 165 the case of survival (time to event outcome), the overall event rate and the mean follow-up 166 time need to be known. In the case of binary outcome, the outcome proportion expected 167 within the model development dataset, based on previous evidence.  168 Tables 4 and 5 give examples of sample size for prediction models of binary and survival 169 outcomes prediction models. 170  171 For less common disease or particular circumstances there may still be a desire to create 172 models to inform practice. A recent published example is of a scoring system to select 173 patients more likely to “benefit” from lung metastasectomy for sarcoma included 135 174 patients.(21) The scoring system has three parameters giving scores of 0-3. What can be 175 lauded in the report is that the authors provide the data. The figure is taken from their paper. 176 The well-used caution “correlation does not mean causation” can be applied. The more 177 important value of r2 is 0.144 indicating that the scores contribute very little, <15%, to the 178 prognosis. It is clear from the graphical display that the scores really do not discriminate 179 usefully between lengths of survival.  180  181 Sarcoma has a predilection for metastasising to the lungs and these patients are often young 182 so there is pressure to do something, anything, to help. The two longest survivors at over 12 183 years who scored 2/3 on the scale will be pointed out repeatedly on clinic visits, generating 184 confirmation bias. What will not be recalled is the harm done by operations of unproven 185 effectiveness on patients the larger number of patients not coming back to clinic. 186  187   188 Table 1 189 Power Alpha Effect size N 80% 0.05 0.35 44 90% 0.05 0.35 56  190 In the case of an effect size of 35% (0.35) and using a two-sample proportions test (Pearson's chi-squared test) 191 the variation in power and sample size can be seen, for the first scenario with assumed 35% survival gain from 192 metastasectomy over control.  193  194 Table2  195 alpha power N Expected events Hazard Ratio Survival Metastasectomy Survival no operation 80% 0.5 36 28 3.269 40% 5% 90% 0.5 48 38 3.269 40% 5%  196 A sample size calculation for the two-sample comparison of survivor functions (log-rank test) using the same 197 assumptions. 198  199 Table 3 200 Power Alpha N Expected events Hazard Ratio Survival Metastasectomy Survival no operation 80% 0.5 656 427 1.314 0.4 0.3 90% 0.5 880 571 1.314 0.4 0.3  201 A sample size calculation for the two-sample comparison of survivor functions (log-rank test) raising the control 202 estimate from 5% to 30%. 203  204   205 Table 4 206 Predictor parameters Outcome prevalence Minimum Sample Size Number of events 10 10% 348 35 20 10% 695 70 30 10% 1042 105     10 40% 369 148 20 40% 519 208 30 40% 778 312  207  208  209  210 Example for a binary outcome where the expected outcome proportion is 10% or 40%  with model parameters 211 10, 20 and 30 in Table 4. 212  213  214  215 Table 5 216 Predictor parameters Overall event rate  Minimum Sample Size Number of outcome events 10 6.5% 1715 231 20 6.5% 3429 462 30 6.5% 5143 692  217  218  219  220  221 Example for a survival outcome where the mean follow-up is 2 years, the overall event rate is 0.065 and the 222 time for model prediction is 2 years with model parameters 10, 20 and 30 in Table 5. 223  224   225  226  227  228   229  230  231  232  233 1. Glasziou P, Chalmers I, Rawlins M, McCulloch P. When are randomised trials unnecessary? 234 Picking signal from noise. BMJ. 2007;334(7589):349-51. 235 2. Fenton H, Finan PJ, Milton R, Shackcloth M, Taylor J, Treasure T, et al. National variation in 236 pulmonary metastasectomy for colorectal cancer. Colorectal Disease. 2020;Accepted for Publication. 237 3. Milosevic M, Edwards J, Tsang D, Dunning J, Shackcloth M, Batchelor T, et al. Pulmonary 238 Metastasectomy in Colorectal Cancer: updated analysis of 93 randomized patients - control survival 239 is much better than previously assumed. Colorectal Dis. 2020;22(10):1314-24. 240 4. Glasziou P, Chalmers I. Research waste is still a scandal-an essay by Paul Glasziou and Iain 241 Chalmers. BMJ. 2018;363:k4645. 242 5. Fiorentino F, Hunt I, Teoh K, Treasure T, Utley M. Pulmonary metastasectomy in colorectal 243 cancer: a systematic review and quantitative synthesis. J R Soc Med. 2010;103(2):60-6. 244 6. Gonzalez M, Poncet A, Combescure C, Robert J, Ris HB, Gervaz P. Risk factors for survival 245 after lung metastasectomy in colorectal cancer patients: a systematic review and meta-analysis. Ann 246 Surg Oncol. 2013;20(2):572-9. 247 7. Handy JR, Bremner RM, Crocenzi TS, Detterbeck FC, Fernando HC, Fidias PM, et al. Expert 248 Consensus Document on Pulmonary Metastasectomy. Ann Thorac Surg. 2019;107(2):631-49. 249 8. Aberg T, Malmberg KA, Nilsson B, Nou E. The effect of metastasectomy: fact or fiction? Ann 250 Thorac Surg. 1980;30(4):378-84. 251 9. Utley M, Treasure T, Linklater K, Moller H. Better out than in?  The resection of pulmonary 252 metastases from colorectal tumours. In: Xie X, Lorca F, Marcon E, editors. Operations Research for 253 Health Care Engineering: Proceedings of the 33rd International Conference on Operational Research 254 Applied to Health Services. Saint-Etienne: Publications de l'Universitaire de Saint-Etienne; 2008. p. 255 493-500. 256 10. Milosevic M, Edwards J, Tsang D, Dunning J, Shackcloth M, Batchelor T, et al. Pulmonary 257 Metastasectomy in Colorectal Cancer (PulMiCC): Updated analysis of 93 randomised patients - 258 control survival is much better than previously assumed. Colorectal Dis. 259 2020;https://onlinelibrary.wiley.com/doi/full/10.1111/codi.15113. 260 11. Treasure T, Farewell V, Macbeth F, Monson K, Williams NR, Brew-Graves C, et al. Pulmonary 261 Metastasectomy versus Continued Active Monitoring in Colorectal Cancer (PulMiCC): a multicentre 262 randomised clinical trial. Trials. 2019;20(1):718. 263 12. Macbeth F, Fallowfield L. The myth of pulmonary metastasectomy. Br J Cancer. 264 2020;123(4):499-500. 265 13. Cox DR. Planning of Experiments: Wiley; 1958. 266 14. Wilson AP, Livesey SA, Treasure T, Gruneberg RN, Sturridge MF. Factors predisposing to 267 wound infection in cardiac surgery. A prospective study of 517 patients. Eur J Cardiothorac Surg. 268 1987;1(3):158-64. 269 15. Nepogodiev D, Bahngu A. Pragmatic multicentre factorial randomised controlled trial testing 270 measures to reduce surgical site infection in low- and middle-income countries: study protocol of 271 the FALCON trial. Colorectal Disease. 2021;IN PRESS. 272 16. Treasure T, MacRae KD. Minimisation: the platinum standard for trials? BMJ. 273 1998;317(7155):362-3. 274 17. Altman DG, Bland JM. Treatment allocation by minimisation. BMJ. 2005;330(7495):843. 275 18. Lee MJ, Hawkins DJ, Bradburn MJ, Lee J, Brown SR, Wilson MJ. Atrial Fibrillation After 276 Resection (AFAR) A progress III Study. Colorectal Disease. 2021;IN PRESS. 277 19. Walker K, Finan PJ, van der Meulen JH. Model for risk adjustment of postoperative mortality 278 in patients with colorectal cancer. Br J Surg. 2015;102(3):269-80. 279 20. Vallance AE, Fearnhead NS, Kuryba A, Hill J, Maxwell-Armstrong C, Braun M, et al. Effect of 280 public reporting of surgeons' outcomes on patient selection, "gaming," and mortality in colorectal 281 cancer surgery in England: population based cohort study. BMJ. 2018;361:k1581. 282 21. Yamamoto H, Yamamoto H, Soh J. A Simple Prognostic Benefit Scoring System for Sarcoma 283 Patients with Pulmonary Metastases: Sarcoma Lung Metastasis Score. Annals of Surgical Oncology. 284 2020. 285 22. Vallance AE, Van Der Meulen J, Kuryba A, Braun M, Jayne DG, Hill J, et al. Socioeconomic 286 differences in selection for liver resection in metastatic colorectal cancer and the impact on survival. 287 Eur J Surg Oncol. 2018;44(10):1588-94. 288  289 

Sample size calculations for randomized controlled trials and for prediction models

https://discovery.ucl.ac.uk/10125600/1/Fiorention%20and%20Treasure%202021%20Colorectal%20Disease.pdf

Sample size calculations for randomized controlled trials and for prediction models

Abstract

Similar works

Full text

Available Versions

UCL Discovery