Survival analysis for AdVerse events with VarYing follow-up times (SAVVY): Rationale and statistical concept of a meta-analytic study

The assessment of safety is an important aspect of the evaluation of new therapies in clinical trials, with analyses of adverse events being an essential part of this. Standard methods for the analysis of adverse events such as the incidence proportion, i.e. the number of patients with a specific adverse event out of all patients in the treatment groups, do not account for both varying follow-up times and competing risks. Alternative approaches such as the Aalen-Johansen estimator of the cumulative incidence function have been suggested. Theoretical arguments and numerical evaluations support the application of these more advanced methodology, but as yet there is to our knowledge only insufficient empirical evidence whether these methods would lead to different conclusions in safety evaluations. The Survival analysis for AdVerse events with VarYing follow-up times (SAVVY) project strives to close this gap in evidence by conducting a meta-analytical study to assess the impact of the methodology on the conclusion of the safety assessment empirically. Here we present the rationale and statistical concept of the empirical study conducted as part of the SAVVY project. The statistical methods are presented in unified notation and examples of their implementation in R and SAS are provided

Survival analysis for AdVerse events with VarYing follow-up times (SAVVY) -- comparison of adverse event risks in randomized controlled trials

Analyses of adverse events (AEs) are an important aspect of the evaluation of experimental therapies. The SAVVY (Survival analysis for AdVerse events with Varying follow-up times) project aims to improve the analyses of AE data in clinical trials through the use of survival techniques appropriately dealing with varying follow-up times, censoring, and competing events (CE). In an empirical study including seventeen randomized clinical trials the effect of varying follow-up times, censoring, and competing events on comparisons of two treatment arms with respect to AE risks is investigated. The comparisons of relative risks (RR) of standard probability-based estimators to the gold-standard Aalen-Johansen estimator or hazard-based estimators to an estimated hazard ratio (HR) from Cox regression are done descriptively, with graphical displays, and using a random effects meta-analysis on AE level. The influence of different factors on the size of the bias is investigated in a meta-regression. We find that for both, avoiding bias and categorization of evidence with respect to treatment effect on AE risk into categories, the choice of the estimator is key and more important than features of the underlying data such as percentage of censoring, CEs, amount of follow-up, or value of the gold-standard RR. There is an urgent need to improve the guidelines of reporting AEs so that incidence proportions are finally replaced by the Aalen-Johansen estimator - rather than by Kaplan-Meier - with appropriate definition of CEs. For RRs based on hazards, the HR based on Cox regression has better properties than the ratio of incidence densities

Survival analysis for AdVerse events with VarYing follow-up times (SAVVY) -- estimation of adverse event risks

The SAVVY project aims to improve the analyses of adverse event (AE) data in clinical trials through the use of survival techniques appropriately dealing with varying follow-up times and competing events (CEs). Although statistical methodologies have advanced, in AE analyses often the incidence proportion, the incidence density, or a non-parametric Kaplan-Meier estimator (KME) are used, which either ignore censoring or CEs. In an empirical study including randomized clinical trials from several sponsor organisations, these potential sources of bias are investigated. The main aim is to compare the estimators that are typically used in AE analysis to the Aalen-Johansen estimator (AJE) as the gold-standard. Here, one-sample findings are reported, while a companion paper considers consequences when comparing treatment groups. Estimators are compared with descriptive statistics, graphical displays and with a random effects meta-analysis. The influence of different factors on the size of the bias is investigated in a meta-regression. Comparisons are conducted at the maximum follow-up time and at earlier evaluation time points. CEs definition does not only include death before AE but also end of follow-up for AEs due to events possibly related to the disease course or the treatment. Ten sponsor organisations provided 17 trials including 186 types of AEs. The one minus KME was on average about 1.2-fold larger than the AJE. Leading forces influencing bias were the amount of censoring and of CEs. As a consequence, the average bias using the incidence proportion was less than 5%. Assuming constant hazards using incidence densities was hardly an issue provided that CEs were accounted for. There is a need to improve the guidelines of reporting risks of AEs so that the KME and the incidence proportion are replaced by the AJE with an appropriate definition of CEs

Surrogatvalidierung durch Korrelation und Surrogate Threshold Effect – Ergebnisse von Simulationsstudien

Background: Progression-free survival (PFS) is often used instead of the patient-relevant endpoint overall survival (OS) in cancer clinical trials. In order for PFS to be accepted as a patient-relevant outcome within the benefit assessment of pharmaceuticals in accordance with the German Social Code, Book Five (SGB V), section 35a, it has to be validated as a surrogate endpoint for OS in the relevant indication. As part of a rapid report the Institute for Quality and Efficiency in Health Care (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen – IQWiG) presented methods for surrogate endpoints validation and recommendations for correlation-based procedures. These methods include the evaluation of the certainty of conclusion of study results and the correlation between estimates of surrogate outcome and patient-relevant outcome on trial-level. The correlation is estimated by sample Pearson correlation coefficient or coefficient of determination and respective confidence interval (CI). Requirements for surrogate validation are a high correlation and a high certainty of conclusion of the study results. In case of medium correlation IQWiG methods propose applying the concept of surrogate threshold effect (STE) to determine thresholds for the estimate of the surrogate endpoint.Methods: In simulation studies we investigate the requirements for a successful surrogate validation when applying a correlation-based approach. Simulation parameters are the estimates of the surrogate and the patient-relevant outcome, the correlation between them, the number of patients and the number of studies. We analyzed different scenarios in order to figure out parameters contributing to high correlation. Furthermore, we investigate requirements of the STE method, allowing conclusions on patient-relevant endpoints by means of surrogate endpoints. Finally, in consideration of IQWiG methods we analyze the challenges of surrogate validation in practical use.Results: Both, simulations of the surrogate validation using correlation-based procedure as well as an analytical derivation show low statistical power despite a medium-sized number of studies and a high true correlation. The power for =5 studies and correlation =0.9 is below 6%. A very high true correlation of =0.95 in at least =25 studies would be required in order to preserve a power of 80%, however this scenario is considered implausible in practice. Further simulations investigating the power of the method of STE showed that only one fifth of the considered scenarios have power above 80%. However, these scenarios included parameter constellations with impractical values regarding number of studies, number of patients and effect estimate of OS. The correlation parameter as well as the parameter of the estimate of PFS barely have an impact on the power of the STE procedure.Conclusion: Our simulations show that in practical use it is quite unlikely to fulfill the condition of high correlation as defined in the rapid report of IQWiG, proposing the lower limit of confidence interval to be crucial. Despite setting the true correlation in the model to a high value, statistical power will be quite small as long as the number of studies remains low or medium which is a realistic assumption in validation of surrogate endpoints within the framework of early benefit assessment. Besides, recommendation to involve certainty of studies in the analysis remains problematic. On closer inspection of the density function of sample correlation coefficient and assuming a given true correlation we can conclude that sample correlation does not depend on the variance of the single estimates but only on sample size (representing the number of studies in the model). Therefore, patient number does not have an impact on the confidence interval of the correlation whether using weight vectors for studies or not. Application of the STE concept according to the requirements described in the rapid report appears to be rather complicated as well. We propose an alternative solution of comparing the value of STE with point estimate of the surrogate endpoint instead of its lower level of confidence interval showing low α-errors in realistic scenarios

Comparing the EQ-5D-5L utility index based on value sets of different countries: impact on the interpretation of clinical study results

Abstract Objective To compare the country-specific value sets of the EQ-5D-5L utility index and to evaluate the impact on the interpretation of clinical study results. Six country value sets from Canada, England, Japan, Korea, Netherlands and Uruguay were obtained from literature. In addition, ten crosswalk value sets were downloaded from the EuroQol.org website. Results For each of the 3125 possible health states the difference between the country with the highest index and the country with the lowest index was calculated. The median difference was 0.417 across the health states. When analyzing multinational clinical studies, country-specific value sets should be used to evaluate treatment effects. Additional country-specific analyses are needed