Denton, CP

Domsic, RT

Fine, MJ

Kwoh, CK

Lucas, M

Medsger, TA

Nihtyanova, SI

Wisniewski, SR

UCL Discovery

REPLY TO LETTER – Arthritis and Rheumatology – Domsic et al 2016 To the Editor: We appreciate the comments on our clinical prediction rule for 5-year mortality for patients with diffuse scleroderma.    We thank Drs. Collignon and Vesely for highlighting many of the issues that we considered in deriving and validating this rule.   We weighed the balance between creating a risk-stratification tool that could easily be applied at the clinical bedside with one that maximized model performance.    In today’s demanding practice environment, time is of the essence.  We reasoned that a tool that could be used quickly without the aid of a web-based calculator or nomogram would be more widely adopted by the rheumatology community.  We recognized that the predictive performance of a model could be decreased by rounding the beta coefficients to quantify the weights for individual prognostic variables.  We acknowledge that our original submission did not provide the data that outlined the performance of our models in each step of their development.    In the original Pittsburgh derived cohort, the area under the curve (AUC) calculated using the original beta-coefficients was 0.80 (95% CI 0.75, 0.81) and only decreased to 0.79 (95% CI 0.74, 0.83) based on beta-coefficients rounded to the nearest integer.   Given the negligible decrement in model performance, we opted to round the betas to the nearest integer.  Creating a 3-level prognostic rule based on these integer values was ultimately responsible for a larger y drop in model performance (AUC decreased to 0.73, 95% CI 0.69, 0.78) in the Pittsburgh derivation cohort.  Remaining consistent with our original goal of creating an easy-to-use prediction rule, we believe the benefits of categorizing a patient as low, moderate or high risk for mortality was worth the trade-off in model performance.   Drs. Collignon and Vesely also reviewed some key considerations in selecting the best methods to identify predictor variables, such as using stepwise selection procedures or pre-filtering variables based on univariate analysis.   Whereas we used univariate variable selection and stepwise multivariable modelling, we applied a p-value of 0.20 to minimize the likelihood of discarding potentially important predictor variables rather than more stringent p-values (e.g. p= 0.5 to 0.10), thereby improving the robustness of the model.  It is also important to point out that stepwise selection is best applied in circumstances where a relatively small number of covariates have a strong relationship with the outcome of interest, as we believe is the case with 5-year mortality our patient population.  We did explore other selection methods (e.g. classification tree analysis), but none were a good fit.  We did not explore the LASSO of Tibshirani technique.   It should be noted that logistic regression has been very commonly used to create many well-known risk prediction rules.    Finally, we wish to clarify that we developed our risk-stratification model to help physicians quantify 5-year prognosis in the initial clinical setting and as an objective measure for clinical research, not as a surrogate endpoint for mortality in clinical trials or other forms of investigation.  For instance, if a patient presents and is low-risk for 5-year mortality then referral for autologous stem cell transplant (which generally has a 10% treatment-related mortality), may not be appropriate.   From a research perspective it could be used to identify potentially eligible patients for clinical trials which may incur some risk, such as an early phase 2 trial.  It could be appropriately used as a stratification tool in clinical trial design.  For example, stratifying enrollment of patients into high/moderate vs low-risk at the onset, so that the study has the appropriate power to analyze these individual groups or combined.  To use this model as a surrogate endpoint would be inappropriate as it was not developed or tested for this.    We agree with Drs Collignon and Vesely’s comments regarding surrogate endpoints, and appreciate their interest in our work.   

Reply [for "Derivation and External Validation of a Prediction Rule for Five-Year Mortality in Patients with Early Diffuse Cutaneous Scleroderma"]

http://discovery.ucl.ac.uk/1514554/1/denton_AR_%20reply%20to%20the%20letter%20to%20the%20editors_Domsic%20et%20al%20Aug%202016.pdf

Reply [for "Derivation and External Validation of a Prediction Rule for Five-Year Mortality in Patients with Early Diffuse Cutaneous Scleroderma"]

Abstract

Similar works

Full text

Available Versions

UCL Discovery