Using classification tree analysis to generate propensity score weights

Rationale, aims and objectivesIn evaluating non‐randomized interventions, propensity scores (PS) estimate the probability of assignment to the treatment group given observed characteristics. Machine learning algorithms have been proposed as an alternative to conventional logistic regression for modelling PS in order to avoid limitations of linear methods. We introduce classification tree analysis (CTA) to generate PS which is a “decision‐tree”‐like classification model that provides accurate, parsimonious decision rules that are easy to display and interpret, reports P values derived via permutation tests, and evaluates cross‐generalizability.MethodUsing empirical data, we identify all statistically valid CTA PS models and then use them to compute strata‐specific, observation‐level PS weights that are subsequently applied in outcomes analyses. We compare findings obtained using this framework to logistic regression and boosted regression, by evaluating covariate balance using standardized differences, model predictive accuracy, and treatment effect estimates obtained using median regression and a weighted CTA outcomes model.ResultsWhile all models had some imbalanced covariates, main‐effects logistic regression yielded the lowest average standardized difference, whereas CTA yielded the greatest predictive accuracy. Nevertheless, treatment effect estimates were generally consistent across all models.ConclusionsAssessing standardized differences in means as a test of covariate balance is inappropriate for machine learning algorithms that segment the sample into two or more strata. Because the CTA algorithm identifies all statistically valid PS models for a sample, it is most likely to identify a correctly specified PS model, and should be considered as an alternative approach to modeling the PS.Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/137726/1/jep12744.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/137726/2/jep12744_am.pd

Derandomized Novelty Detection with FDR Control via Conformal E-values

Conformal prediction and other randomized model-free inference techniques are gaining increasing attention as general solutions to rigorously calibrate the output of any machine learning algorithm for novelty detection. This paper contributes to the field by developing a novel method for mitigating their algorithmic randomness, leading to an even more interpretable and reliable framework for powerful novelty detection under false discovery rate control. The idea is to leverage suitable conformal e-values instead of p-values to quantify the significance of each finding, which allows the evidence gathered from multiple mutually dependent analyses of the same data to be seamlessly aggregated. Further, the proposed method can reduce randomness without much loss of power, partly thanks to an innovative way of weighting conformal e-values based on additional side information carefully extracted from the same data. Simulations with synthetic and real data confirm this solution can be effective at eliminating random noise in the inferences obtained with state-of-the-art alternative techniques, sometimes also leading to higher power.Comment: 19 pages, 11 figure

Utilization of deep learning to quantify fluid volume of neovascular age-related macular degeneration patients based on swept-source OCT imaging: The ONTARIO study.

PURPOSE: To evaluate the predictive ability of a deep learning-based algorithm to determine long-term best-corrected distance visual acuity (BCVA) outcomes in neovascular age-related macular degeneration (nARMD) patients using baseline swept-source optical coherence tomography (SS-OCT) and OCT-angiography (OCT-A) data. METHODS: In this phase IV, retrospective, proof of concept, single center study, SS-OCT data from 17 previously treated nARMD eyes was used to assess retinal layer thicknesses, as well as quantify intraretinal fluid (IRF), subretinal fluid (SRF), and serous pigment epithelium detachments (PEDs) using a novel deep learning-based, macular fluid segmentation algorithm. Baseline OCT and OCT-A morphological features and fluid measurements were correlated using the Pearson correlation coefficient (PCC) to changes in BCVA from baseline to week 52. RESULTS: Total retinal fluid (IRF, SRF and PED) volume at baseline had the strongest correlation to improvement in BCVA at month 12 (PCC = 0.652, p = 0.005). Fluid was subsequently sub-categorized into IRF, SRF and PED, with PED volume having the next highest correlation (PCC = 0.648, p = 0.005) to BCVA improvement. Average total retinal thickness in isolation demonstrated poor correlation (PCC = 0.334, p = 0.189). When two features, mean choroidal neovascular membranes (CNVM) size and total fluid volume, were combined and correlated with visual outcomes, the highest correlation increased to PCC = 0.695 (p = 0.002). CONCLUSIONS: In isolation, total fluid volume most closely correlates with change in BCVA values between baseline and week 52. In combination with complimentary information from OCT-A, an improvement in the linear correlation score was observed. Average total retinal thickness provided a lower correlation, and thus provides a lower predictive outcome than alternative metrics assessed. Clinically, a machine-learning approach to analyzing fluid metrics in combination with lesion size may provide an advantage in personalizing therapy and predicting BCVA outcomes at week 52

Combining machine learning and propensity score weighting to estimate causal effects in multivalued treatments

Rationale, aims and objectivesInterventions with multivalued treatments are common in medical and health research; examples include comparing the efficacy of competing interventions and contrasting various doses of a drug. In recent years, there has been growing interest in the development of methods that estimate multivalued treatment effects using observational data. This paper extends a previously described analytic framework for evaluating binary treatments to studies involving multivalued treatments utilizing a machine learning algorithm called optimal discriminant analysis (ODA).MethodWe describe the differences between regression‐based treatment effect estimators and effects estimated using the ODA framework. We then present an empirical example using data from an intervention including three study groups to compare corresponding effects.ResultsThe regression‐based estimators produced statistically significant mean differences between the two intervention groups, and between one of the treatment groups and controls. In contrast, ODA was unable to discriminate between distributions of any of the three study groups.ConclusionsOptimal discriminant analysis offers an appealing alternative to conventional regression‐based models for estimating effects in multivalued treatment studies because of its insensitivity to skewed data and use of accuracy measures applicable to all prognostic analyses. If these analytic approaches produce consistent treatment effect P values, this bolsters confidence in the validity of the results. If the approaches produce conflicting treatment effect P values, as they do in our empirical example, the investigator should consider the ODA‐derived estimates to be most robust, given that ODA uses permutation P values that require no distributional assumptions and are thus, always valid.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/135025/1/jep12610.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/135025/2/jep12610_am.pd

Feature selection in high-dimensional dataset using MapReduce

This paper describes a distributed MapReduce implementation of the minimum Redundancy Maximum Relevance algorithm, a popular feature selection method in bioinformatics and network inference problems. The proposed approach handles both tall/narrow and wide/short datasets. We further provide an open source implementation based on Hadoop/Spark, and illustrate its scalability on datasets involving millions of observations or features

A survey of cost-sensitive decision tree induction algorithms

The past decade has seen a significant interest on the problem of inducing decision trees that take account of costs of misclassification and costs of acquiring the features used for decision making. This survey identifies over 50 algorithms including approaches that are direct adaptations of accuracy based methods, use genetic algorithms, use anytime methods and utilize boosting and bagging. The survey brings together these different studies and novel approaches to cost-sensitive decision tree learning, provides a useful taxonomy, a historical timeline of how the field has developed and should provide a useful reference point for future research in this field

Machine learning as an ultra-fast alternative to Bayesian retrievals

1Kapteyn Astronomical Institute, University of Groningen, Groningen, The Netherlands 2SRON Netherlands Institute for Space Research, Utrecht, The Netherlands 3Centre for Exoplanet Science, University of Edinburgh, Edinburgh, UK Introduction: Inferring physical and chemical properties of an exoplanet's atmosphere from its transmission spectrum is computationally expensive. A multitude of forward models, sampled from a high dimensional parameter space, need to be compared to the observation. The preferred sampling method is currently Nested Sampling [7], in particular, the MultiNest implementation [2, 3]. It typically requires tens to hundreds of thousands of forward models to converge. Therefore, simpler forward models are usually favoured over longer computation times. A possible workaround is to use machine learning. A machine learning algorithm trained on a grid of forward models and parameter pairs can perform retrievals in seconds. This would make it possible to use complex models that take full advantage of future facilities e.g., JWST. Not only would retrievals of individual exoplanets become much faster, but it would also enable statistical studies of populations of exoplanets. It would also be a valuable tool for retrievability analyses, for example to assess the sensitivity of using different chemical networks. The main obstacle to overcome is being able to predict accurate posterior distributions and error estimates on the retrieved parameters. These need to be as close as possible to their Bayesian counterparts. Methods: Expanding on the 5-parameter grid in [5], we used ARCiS (ARtful modelling Code for exoplanet Science) [6] to generate a grid of 200,000 forward models described by the following parameters: isothermal temperature (T), planetary radius (RP), planetary mass (MP), abundances of water (H2O), ammonia (NH3) and hydrogen cyanide (HCN), and cloud top pressure (Pcloud). The models contain 13 wavelength bins, matching those of WASP-12b's observation with HST/WFC3 [4]. We added normally distributed random noise with σ=50 ppm. We trained a random forest following the details in [5] and a convolutional neural network (CNN). We divided the data into a training set of 190,000 spectra and a test set of 10,000. For the CNN we reserved 19,000 spectra (10%) from the training set for validation. These are needed to update the network weights at each training iteration. The CNN was trained with the loss function introduced in [1] to output a probability distribution. To account for the observational noise, we combined the distributions predicted for multiple noisy copies of the spectrum. To evaluate the performance of the machine learning algorithms, we retrieved all the spectra in the test set and plotted our predictions against the true values for the parameters. We repeated the experiment with only 1,000 spectra for Nested Sampling, reflecting the increased computational overhead of each of these retrievals. We then used a transmission spectrum of WASP-12b observed with HST/WFC3 [4] as a real-world test case. Results: Although the random forest trains faster, the CNN provided better results. Figures 1 and 2 show the predicted versus the true parameters for the CNN and Nested Sampling bulk retrievals. Remarkably, we observe the same structures in both plots. This shows that the CNN is able to learn the relationship between spectral features and parameters. We also found that both the CNN and Nested Sampling provide correct error estimates, with ~60% of predictions within 1σ of the true value, ~98% within 2σ, and virtually all within 3σ. This is in almost perfect agreement with expectation from statistical errors. Figures 3 and 4 show the CNN and Nested Sampling retrieval of WASP-12b. Again, we see very similar results, although the CNN provides broader posterior distributions. Work is ongoing to try to fix this issue. We found that a training set of 180,000 spectra is unnecessarily large, and the same performance can be reached with only 20,000 spectra. This implies that the number of forward model computations needed to train a CNN is smaller than the number needed for a single Bayesian retrieval. If this holds for more complex forward models and higher quality spectra, it would make machine learning an extremely attractive alternative to Nested Sampling. Conclusion: The existing literature on machine learning retrievals of exoplanet atmospheres only has comparisons between machine learning and Nested Sampling for a handful of test cases [1, 5, 8]. In this work we present a comparison of bulk retrievals done with both methods, showing that machine learning can indeed be a viable and fast alternative to Nested Sampling. We are currently working on extending these results to models with equilibrium chemistry and to JWST/NIRSpec simulated spectra. Acknowledgements: This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 860470. References: [1] Cobb, A. D., Himes, M. D., Soboczenski, F., Zorzan, S., O'Beirne, M. D., Baydin, A. G., Gal, Y., Domagal-Goldman, S. D., Arney, G. N., Angerhausen, D. (2019, 5). An Ensemble of Bayesian Neural Networks for Exoplanetary Atmospheric Retrieval. [2] Feroz, F., Hobson, M. P. (2008). Monthly Notices of the Royal Astronomical Society 384 (2). [3] Feroz, F., Hobson, M. P., Bridges, M. (2009). Monthly Notices of the Royal Astronomical Society 398(4). [4] Kreidberg, L., Line, M. R., Bean, J. L., Stevenson, K. B., Desert, J.-M., Madhusudhan, N., Fortney, J. J., Barstow, J. K., Henry, G. W., Williamson, M. H., Showman, A. P. (2015). A DETECTION OF WATER IN THE TRANSMISSION SPECTRUM OF THE HOT JUPITER WASP-12b AND IMPLICATIONS FOR ITS ATMOSPHERIC COMPOSITION. Technical report. [5] Marquez-Neila, P., Fisher, C., Sznitman, R., Heng, K. (2018). Supervised machine learning for analysing spectra of exoplanetary atmospheres. [6] Min, M., Ormel, C. W., Chubb, K., Helling, C., Kawashima, Y. (2020). The ARCiS framework for exoplanet atmospheres. Astronomy & Astrophysics 642. [7] Skilling, J. (2006). Nested sampling for general Bayesian computation. Bayesian Analysis 1 (4). [8] Zingales, T., Waldmann, I. P. (2018). The Astronomical Journal 156 (6)