13 research outputs found
Deep Cox Mixtures for Survival Regression
Survival analysis is a challenging variation of regression modeling because
of the presence of censoring, where the outcome measurement is only partially
known, due to, for example, loss to follow up. Such problems come up frequently
in medical applications, making survival analysis a key endeavor in
biostatistics and machine learning for healthcare, with Cox regression models
being amongst the most commonly employed models. We describe a new approach for
survival analysis regression models, based on learning mixtures of Cox
regressions to model individual survival distributions. We propose an
approximation to the Expectation Maximization algorithm for this model that
does hard assignments to mixture groups to make optimization efficient. In each
group assignment, we fit the hazard ratios within each group using deep neural
networks, and the baseline hazard for each mixture component
non-parametrically.
We perform experiments on multiple real world datasets, and look at the
mortality rates of patients across ethnicity and gender. We emphasize the
importance of calibration in healthcare settings and demonstrate that our
approach outperforms classical and modern survival analysis baselines, both in
terms of discriminative performance and calibration, with large gains in
performance on the minority demographics.Comment: Machine Learning for Healthcare Conference, 202
Diagnosing Model Performance Under Distribution Shift
Prediction models can perform poorly when deployed to target distributions
different from the training distribution. To understand these operational
failure modes, we develop a method, called DIstribution Shift DEcomposition
(DISDE), to attribute a drop in performance to different types of distribution
shifts. Our approach decomposes the performance drop into terms for 1) an
increase in harder but frequently seen examples from training, 2) changes in
the relationship between features and outcomes, and 3) poor performance on
examples infrequent or unseen during training. These terms are defined by
fixing a distribution on while varying the conditional distribution of between training and target, or by fixing the conditional distribution
of while varying the distribution on . In order to do this, we
define a hypothetical distribution on consisting of values common in both
training and target, over which it is easy to compare and thus
predictive performance. We estimate performance on this hypothetical
distribution via reweighting methods. Empirically, we show how our method can
1) inform potential modeling improvements across distribution shifts for
employment prediction on tabular census data, and 2) help to explain why
certain domain adaptation methods fail to improve model performance for
satellite image classification
Evaluating Treatment Prioritization Rules via Rank-Weighted Average Treatment Effects
There are a number of available methods that can be used for choosing whom to
prioritize treatment, including ones based on treatment effect estimation, risk
scoring, and hand-crafted rules. We propose rank-weighted average treatment
effect (RATE) metrics as a simple and general family of metrics for comparing
treatment prioritization rules on a level playing field. RATEs are agnostic as
to how the prioritization rules were derived, and only assesses them based on
how well they succeed in identifying units that benefit the most from
treatment. We define a family of RATE estimators and prove a central limit
theorem that enables asymptotically exact inference in a wide variety of
randomized and observational study settings. We provide justification for the
use of bootstrapped confidence intervals and a framework for testing hypotheses
about heterogeneity in treatment effectiveness correlated with the
prioritization rule. Our definition of the RATE nests a number of existing
metrics, including the Qini coefficient, and our analysis directly yields
inference methods for these metrics. We demonstrate our approach in examples
drawn from both personalized medicine and marketing. In the medical setting,
using data from the SPRINT and ACCORD-BP randomized control trials, we find no
significant evidence of heterogeneous treatment effects. On the other hand, in
a large marketing trial, we find robust evidence of heterogeneity in the
treatment effects of some digital advertising campaigns and demonstrate how
RATEs can be used to compare targeting rules that prioritize estimated risk vs.
those that prioritize estimated treatment benefit
Choosing a Proxy Metric from Past Experiments
In many randomized experiments, the treatment effect of the long-term metric
(i.e. the primary outcome of interest) is often difficult or infeasible to
measure. Such long-term metrics are often slow to react to changes and
sufficiently noisy they are challenging to faithfully estimate in short-horizon
experiments. A common alternative is to measure several short-term proxy
metrics in the hope they closely track the long-term metric -- so they can be
used to effectively guide decision-making in the near-term. We introduce a new
statistical framework to both define and construct an optimal proxy metric for
use in a homogeneous population of randomized experiments. Our procedure first
reduces the construction of an optimal proxy metric in a given experiment to a
portfolio optimization problem which depends on the true latent treatment
effects and noise level of experiment under consideration. We then denoise the
observed treatment effects of the long-term metric and a set of proxies in a
historical corpus of randomized experiments to extract estimates of the latent
treatment effects for use in the optimization problem. One key insight derived
from our approach is that the optimal proxy metric for a given experiment is
not apriori fixed; rather it should depend on the sample size (or effective
noise level) of the randomized experiment for which it is deployed. To
instantiate and evaluate our framework, we employ our methodology in a large
corpus of randomized experiments from an industrial recommendation system and
construct proxy metrics that perform favorably relative to several baselines
Using public clinical trial reports to probe non-experimental causal inference methods
Abstract Background Non-experimental studies (also known as observational studies) are valuable for estimating the effects of various medical interventions, but are notoriously difficult to evaluate because the methods used in non-experimental studies require untestable assumptions. This lack of intrinsic verifiability makes it difficult both to compare different non-experimental study methods and to trust the results of any particular non-experimental study. Methods We introduce TrialProbe, a data resource and statistical framework for the evaluation of non-experimental methods. We first collect a dataset of pseudo “ground truths” about the relative effects of drugs by using empirical Bayesian techniques to analyze adverse events recorded in public clinical trial reports. We then develop a framework for evaluating non-experimental methods against that ground truth by measuring concordance between the non-experimental effect estimates and the estimates derived from clinical trials. As a demonstration of our approach, we also perform an example methods evaluation between propensity score matching, inverse propensity score weighting, and an unadjusted approach on a large national insurance claims dataset. Results From the 33,701 clinical trial records in our version of the ClinicalTrials.gov dataset, we are able to extract 12,967 unique drug/drug adverse event comparisons to form a ground truth set. During our corresponding methods evaluation, we are able to use that reference set to demonstrate that both propensity score matching and inverse propensity score weighting can produce estimates that have high concordance with clinical trial results and substantially outperform an unadjusted baseline. Conclusions We find that TrialProbe is an effective approach for probing non-experimental study methods, being able to generate large ground truth sets that are able to distinguish how well non-experimental methods perform in real world observational data