178 research outputs found
Conformal Drug Property Prediction with Density Estimation under Covariate Shift
In drug discovery, it is vital to confirm the predictions of pharmaceutical
properties from computational models using costly wet-lab experiments. Hence,
obtaining reliable uncertainty estimates is crucial for prioritizing drug
molecules for subsequent experimental validation. Conformal Prediction (CP) is
a promising tool for creating such prediction sets for molecular properties
with a coverage guarantee. However, the exchangeability assumption of CP is
often challenged with covariate shift in drug discovery tasks: Most datasets
contain limited labeled data, which may not be representative of the vast
chemical space from which molecules are drawn. To address this limitation, we
propose a method called CoDrug that employs an energy-based model leveraging
both training data and unlabelled data, and Kernel Density Estimation (KDE) to
assess the densities of a molecule set. The estimated densities are then used
to weigh the molecule samples while building prediction sets and rectifying for
distribution shift. In extensive experiments involving realistic distribution
drifts in various small-molecule drug discovery tasks, we demonstrate the
ability of CoDrug to provide valid prediction sets and its utility in
addressing the distribution shift arising from de novo drug design models. On
average, using CoDrug can reduce the coverage gap by over 35% when compared to
conformal prediction sets not adjusted for covariate shift.Comment: Accepted at NeurIPS 202
Drug Discovery under Covariate Shift with Domain-Informed Prior Distributions over Functions
Accelerating the discovery of novel and more effective therapeutics is an
important pharmaceutical problem in which deep learning is playing an
increasingly significant role. However, real-world drug discovery tasks are
often characterized by a scarcity of labeled data and significant covariate
shift\unicode{x2013}\unicode{x2013}a setting that poses a challenge to
standard deep learning methods. In this paper, we present Q-SAVI, a
probabilistic model able to address these challenges by encoding explicit prior
knowledge of the data-generating process into a prior distribution over
functions, presenting researchers with a transparent and probabilistically
principled way to encode data-driven modeling preferences. Building on a novel,
gold-standard bioactivity dataset that facilitates a meaningful comparison of
models in an extrapolative regime, we explore different approaches to induce
data shift and construct a challenging evaluation setup. We then demonstrate
that using Q-SAVI to integrate contextualized prior knowledge of drug-like
chemical space into the modeling process affords substantial gains in
predictive accuracy and calibration, outperforming a broad range of
state-of-the-art self-supervised pre-training and domain adaptation techniques.Comment: Published in the Proceedings of the 40th International Conference on
Machine Learning (ICML 2023
An Information-theoretical Approach to Semi-supervised Learning under Covariate-shift
A common assumption in semi-supervised learning is that the labeled,
unlabeled, and test data are drawn from the same distribution. However, this
assumption is not satisfied in many applications. In many scenarios, the data
is collected sequentially (e.g., healthcare) and the distribution of the data
may change over time often exhibiting so-called covariate shifts. In this
paper, we propose an approach for semi-supervised learning algorithms that is
capable of addressing this issue. Our framework also recovers some popular
methods, including entropy minimization and pseudo-labeling. We provide new
information-theoretical based generalization error upper bounds inspired by our
novel framework. Our bounds are applicable to both general semi-supervised
learning and the covariate-shift scenario. Finally, we show numerically that
our method outperforms previous approaches proposed for semi-supervised
learning under the covariate shift.Comment: Accepted at AISTATS 202
Semantic Self-adaptation: Enhancing Generalization with a Single Sample
The lack of out-of-domain generalization is a critical weakness of deep
networks for semantic segmentation. Previous studies relied on the assumption
of a static model, i. e., once the training process is complete, model
parameters remain fixed at test time. In this work, we challenge this premise
with a self-adaptive approach for semantic segmentation that adjusts the
inference process to each input sample. Self-adaptation operates on two levels.
First, it fine-tunes the parameters of convolutional layers to the input image
using consistency regularization. Second, in Batch Normalization layers,
self-adaptation interpolates between the training and the reference
distribution derived from a single test sample. Despite both techniques being
well known in the literature, their combination sets new state-of-the-art
accuracy on synthetic-to-real generalization benchmarks. Our empirical study
suggests that self-adaptation may complement the established practice of model
regularization at training time for improving deep network generalization to
out-of-domain data. Our code and pre-trained models are available at
https://github.com/visinf/self-adaptive.Comment: Published in TMLR (July 2023); OpenReview:
https://openreview.net/forum?id=ILNqQhGbLx; Code:
https://github.com/visinf/self-adaptive; Video: https://youtu.be/s4DG65ic0E
Stratified Learning: a general-purpose statistical method for improved learning under Covariate Shift
Covariate shift arises when the labelled training (source) data is not
representative of the unlabelled (target) data due to systematic differences in
the covariate distributions. A supervised model trained on the source data
subject to covariate shift may suffer from poor generalization on the target
data. We propose a novel, statistically principled and theoretically justified
method to improve learning under covariate shift conditions, based on
propensity score stratification, a well-established methodology in causal
inference. We show that the effects of covariate shift can be reduced or
altogether eliminated by conditioning on propensity scores. In practice, this
is achieved by fitting learners on subgroups ("strata") constructed by
partitioning the data based on the estimated propensity scores, leading to
balanced covariates and much-improved target prediction. We demonstrate the
effectiveness of our general-purpose method on contemporary research questions
in observational cosmology, and on additional benchmark examples, matching or
outperforming state-of-the-art importance weighting methods, widely studied in
the covariate shift literature. We obtain the best reported AUC (0.958) on the
updated "Supernovae photometric classification challenge" and improve upon
existing conditional density estimation of galaxy redshift from Sloan Data Sky
Survey (SDSS) data
Conformal Prediction: a Unified Review of Theory and New Challenges
In this work we provide a review of basic ideas and novel developments about
Conformal Prediction -- an innovative distribution-free, non-parametric
forecasting method, based on minimal assumptions -- that is able to yield in a
very straightforward way predictions sets that are valid in a statistical sense
also in in the finite sample case. The in-depth discussion provided in the
paper covers the theoretical underpinnings of Conformal Prediction, and then
proceeds to list the more advanced developments and adaptations of the original
idea.Comment: arXiv admin note: text overlap with arXiv:0706.3188,
arXiv:1604.04173, arXiv:1709.06233, arXiv:1203.5422 by other author
- …