11 research outputs found
Context is Environment
Two lines of work are taking the central stage in AI research. On the one
hand, the community is making increasing efforts to build models that discard
spurious correlations and generalize better in novel test environments.
Unfortunately, the bitter lesson so far is that no proposal convincingly
outperforms a simple empirical risk minimization baseline. On the other hand,
large language models (LLMs) have erupted as algorithms able to learn
in-context, generalizing on-the-fly to eclectic contextual circumstances that
users enforce by means of prompting. In this paper, we argue that context is
environment, and posit that in-context learning holds the key to better domain
generalization. Via extensive theory and experiments, we show that paying
attention to context\unicode{x2013}\unicode{x2013}unlabeled examples as they
arrive\unicode{x2013}\unicode{x2013}allows our proposed In-Context Risk
Minimization (ICRM) algorithm to zoom-in on the test environment risk
minimizer, leading to significant out-of-distribution performance improvements.
From all of this, two messages are worth taking home. Researchers in domain
generalization should consider environment as context, and harness the adaptive
power of in-context learning. Researchers in LLMs should consider context as
environment, to better structure data towards generalization.Comment: 41 Pages, 4 Figure
AdaBest: Minimizing Client Drift in Federated Learning via Adaptive Bias Estimation
In Federated Learning (FL), a number of clients or devices collaborate to
train a model without sharing their data. Models are optimized locally at each
client and further communicated to a central hub for aggregation. While FL is
an appealing decentralized training paradigm, heterogeneity among data from
different clients can cause the local optimization to drift away from the
global objective. In order to estimate and therefore remove this drift,
variance reduction techniques have been incorporated into FL optimization
recently. However, these approaches inaccurately estimate the clients' drift
and ultimately fail to remove it properly. In this work, we propose an adaptive
algorithm that accurately estimates drift across clients. In comparison to
previous works, our approach necessitates less storage and communication
bandwidth, as well as lower compute costs. Additionally, our proposed
methodology induces stability by constraining the norm of estimates for client
drift, making it more practical for large scale FL. Experimental findings
demonstrate that the proposed algorithm converges significantly faster and
achieves higher accuracy than the baselines across various FL benchmarks.Comment: AdaBes
Exploring the forecasting approach for road accidents: Analytical measures with hybrid machine learning
International audienceUrban traffic forecasting models generally follow either a Gaussian Mixture Model (GMM) or a Support Vector Classifier (SVC) to estimate the features of potential road accidents. Although SVC can provide good performances with less data than GMM, it incurs a higher computational cost. This paper proposes a novel framework that combines the descriptive strength of the Gaussian Mixture Model with the high-performance classification capabilities of the Support Vector Classifier. A new approach is presented that uses the mean vectors obtained from the GMM model as input to the SVC. Experimental results show that the approach compares very favorably with baseline statistical methods
Federated Learning for Breast Density Classification: A Real-World Implementation
Building robust deep learning-based models requires large quantities of
diverse training data. In this study, we investigate the use of federated
learning (FL) to build medical imaging classification models in a real-world
collaborative setting. Seven clinical institutions from across the world joined
this FL effort to train a model for breast density classification based on
Breast Imaging, Reporting & Data System (BI-RADS). We show that despite
substantial differences among the datasets from all sites (mammography system,
class distribution, and data set size) and without centralizing data, we can
successfully train AI models in federation. The results show that models
trained using FL perform 6.3% on average better than their counterparts trained
on an institute's local data alone. Furthermore, we show a 45.8% relative
improvement in the models' generalizability when evaluated on the other
participating sites' testing data.Comment: Accepted at the 1st MICCAI Workshop on "Distributed And Collaborative
Learning"; add citation to Fig. 1 & 2 and update Fig.
FL Games: A federated learning framework for distribution shifts
Federated learning aims to train predictive models for data that is
distributed across clients, under the orchestration of a server. However,
participating clients typically each hold data from a different distribution,
whereby predictive models with strong in-distribution generalization can fail
catastrophically on unseen domains. In this work, we argue that in order to
generalize better across non-i.i.d. clients, it is imperative to only learn
correlations that are stable and invariant across domains. We propose FL Games,
a game-theoretic framework for federated learning for learning causal features
that are invariant across clients. While training to achieve the Nash
equilibrium, the traditional best response strategy suffers from high-frequency
oscillations. We demonstrate that FL Games effectively resolves this challenge
and exhibits smooth performance curves. Further, FL Games scales well in the
number of clients, requires significantly fewer communication rounds, and is
agnostic to device heterogeneity. Through empirical evaluation, we demonstrate
that FL Games achieves high out-of-distribution performance on various
benchmarks
Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging
Saliency maps have become a widely used method to make deep learning models more interpretable by providing post-hoc explanations of classifiers through identification of the most pertinent areas of the input medical image. They are increasingly being used in medical imaging to provide clinically plausible explanations for the decisions the neural network makes. However, the utility and robustness of these visualization maps has not yet been rigorously examined in the context of medical imaging. We posit that trustworthiness in this context requires 1) localization utility, 2) sensitivity to model weight randomization, 3) repeatability, and 4) reproducibility. Using the localization information available in two large public radiology datasets, we quantify the performance of eight commonly used saliency map approaches for the above criteria using area under the precision-recall curves (AUPRC) and structural similarity index (SSIM), comparing their performance to various baseline measures. Using our framework to quantify the trustworthiness of saliency maps, we show that all eight saliency map techniques fail at least one of the criteria and are, in most cases, less trustworthy when compared to the baselines. We suggest that their usage in the high-risk domain of medical imaging warrants additional scrutiny and recommend that detection or segmentation models be used if localization is the desired output of the network. Additionally, to promote reproducibility of our findings, we provide the code we used for all tests performed in this work at this link: this https URL
QU-BraTS : MICCAI BraTS 2020 Challenge on QuantifyingUncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results
Deep learning (DL) models have provided the state-of-the-art performance in a wide variety of medical imaging benchmarking challenges, including the Brain Tumor Segmentation (BraTS) challenges. However, the task of focal pathology multi-compartment segmentation (e.g., tumor and lesion sub-regions) is particularly challenging, and potential errors hinder the translation of DL models into clinical workflows. Quantifying the reliability of DL model predictions in the form of uncertainties, could enable clinical review of the most uncertain regions, thereby building trust and paving the way towards clinical translation. Recently, a number of uncertainty estimation methods have been introduced for DL medical image segmentation tasks. Developing scores to evaluate and compare the performance of uncertainty measures will assist the end-user in making more informed decisions. In this study, we explore and evaluate a score developed during the BraTS 2019-2020 task on uncertainty quantification (QU-BraTS), and designed to assess and rank uncertainty estimates for brain tumor multi-compartment segmentation. This score (1) rewards uncertainty estimates that produce high confidence in correct assertions, and those that assign low confidence levels at incorrect assertions, and (2) penalizes uncertainty measures that lead to a higher percentages of under-confident correct assertions. We further benchmark the segmentation uncertainties generated by 14 independent participating teams of QU-BraTS 2020, all of which also participated in the main BraTS segmentation task. Overall, our findings confirm the importance and complementary value that uncertainty estimates provide to segmentation algorithms, and hence highlight the need for uncertainty quantification in medical image analyses. Our evaluation code is made publicly available at https://github.com/RagMeh11/QU-BraT