70 research outputs found
Advancing efficiency and robustness of neural networks for imaging
Enabling machines to see and analyze the world is a longstanding research objective. Advances in computer vision have the potential of influencing many aspects of our lives as they can enable machines to tackle a variety of tasks. Great progress in computer vision has been made, catalyzed by recent progress in machine learning and especially the breakthroughs achieved by deep artificial neural networks.
Goal of this work is to alleviate limitations of deep neural networks that hinder their large-scale adoption for real-world applications. To this end, it investigates methodologies for constructing and training deep neural networks with low computational requirements. Moreover, it explores strategies for achieving robust performance on unseen data. Of particular interest is the application of segmenting volumetric medical scans because of the technical challenges it imposes, as well as its clinical importance. The developed methodologies are generic and of relevance to a broader computer vision and machine learning audience.
More specifically, this work introduces an efficient 3D convolutional neural network architecture, which achieves high performance for segmentation of volumetric medical images, an application previously hindered by high computational requirements of 3D networks. It then investigates sensitivity of network performance on hyper-parameter configuration, which we interpret as overfitting the model configuration to the data available during development. It is shown that ensembling a set of models with diverse configurations mitigates this and improves generalization. The thesis then explores how to utilize unlabelled data for learning representations that generalize better. It investigates domain adaptation and introduces an architecture for adversarial networks tailored for adaptation of segmentation networks. Finally, a novel semi-supervised learning method is proposed that introduces a graph in the latent space of a neural network to capture relations between labelled and unlabelled samples. It then regularizes the embedding to form a compact cluster per class, which improves generalization.Open Acces
On the use of Mahalanobis distance for out-of-distribution detection with neural networks for medical imaging
Implementing neural networks for clinical use in medical applications
necessitates the ability for the network to detect when input data differs
significantly from the training data, with the aim of preventing unreliable
predictions. The community has developed several methods for
out-of-distribution (OOD) detection, within which distance-based approaches -
such as Mahalanobis distance - have shown potential. This paper challenges the
prevailing community understanding that there is an optimal layer, or
combination of layers, of a neural network for applying Mahalanobis distance
for detection of any OOD pattern. Using synthetic artefacts to emulate OOD
patterns, this paper shows the optimum layer to apply Mahalanobis distance
changes with the type of OOD pattern, showing there is no one-fits-all
solution. This paper also shows that separating this OOD detector into multiple
detectors at different depths of the network can enhance the robustness for
detecting different OOD patterns. These insights were validated on real-world
OOD tasks, training models on CheXpert chest X-rays with no support devices,
then using scans with unseen pacemakers (we manually labelled 50% of CheXpert
for this research) and unseen sex as OOD cases. The results inform
best-practices for the use of Mahalanobis distance for OOD detection. The
manually annotated pacemaker labels and the project's code are available at:
https://github.com/HarryAnthony/Mahalanobis-OOD-detection.Comment: Accepted for the Uncertainty for Safe Utilization of Machine Learning
in Medical Imaging (UNSURE 2023) workshop at the MICCAI 202
Post-Deployment Adaptation with Access to Source Data via Federated Learning and Source-Target Remote Gradient Alignment
Deployment of Deep Neural Networks in medical imaging is hindered by
distribution shift between training data and data processed after deployment,
causing performance degradation. Post-Deployment Adaptation (PDA) addresses
this by tailoring a pre-trained, deployed model to the target data distribution
using limited labelled or entirely unlabelled target data, while assuming no
access to source training data as they cannot be deployed with the model due to
privacy concerns and their large size. This makes reliable adaptation
challenging due to limited learning signal. This paper challenges this
assumption and introduces FedPDA, a novel adaptation framework that brings the
utility of learning from remote data from Federated Learning into PDA. FedPDA
enables a deployed model to obtain information from source data via remote
gradient exchange, while aiming to optimize the model specifically for the
target domain. Tailored for FedPDA, we introduce a novel optimization method
StarAlign (Source-Target Remote Gradient Alignment) that aligns gradients
between source-target domain pairs by maximizing their inner product, to
facilitate learning a target-specific model. We demonstrate the method's
effectiveness using multi-center databases for the tasks of cancer metastases
detection and skin lesion classification, where our method compares favourably
to previous work. Code is available at: https://github.com/FelixWag/StarAlignComment: This version was accepted for the Machine Learning in Medical Imaging
(MLMI 2023) workshop at MICCAI 202
Modality Cycles with Masked Conditional Diffusion for Unsupervised Anomaly Segmentation in MRI
Unsupervised anomaly segmentation aims to detect patterns that are distinct
from any patterns processed during training, commonly called abnormal or
out-of-distribution patterns, without providing any associated manual
segmentations. Since anomalies during deployment can lead to model failure,
detecting the anomaly can enhance the reliability of models, which is valuable
in high-risk domains like medical imaging. This paper introduces Masked
Modality Cycles with Conditional Diffusion (MMCCD), a method that enables
segmentation of anomalies across diverse patterns in multimodal MRI. The method
is based on two fundamental ideas. First, we propose the use of cyclic modality
translation as a mechanism for enabling abnormality detection.
Image-translation models learn tissue-specific modality mappings, which are
characteristic of tissue physiology. Thus, these learned mappings fail to
translate tissues or image patterns that have never been encountered during
training, and the error enables their segmentation. Furthermore, we combine
image translation with a masked conditional diffusion model, which attempts to
`imagine' what tissue exists under a masked area, further exposing unknown
patterns as the generative model fails to recreate them. We evaluate our method
on a proxy task by training on healthy-looking slices of BraTS2021
multi-modality MRIs and testing on slices with tumors. We show that our method
compares favorably to previous unsupervised approaches based on image
reconstruction and denoising with autoencoders and diffusion models.Comment: Accepted in Multiscale Multimodal Medical Imaging workshop in MICCAI
202
Joint Optimization of Class-Specific Training- and Test-Time Data Augmentation in Segmentation
This paper presents an effective and general data augmentation framework for
medical image segmentation. We adopt a computationally efficient and
data-efficient gradient-based meta-learning scheme to explicitly align the
distribution of training and validation data which is used as a proxy for
unseen test data. We improve the current data augmentation strategies with two
core designs. First, we learn class-specific training-time data augmentation
(TRA) effectively increasing the heterogeneity within the training subsets and
tackling the class imbalance common in segmentation. Second, we jointly
optimize TRA and test-time data augmentation (TEA), which are closely connected
as both aim to align the training and test data distribution but were so far
considered separately in previous works. We demonstrate the effectiveness of
our method on four medical image segmentation tasks across different scenarios
with two state-of-the-art segmentation models, DeepMedic and nnU-Net. Extensive
experimentation shows that the proposed data augmentation framework can
significantly and consistently improve the segmentation performance when
compared to existing solutions. Code is publicly available.Comment: Accepted by IEEE Transactions on Medical Imagin
Context label learning: improving background class representations in semantic segmentation
Background samples provide key contextual information for segmenting regions of interest (ROIs). However, they always cover a diverse set of structures, causing difficulties for the segmentation model to learn good decision boundaries with high sensitivity and precision. The issue concerns the highly heterogeneous nature of the background class, resulting in multi-modal distributions. Empirically, we find that neural networks trained with heterogeneous background struggle to map the corresponding contextual samples to compact clusters in feature space. As a result, the distribution over background logit activations may shift across the decision boundary, leading to systematic over-segmentation across different datasets and tasks. In this study, we propose context label learning (CoLab) to improve the context representations by decomposing the background class into several subclasses. Specifically, we train an auxiliary network as a task generator, along with the primary segmentation model, to automatically generate context labels that positively affect the ROI segmentation accuracy. Extensive experiments are conducted on several challenging segmentation tasks and datasets. The results demonstrate that CoLab can guide the segmentation model to map the logits of background samples away from the decision boundary, resulting in significantly improved segmentation accuracy. Code is available at https://github.com/ZerojumpLine/CoLab
As firm as their foundations: can open-sourced foundation models be used to create adversarial examples for downstream tasks?
Foundation models pre-trained on web-scale vision-language data, such as CLIP, are widely used as cornerstones of powerful machine learning systems. While pre-training offers clear advantages for downstream learning, it also endows downstream models with shared adversarial vulnerabilities that can be easily identified through the open-sourced foundation model. In this work, we expose such vulnerabilities among CLIP’s downstream models and show that foundation models can serve as a basis for attacking their downstream systems. In particular, we propose a simple yet alarmingly effective adversarial attack strategy termed Patch Representation Misalignment (PRM). Solely based on open-sourced CLIP vision encoders, this method can produce highly effective adversaries that simultaneously fool more than 20 downstream models spanning 4 common vision-language tasks (semantic segmentation, object detection, image captioning and visual question-answering). Our findings highlight the concerning safety risks introduced by the extensive usage of publicly available foundational models in the development of downstream systems, calling for extra caution in these scenarios
- …