One-shot Localization and Segmentation of Medical Images with Foundation Models

Recent advances in Vision Transformers (ViT) and Stable Diffusion (SD) models with their ability to capture rich semantic features of the image have been used for image correspondence tasks on natural images. In this paper, we examine the ability of a variety of pre-trained ViT (DINO, DINOv2, SAM, CLIP) and SD models, trained exclusively on natural images, for solving the correspondence problems on medical images. While many works have made a case for in-domain training, we show that the models trained on natural images can offer good performance on medical images across different modalities (CT,MR,Ultrasound) sourced from various manufacturers, over multiple anatomical regions (brain, thorax, abdomen, extremities), and on wide variety of tasks. Further, we leverage the correspondence with respect to a template image to prompt a Segment Anything (SAM) model to arrive at single shot segmentation, achieving dice range of 62%-90% across tasks, using just one image as reference. We also show that our single-shot method outperforms the recently proposed few-shot segmentation method - UniverSeg (Dice range 47%-80%) on most of the semantic segmentation tasks(six out of seven) across medical imaging modalities.Comment: Accepted at NeurIPS 2023 R0-FoMo Worksho

Investigating neuromagnetic brain responses against chromatic flickering stimuli by wavelet entropies

BACKGROUND: Photosensitive epilepsy is a type of reflexive epilepsy triggered by various visual stimuli including colourful ones. Despite the ubiquitous presence of colorful displays, brain responses against different colour combinations are not properly studied. METHODOLOGY/PRINCIPAL FINDINGS: Here, we studied the photosensitivity of the human brain against three types of chromatic flickering stimuli by recording neuromagnetic brain responses (magnetoencephalogram, MEG) from nine adult controls, an unmedicated patient, a medicated patient, and two controls age-matched with patients. Dynamical complexities of MEG signals were investigated by a family of wavelet entropies. Wavelet entropy is a newly proposed measure to characterize large scale brain responses, which quantifies the degree of order/disorder associated with a multi-frequency signal response. In particular, we found that as compared to the unmedicated patient, controls showed significantly larger wavelet entropy values. We also found that Renyi entropy is the most powerful feature for the participant classification. Finally, we also demonstrated the effect of combinational chromatic sensitivity on the underlying order/disorder in MEG signals. CONCLUSIONS/SIGNIFICANCE: Our results suggest that when perturbed by potentially epileptic-triggering stimulus, healthy human brain manages to maintain a non-deterministic, possibly nonlinear state, with high degree of disorder, but an epileptic brain represents a highly ordered state which making it prone to hyper-excitation. Further, certain colour combination was found to be more threatening than other combinations

Geometric Distortion

interlaced q-space sampling in diffusion MR

Temporal analysis.

<p>Same as <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0007173#pone-0007173-g003" target="_blank">Fig. 3</a> but for Red/Green flickering.</p

Systematic illustration of wavelet entropy method.

<p>MEG signal was first transformed to mutiresolution time-frequency domain by wavelet transformation. Then the values were windowed and corresponding energy was computed in all resolution. Finally wavelet entropy was computed.</p

Spatial analysis.

<p>Entropy values (mean±s.e.) in seven broad cortical regions for Red/Blue flickering stimulus for adult controls, one unmedicated patient, one medicated patient, and two further controls age-matched with patients. Then mean entropy value across time windows excluding the first window was calculated within each category. Note lower entropy values for unmedicated patients and higher entropy values for control participants across cortical regions.</p

Temporal analysis.

<p>Temporal evolution of entropy values (mean±s.e.) for Red/Blue flicker. <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0007173#s3" target="_blank">Results</a> were first averaged over all the sensors for each participant. Each point represents the center of each window which was 400 ms long. The onset of visual stimulus was indicated by a vertical line. Note similar entropy values for all the participants at the first window ranging from 200 ms prestimulus to 200 ms post-stimulus. However, entropy values reduced drastically in the patient at later stages of poststimulus processing.</p