3 research outputs found
Leveraging Multimodal Fusion for Enhanced Diagnosis of Multiple Retinal Diseases in Ultra-wide OCTA
Ultra-wide optical coherence tomography angiography (UW-OCTA) is an emerging
imaging technique that offers significant advantages over traditional OCTA by
providing an exceptionally wide scanning range of up to 24 x 20 ,
covering both the anterior and posterior regions of the retina. However, the
currently accessible UW-OCTA datasets suffer from limited comprehensive
hierarchical information and corresponding disease annotations. To address this
limitation, we have curated the pioneering M3OCTA dataset, which is the first
multimodal (i.e., multilayer), multi-disease, and widest field-of-view UW-OCTA
dataset. Furthermore, the effective utilization of multi-layer ultra-wide
ocular vasculature information from UW-OCTA remains underdeveloped. To tackle
this challenge, we propose the first cross-modal fusion framework that
leverages multi-modal information for diagnosing multiple diseases. Through
extensive experiments conducted on our openly available M3OCTA dataset, we
demonstrate the effectiveness and superior performance of our method, both in
fixed and varying modalities settings. The construction of the M3OCTA dataset,
the first multimodal OCTA dataset encompassing multiple diseases, aims to
advance research in the ophthalmic image analysis community
Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation
In this paper, we examine the recent Segment Anything Model (SAM) on medical
images, and report both quantitative and qualitative zero-shot segmentation
results on nine medical image segmentation benchmarks, covering various imaging
modalities, such as optical coherence tomography (OCT), magnetic resonance
imaging (MRI), and computed tomography (CT), as well as different applications
including dermatology, ophthalmology, and radiology. Those benchmarks are
representative and commonly used in model development. Our experimental results
indicate that while SAM presents remarkable segmentation performance on images
from the general domain, its zero-shot segmentation ability remains restricted
for out-of-distribution images, e.g., medical images. In addition, SAM exhibits
inconsistent zero-shot segmentation performance across different unseen medical
domains. For certain structured targets, e.g., blood vessels, the zero-shot
segmentation of SAM completely failed. In contrast, a simple fine-tuning of it
with a small amount of data could lead to remarkable improvement of the
segmentation quality, showing the great potential and feasibility of using
fine-tuned SAM to achieve accurate medical image segmentation for a precision
diagnostics. Our study indicates the versatility of generalist vision
foundation models on medical imaging, and their great potential to achieve
desired performance through fine-turning and eventually address the challenges
associated with accessing large and diverse medical datasets in support of
clinical diagnostics.Comment: Published in Diagnostic
Large AI Models in Health Informatics: Applications, Challenges, and the Future
Large AI models, or foundation models, are models recently emerging with
massive scales both parameter-wise and data-wise, the magnitudes of which can
reach beyond billions. Once pretrained, large AI models demonstrate impressive
performance in various downstream tasks. A prime example is ChatGPT, whose
capability has compelled people's imagination about the far-reaching influence
that large AI models can have and their potential to transform different
domains of our lives. In health informatics, the advent of large AI models has
brought new paradigms for the design of methodologies. The scale of multi-modal
data in the biomedical and health domain has been ever-expanding especially
since the community embraced the era of deep learning, which provides the
ground to develop, validate, and advance large AI models for breakthroughs in
health-related areas. This article presents a comprehensive review of large AI
models, from background to their applications. We identify seven key sectors in
which large AI models are applicable and might have substantial influence,
including 1) bioinformatics; 2) medical diagnosis; 3) medical imaging; 4)
medical informatics; 5) medical education; 6) public health; and 7) medical
robotics. We examine their challenges, followed by a critical discussion about
potential future directions and pitfalls of large AI models in transforming the
field of health informatics.Comment: This article has been accepted for publication in IEEE Journal of
Biomedical and Health Informatic