927 research outputs found
ISM In-Space Manufacturing
Develop and enable the technologies, materials, and processes required to provide affordable, sustainable on-demand manufacturing, recycling, and repair during Exploration Missions
Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Multimodal learning seeks to utilize data from multiple sources to improve
the overall performance of downstream tasks. It is desirable for redundancies
in the data to make multimodal systems robust to missing or corrupted
observations in some correlated modalities. However, we observe that the
performance of several existing multimodal networks significantly deteriorates
if one or multiple modalities are absent at test time. To enable robustness to
missing modalities, we propose simple and parameter-efficient adaptation
procedures for pretrained multimodal networks. In particular, we exploit
low-rank adaptation and modulation of intermediate features to compensate for
the missing modalities. We demonstrate that such adaptation can partially
bridge performance drop due to missing modalities and outperform independent,
dedicated networks trained for the available modality combinations in some
cases. The proposed adaptation requires extremely small number of parameters
(e.g., fewer than 0.7% of the total parameters in most experiments). We conduct
a series of experiments to highlight the robustness of our proposed method
using diverse datasets for RGB-thermal and RGB-Depth semantic segmentation,
multimodal material segmentation, and multimodal sentiment analysis tasks. Our
proposed method demonstrates versatility across various tasks and datasets, and
outperforms existing methods for robust multimodal learning with missing
modalities.Comment: 18 pages, 3 figures, 11 table
Brief Note: Fishes of the Upper Portage River, Ohio, 1973-1975
Author Institution: Aqua Tech Environmental Consultants, Inc. ; Bowling Green State University, Department of Biological Sciences ; Aqua Tech Environmental Consultants, Inc
- …