Robust Deep Multi-Modal Sensor Fusion using Fusion Weight Regularization and Target Learning

Sensor fusion has wide applications in many domains including health care and autonomous systems. While the advent of deep learning has enabled promising multi-modal fusion of high-level features and end-to-end sensor fusion solutions, existing deep learning based sensor fusion techniques including deep gating architectures are not always resilient, leading to the issue of fusion weight inconsistency. We propose deep multi-modal sensor fusion architectures with enhanced robustness particularly under the presence of sensor failures. At the core of our gating architectures are fusion weight regularization and fusion target learning operating on auxiliary unimodal sensing networks appended to the main fusion model. The proposed regularized gating architectures outperform the existing deep learning architectures with and without gating under both clean and corrupted sensory inputs resulted from sensor failures. The demonstrated improvements are particularly pronounced when one or more multiple sensory modalities are corrupted.Comment: 8 page

Auxiliary Loss Weighting for Robust Multi-Modal Sensor Fusion with Deep Neural Networks

The major focus of this research is on sensor fusion. Sensor fusion means to combine multiple sensory input or data from different source such that the performance is better than the best performance would be when those different data were used individually. As we know, in the world of AI, sensors are really important. Traditionally, we treat these data as they are separated. Doing so may deliver good performance when the sensors are exempt from noise and malfunction problems. However, if sensor failure appears, the performance will drop. Sensor fusion is a solution for the above situation. Recently, deep neural networks have been rigorously studied for sensor fusion applications such as autonomous driving and robot control. Among these studies, various gated neural network architectures were proposed, which have improved the existing classical convolutional neural networks (CNNs). Several problems existed for those gated neural network architectures. In this research, some of them are described. Then, to solve those problems, a further optimized gated architecture, a gated CNN with auxiliary paths, was proposed. The major focus of this thesis work is on auxiliary loss weighting, a technique to further regulate the gated CNNs with auxiliary paths and improve their performance. The CAD-60 dataset is utilized as a benchmark to demonstrate the significant performance improvements through the proposed architecture and its robustness in the presence of sensor noise and failures

Hardware Testbed and Deep Neural Networks for Multi-Modal Sensor Fusion

Deep neural networks (DNN) have been widely applied in sensor fusion, providing an end-to-end solution for fusion of features extracted from multiple sensory inputs. A class of new sensor fusion networks based on DNN called gating architectures proposed in recent years improves the prediction performances over the conventional fusion mechanisms employed in convolutional neural networks (CNNs). However, experimental results show that the gating architectures are not always robust and sometimes even underperform conventional fusion methods. In this work, the limitations of existing gating architectures are discussed and analyzed. Through experiments, we demonstrate that gating architectures fail to learn correct fusion weights for sensory inputs, showing the inconsistency between fusion weights and corresponding qualities of sensory inputs, and hence limit the prediction performance. We propose an improved fusion architecture by introducing the auxiliary path model to regulate the fusion weights in the gating architecture. We also provide in-depth studies on the regularization mechanisms to show that the improvements on performances are achieved by the more robustly learnt fusion weights. Evaluations are performed under two different public datasets. We generate comprehensive sensor failure schemes, where the proposed architecture significantly outperforms a baseline non-gating architecture and one existing gating architecture. We also build up a sensor fusion hardware platform: a robot car, which is equipped with multiple sensors. The robot will be further developed and adopted as a hardware platform for evaluating the proposed sensor fusion architecture