3 research outputs found
Human Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of Depth and Inertial Sensors
Multimodal fusion frameworks for Human Action Recognition (HAR) using depth
and inertial sensor data have been proposed over the years. In most of the
existing works, fusion is performed at a single level (feature level or
decision level), missing the opportunity to fuse rich mid-level features
necessary for better classification. To address this shortcoming, in this
paper, we propose three novel deep multilevel multimodal fusion frameworks to
capitalize on different fusion strategies at various stages and to leverage the
superiority of multilevel fusion. At input, we transform the depth data into
depth images called sequential front view images (SFIs) and inertial sensor
data into signal images. Each input modality, depth and inertial, is further
made multimodal by taking convolution with the Prewitt filter. Creating
"modality within modality" enables further complementary and discriminative
feature extraction through Convolutional Neural Networks (CNNs). CNNs are
trained on input images of each modality to learn low-level, high-level and
complex features. Learned features are extracted and fused at different stages
of the proposed frameworks to combine discriminative and complementary
information. These highly informative features are served as input to a
multi-class Support Vector Machine (SVM). We evaluate the proposed frameworks
on three publicly available multimodal HAR datasets, namely, UTD Multimodal
Human Action Dataset (MHAD), Berkeley MHAD, and UTD-MHAD Kinect V2.
Experimental results show the supremacy of the proposed fusion frameworks over
existing methods.Comment: 10 pages, 13 figure
Vision and Inertial Sensing Fusion for Human Action Recognition : A Review
Human action recognition is used in many applications such as video
surveillance, human computer interaction, assistive living, and gaming. Many
papers have appeared in the literature showing that the fusion of vision and
inertial sensing improves recognition accuracies compared to the situations
when each sensing modality is used individually. This paper provides a survey
of the papers in which vision and inertial sensing are used simultaneously
within a fusion framework in order to perform human action recognition. The
surveyed papers are categorized in terms of fusion approaches, features,
classifiers, as well as multimodality datasets considered. Challenges as well
as possible future directions are also stated for deploying the fusion of these
two sensing modalities under realistic conditions.Comment: 14 pages,4 figures,2 tables. Submitted to IEEE Sensors Journa
A Survey of Data Fusion in Smart City Applications
The advancement of various research sectors such as Internet of Things (IoT),
Machine Learning, Data Mining, Big Data, and Communication Technology has shed
some light in transforming an urban city integrating the aforementioned
techniques to a commonly known term - Smart City. With the emergence of smart
city, plethora of data sources have been made available for wide variety of
applications. The common technique for handling multiple data sources is data
fusion, where it improves data output quality or extracts knowledge from the
raw data. In order to cater evergrowing highly complicated applications,
studies in smart city have to utilize data from various sources and evaluate
their performance based on multiple aspects. To this end, we introduce a
multi-perspectives classification of the data fusion to evaluate the smart city
applications. Moreover, we applied the proposed multi-perspectives
classification to evaluate selected applications in each domain of the smart
city. We conclude the paper by discussing potential future direction and
challenges of data fusion integration.Comment: Accepted and To be published in Elsevier Information Fusio