33,657 research outputs found
Deep combination of radar with optical data for gesture recognition: role of attention in fusion architectures
Multimodal time series classification is an important aspect of human gesture recognition, in which limitations of individual sensors can be overcome by combining data from multiple modalities. In a deep learning pipeline, the attention mechanism further allows for a selective, contextual concentration on relevant features. However, while the standard attention mechanism is an effective tool when working with Natural Language Processing (NLP), it is not ideal when working with temporally- or spatially-sparse multi-modal data. In this paper, we present a novel attention mechanism, Multi-Modal Attention Preconditioning (MMAP). We first demonstrate that MMAP outperforms regular attention for the task of classification of modalities involving temporal and spatial sparsity and secondly investigate the impact of attention in the fusion of radar and optical data for gesture recognition via three specific modalities: dense spatiotemporal optical data, spatially sparse/temporally dense kinematic data, and sparse spatiotemporal radar data. We explore the effect of attention on early, intermediate, and late fusion architectures and compare eight different pipelines in terms of accuracy and their ability to preserve detection accuracy when modalities are missing. Results highlight fundamental differences between late and intermediate attention mechanisms in respect to the fusion of radar and optical data
A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community
In recent years, deep learning (DL), a re-branding of neural networks (NNs),
has risen to the top in numerous areas, namely computer vision (CV), speech
recognition, natural language processing, etc. Whereas remote sensing (RS)
possesses a number of unique challenges, primarily related to sensors and
applications, inevitably RS draws from many of the same theories as CV; e.g.,
statistics, fusion, and machine learning, to name a few. This means that the RS
community should be aware of, if not at the leading edge of, of advancements
like DL. Herein, we provide the most comprehensive survey of state-of-the-art
RS DL research. We also review recent new developments in the DL field that can
be used in DL for RS. Namely, we focus on theories, tools and challenges for
the RS community. Specifically, we focus on unsolved challenges and
opportunities as it relates to (i) inadequate data sets, (ii)
human-understandable solutions for modelling physical phenomena, (iii) Big
Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and
learning algorithms for spectral, spatial and temporal data, (vi) transfer
learning, (vii) an improved theoretical understanding of DL systems, (viii)
high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote
Sensin
IOD-CNN: Integrating Object Detection Networks for Event Recognition
Many previous methods have showed the importance of considering semantically
relevant objects for performing event recognition, yet none of the methods have
exploited the power of deep convolutional neural networks to directly integrate
relevant object information into a unified network. We present a novel unified
deep CNN architecture which integrates architecturally different, yet
semantically-related object detection networks to enhance the performance of
the event recognition task. Our architecture allows the sharing of the
convolutional layers and a fully connected layer which effectively integrates
event recognition, rigid object detection and non-rigid object detection.Comment: submitted to IEEE International Conference on Image Processing 201
Deep learning in remote sensing: a review
Standing at the paradigm shift towards data-intensive science, machine
learning techniques are becoming increasingly important. In particular, as a
major breakthrough in the field, deep learning has proven as an extremely
powerful tool in many fields. Shall we embrace deep learning as the key to all?
Or, should we resist a 'black-box' solution? There are controversial opinions
in the remote sensing community. In this article, we analyze the challenges of
using deep learning for remote sensing data analysis, review the recent
advances, and provide resources to make deep learning in remote sensing
ridiculously simple to start with. More importantly, we advocate remote sensing
scientists to bring their expertise into deep learning, and use it as an
implicit general model to tackle unprecedented large-scale influential
challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin
A Hybrid Approach for Data Analytics for Internet of Things
The vision of the Internet of Things is to allow currently unconnected
physical objects to be connected to the internet. There will be an extremely
large number of internet connected devices that will be much more than the
number of human being in the world all producing data. These data will be
collected and delivered to the cloud for processing, especially with a view of
finding meaningful information to then take action. However, ideally the data
needs to be analysed locally to increase privacy, give quick responses to
people and to reduce use of network and storage resources. To tackle these
problems, distributed data analytics can be proposed to collect and analyse the
data either in the edge or fog devices. In this paper, we explore a hybrid
approach which means that both innetwork level and cloud level processing
should work together to build effective IoT data analytics in order to overcome
their respective weaknesses and use their specific strengths. Specifically, we
collected raw data locally and extracted features by applying data fusion
techniques on the data on resource constrained devices to reduce the data and
then send the extracted features to the cloud for processing. We evaluated the
accuracy and data consumption over network and thus show that it is feasible to
increase privacy and maintain accuracy while reducing data communication
demands.Comment: Accepted to be published in the Proceedings of the 7th ACM
International Conference on the Internet of Things (IoT 2017
- …