759 research outputs found
Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data
There are threefold challenges in emotion recognition. First, it is difficult
to recognize human's emotional states only considering a single modality.
Second, it is expensive to manually annotate the emotional data. Third,
emotional data often suffers from missing modalities due to unforeseeable
sensor malfunction or configuration issues. In this paper, we address all these
problems under a novel multi-view deep generative framework. Specifically, we
propose to model the statistical relationships of multi-modality emotional data
using multiple modality-specific generative networks with a shared latent
space. By imposing a Gaussian mixture assumption on the posterior approximation
of the shared latent variables, our framework can learn the joint deep
representation from multiple modalities and evaluate the importance of each
modality simultaneously. To solve the labeled-data-scarcity problem, we extend
our multi-view model to semi-supervised learning scenario by casting the
semi-supervised classification problem as a specialized missing data imputation
task. To address the missing-modality problem, we further extend our
semi-supervised multi-view model to deal with incomplete data, where a missing
view is treated as a latent variable and integrated out during inference. This
way, the proposed overall framework can utilize all available (both labeled and
unlabeled, as well as both complete and incomplete) data to improve its
generalization ability. The experiments conducted on two real multi-modal
emotion datasets demonstrated the superiority of our framework.Comment: arXiv admin note: text overlap with arXiv:1704.07548, 2018 ACM
Multimedia Conference (MM'18
Deep learning for the early detection of harmful algal blooms and improving water quality monitoring
Climate change will affect how water sources are managed and monitored. The frequency of algal blooms will increase with climate change as it presents favourable conditions for the reproduction of phytoplankton. During monitoring, possible sensory failures in monitoring systems result in partially filled data which may affect critical systems. Therefore, imputation becomes necessary to decrease error and increase data quality. This work investigates two issues in water quality data analysis: improving data quality and anomaly detection. It consists of three main topics: data imputation, early algal bloom detection using in-situ data and early algal bloom detection using multiple modalities.The data imputation problem is addressed by experimenting with various methods with a water quality dataset that includes four locations around the North Sea and the Irish Sea with different characteristics and high miss rates, testing model generalisability. A novel neural network architecture with self-attention is proposed in which imputation is done in a single pass, reducing execution time. The self-attention components increase the interpretability of the imputation process at each stage of the network, providing knowledge to domain experts.After data curation, algal activity is predicted using transformer networks, between 1 to 7 days ahead, and the importance of the input with regard to the output of the prediction model is explained using SHAP, aiming to explain model behaviour to domain experts which is overlooked in previous approaches. The prediction model improves bloom detection performance by 5% on average and the explanation summarizes the complex structure of the model to input-output relationships. Performance improvements on the initial unimodal bloom detection model are made by incorporating multiple modalities into the detection process which were only used for validation purposes previously. The problem of missing data is also tackled by using coordinated representations, replacing low quality in-situ data with satellite data and vice versa, instead of imputation which may result in biased results
Automatic Sensor-free Affect Detection: A Systematic Literature Review
Emotions and other affective states play a pivotal role in cognition and,
consequently, the learning process. It is well-established that computer-based
learning environments (CBLEs) that can detect and adapt to students' affective
states can enhance learning outcomes. However, practical constraints often pose
challenges to the deployment of sensor-based affect detection in CBLEs,
particularly for large-scale or long-term applications. As a result,
sensor-free affect detection, which exclusively relies on logs of students'
interactions with CBLEs, emerges as a compelling alternative. This paper
provides a comprehensive literature review on sensor-free affect detection. It
delves into the most frequently identified affective states, the methodologies
and techniques employed for sensor development, the defining attributes of
CBLEs and data samples, as well as key research trends. Despite the field's
evident maturity, demonstrated by the consistent performance of the models and
the application of advanced machine learning techniques, there is ample scope
for future research. Potential areas for further exploration include enhancing
the performance of sensor-free detection models, amassing more samples of
underrepresented emotions, and identifying additional emotions. There is also a
need to refine model development practices and methods. This could involve
comparing the accuracy of various data collection techniques, determining the
optimal granularity of duration, establishing a shared database of action logs
and emotion labels, and making the source code of these models publicly
accessible. Future research should also prioritize the integration of models
into CBLEs for real-time detection, the provision of meaningful interventions
based on detected emotions, and a deeper understanding of the impact of
emotions on learning
Human Observer and Automatic Assessment of Movement Related Self-Efficacy in Chronic Pain: from Exercise to Functional Activity
Clinicians tailor intervention in chronic pain rehabilitation to movement related self-efficacy (MRSE). This motivates us to investigate automatic MRSE estimation in this context towards the development of technology that is able to provide appropriate support in the absence of a clinician. We first explored clinical observer estimation, which showed that body movement behaviours, rather than facial expressions or engagement behaviours, were more pertinent to MRSE estimation during physical activity instances. Based on our findings, we built a system that estimates MRSE from bodily expressions and bodily muscle activity captured using wearable sensors. Our results (F1 scores of 0.95 and 0.78 in two physical exercise types) provide evidence of the feasibility of automatic MRSE estimation to support chronic pain physical rehabilitation. We further explored automatic estimation of MRSE with a reduced set of low-cost sensors to investigate the possibility of embedding such capabilities in ubiquitous wearable devices to support functional activity. Our evaluation for both exercise and functional activity resulted in F1 score of 0.79. This result suggests the possibility of (and calls for more studies on) MRSE estimation during everyday functioning in ubiquitous settings. We provide a discussion of the implication of our findings for relevant areas
MultiWave: Multiresolution Deep Architectures through Wavelet Decomposition for Multivariate Time Series Prediction
The analysis of multivariate time series data is challenging due to the
various frequencies of signal changes that can occur over both short and long
terms. Furthermore, standard deep learning models are often unsuitable for such
datasets, as signals are typically sampled at different rates. To address these
issues, we introduce MultiWave, a novel framework that enhances deep learning
time series models by incorporating components that operate at the intrinsic
frequencies of signals. MultiWave uses wavelets to decompose each signal into
subsignals of varying frequencies and groups them into frequency bands. Each
frequency band is handled by a different component of our model. A gating
mechanism combines the output of the components to produce sparse models that
use only specific signals at specific frequencies. Our experiments demonstrate
that MultiWave accurately identifies informative frequency bands and improves
the performance of various deep learning models, including LSTM, Transformer,
and CNN-based models, for a wide range of applications. It attains top
performance in stress and affect detection from wearables. It also increases
the AUC of the best-performing model by 5% for in-hospital COVID-19 mortality
prediction from patient blood samples and for human activity recognition from
accelerometer and gyroscope data. We show that MultiWave consistently
identifies critical features and their frequency components, thus providing
valuable insights into the applications studied.Comment: Published in the Conference on Health, Inference, and Learning (CHIL
2023
- …