Toward Foundation Models for Earth Monitoring: Generalizable Deep Learning Models for Natural Hazard Segmentation

Climate change results in an increased probability of extreme weather events that put societies and businesses at risk on a global scale. Therefore, near real-time mapping of natural hazards is an emerging priority for the support of natural disaster relief, risk management, and informing governmental policy decisions. Recent methods to achieve near real-time mapping increasingly leverage deep learning (DL). However, DL-based approaches are designed for one specific task in a single geographic region based on specific frequency bands of satellite data. Therefore, DL models used to map specific natural hazards struggle with their generalization to other types of natural hazards in unseen regions. In this work, we propose a methodology to significantly improve the generalizability of DL natural hazards mappers based on pre-training on a suitable pre-task. Without access to any data from the target domain, we demonstrate this improved generalizability across four U-Net architectures for the segmentation of unseen natural hazards. Importantly, our method is invariant to geographic differences and differences in the type of frequency bands of satellite data. By leveraging characteristics of unlabeled images from the target domain that are publicly available, our approach is able to further improve the generalization behavior without fine-tuning. Thereby, our approach supports the development of foundation models for earth monitoring with the objective of directly segmenting unseen natural hazards across novel geographic regions given different sources of satellite imagery.Comment: Accepted at IEEE International Geoscience and Remote Sensing Symposium (IGARSS 2023

Multimodal Affect and Aesthetic Experience

The term “aesthetic experience” corresponds to the inner state of a person exposed to form and content of artistic objects. Exploring certain aesthetic values of artistic objects, as well as interpreting the aesthetic experience of people when exposed to art can contribute towards understanding (a) art and (b) people’s affective reactions to artwork. Focusing on different types of artistic content, such as movies, music, urban art and other artwork, the goal of this workshop is to enhance the interdisciplinary collaboration between affective computing and aesthetics researchers

Hybrid Virtual Coaching and Telemonitoring in COPD Management: The CAir Randomised Controlled Study

Objective: To investigate the effectiveness of 12-weeks hybrid virtual coaching on health-related quality-of-life (HrQoL) in patients with stable COPD. Methods: We equipped all patients with a CAir Desk for telemonitoring, the intervention group additionally received hybrid virtual coaching through the built-in smartphone. The multimodal intervention based on the Living well with COPD programme, containing educational content, physical activity coaching, and home-based exercises. Primary outcome was HrQoL as measured by the SGRQ. Secondary outcomes were symptom burden, physical activity, functional exercise capacity, and lung function. Between-group differences were calculated using linear regression models, controlling for baseline differences. Results: We included 30 participants with COPD (13/17 women/men; 63 [9] years; FEV$_{1}$ 54 [22] % predicted), 24 (80%) completed the study. SGRQ improved in both groups (intervention: -4.5 [20.1]; control: -2.7 [7.4] points) without statistically significant or clinically relevant between-group differences (B = -2.5 points, 95% CI = -24.3, 19.3, p = 0.81). Physical activity increased only in the intervention group (313 [1561] vs -364 [2399] steps) without statistically significant but clinically relevant between-group difference (B = 2147 steps, 95% CI = -86, 4395, p = 0.06). Symptom burden decreased in both groups (-4.2 [6.7] vs -1.0 [2.8] points) without statistically significant but clinically relevant between-group difference (B = -3.0 points, 95% CI = -10.8, 5.0, p = 0.43). Conclusion: Twelve-weeks hybrid virtual coaching did not improve HrQoL more than telemonitoring only in patients with stable COPD. The intervention group improved their physical activity and symptom burden clinically relevant more than the control group

Multimodal Remote Home Monitoring of Lung Transplant Recipients during COVID-19 Vaccinations: Usability Pilot Study of the COVIDA Desk Incorporating Wearable Devices

Background and Objectives: Remote patient monitoring (RPM) of vital signs and symptoms for lung transplant recipients (LTRs) has become increasingly relevant in many situations. Nevertheless, RPM research integrating multisensory home monitoring in LTRs is scarce. We developed a novel multisensory home monitoring device and tested it in the context of COVID-19 vaccinations. We hypothesize that multisensory RPM and smartphone-based questionnaire feedback on signs and symptoms will be well accepted among LTRs. To assess the usability and acceptability of a remote monitoring system consisting of wearable devices, including home spirometry and a smartphone-based questionnaire application for symptom and vital sign monitoring using wearable devices, during the first and second SARS-CoV-2 vaccination. Materials and Methods: Observational usability pilot study for six weeks of home monitoring with the COVIDA Desk for LTRs. During the first week after the vaccination, intensive monitoring was performed by recording data on physical activity, spirometry, temperature, pulse oximetry and self-reported symptoms, signs and additional measurements. During the subsequent days, the number of monitoring assessments was reduced. LTRs reported on their perceptions of the usability of the monitoring device through a purpose-designed questionnaire. Results: Ten LTRs planning to receive the first COVID-19 vaccinations were recruited. For the intensive monitoring study phase, LTRs recorded symptoms, signs and additional measurements. The most frequent adverse events reported were local pain, fatigue, sleep disturbance and headache. The duration of these symptoms was 5–8 days post-vaccination. Adherence to the main monitoring devices was high. LTRs rated usability as high. The majority were willing to continue monitoring. Conclusions: The COVIDA Desk showed favorable technical performance and was well accepted by the LTRs during the vaccination phase of the pandemic. The feasibility of the RPM system deployment was proven by the rapid recruitment uptake, technical performance (i.e., low number of errors), favorable user experience questionnaires and detailed individual user feedback

Fine-tuning of Geospatial Foundation Models for Aboveground Biomass Estimation

Global vegetation structure mapping is critical for understanding the global carbon cycle and maximizing the efficacy of nature-based carbon sequestration initiatives. Moreover, vegetation structure mapping can help reduce the impacts of climate change by, for example, guiding actions to improve water security, increase biodiversity and reduce flood risk. Global satellite measurements provide an important set of observations for monitoring and managing deforestation and degradation of existing forests, natural forest regeneration, reforestation, biodiversity restoration, and the implementation of sustainable agricultural practices. In this paper, we explore the effectiveness of fine-tuning of a geospatial foundation model to estimate above-ground biomass (AGB) using space-borne data collected across different eco-regions in Brazil. The fine-tuned model architecture consisted of a Swin-B transformer as the encoder (i.e., backbone) and a single convolutional layer for the decoder head. All results were compared to a U-Net which was trained as the baseline model Experimental results of this sparse-label prediction task demonstrate that the fine-tuned geospatial foundation model with a frozen encoder has comparable performance to a U-Net trained from scratch. This is despite the fine-tuned model having 13 times less parameters requiring optimization, which saves both time and compute resources. Further, we explore the transfer-learning capabilities of the geospatial foundation models by fine-tuning on satellite imagery with sparse labels from different eco-regions in Brazil

Recognizing film aesthetics, spectators' affect and aesthetic emotions from multimodal signals

Even though aesthetic experiences area common in our lives, processes involved in aesthetic experience are not fully understood. Moreover, there is no comprehensive theory that explains and defines the concept of aesthetic experience in art. The challenge of studies on aesthetic experiences is to understand different stages of aesthetic information processing, such as perceptual analysis, cognitive processes, and evaluation resulting in aesthetic judgments and emotions. The main goal of this thesis is to analyze film aesthetic experience evoked in spectators. In particular, we aim to detect aesthetic highlights in movies, as well as recognize induced emotions and aesthetic emotions elicited in spectators. The outcomes of the research on induced emotions, aesthetic emotions, and aesthetic highlights allow researchers to better understand processes involved in film aesthetic experience and can be used for emotional and aesthetic scene detection, emotional and aesthetic scene design, video summarization, and prediction of affective and aesthetic content

A reconfigurable integrated electronic tongue and its use in accelerated analysis of juices and wines

Potentiometric electronic tongues (ETs) leveraging trends in miniaturization and internet of things (IoT) bear promise for facile mobile chemical analysis of complex multicomponent liquids, such as beverages. In this work, hand-crafted feature extraction from the transient potentiometric response of an array of low-selective miniaturized polymeric sensors is combined with a data pipeline for deployment of trained machine learning models on a cloud back-end or edge device. The sensor array demonstrated sensitivity to different organic acids and exhibited interesting performance for the fingerprinting of fruit juices and wines, including differentiation of samples through supervised learning based on sensory descriptors and prediction of consumer acceptability of aged juice samples. Product authentication, quality control and support of sensory evaluation are some of the applications that are expected to benefit from integrated electronic tongues that facilitate the characterization of complex properties of multi-component liquids