Deep learning, remote sensing and visual analytics to support automatic flood detection

Floods can have devastating consequences on people, infrastructure, and the ecosystem. Satellite imagery has proven to be an efficient instrument in supporting disaster management authorities during flood events. In contrast to optical remote sensing technology, Synthetic Aperture Radar (SAR) can penetrate clouds, and authorities can use SAR images even during cloudy circumstances. A challenge with SAR is the accurate classification and segmentation of flooded areas from SAR imagery. Recent advancements in deep learning algorithms have demonstrated the potential of deep learning for image segmentation demonstrated. Our research adopted deep learning algorithms to classify and segment flooded areas in SAR imagery. We used UNet and Feature Pyramid Network (FPN), both based on EfficientNet-B7 implementation, to detect flooded areas in SAR imaginary of Nebraska, North Alabama, Bangladesh, Red River North, and Florence. We evaluated both deep learning methods' predictive accuracy and will present the evaluation results at the conference. In the next step of our research, we develop an XAI toolbox to support the interpretation of detected flooded areas and algorithmic decisions of the deep learning methods through interactive visualizations

When XR Meets AI: Integrating Interactive Machine Learning with an XR Musical Instrument

This paper explores the integration of artificial intelligence (AI) with extended reality (XR) through the development of Netz, an XR musical instrument (XRMI) designed to enhance musical expression and instrument interaction via deep learning. Netz utilises algorithms to map physical gestures to digital controls and allows for customisable control schemes, improving the accuracy of gesture interpretation and the overall musical experience. The instrument was developed through a participatory design process involving a professional keyboard player and music producer over three phases: exploration, making, and performance & refinement. Initial challenges with traditional computational approaches to hand-pose classification were overcome by incorporating an interactive machine learning (IML) model, enabling personalised gesture control. Evaluation included user tasks and thematic analysis of interviews, highlighting improved interaction and the potential of AI to augment musical performance in XR. The study is limited by a single participant evaluation. Future work will involve a wider range of musicians to assess the generalisability of our findings

Deep Interactive Region Segmentation and Captioning

With recent innovations in dense image captioning, it is now possible to describe every object of the scene with a caption while objects are determined by bounding boxes. However, interpretation of such an output is not trivial due to the existence of many overlapping bounding boxes. Furthermore, in current captioning frameworks, the user is not able to involve personal preferences to exclude out of interest areas. In this paper, we propose a novel hybrid deep learning architecture for interactive region segmentation and captioning where the user is able to specify an arbitrary region of the image that should be processed. To this end, a dedicated Fully Convolutional Network (FCN) named Lyncean FCN (LFCN) is trained using our special training data to isolate the User Intention Region (UIR) as the output of an efficient segmentation. In parallel, a dense image captioning model is utilized to provide a wide variety of captions for that region. Then, the UIR will be explained with the caption of the best match bounding box. To the best of our knowledge, this is the first work that provides such a comprehensive output. Our experiments show the superiority of the proposed approach over state-of-the-art interactive segmentation methods on several well-known datasets. In addition, replacement of the bounding boxes with the result of the interactive segmentation leads to a better understanding of the dense image captioning output as well as accuracy enhancement for the object detection in terms of Intersection over Union (IoU).Comment: 17, pages, 9 figure