WiSARD: A Labeled Visual and Thermal Image Dataset for Wilderness Search and Rescue

Sensor-equipped unoccupied aerial vehicles (UAVs) have the potential to help reduce search times and alleviate safety risks for first responders carrying out Wilderness Search and Rescue (WiSAR) operations, the process of finding and rescuing person(s) lost in wilderness areas. Unfortunately, visual sensors alone do not address the need for robustness across all the possible terrains, weather, and lighting conditions that WiSAR operations can be conducted in. The use of multi-modal sensors, specifically visual-thermal cameras, is critical in enabling WiSAR UAVs to perform in diverse operating conditions. However, due to the unique challenges posed by the wilderness context, existing dataset benchmarks are inadequate for developing vision-based algorithms for autonomous WiSAR UAVs. To this end, we present WiSARD, a dataset with roughly 56,000 labeled visual and thermal images collected from UAV flights in various terrains, seasons, weather, and lighting conditions. To the best of our knowledge, WiSARD is the first large-scale dataset collected with multi-modal sensors for autonomous WiSAR operations. We envision that our dataset will provide researchers with a diverse and challenging benchmark that can test the robustness of their algorithms when applied to real-world (life-saving) applications

Capacitive sensor to detect fallen humans in conditions of low visibility

This paper examines the potential for a capacitive sensor to be used as part of a system to detect fallen humans at very close range. Previous research suggests that a robotic system incorporating a low cost capacitive sensor could potentially distinguish between different materials. The work reported in this paper stemmed from an attempt to determine the true extent to which such a system might reliably differentiate between fallen humans and other objects. The work is motivated by the fact that there are several different emergency circumstances in which such a system might save lives if it could reliably detect immobile humans. These scenarios include situations where older people have fallen and are unable to move or raise an alert, and circumstances where people have been overcome by smoke in a burning building. Current sensing systems are typically unsuitable in conditions of low visibility such as smoke filled rooms. This analysis focused specifically on the potential for a robot equipped with a capacitive sensing system to identify an immobile human in a low visibility emergency scenario. It is concluded that further work would be required to determine whether this type of capacitive sensing system is genuinely suitable for this task

무인비행체 탑재 열화상 및 실화상 이미지를 활용한 야생동물 탐지 가능성 연구

학위논문(석사) -- 서울대학교대학원 : 환경대학원 환경조경학과, 2022.2. 송영근.야생동물의 탐지와 모니터링을 위해, 현장 직접 관찰, 포획-재포획과 같은 전통적 조사 방법이 다양한 목적으로 수행되어왔다. 하지만, 이러한 방법들은 많은 시간과 상대적으로 비싼 비용이 필요하며, 신뢰 가능한 탐지 결과를 얻기 위해선 숙련된 현장 전문가가 필요하다. 게다가, 전통적인 현장 조사 방법은 현장에서 야생동물을 마주치는 등 위험한 상황에 처할 수 있다. 이에 따라, 카메라 트래핑, GPS 추적, eDNA 샘플링과 같은 원격 조사 방법이 기존의 전통적 조사방법을 대체하며 더욱 빈번히 사용되기 시작했다. 하지만, 이러한 방법들은 여전히 목표로 하는 대상의 전체 면적과, 개별 개체를 탐지할 수 없다는 한계를 가지고 있다. 이러한 한계를 극복하기 위해, 무인비행체 (UAV, Unmanned Aerial Vehicle)가 야생동물 탐지의 대중적인 도구로 자리매김하고 있다. UAV의 가장 큰 장점은, 선명하고 촘촘한 공간 및 시간해상도와 함께 전체 연구 지역에 대한 동물 탐지가 가능하다는 것이다. 이에 더해, UAV를 사용함으로써, 접근하기 어려운 지역이나 위험한 곳에 대한 조사가 가능해진다. 하지만, 이러한 이점 외에, UAV의 단점도 명확히 존재한다. 대상지, 비행 속도 및 높이 등과 같이 UAV를 사용하는 환경에 따라, 작은 동물, 울창한 숲속에 있는 개체, 빠르게 움직이는 동물을 탐지하는 것이 제한된다. 또한, 기상환경에 따라서도 비행이 불가할 수 있고, 배터리 용량으로 인한 비행시간의 제한도 존재한다. 하지만, 정밀한 탐지가 불가능하더라도, 이와 관련 연구가 꾸준히 수행되고 있으며, 선행연구들은 육상 및 해상 포유류, 조류, 그리고 파충류 등을 탐지하는 데에 성공하였다. UAV를 통해 얻어지는 가장 대표적인 데이터는 실화상 이미지이다. 이를 사용해 머신러닝 및 딥러닝 (ML-DL, Machine Learning and Deep Learning) 방법이 주로 사용되고 있다. 이러한 방법은 상대적으로 정확한 탐지 결과를 보여주지만, 특정 종을 탐지할 수 있는 모델의 개발을 위해선 최소한 천 장의 이미지가 필요하다. 실화상 이미지 외에도, 열화상 이미지 또한 UAV를 통해 획득 될 수 있다. 열화상 센서 기술의 개발과 센서 가격의 하락은 많은 야생동물 연구자들의 관심을 사로잡았다. 열화상 카메라를 사용하면 동물의 체온과 주변환경과의 온도 차이를 통해 정온동물을 탐지하는 것이 가능하다. 하지만, 새로운 데이터가 사용되더라도, 여전히 ML-DL 방법이 동물 탐지에 주로 사용되고 있으며, 이러한 방법은 UAV를 활용한 야생동물의 실시간 탐지를 제한한다. 따라서, 본 연구는 열화상과 실화상 이미지를 활용한 동물 자동 탐지 방법의 개발과, 개발된 방법이 이전 방법들의 평균 이상의 정확도와 함께 현장에서 실시간으로 사용될 수 있도록 하는 것을 목표로 한다.For wildlife detection and monitoring, traditional methods such as direct observation and capture-recapture have been carried out for diverse purposes. However, these methods require a large amount of time, considerable expense, and field-skilled experts to obtain reliable results. Furthermore, performing a traditional field survey can result in dangerous situations, such as an encounter with wild animals. Remote monitoring methods, such as those based on camera trapping, GPS collars, and environmental DNA sampling, have been used more frequently, mostly replacing traditional survey methods, as the technologies have developed. But these methods still have limitations, such as the lack of ability to cover an entire region or detect individual targets. To overcome those limitations, the unmanned aerial vehicle (UAV) is becoming a popular tool for conducting a wildlife census. The main benefits of UAVs are able to detect animals remotely covering a wider region with clear and fine spatial and temporal resolutions. In addition, by operating UAVs investigate hard to access or dangerous areas become possible. However, besides these advantages, the limitations of UAVs clearly exist. By UAV operating environments such as study site, flying height or speed, the ability to detect small animals, targets in the dense forest, tracking fast-moving animals can be limited. And by the weather, operating UAV is unable, and the flight time is limited by the battery matters. Although detailed detection is unavailable, related researches are developing and previous studies used UAV to detect terrestrial and marine mammals, avian and reptile species. The most common type of data acquired by UAVs is RGB images. Using these images, machine-learning and deep-learning (ML–DL) methods were mainly used for wildlife detection. ML–DL methods provide relatively accurate results, but at least 1,000 images are required to develop a proper detection model for specific species. Instead of RGB images, thermal images can be acquired by a UAV. The development of thermal sensor technology and sensor price reduction has attracted the interest of wildlife researchers. Using a thermal camera, homeothermic animals can be detected based on the temperature difference between their bodies and the surrounding environment. Although the technology and data are new, the same ML–DL methods were typically used for animal detection. These ML-DL methods limit the use of UAVs for real-time wildlife detection in the field. Therefore, this paper aims to develop an automated animal detection method with thermal and RGB image datasets and to utilize it under in situ conditions in real-time while ensuring the average-above detection ability of previous methods.Abstract I Contents IV List of Tables VII List of Figures VIII Chapter 1. Introduction 1 1.1 Research background 1 1.2 Research goals and objectives 10 1.2.1 Research goals 10 1.2.2 Research objectives 11 1.3 Theoretical background 13 1.3.1 Concept of the UAV 13 1.3.2 Concept of the thermal camera 13 Chapter 2. Methods 15 2.1 Study site 15 2.2 Data acquisition and preprocessing 16 2.2.1 Data acquisition 16 2.2.2 RGB lens distortion correction and clipping 19 2.2.3 Thermal image correction by fur color 21 2.2.4 Unnatural object removal 22 2.3 Animal detection 24 2.3.1 Sobel edge creation and contour generation 24 2.3.2 Object detection and sorting 26 Chapter 3. Results 30 3.1 Number of counted objects 31 3.2 Time costs of image types 33 Chapter 4. Discussion 36 4.1 Reference comparison 36 4.2 Instant detection 40 4.3 Supplemental usage 41 4.4 Utility of thermal sensors 42 4.5 Applications in other fields 43 Chapter 5. Conclusions 47 References 49 Appendix: Glossary 61 초록 62석

Detection of bodies in maritime rescue operations using Unmanned Aerial Vehicles with multispectral cameras

In this study, we use unmanned aerial vehicles equipped with multispectral cameras to search for bodies in maritime rescue operations. A series of flights were performed in open‐water scenarios in the northwest of Spain, using a certified aquatic rescue dummy in dangerous areas and real people when the weather conditions allowed it. The multispectral images were aligned and used to train a convolutional neural network for body detection. An exhaustive evaluation was performed to assess the best combination of spectral channels for this task. Three approaches based on a MobileNet topology were evaluated, using (a) the full image, (b) a sliding window, and (c) a precise localization method. The first method classifies an input image as containing a body or not, the second uses a sliding window to yield a class for each subimage, and the third uses transposed convolutions returning a binary output in which the body pixels are marked. In all cases, the MobileNet architecture was modified by adding custom layers and preprocessing the input to align the multispectral camera channels. Evaluation shows that the proposed methods yield reliable results, obtaining the best classification performance when combining green, red‐edge, and near‐infrared channels. We conclude that the precise localization approach is the most suitable method, obtaining a similar accuracy as the sliding window but achieving a spatial localization close to 1 m. The presented system is about to be implemented for real maritime rescue operations carried out by Babcock Mission Critical Services Spain.This study was performed in collaboration with BabcockMCS Spain and funded by the Galicia Region Government through the Civil UAVs Initiative program, the Spanish Government’s Ministry of Economy, Industry, and Competitiveness through the RTC‐2014‐1863‐8 and INAER4‐14Y (IDI‐20141234) projects, and the grant number 730897 under the HPC‐EUROPA3 project supported by Horizon 2020

A Comprehensive Review on Computer Vision Analysis of Aerial Data

With the emergence of new technologies in the field of airborne platforms and imaging sensors, aerial data analysis is becoming very popular, capitalizing on its advantages over land data. This paper presents a comprehensive review of the computer vision tasks within the domain of aerial data analysis. While addressing fundamental aspects such as object detection and tracking, the primary focus is on pivotal tasks like change detection, object segmentation, and scene-level analysis. The paper provides the comparison of various hyper parameters employed across diverse architectures and tasks. A substantial section is dedicated to an in-depth discussion on libraries, their categorization, and their relevance to different domain expertise. The paper encompasses aerial datasets, the architectural nuances adopted, and the evaluation metrics associated with all the tasks in aerial data analysis. Applications of computer vision tasks in aerial data across different domains are explored, with case studies providing further insights. The paper thoroughly examines the challenges inherent in aerial data analysis, offering practical solutions. Additionally, unresolved issues of significance are identified, paving the way for future research directions in the field of aerial data analysis.Comment: 112 page

Planar PØP: feature-less pose estimation with applications in UAV localization

© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.We present a featureless pose estimation method that, in contrast to current Perspective-n-Point (PnP) approaches, it does not require n point correspondences to obtain the camera pose, allowing for pose estimation from natural shapes that do not necessarily have distinguished features like corners or intersecting edges. Instead of using n correspondences (e.g. extracted with a feature detector) we will use the raw polygonal representation of the observed shape and directly estimate the pose in the pose-space of the camera. This method compared with a general PnP method, does not require n point correspondences neither a priori knowledge of the object model (except the scale), which is registered with a picture taken from a known robot pose. Moreover, we achieve higher precision because all the information of the shape contour is used to minimize the area between the projected and the observed shape contours. To emphasize the non-use of n point correspondences between the projected template and observed contour shape, we call the method Planar PØP. The method is shown both in simulation and in a real application consisting on a UAV localization where comparisons with a precise ground-truth are provided.Peer ReviewedPostprint (author's final draft