Smartphone Based Object Detection for Shark Spotting

Given concern over shark attacks in coastal regions, the recent use of unmanned aerial vehicles (UAVs), or drones, has increased to ensure the safety of beachgoers. However, much of city officials\u27 process remains manual, with drone operation and review of footage still playing a significant role. In pursuit of a more automated solution, researchers have turned to the usage of neural networks to perform detection of sharks and other marine life. For on-device solutions, this has historically required assembling individual hardware components to form an embedded system to utilize the machine learning model. This means that the camera, neural processing unit, and central processing unit are purchased and assembled separately, requiring specific drivers and involves a lengthy setup process. Addressing these issues, we look to the usage of smartphones as a novel integrated solution for shark detection. This paper looks at using an iPhone 14 Pro as the driving force for a YOLOv5 based model, and comparing our results to previous literature in shark-based object detection. We find that our system outperforms previous methods at both higher throughput and increased accuracy

Detectability of dolphins and turtles from Unoccupied Aerial Vehicle (UAV) survey imagery

For many decades occupied aircraft with trained observers have conducted aerial surveys of marine megafauna to estimate population size and dynamics. Recent technological advances mean that unoccupied aerial vehicles (UAVs) now provide a potential alternative to occupied surveys, eliminating some of the disadvantages of occupied surveys such as risk to human life, weather constraints and cost. In this study, data collected from an occupied aircraft (at 500 ft) and a UAV (at 1400 ft) flown at the same time, deployed for counting dugongs, were compared for detecting dolphins and turtles within Shark Bay, Western Australia. The UAV images were manually reviewed post hoc to count the animals sighted and the environmental conditions (visibility, sea state, cloud cover and glare) had been classified by the occupied teams’ data for each image. The UAV captured more sightings (174 dolphins and 368 turtles) than were recorded by the flight team (93 dolphins and 312 turtles). Larger aggregations (>10 animals) were also found in the UAV images (5 aggregations of dolphins and turtles) compared to the occupied teams sightings (0 dolphins and 3 aggregations of turtles). A generalised linear mixed model determined that turtle detection was significantly affected by visibility, while cloud cover, sea state and visibility significantly affected dolphin detection in both platforms. An expert survey of 120 images was also conducted to determine the image ground sampling distance (GSD; four levels from 1.7 to 3.5 cm/pixel) needed to identify dolphin and turtles to species. At 3 cm/pixel only 40% of the dolphins and turtles were identified to species with a reasonable level of certainty (>75% certainty). This study demonstrated that UAVs can be successfully deployed for detecting dolphins and turtles and that a GSD of 1.7 – 3cm/pixel is too low resolution to effectively identify dolphin and turtle species. Overcoming the limitations imposed on UAVs such as aviator regulatory bodies and payload capabilities will make UAVs a pivotal tool for future research, conservation, and management

Automated shark detection using computer vision

With the technological advancements of UAVs, researchers are finding more ways to harness their capabilities to reduce expenses in everyday society. Machine vision is at the forefront of this research and in particular image recognition. Training a machine to identify objects and di↵erentiate them from others plays an integral role in the advancement of artificial intelligence. This project aims to design an algorithm capable of automatically detecting sharks from a UAV. Testing is performed by post-processing aerial footage of sharks taken from helicopters and drones, and analysing the reliability of the algorithm. Initially this research project involved analysing aerial photography of sharks, dissecting the images into the individual colour channels that made up the RGB and HSV colour spaces and identifying methods to detect the shark blobs. Once an adaptive threshold of the brightness channel was designed, filters were curated specific to the environments presented in the obtained aerial footage to reject false positives. These methods were considerably successful in both rejecting false positives and consistently detecting the sharks in the video feed. The methods produced in this dissertation leave room for future work in the shark detection field. By acquiring more reliable data, improvements such as using a kalman filter to detect and track moving blobs could be implemented to produce a robust shark detection and tracking system

Towards On-Device Detection of Sharks with Drones

Recent years have seen several projects across the globe using drones to detect sharks, including several high profile projects around alerting beach authorities to keep people safe. However, so far many of these attempts have used cloud-based machine learning solutions for the detection component, which complicates setup and limits their use geographically to areas with internet connection. An on-device (or on-controller) shark detector would offer greater freedom for researchers searching for and tracking sharks in the field, but such a detector would need to operate under reduced resource constraints. To this end we look at SSD MobileNet, a popular object detection architecture that targets edge devices by sacrificing some accuracy. We look at the results of SSD MobileNet in detecting sharks from a data set of aerial images created by a collaboration between Cal Poly and CSU Long Beach’s Shark Lab. We conclude that SSD MobileNet does suffer from some accuracy issues with smaller objects in particular, and we note the importance of customized anchor box configuration

Adaptive detection tracking system for autonomous UAV maritime patrolling

Nowadays, Unmanned Aerial Vehicles (UAVs) are considered reliable systems, suitable for several autonomous applications, especially for target detection and tracking. Although significant developments were achieved in object detection systems over the last decades using the deep learning technique known as Convolutional Neural Networks (CNN), there are still research gaps in this area. In this paper, we present a new object detection-tracking algorithm that can be used on low power consuming processing boards. In particular, we analysed a specific application scenario in which a UAV patrols coastlines and autonomously classifies different kind of marine objects. Current state of the art solutions propose centralised architectures or flying systems with human in the loop, making the whole system poorly efficient and not scalable. On the contrary, applying a Deep Learning detection system that runs on commercial Graphics Processing Units (GPUs) makes UAVs potentially more efficient than humans (especially for dull tasks like coastline patrolling) and the whole system becomes easily scalable because each UAV can fly independently and the Ground Control Station does not represent a bottleneck. To deal with this task, a database consisting of more than 115000 images was created to train and test several CNN architectures. Furthermore, an adaptive detection-tracking algorithm was introduced to make the whole system faster by optimizing the balancing between detecting new objects and tracking existing targets. The proposed solution is based on the measure of the tracking confidence and the frame similarity, by means of the Structural SIMilarity (SSIM) index, computed both globally and locally. Finally, the developed algorithms were tested on a realistic scenario by means of a UAV test-bed

Whale counting in satellite and aerial images with deep learning

Despite their interest and threat status, the number of whales in world’s oceans remains highly uncertain. Whales detection is normally carried out from costly sighting surveys, acoustic surveys or through high-resolution images. Since deep convolutional neural networks (CNNs) are achieving great performance in several computer vision tasks, here we propose a robust and generalizable CNN-based system for automatically detecting and counting whales in satellite and aerial images based on open data and tools. In particular, we designed a two-step whale counting approach, where the first CNN finds the input images with whale presence, and the second CNN locates and counts each whale in those images. A test of the system on Google Earth images in ten global whale-watching hotspots achieved a performance (F1-measure) of 81% in detecting and 94% in counting whales. Combining these two steps increased accuracy by 36% compared to a baseline detection model alone. Applying this cost-effective method worldwide could contribute to the assessment of whale populations to guide conservation actions. Free and global access to high-resolution imagery for conservation purposes would boost this process.S.T. was supported by the Ramón y Cajal Programme of the Spanish government (RYC-2015-18136). S.T., E.G., and F.H. were supported by the Spanish Ministry of Science under the project TIN2017-89517-P. D. A-S. received support from European LIFE Project ADAPTAMED LIFE14 CCA/ES/000612, and from ERDF and Andalusian Government under the project GLOCHARID. D.A.-S. received support from NASA Work Programme on Group on Earth Observations - Biodiversity Observation Network (GEOBON) under grant 80NSSC18K0446, from project ECOPOTENTIAL, funded by European Union Horizon 2020 Research and Innovation Programme under grant agreement No. 641762, and from the Spanish Ministry of Science under project CGL2014-61610-EXP and grant JC2015-00316. M.R. received support from International mobility grant for prestigious researchers by (CEIMAR) International Campus of Excellence of the Sea

Going batty: the challenges and opportunities of using drones to monitor the behaviour and habitat use of rays

The way an animal behaves in its habitat provides insight into its ecological role. As such, collecting robust, accurate datasets in a time-efficient manner is an ever-present pressure for the field of behavioural ecology. Faced with the shortcomings and physical limitations of traditional ground-based data collection techniques, particularly in marine studies, drones offer a low-cost and efficient approach for collecting data in a range of coastal environments. Despite drones being widely used to monitor a range of marine animals, they currently remain underutilised in ray research. The innovative application of drones in environmental and ecological studies has presented novel opportunities in animal observation and habitat assessment, although this emerging field faces substantial challenges. As we consider the possibility to monitor rays using drones, we face challenges related to local aviation regulations, the weather and environment, as well as sensor and platform limitations. Promising solutions continue to be developed, however, growing the potential for drone-based monitoring of behaviour and habitat use of rays. While the barriers to enter this field may appear daunting for researchers with little experience with drones, the technology is becoming increasingly accessible, helping ray researchers obtain a wide range of highly useful data

Factors Affecting Shark Detection from Drone Patrols in Southeast Queensland, Eastern Australia

Drones enable the monitoring for sharks in real-time, enhancing the safety of ocean users with minimal impact on marine life. Yet, the effectiveness of drones for detecting sharks (especially potentially dangerous sharks; i.e., white shark, tiger shark, bull shark) has not yet been tested at Queensland beaches. To determine effectiveness, it is necessary to understand how environmental and operational factors affect the ability of drones to detect sharks. To assess this, we utilised data from the Queensland SharkSmart drone trial, which operated at five southeast Queensland beaches for 12 months in 2020–2021. The trial conducted 3369 flights, covering 1348 km and sighting 174 sharks (48 of which were >2 m in length). Of these, eight bull sharks and one white shark were detected, leading to four beach evacuations. The shark sighting rate was 3% when averaged across all beaches, with North Stradbroke Island (NSI) having the highest sighting rate (17.9%) and Coolum North the lowest (0%). Drone pilots were able to differentiate between key shark species, including white, bull and whaler sharks, and estimate total length of the sharks. Statistical analysis indicated that location, the sighting of other fauna, season and flight number (proxy for time of day) influenced the probability of sighting shark

Modelagem de abundância com drones : detectabilidade, desenho amostral e revisão automática de imagens em um estudo com cervos-do-pantanal

Entender como a abundância de uma espécie se distribui no espaço e/ou no tempo é uma questão fundamental em ecologia e conservação e ajuda, por exemplo, a elucidar relações entre a heterogeneidade de paisagens e populações ou compreender influência de predação na distribuição de indivíduos. Informações de tamanho populacional também são essenciais para avaliar risco de extinção, monitorar populações ameaçadas e planejar ações de conservação. Modelar a abundância de cervos-do-pantanal (Blastocerus dichotomus), sendo um grande herbívoro da América do Sul, pode ser importante para entender relações da espécie com a variação espacial da produtividade primária, das áreas úmidas que a espécie ocupa e do seu principal predador, a onça- pintada. Além disso, por estar ameaçado de extinção, estimar a abundância de cervos pode contribuir para avaliar populações relictuais da espécie, assim como monitorar populações após grandes eventos, como os incêndios de 2020 no Pantanal. Porém, acessar estimativas de abundância confiáveis de maneira eficiente requer métodos robustos que levem em conta os possíveis erros nas contagens e que forneçam as estimativas em tempo hábil, além de um desenho amostral otimizado para aproveitar os recursos geralmente escassos. Os drones tem aparecido como uma ferramenta versátil e custo-efetiva para amostragem de populações animais e vêm sendo aplicados para várias espécies diferentes nos mais variados contextos ecológicos. Como um método emergente, o uso de drones na ecologia fornece oportunidades para explorar novas possibilidades de amostragem e análise de dados, ao mesmo tempo em que pode apresentar novos desafios. Nesta tese, i) exploro oportunidade e desafios na utilização de drones para modelagem de abundância de animais, abordando questões de erros de detecção, desenho amostral e como lidar com os grandes bancos de imagens gerados; e ii) aplico os métodos desenvolvidos para estudar a variação na abundância de cervo-do- pantanal, assim como estabelecer uma abordagem para monitoramento robusto e efetivo dessa espécie. Assim, no primeiro capítulo, conduzo uma revisão na literatura descrevendo os potenciais erros de detecção que podem enviesar estimativas de abundância com drones, buscando soluções atuais para lidar com esses erros e identificando lacunas que precisam de desenvolvimento. Nessa revisão, destaco o potencial dos modelos hierárquicos para estimar abundância em amostragens com drone. No segundo capítulo, aplico amostragens espaço-temporalmente replicadas com drone, analisadas com modelos hierárquicos N-mixture, para entender o efeito de processos topo-base (distribuição de onças-pintadas) e base-topo (disponibilidade de forragem de qualidade e corpos d’água) na distribuição da abundância de cervos-do- pantanal. Nesse estudo, encontrei que, na época seca, os cervos se concentram em áreas de alta qualidade (maior disponibilidade de forragem e próximas a corpos d’água), mesmo sendo a região em que é esperado maior efeito da predação. No capítulo 3, em um estudo com simulações, avalio o desempenho de modelos N-mixture para estimativas de abundância a partir de amostragens espaço-temporalmente replicadas, explorando otimização de esforço amostral e o impacto de um protocolo com observadores duplos na acurácia das estimativas. No capítulo 4, desenvolvo uma abordagem para estimar abundância com drone usando observadores múltiplos na revisão das imagens, sendo um dos observadores baseado em um processamento semiautomático usando algoritmos de inteligência artificial. Nesse estudo, exploro técnicas de aprendizado profundo de máquina, com redes neurais convolucionais, acessíveis para ecólogos, treinando algoritmos para detectar cervos nas imagens de drone. Além de ajudar a elucidar questões sobre as relações do cervo-do-pantanal com aspectos diferentes da paisagem do Pantanal, as abordagens exploradas e desenvolvidas aqui têm um grande potencial de aplicação, ajudando a estabelecer os drones como uma ferramenta eficiente para modelagem e monitoramento populacional de diversas espécies animais, e particularmente de cervos.Understanding how abundance distributes in space and/or time is a fundamental question in ecology and conservation, and it helps, for example, to elucidate relationships between landscape heterogeneity or predation and populations. Information on the population size also is essential to evaluate extinction risk, monitor threatened species and plan conservation actions. Abundance modeling of marsh deer (Blastocerus dichotomus), as a large herbivore of South America, may be important to understand the relationships of this species with spatial variation in primary productivity, in the availability of wetlands that the species inhabits, and in the distribution of its main predator, the jaguar. Moreover, since marsh deer threatened to extinction, estimating its abundance can be contribute in assessments of relictual populations, as well as in monitoring the species after big events, such as the Pantanal 2020 megafires. However, efficiently assessing reliable abundance estimates require robust methods that account for possible sources of error in counts while providing the estimates timely. An optimized sampling design is also important, in order to make the best use of the usual scarce resources. Drones have raised as a versatile and cost- effective tool for sampling animal populations, and they have been applied for several species in a wide variety of ecological contexts. As being an emergent method, the use of drones in ecology provides opportunities to explore novel possibilities of sampling and analyzing data, while potentially presenting new challenges. In this thesis I: i) explore opportunities and challenges in the use of drones for animal abundance modeling, approaching issues about detection errors, sampling design and how to deal with the huge image sets generated from drone flights; and ii) apply the developed methods to study the spatial variation in marsh deer abundance and to establish an approach to monitor this species robustly and efficiently. Thus, in the first chapter, I carry on a literature review describing potential sources of errors that may bias abundance estimation with drones and the current solutions to address them, identifying gaps that need development. In this review, I highlight the potential of hierarchical models for abundance estimations from drone-based surveys. In the second chapter, I apply spatiotemporally replicated drone surveys, analyzed with N-mixture models, to understand the influence of bottom-up (forage and water) and top-down (jaguar density) variables on the spatial variation of marsh deer local abundance. In such study, I found that, in the dry season, the deer concentrate in high quality areas (high-quality forage available and close to water bodies), even these regions being expected to present higher predation risks. In chapter 3, in a simulation study, I evaluate the performance of N- mixture models for abundance estimation from spatiotemporally replicated surveys, exploring optimization of sampling effort and the impact of a double-observer protocol on estimation accuracy. In chapter 4, I develop a pipeline to estimate abundance from drone-based surveys using a multiple-observer protocol in which one of the observer is a semiautomated procedure based on deep learning algorithms. In such study, I explore deep learning techniques with convolutional neural networks that are accessible for ecologists, and train algorithms to detect marsh deer in drone imagery. Besides helping to elucidate questions about the relationships of marsh deer with landscape variables in Pantanal, the approaches explored and developed here have a great potential of application in order to establish drones as an efficient technique for population modeling and monitoring of several wildlife species, and particularly the marsh deer

Computer vision for plant and animal inventory

The population, composition, and spatial distribution of the plants and animals in certain regions are always important data for natural resource management, conservation and farming. The traditional ways to acquire such data require human participation. The procedure of data processing by human is usually cumbersome, expensive and time-consuming. Hence the algorithms for automatic animal and plant inventory show their worth and become a hot topic. We propose a series of computer vision methods for automated plant and animal inventory, to recognize, localize, categorize, track and count different objects of interest, including vegetation, trees, fishes and livestock animals. We make use of different sensors, hardware platforms, neural network architectures and pipelines to deal with the varied properties and challenges of these objects. (1) For vegetation analysis, we propose a fast multistage method to estimate the coverage. The reference board is localized based on its edge and texture features. And then a K-means color model of the board is generated. Finally, the vegetation is segmented at pixel level using the color model. The proposed method is robust to lighting condition changes. (2) For tree counting in aerial images, we propose a novel method called density transformer, or DENT, to learn and predict the density of the trees at different positions. DENT uses an efficient multi-receptive field network to extract visual features from different positions. A transformer encoder is applied to filter and transfer useful contextual information across different spatial positions. DENT significantly outperformed the existing state-of-art CNN detectors and regressors on both the dataset built by ourselves and an existing cross-site dataset. (3) We propose a framework of fish classification system using boat cameras. The framework contains two branches. A branch extracts the contextual information from the whole image. The other branch localizes all the individual fish and normalizes their poses. The classification results from the two branches are weighted based on the clearness of the image and the familiarness of the context. Our system achieved the top 1 percent rank in the competition of The Nature Conservancy Fisheries Monitoring. (4) We also propose a video-based pig counting algorithm using an inspection robot. We adopt a novel bottom-up keypoint tracking method and a novel spatial-aware temporal response filtering method to count the pigs. The proposed approach outperformed the other methods and even human competitors in the experiments.Includes bibliographical references