Automated classification of three-dimensional reconstructions of coral reefs using convolutional neural networks

© The Author(s), 2020. This article is distributed under the terms of the Creative Commons Attribution License. The definitive version was published in Hopkinson, B. M., King, A. C., Owen, D. P., Johnson-Roberson, M., Long, M. H., & Bhandarkar, S. M. Automated classification of three-dimensional reconstructions of coral reefs using convolutional neural networks. PLoS One, 15(3), (2020): e0230671, doi: 10.1371/journal.pone.0230671.Coral reefs are biologically diverse and structurally complex ecosystems, which have been severally affected by human actions. Consequently, there is a need for rapid ecological assessment of coral reefs, but current approaches require time consuming manual analysis, either during a dive survey or on images collected during a survey. Reef structural complexity is essential for ecological function but is challenging to measure and often relegated to simple metrics such as rugosity. Recent advances in computer vision and machine learning offer the potential to alleviate some of these limitations. We developed an approach to automatically classify 3D reconstructions of reef sections and assessed the accuracy of this approach. 3D reconstructions of reef sections were generated using commercial Structure-from-Motion software with images extracted from video surveys. To generate a 3D classified map, locations on the 3D reconstruction were mapped back into the original images to extract multiple views of the location. Several approaches were tested to merge information from multiple views of a point into a single classification, all of which used convolutional neural networks to classify or extract features from the images, but differ in the strategy employed for merging information. Approaches to merging information entailed voting, probability averaging, and a learned neural-network layer. All approaches performed similarly achieving overall classification accuracies of ~96% and >90% accuracy on most classes. With this high classification accuracy, these approaches are suitable for many ecological applications.This study was funded by grants from the Alfred P. Sloan Foundation (BMH, BR2014-049; https://sloan.org), and the National Science Foundation (MHL, OCE-1657727; https://www.nsf.gov). The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript

Supplementary report to the final report of the coral reef expert group: S6. Novel technologies in coral reef monitoring

[Extract] This report summarises a review of current technological advances applicable to coral reef monitoring, with a focus on the Great Barrier Reef Marine Park (the Marine Park). The potential of novel technologies to support coral reef monitoring within the Reef 2050 Integrated Monitoring and Reporting Program (RIMReP) framework was evaluated based on their performance, operational maturity and compatibility with traditional methods. Given the complexity of this evaluation, this exercise was systematically structured to address the capabilities of technologies in terms of spatial scales and ecological indicators, using a ranking system to classify expert recommendations.An accessible copy of this report is not yet available from this repository, please contact [email protected] for more information

Deep learning based deep-sea automatic image enhancement and animal species classification

The automatic classification of marine species based on images is a challenging task for which multiple solutions have been increasingly provided in the past two decades. Oceans are complex ecosystems, difficult to access, and often the images obtained are of low quality. In such cases, animal classification becomes tedious. Therefore, it is often necessary to apply enhancement or pre-processing techniques to the images, before applying classification algorithms. In this work, we propose an image enhancement and classification pipeline that allows automated processing of images from benthic moving platforms. Deep-sea (870 m depth) fauna was targeted in footage taken by the crawler “Wally” (an Internet Operated Vehicle), within the Ocean Network Canada (ONC) area of Barkley Canyon (Vancouver, BC; Canada). The image enhancement process consists mainly of a convolutional residual network, capable of generating enhanced images from a set of raw images. The images generated by the trained convolutional residual network obtained high values in metrics for underwater imagery assessment such as UIQM (~ 2.585) and UCIQE (2.406). The highest SSIM and PSNR values were also obtained when compared to the original dataset. The entire process has shown good classification results on an independent test data set, with an accuracy value of 66.44% and an Area Under the ROC Curve (AUROC) value of 82.91%, which were subsequently improved to 79.44% and 88.64% for accuracy and AUROC respectively. These results obtained with the enhanced images are quite promising and superior to those obtained with the non-enhanced datasets, paving the strategy for the on-board real-time processing of crawler imaging, and outperforming those published in previous papers.This work was developed at Deusto Seidor S.A. (01015, Vitoria-Gasteiz, Spain) within the framework of the Tecnoterra (ICM-CSIC/UPC) and the following project activities: ARIM (Autonomous Robotic sea-floor Infrastructure for benthopelagic Monitoring); MarTERA ERA-Net Cofund; Centro para el Desarrollo Tecnológico Industrial, CDTI; and RESBIO (TEC2017-87861-R; Ministerio de Ciencia, Innovación y Universidades). This work was supported by the Centro para el Desarrollo Tecnológico Industrial (CDTI) (Grant No. EXP 00108707 / SERA-20181020)

Semantic Segmentation for Real-World Applications

En visión por computador, la comprensión de escenas tiene como objetivo extraer información útil de una escena a partir de datos de sensores. Por ejemplo, puede clasificar toda la imagen en una categoría particular o identificar elementos importantes dentro de ella. En este contexto general, la segmentación semántica proporciona una etiqueta semántica a cada elemento de los datos sin procesar, por ejemplo, a todos los píxeles de la imagen o, a todos los puntos de la nube de puntos. Esta información es esencial para muchas aplicaciones de visión por computador, como conducción, aplicaciones médicas o robóticas. Proporciona a los ordenadores una comprensión sobre el entorno que es necesaria para tomar decisiones autónomas.El estado del arte actual de la segmentación semántica está liderado por métodos de aprendizaje profundo supervisados. Sin embargo, las condiciones del mundo real presentan varias restricciones para la aplicación de estos modelos de segmentación semántica. Esta tesis aborda varios de estos desafíos: 1) la cantidad limitada de datos etiquetados disponibles para entrenar modelos de aprendizaje profundo, 2) las restricciones de tiempo y computación presentes en aplicaciones en tiempo real y/o en sistemas con poder computacional limitado, y 3) la capacidad de realizar una segmentación semántica cuando se trata de sensores distintos de la cámara RGB estándar.Las aportaciones principales en esta tesis son las siguientes:1. Un método nuevo para abordar el problema de los datos anotados limitados para entrenar modelos de segmentación semántica a partir de anotaciones dispersas. Los modelos de aprendizaje profundo totalmente supervisados lideran el estado del arte, pero mostramos cómo entrenarlos usando solo unos pocos píxeles etiquetados. Nuestro enfoque obtiene un rendimiento similar al de los modelos entrenados con imágenescompletamente etiquetadas. Demostramos la relevancia de esta técnica en escenarios de monitorización ambiental y en dominios más generales.2. También tratando con datos de entrenamiento limitados, proponemos un método nuevo para segmentación semántica semi-supervisada, es decir, cuando solo hay una pequeña cantidad de imágenes completamente etiquetadas y un gran conjunto de datos sin etiquetar. La principal novedad de nuestro método se basa en el aprendizaje por contraste. Demostramos cómo el aprendizaje por contraste se puede aplicar a la tarea de segmentación semántica y mostramos sus ventajas, especialmente cuando la disponibilidad de datos etiquetados es limitada logrando un nuevo estado del arte.3. Nuevos modelos de segmentación semántica de imágenes eficientes. Desarrollamos modelos de segmentación semántica que son eficientes tanto en tiempo de ejecución, requisitos de memoria y requisitos de cálculo. Algunos de nuestros modelos pueden ejecutarse en CPU a altas velocidades con alta precisión. Esto es muy importante para configuraciones y aplicaciones reales, ya que las GPU de gama alta nosiempre están disponibles.4. Nuevos métodos de segmentación semántica con sensores no RGB. Proponemos un método para la segmentación de nubes de puntos LiDAR que combina operaciones de aprendizaje eficientes tanto en 2D como en 3D. Logra un rendimiento de segmentación excepcional a velocidades realmente rápidas. También mostramos cómo mejorar la robustez de estos modelos al abordar el problema de sobreajuste y adaptaciónde dominio. Además, mostramos el primer trabajo de segmentación semántica con cámaras de eventos, haciendo frente a la falta de datos etiquetados.Estas contribuciones aportan avances significativos en el campo de la segmentación semántica para aplicaciones del mundo real. Para una mayor contribución a la comunidad cientfíica, hemos liberado la implementación de todas las soluciones propuestas.----------------------------------------In computer vision, scene understanding aims at extracting useful information of a scene from raw sensor data. For instance, it can classify the whole image into a particular category (i.e. kitchen or living room) or identify important elements within it (i.e., bottles, cups on a table or surfaces). In this general context, semantic segmentation provides a semantic label to every single element of the raw data, e.g., to all image pixels or to all point cloud points.This information is essential for many applications relying on computer vision, such as AR, driving, medical or robotic applications. It provides computers with understanding about the environment needed to make autonomous decisions, or detailed information to people interacting with the intelligent systems. The current state of the art for semantic segmentation is led by supervised deep learning methods.However, real-world scenarios and conditions introduce several challenges and restrictions for the application of these semantic segmentation models. This thesis tackles several of these challenges, namely, 1) the limited amount of labeled data available for training deep learning models, 2) the time and computation restrictions present in real time applications and/or in systems with limited computational power, such as a mobile phone or an IoT node, and 3) the ability to perform semantic segmentation when dealing with sensors other than the standard RGB camera.The general contributions presented in this thesis are following:A novel approach to address the problem of limited annotated data to train semantic segmentation models from sparse annotations. Fully supervised deep learning models are leading the state-of-the-art, but we show how to train them by only using a few sparsely labeled pixels in the training images. Our approach obtains similar performance than models trained with fully-labeled images. We demonstrate the relevance of this technique in environmental monitoring scenarios, where it is very common to have sparse image labels provided by human experts, as well as in more general domains. Also dealing with limited training data, we propose a novel method for semi-supervised semantic segmentation, i.e., when there is only a small number of fully labeled images and a large set of unlabeled data. We demonstrate how contrastive learning can be applied to the semantic segmentation task and show its advantages, especially when the availability of labeled data is limited. Our approach improves state-of-the-art results, showing the potential of contrastive learning in this task. Learning from unlabeled data opens great opportunities for real-world scenarios since it is an economical solution. Novel efficient image semantic segmentation models. We develop semantic segmentation models that are efficient both in execution time, memory requirements, and computation requirements. Some of our models able to run in CPU at high speed rates with high accuracy. This is very important for real set-ups and applications since high-end GPUs are not always available. Building models that consume fewer resources, memory and time, would increase the range of applications that can benefit from them. Novel methods for semantic segmentation with non-RGB sensors.We propose a novel method for LiDAR point cloud segmentation that combines efficient learning operations both in 2D and 3D. It surpasses state-of-the-art segmentation performance at really fast rates. We also show how to improve the robustness of these models tackling the overfitting and domain adaptation problem. Besides, we show the first work for semantic segmentation with event-based cameras, coping with the lack of labeled data. To increase the impact of this contributions and ease their application in real-world settings, we have made available an open-source implementation of all proposed solutions to the scientific community.<br /

Autonomous Systems for the Environmental Characterization of Lagoons

This chapter reviews the state of the art in robotics and autonomous systems (RAS) for monitoring the environmental characteristics of lagoons, as well as potential future uses of such technologies that could contribute to enhancing current monitoring programmes. Particular emphasis will be given to unmanned aerial vehicles (UAVs), autonomous under water vehicles (AUVs), remotely operated underwater vehicles (ROVs) and (semi-)autonomous boats. Recent technological advances in UAVs, AUVs and ROVs have demonstrated that high-resolution data (e.g. 0.4 cm imagery resolution) can be gathered when bespoke sensors are incorporated within these platforms. This in turn enables the accurate quantification of key metrics within lagoon environments, such as coral morphometries. For example, coral height and width can now be estimated remotely with errors below 12.6 and 14.7 cm, respectively. The chapter will explore how the use of such technologies in combination could improve the understanding of lagoon environments through increased knowledge of the spatial and temporal variations of parameters of interest. Within this context, both advantages and limitations of the proposed approaches will be highlighted and described from operational, logistical, and regulatory considerations. The chapter will be based on recent peer-reviewed research outputs obtained by the authors

Coral Colony-Scale Rugosity Metrics and Applications for Assessing Temporal Trends in the Structural Complexity of Coral Reefs.

Globally, coral reefs are experiencing reductions in structural complexity, primarily due to a loss of key reef building taxa. Monitoring these changes is difficult due to the time-consuming nature of in-situ measurements and lack of data concerning coral genus-specific contributions to reef structure. This research aimed to develop a new technique that uses coral colony level data to quantify reef rugosity (a 3-dimensional measure of reef structure) from three sources of coral survey data: 2D video imagery, line intercept data and UAV imagery. A database of coral colony rugosity data, comparing coral colony planar and contour length for 40 coral genera, 14 morphotypes and 9 abiotic reef substrates, was created using measurements from the Great Barrier Reef and Natural History Museum. Mean genus rugosity was identified as a key trait related to coral life history strategy. Linear regression analyses (y = mx) revealed statistically significant (p < 0.05) relationships between coral colony size and rugosity for every coral genus, morphotype and substrate. The gradient governing these relationships was unique for each coral taxa, ranging from mean = 1.23, for (encrusting) Acanthastrea, to m = 3.84, for (vase-shape) Merulina. These gradients were used as conversion factors to calculate reef rugosity from linear distances measured in video transects of both artificial reefs, used as a control test, and in-situ natural coral reefs, using Kinovea software. This calculated, ‘virtual’ rugosity had a strong, positive relationship with in-situ microscale rugosity (r2 = 0.96) measured from the control transects, but not with that measured at the meso-scale in natural, highly heterogeneous reef environments (r2 < 0.2). This showed that the technique can provide accurate rugosity information when considered at the coral colony level. The conversion factors were also applied to historic line intercept data from the Seychelles, where temporal changes in calculated rugosity were consistent with changes in coral cover between 2008 and 2017. Finally, on application to 2,283 corals digitised from UAV imagery of the Maldives, the conversion factors enabled calculation of rugosity for three 100 m2 reef areas and prediction of how this rugosity will decrease during two future scenarios of coral reef degradation and community change. The study highlights that the application of genera-specific coral rugosity data to both new and existing coral reef survey datasets could be a valuable tool for monitoring reef structural complexity over large spatial scales

Semantic Segmentation from Sparse Labeling Using Multi-Level Superpixels

Semantic segmentation is a challenging problemthat can benefit numerous robotics applications, since it pro-vides information about the content at every image pixel.Solutions to this problem have recently witnessed a boost onperformance and results thanks to deep learning approaches.Unfortunately, common deep learning models for semanticsegmentation present several challenges which hinder real lifeapplicability in many domains. A significant challenge is theneed of pixel level labeling on large amounts of trainingimages to be able to train those models, which implies avery high cost. This work proposes and validates a simplebut effective approach to train dense semantic segmentationmodels from sparsely labeled data. Labeling only a few pixelsper image reduces the human interaction required. We findmany available datasets, e.g., environment monitoring data, thatprovide this kind of sparse labeling. Our approach is basedon augmenting the sparse annotation to a dense one with theproposed adaptive superpixel segmentation propagation. Weshow that this label augmentation enables effective learning ofstate-of-the-art segmentation models, getting similar results tothose models trained with dense ground-truth

Going batty: the challenges and opportunities of using drones to monitor the behaviour and habitat use of rays

The way an animal behaves in its habitat provides insight into its ecological role. As such, collecting robust, accurate datasets in a time-efficient manner is an ever-present pressure for the field of behavioural ecology. Faced with the shortcomings and physical limitations of traditional ground-based data collection techniques, particularly in marine studies, drones offer a low-cost and efficient approach for collecting data in a range of coastal environments. Despite drones being widely used to monitor a range of marine animals, they currently remain underutilised in ray research. The innovative application of drones in environmental and ecological studies has presented novel opportunities in animal observation and habitat assessment, although this emerging field faces substantial challenges. As we consider the possibility to monitor rays using drones, we face challenges related to local aviation regulations, the weather and environment, as well as sensor and platform limitations. Promising solutions continue to be developed, however, growing the potential for drone-based monitoring of behaviour and habitat use of rays. While the barriers to enter this field may appear daunting for researchers with little experience with drones, the technology is becoming increasingly accessible, helping ray researchers obtain a wide range of highly useful data

PAPARA(ZZ)I : An open-source software interface for annotating photographs of the deep-sea

PAPARA(ZZ)I is a lightweight and intuitive image annotation program developed for the study of benthic megafauna. It offers functionalities such as free, grid and random point annotation. Annotations may be made following existing classification schemes for marine biota and substrata or with the use of user defined, customised lists of keywords, which broadens the range of potential application of the software to other types of studies (e.g. marine litter distribution assessment). If Internet access is available, PAPARA(ZZ)I can also query and use standardised taxa names directly from the World Register of Marine Species (WoRMS). Program outputs include abundances, densities and size calculations per keyword (e.g. per taxon). These results are written into text files that can be imported into spreadsheet programs for further analyses. PAPARA(ZZ)I is open-source and is available at http://papara-zz-i.github.io. Compiled versions exist for most 64-bit operating systems: Windows, Mac OS X and Linux

Needs and gaps in optical underwater technologies and methods for the investigation of marine animal forest 3D-structural complexity

Marine animal forests are benthic communities dominated by sessile suspension feeders (such as sponges, corals, and bivalves) able to generate three-dimensional (3D) frameworks with high structural complexity. The biodiversity and functioning of marine animal forests are strictly related to their 3D complexity. The present paper aims at providing new perspectives in underwater optical surveys. Starting from the current gaps in data collection and analysis that critically limit the study and conservation of marine animal forests, we discuss the main technological and methodological needs for the investigation of their 3D structural complexity at different spatial and temporal scales. Despite recent technological advances, it seems that several issues in data acquisition and processing need to be solved, to properly map the different benthic habitats in which marine animal forests are present, their health status and to measure structural complexity. Proper precision and accuracy should be chosen and assured in relation to the biological and ecological processes investigated. Besides, standardized methods and protocols are strictly necessary to meet the FAIR (findability, accessibility, interoperability, and reusability) data principles for the stewardship of habitat mapping and biodiversity, biomass, and growth data