RecoMIA - Recommendations for Marine Image Annotation: Lessons Learned and Future Directions

Marine imaging is transforming into a sensor technology applied for high throughput sampling. In the context of habitat mapping, imaging establishes thereby an important bridge technology regarding the spatial resolution and information content between physical sampling gear (e.g., box corer, multi corer) on the one end and hydro-acoustic sensors on the other end of the spectrum of sampling methods. In contrast to other scientific imaging domains, such as digital pathology, there are no protocols and reports available that guide users (often referred to as observers) in the non-trivial process of assigning semantic categories to whole images, regions, or objects of interest (OOI), which is referred to as annotation. These protocols are crucial to facilitate image analysis as a robust scientific method. In this article we will review the past observations in manual Marine Image Annotations (MIA) and provide (a) a guideline for collecting manual annotations, (b) definitions for annotation quality, and (c) a statistical framework to analyze the performance of human expert annotations and to compare those to computational approaches

RecoMIA - Recommendations for marine image annotation: Lessons learned and future directions

Schoening T, Osterloff J, Nattkemper TW. RecoMIA - Recommendations for marine image annotation: Lessons learned and future directions. Frontiers in Marine Science. 2016;3: 59.Marine imaging is transforming into a sensor technology applied for high throughput sampling. In the context of habitat mapping, imaging establishes thereby an important bridge technology regarding the spatial resolution and information content between physical sampling gear (e.g., box corer, multi corer) on the one end and hydro-acoustic sensors on the other end of the spectrum of sampling methods. In contrast to other scientific imaging domains, such as digital pathology, there are no protocols and reports available that guide users (often referred to as observers) in the non-trivial process of assigning semantic categories to whole images, regions, or objects of interest (OOI), which is referred to as annotation. These protocols are crucial to facilitate image analysis as a robust scientific method. In this article we will review the past observations in manual Marine Image Annotations (MIA) and provide (a) a guideline for collecting manual annotations, (b) definitions for annotation quality, and (c) a statistical framework to analyze the performance of human expert annotations and to compare those to computational approaches

Automated detection in benthic images for megafauna classification and marine resource exploration: supervised and unsupervised methods for classification and regression tasks in benthic images with efficient integration of expert knowledge

Schoening T. Automated detection in benthic images for megafauna classification and marine resource exploration: supervised and unsupervised methods for classification and regression tasks in benthic images with efficient integration of expert knowledge. Bielefeld: Universitätsbibliothek Bielefeld; 2015.Image acquisition of deep sea floors allows to cast a glance on an extraordinary environment. Exploring the rarely known geology and biology of the deep sea regularly questions the scientific understanding of occurring conditions, processes and changes. Increasing sampling efforts, by both more frequent image acquisition as well as widespread monitoring of large areas, currently refine the scientific models about this environment. Accompanied by the sampling efforts, novel challenges emerge for the image based marine research. These include growing data volume, growing data variety and increased velocity at which data is acquired. Apart from the included technical challenges, the fundamental problem is to add semantics to the acquired data to extract further meaning and gain derived knowledge. Manual analysis of the data in terms of manually annotating images (e.g. annotating occurring species to gain species interaction knowledge) is an intricate task and has become infeasible due to the huge data volumes. The combination of data and interpretation challenges calls for automated approaches based on pattern recognition and especially computer vision methods. These methods have been applied in other fields to add meaning to visual data but have rarely been applied to the peculiar case of marine imaging. First of all, the physical factors of the environment constitute a unique computer vision challenge and require special attention in adapting the methods. Second, the impossibility to create a reliable reference gold standard from multiple field expert annotations challenges the development and evaluation of automated, pattern recognition based approaches. In this thesis, novel automated methods to add semantics to benthic images are presented that are based on common pattern recognition techniques. Three major benthic computer vision scenarios are addressed: the detection of laser points for scale quantification, the detection and classification of benthic megafauna for habitat composition assessments and the detection and quantity estimation of benthic mineral resources for deep sea mining. All approaches to address these scenarios are fitted to the peculiarities of the marine environment. The primary paradigm, that guided the development of all methods, was to design systems that can be operated by field experts without knowledge about the applied pattern recognition methods. Therefore, the systems have to be generally applicable to arbitrary image based detection scenarios. This in turn makes them applicable in other computer vision fields outside the marine environment as well. By tuning system parameters automatically from field expert annotations and applying methods that cope with errors in those annotations, the limitations of inaccurate gold standards can be bypassed. This allows to use the developed systems to further refine the scientific models based on automated image analysis

Computer Vision for Marine Environmental Monitoring

Osterloff J. Computer Vision for Marine Environmental Monitoring. Bielefeld: Universität Bielefeld; 2018.Ocean exploration using imaging techniques has recently become very popular as camera systems became affordable and technique developed further. Marine imaging provides a unique opportunity to monitor the marine environment. The visual exploration using images allows to explore the variety of fauna, flora and geological structures of the marine environment. This monitoring creates a bottleneck as a manual evaluation of the large amounts of underwater image data is very time consuming. Information encapsulated in the images need to be extracted so that they can be included in statistical analyzes. Objects of interest (OOI) have to be localized and identified in the recorded images. In order to overcome the bottleneck, computer vision (CV) is applied in this thesis to extract the image information (semi-) automatically. A pre-evaluation of the images by marking OOIs manually, i.e. the manual annotation process, is necessary to provide examples for the applied CV methods. Five major challenges are identified in this thesis to apply of CV for marine environmental monitoring. The challenges can be grouped into challenges caused by underwater image acquisition and by the use of manual annotations for machine learning (ML). The image acquisition challenges are the optical properties challenge, e.g. a wavelength dependent attenuation underwater, and the dynamics of these properties, as different amount of matter in the water column affect colors and illumination in the images. The manual annotation challenges for applying ML for underwater images are, the low number of available manual annotations, the quality of the annotations in terms of correctness and reproducibility and the spatial uncertainty of them. The latter is caused by allowing a spatial uncertainty to speed up the manual annotation process e.g. using point annotations instead of fully outlining OOIs on a pixel level. The challenges are resolved individually in four different new CV approaches. The individual CV approaches allow to extract new biologically relevant information from time-series images recorded underwater. Manual annotations provide the ground truth for the CV systems and therefore for the included ML. Placing annotations manually in underwater images is a challenging task. In order to assess the quality in terms of correctness and reproducibility a detailed quality assessment for manual annotations is presented. This includes the computation of a gold standard to increase the quality of the ground truth for the ML. In the individually tailored CV systems, different ML algorithms are applied and adapted for marine environmental monitoring purposes. Applied ML algorithms cover a broad variety from unsupervised to supervised methods, including deep learning algorithms. Depending on the biologically motivated research question, systems are evaluated individually. The first two CV systems are developed for the _in-situ_ monitoring of the sessile species _Lophelia pertusa_. Visual information of the cold-water coral is extracted automatically from time-series images recorded by a fixed underwater observatory (FUO) located at 260 m depth and 22 km off the Norwegian coast. Color change of a cold water coral reef over time is quantified and the polyp activity of the imaged coral is estimated (semi-) automatically. The systems allow for the first time to document an _in-situ_ change of color of a _Lophelia pertusa_ coral reef and to estimate the polyp activity for half a year with a temporal resolution of one hour. The third CV system presented in this thesis allows to monitor the mobile species shrimp _in-situ_. Shrimp are semitransparent creating additional challenges for localization and identification in images using CV. Shrimp are localized and identified in time-series images recorded by the same FUO. Spatial distribution and temporal occurrence changes are observed by comparing two different time periods. The last CV system presented in this thesis is developed to quantify the impact of sedimentation on calcareous algae samples in a _wet-lab_ experiment. The size and color change of the imaged samples over time can be quantified using a consumer camera and a color reference plate placed in the field of view for each recorded image. Extracting biologically relevant information from underwater images is only the first step for marine environmental monitoring. The extracted image information, like behavior or color change, needs to be related to other environmental parameters. Therefore, also data science methods are applied in this thesis to unveil some of the relations between individual species' information extracted semi-automatically from underwater images and other environmental parameters

The sustainability of deep-sea fishing in Greenland from a benthic ecosystem perspective: the nature of habitats, impacts of trawling and the effectiveness of governance

The deep sea (>200 m) is the world’s least explored and largest biome, covering ~65% of the earth’s surface, it is increasingly subject to anthropogenic disturbance from fishing. The offshore Greenland halibut (Reinhardtius hippoglossoides) fishery, west Greenland, employs demersal trawl gear at depths of 800-1,400 m. Recent Marine Stewardship Council (MSC) certification of this fishery highlighted the paucity of knowledge of benthic habitats and trawling impacts. This interdisciplinary thesis employs a benthic video sled to investigate deep-sea habitats and trawling impacts and conducts a critical analysis of the fishery’s governance, with reference to the role of the MSC certification. The results provide new insights into this poorly known region of the Northwest Atlantic, including identifying four candidate vulnerable marine ecosystems (VMEs). Imagery obtained demonstrates that chronic trawling has had extensive impacts on the seafloor, which are significantly associated with the benthic communities observed. Further, trawling effort is shown to have a significant negative association with the abundance of some VME indicator taxa. The governance case study finds an effective system of state-led governance, supported by scientific, certification and industry actors. Outcomes directly attributable to engagement with the MSC certification include the introduction of a management plan and new benthic research programmes. However, questions are raised about the MSC certification, providing case study examples of existing criticisms. Assessments are weak with respect to benthic habitats and overreliant on the definitive, expert judgement of Conformity Assessment Bodies (CABs), whose independence is questioned. The assurance offered by the MSC certification in terms of the sustainability of trawling impacts on benthic ecosystems is found to seriously lack credibility. Findings are of direct relevance to the management of deep-sea fisheries in Greenland and elsewhere. Widely applicable critical insights into deep-sea fishery governance are presented, including into the role of eco-labels as a market-mechanism to promote sustainable fishery management

BIIGLE Tools - A Web 2.0 Approach for Visual Bioimage Database Mining

Schoening T, Ehnert N, Ontrup J, Nattkemper TW. BIIGLE Tools - A Web 2.0 Approach for Visual Bioimage Database Mining. In: IV 09 - Intl. Conference on Information Visualization. IEEE; 2009.In this paper we want to discuss the usage of web 2.0 techniques to realize information visualization based exploration and annotation of huge volume, semi-structured data, and in particular high throughput bioimage series. To this end, we developed a toolbox for a graphical representation of different displays in a browser context which can be used for image database exploration in a link & brush fashion. The web based approach proved to be capable of information visualization tasks and supports collaboration of several users at arbitrary locations