27 research outputs found
Maximizing citizen scientistsâ contribution to automated species recognition
publishedVersio
Recognizability bias in citizen science photographs
Citizen science and automated collection methods increasinglydepend on image recognition to provide the amountsof observational data research and management needs.Recognition models, meanwhile, also require large amounts ofdata from these sources, creating a feedback loop between themethods and tools. Species that are harder to recognize, bothfor humans and machine learning algorithms, are likely to beunder-reported, and thus be less prevalent in the trainingdata. As a result, the feedback loop may hamper trainingmostly for species that already pose the greatest challenge. Inthis study, we trained recognition models for various taxa, andfound evidence for aârecognizability biasâ, where species thatare more readily identified by humans and recognitionmodels alike are more prevalent in the available image data.This pattern is present across multiple taxa, and does notappear to relate to differences in picture quality, biologicaltraits or data collection metrics other than recognizability. Thishas implications for the expected performance of futuremodels trained with more data, including such challenging species. citizen science, image recognition, machinelearning, recognizability, artificial intelligence/environmental science/ecology, Ecology, conservation and global change biologypublishedVersio
COOD:Combined out-of-distribution detection using multiple measures for anomaly & novel class detection in large-scale hierarchical classification
High-performing out-of-distribution (OOD) detection, both anomaly and novel class, is an important prerequisite for the practical use of classification models. In this paper, we focus on the species recognition task in images concerned with large databases, a large number of fine-grained hierarchical classes, severe class imbalance, and varying image quality. We propose a framework for combining individual OOD measures into one combined OOD (COOD) measure using a supervised model. The individual measures are several existing state-of-the-art measures and several novel OOD measures developed with novel class detection and hierarchical class structure in mind. COOD was extensively evaluated on three large-scale (500k+ images) biodiversity datasets in the context of anomaly and novel class detection. We show that COOD outperforms individual, including state-of-the-art, OOD measures by a large margin in terms of TPR@1% FPR in the majority of experiments, e.g., improving detecting ImageNet images (OOD) from 54.3% to 85.4% for the iNaturalist 2018 dataset. SHAP (feature contribution) analysis shows that different individual OOD measures are essential for various tasks, indicating that multiple OOD measures and combinations are needed to generalize. Additionally, we show that explicitly considering ID images that are incorrectly classified for the original (species) recognition task is important for constructing high-performing OOD detection methods and for practical applicability. The framework can easily be extended or adapted to other tasks and media modalities
Deep Learning from Label Proportions for Emphysema Quantification
We propose an end-to-end deep learning method that learns to estimate
emphysema extent from proportions of the diseased tissue. These proportions
were visually estimated by experts using a standard grading system, in which
grades correspond to intervals (label example: 1-5% of diseased tissue). The
proposed architecture encodes the knowledge that the labels represent a
volumetric proportion. A custom loss is designed to learn with intervals. Thus,
during training, our network learns to segment the diseased tissue such that
its proportions fit the ground truth intervals. Our architecture and loss
combined improve the performance substantially (8% ICC) compared to a more
conventional regression network. We outperform traditional lung densitometry
and two recently published methods for emphysema quantification by a large
margin (at least 7% AUC and 15% ICC), and achieve near-human-level performance.
Moreover, our method generates emphysema segmentations that predict the spatial
distribution of emphysema at human level.Comment: Accepted to MICCAI 201
Adversarial Attack Vulnerability of Medical Image Analysis Systems: Unexplored Factors
Adversarial attacks are considered a potentially serious security threat for
machine learning systems. Medical image analysis (MedIA) systems have recently
been argued to be vulnerable to adversarial attacks due to strong financial
incentives and the associated technological infrastructure.
In this paper, we study previously unexplored factors affecting adversarial
attack vulnerability of deep learning MedIA systems in three medical domains:
ophthalmology, radiology, and pathology. We focus on adversarial black-box
settings, in which the attacker does not have full access to the target model
and usually uses another model, commonly referred to as surrogate model, to
craft adversarial examples. We consider this to be the most realistic scenario
for MedIA systems.
Firstly, we study the effect of weight initialization (ImageNet vs. random)
on the transferability of adversarial attacks from the surrogate model to the
target model. Secondly, we study the influence of differences in development
data between target and surrogate models. We further study the interaction of
weight initialization and data differences with differences in model
architecture. All experiments were done with a perturbation degree tuned to
ensure maximal transferability at minimal visual perceptibility of the attacks.
Our experiments show that pre-training may dramatically increase the
transferability of adversarial examples, even when the target and surrogate's
architectures are different: the larger the performance gain using
pre-training, the larger the transferability. Differences in the development
data between target and surrogate models considerably decrease the performance
of the attack; this decrease is further amplified by difference in the model
architecture. We believe these factors should be considered when developing
security-critical MedIA systems planned to be deployed in clinical practice.Comment: First three authors contributed equall
Supporting citizen scientists with automatic species identification using deep learning image recognition models
Volunteers, researchers and citizen scientists are important contributors to observation and monitoring databases. Their contributions thus become part of a global digital data pool, that forms the basis for important and powerful tools for conservation, research, education and policy. With the data contributed by citizen scientists also come concerns about data completeness and quality. For data generated by citizen scientists taxonomic bias effects, where certain species (groups) are underrepresented in observations, are even stronger than for professionally collected data. Identification tools that help citizen scientists to access more difficult, underrepresented groups, can help to close this gap.
We are exploring the possibilities of using artificial intelligence for automatic species identification as a tool to support the registration of field observations. Our aim is to offer nature enthusiasts the possibility of automatically identifying species, based on photos they have taken as part of an observation. Furthermore, by allowing them to register these identifications as part of the observation, we aim to enhance the completeness and quality of the observation database. We will demonstrate the use of automatic species recognition as part of the process of observation registration, using a recognition model that is based on deep learning techniques.
We investigated the automatic species recognition using deep learning models trained with observation data of the popular website Observation.org (https://observation.org/). At Observation.org data quality is ensured by a review process of all observations by experts. Using the pictures and corresponding validated metadata from their database, models were developed covering several species groups. These techniques were based on earlier work that culminated in ObsIdentify, an free offline mobile app for identifying species based on pictures taken in the field. The models are also made available as an API web service, which allows for identification by offering a photo through common HTTP-communication - essentially like uploading it through a webpage. This web service was implemented in the observation entry workflows of Observation.org. By providing an automatically generated taxonomic identification with each image, we expect to stimulate existing citizen scientists to generate a larger quantity of and more biodiverse observations. Additionally we hope to motivate new citizen scientists to start contributing.
Additionally, we investigated the use of image recognition for the identification of additional species in the photo other than the primary subject, for example the identification of the host plant in photos of insects. The Observation.org database contains many of such photos which are associated with a single species observation, while additional, other species are also present in the photo, but are unidentified. Combining object detection to detect individual species with species recognition models opens up the possibility of automatically identifying and counting these species, enhancing the quality of the observations. In the presentation we will present the initial results of this application of deep learning technology, and discuss the possibilities and challenges
Species Distribution Modelling Using Deep Learning
Species distribution modelling, or ecological niche modelling, is a collection of techniques for the construction of correlative models based on the combination of species occurrences and GIS data. Using such models, a variety of research questions in biodiversity science can be investigated, among which are the assessment of habitat suitability around the globe (e.g. in the case of invasive species), the response of species to alternative climatic regimes (e.g. by forecasting climate change scenarios, or by hindcasting into palaeoclimates), and the overlap of species in niche space. The algorithms used for the construction of such models include maximum entropy, neural networks, and random forests. Recent advances both in computing power and in algorithm development raise the possibility that deep learning techniques will provide valuable additions to these existing approaches. Here, we present our recent findings in the development of workflows to apply deep learning to species distribution modelling, and discuss the prospects for the large-scale application of deep learning in web service infrastructures to analyze the growing corpus of species occurrence data in biodiversity information facilities
Using Deep Learning in Collection Management to Reduce the Taxonomistâs Workload
The completeness and quality of the information in natural history museum collections is essential to support its use, such as in collection management. Currently, the accuracy of the taxonomic information largely depends on expert provided metadata, such as species identification. At present an increase in the use of digitization techniques coincides with a dwindling of the number of taxonomic specialists, creating a growing backlog in specimen identifications.
We are investigating the role of artificial intelligence for automatic species identification in supporting collection management. When identifying collection specimens, common species are predominantly present, taking up a large amount of the expert's time, who has to deal with a relatively easy, repetitive task. Therefore, one of our aims is to use human expertise where it is most needed, for complex tasks, and use properly validated computational methods for repetitive, less difficult identifications. To this end, we demonstrate the use of automatic species identification in digitization workflows, using deep learning based image recognition.
We investigated potential gains in the identification process of a large digitization project of papered Lepidoptera (>500,000 specimens). In this ongoing project, volunteers unpack, register and photograph the unmounted butterflies and repack them sustainably, still unmounted. Using only the individual images made by volunteers, taxonomic experts identify the specimens. Considering that the speed of digitization currently exceeds that of identification, a growing backlog of yet-to-be-identified specimens has formed, limiting the speed of publication of this biodiversity information. The test case for image recognition concerns specimens of the families Papilionidae and Lycaenidae, mostly collected in Indonesia.
By allowing the volunteers to provide an automatically generated identification with each image, we enable the taxonomic specialists to quickly validate the more easily identifiable specimens. This reduces their workload, allows them to focus on the more demanding specimens and increases the rate of specimen identification. We demonstrate how to combine computer and human decisions to ensure both high data quality standards and reduction of expert time
Smart Insect Cameras
Recent studies have shown a worrying decline in the quantity and diversity of insects at a number of locations in Europe (Hallmann et al. 2017) and elsewhere (Lister and Garcia 2018). Although the downward trend that these studies show is clear, they are limited to certain insect groups and geographical locations. Most available studies (see overview in SĂĄnchez-Bayo and Wyckhuys 2019) were performed in nature reserves, leaving rural and urban areas largely understudied. Most studies are based on the long-term collaborative efforts of entomologists and volunteers performing labor-intensive repeat measurements, inherently limiting the number of locations that can be monitored.
We propose a monitoring network for insects in the Netherlands, consisting of a large number of smart insect cameras spread across nature, rural, and urban areas. The aim of the network is to provide a labor-extensive continuous monitoring of different insect groups. In addition, we aimed to develop the cameras at a relatively cheap price point so that cameras can be installed at a large number of locations and encourage participation by citizen science enthusiasts. The cameras are made smart with image processing, consisting of image enhancement, insect detection and species identification being performed, using deep learning based algorithms. The cameras take pictures of a screen, measuring ca. 30Ă40 cm, every 10 seconds, capturing insects that have landed on the screen (Fig. 1). Several screen setups were evaluated. Vertical screens were used to attract flying insects. Different screen colors and lighting at night, to attract night flying insects such as moths, were used. In addition two horizontal screen orientations were used (1) to emulate pan traps to attract several pollinator species (bees and hoverflies) and (2) to capture ground-based insects and arthropods such as beetles and spiders.
Time sequences of images were analyzed semi-automatically, in the following way. First, single insects are outlined and cropped using boxes at every captured image. Then the cropped single insects in every image were preliminarily identified, using a previously developed deep-learning-based automatic species identification software, Nature Identification API (https://identify.biodiversityanalysis.nl). In the next step, single insects were linked between consecutive images using a tracking algorithm that uses screen position and the preliminary identifications. This step yields for every individual insect a linked series of outlines and preliminary identifications. The preliminary identifications for individual insects can differ between multiple captured images and were therefore combined into one identification using a fusing algorithm. The result of the algorithm is a series of tracks of individual insects with species identifications, which can be subsequently translated into an estimate of the counts of insects per species or species complexes.
Here we show the first set of results acquired during the spring and summer of 2019. We will discuss practical experiences with setting up cameras in the field, including the effectiveness of the different set-ups. We will also show the effectiveness of using automatic species identification in the type of images that were acquired (see attached figure) and discuss to what extent individual species can be identified reliably. Finally, we will discuss the ecological information that can be extracted from the smart insect cameras
Machine Learning Model for Identifying Dutch/Belgian Biodiversity
The potential of citizen scientists to contribute to information about occurrences of species and other biodiversity questions is large because of the ubiquitous presence of organisms and friendly nature of the subject. Online platforms that collect observations of species from the public have existed for several years now. They have seen a rapid growth recently, partly due to the widespread availability of mobile phones. These online platforms, and many scientific studies as well, suffer from a taxonomic bias: the effect that certain species groups are overrepresented in the data (Troudet et al. 2017). One of the reasons for this bias is that the accurate identification of species, by non-experts and experts, has been limited by the large number of species that exist. Even in the geographically limited area of the Netherlands and Belgium, the number of species that are regularly observed are in the thousands. This makes the ability to identify all those species difficult or impossible for an individual.
Recent advances in species identification powered by deep learning, based on images (Norouzzadeh et al. 2018), suggest a large potential for a new set of digital tools that can help the public (and experts) to identify species automatically. The online observation platform Observation.org has collected over 93 million occurrences in the Netherlands and Belgium in the last 15 years. About 20% of these occurrences are supported by photographs, giving a rich database of 17 million photographs covering all major species groups (e.g., birds, mammals, plants, insects, fungi). Most of the observations with photos were validated by human experts at Observation.org, creating a unique database suitable for machine learning. We have developed a deep learning-based species identification model using this database containing 13,767 species, 1,530 species-groups, 734 subspecies and 117 hybrids. The model is made available to the public through a web service (https://identify.biodiversityanalysis.nl) and through a set of mobile apps (ObsIdentify).
In this talk we will discuss our technical approach for dealing with the large number of species in a deep learning model. We will evaluate the results in terms of performance for different species groups and what this could mean to address part of the taxonomic bias. We will also consider limitations of (image-based) automated species identification and determine venues to further improve identification. We will illustrate how the web service and mobile apps are applied to support citizen scientists and the observation validation workflows at Observation.org. Finally, we will examine the potential of these methods to provide large scale automated analysis of biodiversity data