A crowdsourcing semi-automatic image segmentation platform for cell biology

State-of-the-art computer-vision algorithms rely on big and accurately annotated data, which are expensive, laborious and time-consuming to generate. This task is even more challenging when it comes to microbiological images, because they require specialized expertise for accurate annotation. Previous studies show that crowdsourcing and assistive-annotation tools are two potential solutions to address this challenge. In this work, we have developed a web-based platform to enable crowdsourcing annotation of image data; the platform is powered by a semi-automated assistive tool to support non-expert annotators to improve the annotation efficiency. The behavior of annotators with and without the assistive tool is analyzed, using biological images of different complexity. More specifically, non-experts have been asked to use the platform to annotate microbiological images of gut parasites, which are compared with annotations by experts. A quantitative evaluation is carried out on the results, confirming that the assistive tools can noticeably decrease the non-expert annotation�s cost (time, click, interaction, etc.) while preserving or even improving the annotation�s quality. The annotation quality of non-experts has been investigated using IOU (intersection of union), precision and recall; based on this analysis we propose some ideas on how to better design similar crowdsourcing and assistive platforms

A Survey of Crowdsourcing in Medical Image Analysis

Rapid advances in image processing capabilities have been seen across many domains, fostered by the application of machine learning algorithms to "big-data". However, within the realm of medical image analysis, advances have been curtailed, in part, due to the limited availability of large-scale, well-annotated datasets. One of the main reasons for this is the high cost often associated with producing large amounts of high-quality meta-data. Recently, there has been growing interest in the application of crowdsourcing for this purpose; a technique that a technique that is well established in a number of disciplines, including astronomy, ecology and meteorology for creating large-scale datasets across a range of disciplines, from computer vision to astrophysics. Despite the growing popularity of this approach, there has not yet been a comprehensive literature review to provide guidance to researchers considering using crowdsourcing methodologies in their own medical imaging analysis. In this survey, we review studies applying crowdsourcing to the analysis of medical images, published prior to July 2018. We identify common approaches and challenges and provide recommendations to researchers implementing crowdsourcing for medical imaging tasks. Finally, we discuss future opportunities for development within this emerging domain

Large-scale medical image annotation with quality-controlled crowdsourcing

Accurate annotations of medical images are essential for various clinical applications. The remarkable advances in machine learning, especially deep learning based techniques, show great potential for automatic image segmentation. However, these solutions require a huge amount of accurately annotated reference data for training. Especially in the domain of medical image analysis, the availability of domain experts for reference data generation is becoming a major bottleneck for machine learning applications. In this context, crowdsourcing has gained increasing attention as a tool for low-cost and large-scale data annotation. As a method to outsource cognitive tasks to anonymous non-expert workers over the internet, it has evolved into a valuable tool for data annotation in various research fields. Major challenges in crowdsourcing remain the high variance in the annotation quality as well as the lack of domain specific knowledge of the individual workers. Current state-of-the-art methods for quality control usually induce further costs, as they rely on a redundant distribution of tasks or perform additional annotations on tasks with already known reference outcome. Aim of this thesis is to apply common crowdsourcing techniques for large-scale medical image annotation and create a cost effective quality control method for crowd-sourced image annotation. The problem of large-scale medical image annotation is addressed by introducing a hybrid crowd-algorithm approach that allowed expert-level organ segmentation in CT scans. A pilot study performed on the case of liver segmentation in abdominal CT scans showed that the proposed approach is able to create organ segmentations matching the quality of those create by medical experts. Recording the behavior of individual non-expert online workers during the annotation process in clickstreams enabled the derivation of an annotation quality measure that could successfully be used to merge crowd-sourced segmentations. A comprehensive validation study performed with various object classes from publicly available data sets demonstrated that the presented quality control measure generalizes well over different object classes and clearly outperforms state-of-the-art methods in terms of costs and segmentation quality. In conclusion, the methods introduced in this thesis are an essential contribution to reduce the annotation costs and further improve the quality of crowd-sourced image segmentation

ImageNet Large Scale Visual Recognition Challenge

The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements.Comment: 43 pages, 16 figures. v3 includes additional comparisons with PASCAL VOC (per-category comparisons in Table 3, distribution of localization difficulty in Fig 16), a list of queries used for obtaining object detection images (Appendix C), and some additional reference

New forms of collaborative innovation and production on the internet : an interdisciplinary perspective

Contents Introduction 1 New forms of collaborative innovation and production on the Internet : Volker Wittke and Heidemarie Hanekop Interdisciplinary perspectives on collaborative innovation and production: Conceptual debates 2 Customer Co-Creation: Open Innovation with Customers : Frank Piller, Christoph Ihl and Alexander Vossen 3 Governing Social Production : Niva Elkin-Koren 4 Trust Management in Online Communities : Audun Jøsang 5 Building a reputation system for Wikipedia : Christian Damsgaard Jensen 6 Cooperation in Wikipedia from a Network Perspective : Christian Stegbauer Firm driven collaborative innovation and production: Case studies 7 Managing a New Consumer Culture: “Working Consumers” in Web 2.0 as a Source of Corporate Feedback : Sabine Hornung, Frank Kleemann and G. Günter Voß 8 Prosuming, or when customers turn collaborators: coordination and motivation of customer contribution : Birgit Blättel-Mink, Raphael Menez, Dirk Dalichau, Daniel Kahnert 9 Role Confusion in Open Innovation Intermediary Arenas : Tobias Fredberg, Maria Elmquist, Susanne Ollila, Anna Yström List of Contributor

New forms of collaborative innovation and production on the internet

The Internet has enabled new forms of large-scale collaboration. Voluntary contributions by large numbers of users and co-producers lead to new forms of production and innovation, as seen in Wikipedia, open source software development, in social networks or on user-generated content platforms as well as in many firm-driven Web 2.0 services. Large-scale collaboration on the Internet is an intriguing phenomenon for scholarly debate because it challenges well established insights into the governance of economic action, the sources of innovation, the possibilities of collective action and the social, legal and technical preconditions for successful collaboration. Although contributions to the debate from various disciplines and fine-grained empirical studies already exist, there still is a lack of an interdisciplinary approach

New forms of collaborative innovation and production on the internet

The Internet has enabled new forms of large-scale collaboration. Voluntary contributions by large numbers of users and co-producers lead to new forms of production and innovation, as seen in Wikipedia, open source software development, in social networks or on user-generated content platforms as well as in many firm-driven Web 2.0 services. Large-scale collaboration on the Internet is an intriguing phenomenon for scholarly debate because it challenges well established insights into the governance of economic action, the sources of innovation, the possibilities of collective action and the social, legal and technical preconditions for successful collaboration. Although contributions to the debate from various disciplines and fine-grained empirical studies already exist, there still is a lack of an interdisciplinary approach

New forms of collaborative innovation and production on the internet - an interdisciplinary perspective

The Internet has enabled new forms of large-scale collaboration. Voluntary contributions by large numbers of users and co-producers lead to new forms of production and innovation, as seen in Wikipedia, open source software development, in social networks or on user-generated content platforms as well as in many firm-driven Web 2.0 services. Large-scale collaboration on the Internet is an intriguing phenomenon for scholarly debate because it challenges well established insights into the governance of economic action, the sources of innovation, the possibilities of collective action and the social, legal and technical preconditions for successful collaboration. Although contributions to the debate from various disciplines and fine-grained empirical studies already exist, there still is a lack of an interdisciplinary approach