Truth Discovery in Crowdsourced Detection of Spatial Events

Postprin

Globally Optimal Crowdsourcing Quality Management

We study crowdsourcing quality management, that is, given worker responses to a set of tasks, our goal is to jointly estimate the true answers for the tasks, as well as the quality of the workers. Prior work on this problem relies primarily on applying Expectation-Maximization (EM) on the underlying maximum likelihood problem to estimate true answers as well as worker quality. Unfortunately, EM only provides a locally optimal solution rather than a globally optimal one. Other solutions to the problem (that do not leverage EM) fail to provide global optimality guarantees as well. In this paper, we focus on filtering, where tasks require the evaluation of a yes/no predicate, and rating, where tasks elicit integer scores from a finite domain. We design algorithms for finding the global optimal estimates of correct task answers and worker quality for the underlying maximum likelihood problem, and characterize the complexity of these algorithms. Our algorithms conceptually consider all mappings from tasks to true answers (typically a very large number), leveraging two key ideas to reduce, by several orders of magnitude, the number of mappings under consideration, while preserving optimality. We also demonstrate that these algorithms often find more accurate estimates than EM-based algorithms. This paper makes an important contribution towards understanding the inherent complexity of globally optimal crowdsourcing quality management

Optimization of the detection of microbes in blood from immunocompromised patients with haematological malignancies

AbstractThe present study aimed to improve the rate of detection of blood-borne microbes by using PCRs with pan-bacterial and Candida specificity. Seventeen per cent of the blood samples (n = 178) collected from 107 febrile patients with haematological malignancies were positive using standard culture (BacT/Alert system). Candida PCR was positive in 12 patients, only one of whom scored culture-positive. Bacterial PCR using fresh blood samples was often negative, but the detection rate increased when the blood was pre-incubated for 2 days. These data indicate that PCR assays might be a complement for the detection of blood-borne opportunists in immunocompromised haematology patients

Inter-expert and intra-expert reliability in sleep spindle scoring

Objectives: To measure the inter-expert and intra-expert agreement in sleep spindle scoring, and to quantify how many experts are needed to build a reliable dataset of sleep spindle scorings. Methods: The EEG dataset was comprised of 400 randomly selected 115 s segments of stage 2 sleep from 110 sleeping subjects in the general population (57 ± 8, range: 42–72 years). To assess expert agreement, a total of 24 Registered Polysomnographic Technologists (RPSGTs) scored spindles in a subset of the EEG dataset at a single electrode location (C3-M2). Intra-expert and inter-expert agreements were calculated as F_1-scores, Cohen’s kappa (κ), and intra-class correlation coefficient (ICC). Results: We found an average intra-expert F_1-score agreement of 72 ± 7% (κ: 0.66 ± 0.07). The average inter-expert agreement was 61 ± 6% (κ: 0.52 ± 0.07). Amplitude and frequency of discrete spindles were calculated with higher reliability than the estimation of spindle duration. Reliability of sleep spindle scoring can be improved by using qualitative confidence scores, rather than a dichotomous yes/no scoring system. Conclusions: We estimate that 2–3 experts are needed to build a spindle scoring dataset with ‘substantial’ reliability (κ: 0.61–0.8), and 4 or more experts are needed to build a dataset with ‘almost perfect’ reliability (κ: 0.81–1). Significance: Spindle scoring is a critical part of sleep staging, and spindles are believed to play an important role in development, aging, and diseases of the nervous system

Efficient crowdsourcing for multi-class labeling

Crowdsourcing systems like Amazon's Mechanical Turk have emerged as an effective large-scale human-powered platform for performing tasks in domains such as image classification, data entry, recommendation, and proofreading. Since workers are low-paid (a few cents per task) and tasks performed are monotonous, the answers obtained are noisy and hence unreliable. To obtain reliable estimates, it is essential to utilize appropriate inference algorithms (e.g. Majority voting) coupled with structured redundancy through task assignment. Our goal is to obtain the best possible trade-off between reliability and redundancy. In this paper, we consider a general probabilistic model for noisy observations for crowd-sourcing systems and pose the problem of minimizing the total price (i.e. redundancy) that must be paid to achieve a target overall reliability. Concretely, we show that it is possible to obtain an answer to each task correctly with probability 1-ε as long as the redundancy per task is O((K/q) log (K/ε)), where each task can have any of the $K$ distinct answers equally likely, q is the crowd-quality parameter that is defined through a probabilistic model. Further, effectively this is the best possible redundancy-accuracy trade-off any system design can achieve. Such a single-parameter crisp characterization of the (order-)optimal trade-off between redundancy and reliability has various useful operational consequences. Further, we analyze the robustness of our approach in the presence of adversarial workers and provide a bound on their influence on the redundancy-accuracy trade-off. Unlike recent prior work [GKM11, KOS11, KOS11], our result applies to non-binary (i.e. K>2) tasks. In effect, we utilize algorithms for binary tasks (with inhomogeneous error model unlike that in [GKM11, KOS11, KOS11]) as key subroutine to obtain answers for K-ary tasks. Technically, the algorithm is based on low-rank approximation of weighted adjacency matrix for a random regular bipartite graph, weighted according to the answers provided by the workers.National Science Foundation (U.S.

Limitations of Majority Agreement in Crowdsourced Image Interpretation

Crowdsourcing can efficiently complete tasks that are difficult to automate, but the quality of crowdsourced data is tricky to evaluate. Algorithms to grade volunteer work often assume that all tasks are similarly difficult, an assumption that is frequently false. We use a cropland identification game with over 2,600 participants and 165,000 unique tasks to investigate how best to evaluate the difficulty of crowdsourced tasks and to what extent this is possible based on volunteer responses alone. Inter-volunteer agreement exceeded 90% for about 80% of the images and was negatively correlated with volunteer-expressed uncertainty about image classification. A total of 343 relatively difficult images were independently classified as cropland, non-cropland or impossible by two experts. The experts disagreed weakly (one said impossible while the other rated as cropland or non-cropland) on 27% of the images, but disagreed strongly (cropland vs. non-cropland) on only 7%. Inter-volunteer disagreement increased significantly with inter-expert disagreement. While volunteers agreed with expert classifications for most images, over 20% would have been mis-categorized if only the volunteers’ majority vote was used. We end with a series of recommendations for managing the challenges posed by heterogeneous tasks in crowdsourcing campaigns

Challenging the heterogeneity of disease presentation in malignant melanoma-impact on patient treatment

There is an increasing global interest to support research areas that can assist in understanding disease and improving patient care. The National Cancer Institute (NIH) has identified precision medicine-based approaches as key research strategies to expedite advances in cancer research. The Cancer Moonshot program ( https://www.cancer.gov/research/key-initiatives/moonshot-cancer-initiative ) is the largest cancer program of all time, and has been launched to accelerate cancer research that aims to increase the availability of therapies to more patients and, ultimately, to eradicate cancer. Mass spectrometry-based proteomics has been extensively used to study the molecular mechanisms of cancer, to define molecular subtypes of tumors, to map cancer-associated protein interaction networks and post-translational modifications, and to aid in the development of new therapeutics and new diagnostic and prognostic tests. To establish the basis for our melanoma studies, we have established the Southern Sweden Malignant Melanoma Biobank. Tissues collected over many years have been accurately characterized with respect to the tumor and patient information. The extreme variability displayed in the protein profiles and the detection of missense mutations has confirmed the complexity and heterogeneity of the disease. It is envisaged that the combined analysis of clinical, histological, and proteomic data will provide patients with a more personalized medical treatment. With respect to disease presentation, targeted treatment and medical mass spectrometry analysis and imaging, this overview report will outline and summarize the current achievements and status within malignant melanoma. We present data generated by our cancer research center in Lund, Sweden, where we have built extensive capabilities in biobanking, proteogenomics, and patient treatments over an extensive time period

Plasma proteomic profiling in postural orthostatic tachycardia syndrome (POTS) reveals new disease pathways

Postural orthostatic tachycardia syndrome (POTS) is a cardiovascular autonomic disorder characterized by excessive heart rate increase on standing, leading to debilitating symptoms with limited therapeutic possibilities. Proteomics is a large-scale study of proteins that enables a systematic unbiased view on disease and health, allowing stratification of patients based on their protein background. The aim of the present study was to determine plasma protein biomarkers of POTS and to reveal proteomic pathways differentially regulated in POTS. We performed an age- and sex-matched, case–control study in 130 individuals (case–control ratio 1:1) including POTS and healthy controls. Mean age in POTS was 30 ± 9.8 years (84.6% women) versus controls 31 ± 9.8 years (80.0% women). We analyzed plasma proteins using data-independent acquisition (DIA) mass spectrometry. Pathway analysis of significantly differently expressed proteins was executed using a cutoff log2 fold change set to 1.2 and false discovery rate (p-value) of < 0.05. A total of 393 differential plasma proteins were identified. Label-free quantification of DIA-data identified 30 differentially expressed proteins in POTS compared with healthy controls. Pathway analysis identified the strongest network interactions particularly for proteins involved in thrombogenicity and enhanced platelet activity, but also inflammation, cardiac contractility and hypertrophy, and increased adrenergic activity. Our observations generated by the first use a label-free unbiased quantification reveal the proteomic footprint of POTS in terms of a hypercoagulable state, proinflammatory state, enhanced cardiac contractility and hypertrophy, skeletal muscle expression, and adrenergic activity. These findings support the hypothesis that POTS may be an autoimmune, inflammatory and hyperadrenergic disorder

Selected reactive oxygen species and antioxidant enzymes in common bean after Pseudomonas syringae pv. phaseolicola and Botrytis cinerea infection

Phaseolus vulgaris cv. Korona plants were inoculated with the bacteria Pseudomonas syringae pv. phaseolicola (Psp), necrotrophic fungus Botrytis cinerea (Bc) or with both pathogens sequentially. The aim of the experiment was to determine how plants cope with multiple infection with pathogens having different attack strategy. Possible suppression of the non-specific infection with the necrotrophic fungus Bc by earlier Psp inoculation was examined. Concentration of reactive oxygen species (ROS), such as superoxide anion (O2 -) and H2O2 and activities of antioxidant enzymes such as superoxide dismutase (SOD), catalase (CAT) and peroxidase (POD) were determined 6, 12, 24 and 48 h after inoculation. The measurements were done for ROS cytosolic fraction and enzymatic cytosolic or apoplastic fraction. Infection with Psp caused significant increase in ROS levels since the beginning of experiment. Activity of the apoplastic enzymes also increased remarkably at the beginning of experiment in contrast to the cytosolic ones. Cytosolic SOD and guaiacol peroxidase (GPOD) activities achieved the maximum values 48 h after treatment. Additional forms of the examined enzymes after specific Psp infection were identified; however, they were not present after single Bc inoculation. Subsequent Bc infection resulted only in changes of H2O2 and SOD that occurred to be especially important during plant–pathogen interaction. Cultivar Korona of common bean is considered to be resistant to Psp and mobilises its system upon infection with these bacteria. We put forward a hypothesis that the extent of defence reaction was so great that subsequent infection did not trigger significant additional response