Task execution timeliness, i.e., the completion of a task within a given time frame, is a known open issue in crowdsourcing. While running tasks on crowdsourcing platforms a requester experiences long tails in execution caused by abandoned assignments (those left by workers unfinished), which become available for other workers only after some expiration time (e.g., 30 minutes in CrowdFlower). These abandoned assignments result in significant delays and a poor predictability of the overall task execution time. In this paper, we propose an approach and an implementation called ReLauncher to identify such abandoned assignments and relaunch them for other workers. We evaluate our implementation with an experiment on CrowdFlower that provides substantive evidence for a significant execution speed improvement with an average extra cost of about 10%
The internet enables us to collect and store unprecedented amounts of data. We need better models for processing, analyzing, and making conclusions from the data. In this work, crowdsourcing is presented as a viable option for collecting data, extracting patterns and insights from big data. Humans in collaboration, when provided with appropriate tools, can collectively see patterns, extract insights and draw conclusions from data. We study different models and algorithms for crowdsourcing discovery.In each section in this dissertation a problem is proposed, the importance of it is discussed, solutions are proposed and evaluated. Crowdsourcing is the unifying theme for the projects that are presented in this dissertation. In the first half of the dissertation we study different aspects of crowdsourcing like pricing, completion times, incentives, and consistency with in-lab and controlled experiments. In the second half of the dissertation we focus on Opinion Space\footnote{opinion.berkeley.edu} and the algorithms and models that we designed for collecting innovative ideas from participants. This dissertation specifically studies how to use crowdsourcing to discover patterns and innovative ideas.We start by looking at the CONE Welder project\footnote{Available at http://cone.berkeley.edu/ from 2008 to 2011} which uses a robotic camera in a remote location to study the effect of climate change on the migration of birds. In CONE, an amateur birdwatcher can operate a robotic camera at a remote location from within her web browser. She can take photos of different bird species and classify different birds using the user interface in CONE. This allowed us to compare the species presented in the area from 2008 to 2011 with the species presented in the area that are reported by Blacklock in 1984 \cite{Blacklock:1984}. Citizen scientists found eight avian species previously unknown to have breeding populations within the region. CONE is an example of using crowdsourcing for discovering new migration patterns.Crowdsourcing can also be used to collect data on human motor movement. Fitts' law is a classical model to predict the average movement time for a human motor motion. It has been traditionally used in the field of human-computer interaction (HCI) as a model that explains the movement time from an origin to a target by a pointing device and it is a logarithmic function of the width of the target (W) and the distance of the pointer to the target (A). In the next project we first present the square-root variant of the Fitts' law similar to Meyer et el. \cite{meyer1988optimality}. To evaluate this model we performed two sets of studies, one uncontrolled and crowdsourced study and one in-lab controlled study with 46 participants. We show that the data collected from the crowdsourced experiment accurately follows the results from the in-lab experiments. For Homogeneous Targets the Square-Root model (T=a+bWA) results in a smaller ERMS error than the two other control models, LOG (T=a+blogW2A) and LOG' (T=a+blogWA+1) for A/W<10. Similarly for Heterogeneous Targets the Square-Root model results in a significantly smaller ERMS error when compared to the LOG model for A/W<10. The LOG model resulted in significantly smaller ERMS error in the A/W>15. In the Heterogeneous Targets the LOG' model consistently resulted in a significantly smaller error for 0OpinionSpaceisasystemthatdirectlyelicitsopinionsfromparticipantsforideageneration.Itusesbothnumericalandtextualdataandwelookatmethodstocombinethesetwosetsofdata.CanonicalCorrelationAnalysis,CCA,isusedasamethodtocombineboththetextualandnumericalinputsfromparticipants.CCAseekstofindlineartransformationmatricesthatmaximizethelowerdimensioncorrelationbetweentheprojectionofnumericalrating(Xw_x)andtextualcommentsontothetwodimensionalspace(Yw_y).Inotherwordsitseekstosolvethefollowingproblemargmax_{w_x,w_y} corr(Xw_x, Yw_y)inwhichXandYarerepresentationsofthenumericalratingandtextualcommentsofparticipantsinhighdimensionsandXw_xandYw_yaretheirlowerdimensionrepresentations.Byusingparticipants′numericalfeedbacksoneachothers′comments,wethendevelopanevaluationframeworktocomparedifferentdimensionalityreductionmethods.InourevaluationframeworkadimensionalityreductionisthemostappropriateforOpinionSpacewhenthevalueof\gamma = -corr(r,D)hasthelargestvalue.In\gamma = -corr(R,D),Risthesetofr_{ij}values.r_{ij}istheratingthattheparticipantiisgivingtothetextualopinionofparticipantj.SimilarlyDisthesetthatcontainsd_{ij}values.d_{ij}istheEuclideandistancebetweenthelocationsofparticipantiandj.InthisdissertationweprovidesupportingargumentastowhythisevaluationframeworkisappropriateforOpinionSpace.WehavecompareddifferentvariationsofCCAandPCAdimensionalityreductionsondifferentdatasets.Ourresultssuggeststhatthe\gammavaluesforCCAareatleast\%169largerthanthe\gammavaluesofPCA,makingCCAamoreappropriatedimensionalityreductionmodelforOpinionSpace.Aproductreviewonanonlineretailerwebsiteisoftenaccompaniedwithnumericalratingsfortheproductondifferentscales,atextualreviewandsometimesinformationonwhetherornotthereviewishelpful.GeneralizedSentimentAnalysislooksatthecorrelationbetweenthetextualcommentandnumericalratingandusesthattoinferthenumericalratingsondifferentscalesfromthetextualcomment.WeprovidetheformulationsforusingCCAforsolvingsuchaproblem.WecompareourCCAmodelwithSupportVectorMachine,LinearRegression,andothertraditionalmachinelearningmodelsandhighlightthestrengthsandweaknessesofthismodel.WefoundthattrainingtheCCAformulationissignificantlyfasterthanSVMwhichistraditionallyusedinthiscontext(thefastesttrainingtimeforSVMinLibSVMwas1,126secondswhileCCAtookonly33secondsfortraining).WealsoobservedthattheMeanSquaredErrorforCCAwassmallerthanothercompetingmodels(TheMSEforCCAwithtf−idffeatureswas1.69whilethisvalueforSVMwas2.28).Linearregressionwasmoresensitivetothefeaturizationmethod.ItresultedinlargerMSEwhenusedonmultinomial(MSE = 8.88)andBernoullifeatures(MSE = 4.21)butsmallerMSEwhentf−idfweightswereused(MSE=1.47$)
The scientific field study of wildlife often requires vigilant observation of detailed animal behavior over extended periods. In remote and inhospitable locations, observation can be an arduous, expensive, and dangerous experience for field scientists. We are developing a new class of networked teleoperated robotic “observatories ” that allows “citizen scientists ” and professional scientists to remotely observe, record, and index animal activity and behaviors via the internet. This paper describes CONE-Welder, installed at the Rob & Bessie Welder Wildlife Foundation in Texas to gather photographic and quantitative data for a biological study of avian activity and hypothesized range change for selected subtropical bird species. Since the system was deployed on 12 May 2008, over 600 users (“players”) have participated online. Players have requested over 2.2 million camera frames and captured over 29,000 photographs. Within these photos, citizen scientists have classified 74 unique species, including eight avian species previously unknown to have breeding populations within the region. The collected dataset quantifies seasonal presence of birds of particular interest, e.g., the Green Jay (Cyanocorax incas). This paper describes the system architecture, the game interface that provides incentives for player participation, and initial data collected. CONE-Welder is available online at