A paid open access option is available for this journal. Authors own final version only can be archived Publisher's version/PDF cannot be used On author's website immediately On any open access repository after 12 months from publication Published source must be acknowledged Must link to publisher version Set phrase to accompany link to published version (see policy) Articles in some journals can be made Open Access on payment of additional chargePublisher PD

Corsar, David

Edwards, Pete

Markovic, Milan

Pan, Jeff Z

English

Milan Markovic

Peter Edwards

David Corsar

Jeff Z. Pan

Crossref

DEMO: Managing the Provenance of Crowdsourced Disruption Reports

Research Archive

DEMO: Managing the Provenance ofCrowdsourced Disruption Reports?Milan Markovic, Peter Edwards, David Corsar, and Jeff Z. PanComputing Science & dot.rural Digital Economy Hub, University of Aberdeen,Aberdeen, AB24 5UA{m.markovic,p.edwards,dcorsar,j.z.pan} @ abdn.ac.ukAbstract. Human computation systems that outsource tasks to thecrowd often have to address issues associated with the quality of con-tributions. We are exploring the potential role of provenance to facilitateprocesses such as quality assessment within such systems. In this demo wepresent an application for managing traffic disruption reports generatedby the crowd, and outline the technologies used to integrate provenance,linked data, and streams.1 IntroductionPart of the original vision for the World Wide Web described by Berners-Leeand Fischetti in Weaving the Web [2] was the creation of a human networkthat would make it possible to create abstract social machines on the Web.These machines are described as: “processes in which the people do the creativework and the machine does the administration...”. This is very similar to thehuman-based computation concept [5], where certain steps of a computationalprocess are outsourced to humans. Both these visions of the web create a needfor an infrastructure to handle the incorporation of human elements within alarger social computing ecosystem. Hendler [3] noted that early social machinesalready exist on the Web in the form of interactive applications (e.g. Wikipedia1).Hendler also highlighted that these applications are limited as their functionsare largely isolated from one another (e.g. they are unable to easily share data).We argue that this limitation could be addressed by emerging practices suchas linked data [1] - a set of principles for consuming and publishing machine-readable data on the web.One of the ways of obtaining human input is through harvesting of so-calledcollective intelligence via crowdsourcing methods. Systems using crowdsourcingtypically rely on large, diverse crowds, where the number of error generatingindividuals is small, resulting in minimal effect to overall system performance.However, in situations where the crowd size is small (perhaps because of the? The research described here is supported by the award made by the RCUK Dig-ital Economy programme to the dot.rural Digital Economy Hub; award reference:EP/G066051/11 http://wikipedia.orgnature of the task or a limited population of potential participants) the po-tential for adverse effects caused by unreliable individuals is significant. Withinsuch systems it is therefore critical to reason about the quality of contribu-tions. We propose a solution to facilitate such reasoning operations based on themaintenance of a provenance record within the crowdsourcing system. In thiscontext provenance would mean a record of the data generated/maintained bythe crowd and the process(es) involved. Another important characteristic of suchapplications is their dynamic nature, with participants creating, maintaining orvalidating data continuously over time. We argue that participant interactionsshould therefore be modelled as a continuous stream of data elements publishedin compliance with the Linked Data Principles [4]. This necessitates a provenancesolution able to interoperate with such streams.2 Application ScenarioTravel disruption is not easy to predict and even monitoring of disruption posessome challenges (e.g. how to obtain information from the site of an incident). Acrowdsourcing application able to gather, manage, and assess disruption reportswould provide an obvious solution. For example, consider a system that allowsparticipants to report travel disruption events (e.g. an accident on a particu-lar route) from their mobile device. In addition, they are able to perform othertasks such as the creation of links between disruption reports or validation ofdata provided by others (evaluation). By linking here we mean the identificationof relationships between disruption reports (e.g. queuing traffic caused by anincident five miles ahead). However, this data alone does not provide importantcontextual detail such as who created it, who performed a maintenance opera-tion, when and how it was performed - all of which are useful when assessingthe credibility of participants and the data they contribute. We argue that theprovenance record should be able to provide this context, by capturing informa-tion about participants and their activities. For example, user John linked twodisruption reports as related, but in the past links created by him have alwaysbeen subsequently reported as incorrect by others.A disruption event report is likely to trigger a stream of data relating to thisevent (such as other disruption reports, or validation reports). It is thereforeentirely natural to represent these data as a stream of elements, with participantscontributing to a stream about a particular event (e.g. an incident on route A90).A system utilising the crowd to manage travel disruption would thus need tobe built around a set of such streams. Capturing the provenance of a streamobject (e.g. the disruption report that initiated the stream) and the provenanceof stream elements (e.g. who created a specific data element, or created a linkbetween elements) would provide additional context to support reasoning aboutthe quality of the data on the stream.External DatasetsTriple Store Web ServerData Data CreationServiceStreamServiceProvenance ServiceData QueryServiceData MaintenanceServiceRoad NetworkReport Assessment  ServiceCrowdFig. 1. Provenance-enabled travel disruption system architecture.3 System ArchitectureWe have constructed a system that is able to gather and manage disruptionreports from the crowd, and to capture the provenance of these activities; thearchitecture of this system is shown in Figure 1. A mobile client application(Figure 2) was built using the jQuery Mobile2 and OpenLayers3 library. Theclient collects information from the crowd and communicates crowdsourced re-sults back to users. It is optimised for use on touch screen mobile devices andsupports the following functionality: creation of disruption reports, creation ofvalidation reports, creation of reports about relationships (links) between twodisruption reports, visualisation of other reports and their links.Fig. 2. A mobile client application.The server-side framework was built as a set of RESTful web services. Thedata within the system is stored in a TDB4 triple store, and accessed via a2 http://jquerymobile.com/3 http://openlayers.org/4 http://jena.apache.org/documentation/tdb/Fuseki5 SPARQL6 endpoint. The framework is responsible for managing reportscreated by the mobile client application, managing the provenance of operationswithin the system, and the storage of this data (both the reports and theirprovenance). The stream service manages the creation of streams (as an orderedsequence of elements) and is responsible for handling stream operations such as:registering/unregistering queries; inserting reports (with associated provenance);and closing/deleting streams. Stream elements consist of the Unique ResourceIdentifier that refers to the data stored in a triple store. The provenance ser-vice generates annotations and is described further in section 3.2. The reportassessment service (described in section 3.3) assesses and annotates reports gen-erated by the crowd. Disruption reports are represented using a travel disruptionontology, describing a set of concepts from the domain of transport and traveldisruption; this ontology was developed following a review of a number of UKtravel information services.3.1 Data IntegrationTo provide additional contextual information about a report, the system auto-matically records a timestamp, user location (from the phone’s GPS receiver),the error associated with the location, and the result of reverse geocoding thelocation. Each report is thus a combination of this data and the data directlycontributed by the participant. When the system receives a report, if a streamalready exists for reports in that location (e.g. the street/road), then the re-port is added to that stream; otherwise, a new stream is created. Some reports(e.g. validation) explicitly state the relationship to an existing report, and aretherefore added to the same stream as the report to which they refer.3.2 Provenance InformationThree components within the system either manage or use provenance infor-mation: the provenance service, the stream service, and the report assessmentservice. Two types of provenance annotations are generated by the provenanceservice and are stored in a triple store: data provenance and stream provenance.Data provenance is generated in response to a number of events: when data iscreated; when data arrives from the client; or when links between disruption re-ports are created. The data provenance record7 then contains information suchas the agent that created the report, the activities involved in creating the re-port (e.g. acquiring the agent’s location, uploading from the client application,and subsequent processing by the web service), and the entities used/generatedby these activities. Stream provenance is generated in response to: creation ofa new stream; closing a stream; and data being added to a stream. The stream5 http://jena.apache.org/documentation/serving data/6 http://www.w3.org/TR/rdf-sparql-query/7 Expressed using terms from the provenance model being developed by the W3CProvenance Working Group (http://www.w3.org/2011/prov)provenance record then contains information such as the activities that triggeredthe creation/closing of a stream, the activities that added elements to a stream,and the entities used by those activities (e.g. the report that was received).3.3 Report AssessmentThe report assessment service performs evaluation of the submitted reports us-ing the provenance record and other contextual information (other reports, andlinks between reports). Currently we have implemented two prototype metricswithin the service. The first metric is based on the distance between the reporteddisruption and the participant providing the report: the greater the distance be-tween the location of the incident being reported and the location of the userreporting it, the lower the reliability of that report. The second metric uses asimple reputation model for the report creator, based on how previous reportsgenerated by that user have been validated by others. For example, if a user cre-ates a disruption report, which is later validated as correct by others, that user’sreputation for creating disruption reports will increase. However, if others claimthe report is incorrect, then the reputation decreases. As every disruption eventhas a limited lifespan, only validation reports received within that time periodshould be used when building the reputation of a participant. However, as eachdisruption event has a different duration, we are investigating how to incorporatecrowdsourcing the end of disruption events into the client application.During the demonstration the following features of the client application andtravel disruption framework will be presented: observation (disruption report)contribution via the client application to highlight our ontology-driven solutionfor creation of the reports and generation of associated provenance data; vali-dation of disruption reports via the client application; creation of links betweenrelated disruption reports and provenance associated with this process; reason-ing with provenance within the system to identify unreliable disruption reports;and visualisation of the crowdsourced results via the client application.References1. T. Berners-Lee. Linked data. http://www.w3.org/DesignIssues/LinkedData.html,accessed:10/03/2012.2. T. Berners-Lee and M. Fischetti. Weaving the Web: The original design and ultimatedestiny of the World Wide Web. Harper Collins, NY, 1999.3. J. Hendler and T. Berners-Lee. From the semantic web to social machines: Aresearch challenge for AI on the world wide web. Artificial Intelligence, 174(2):156–161, 2009.4. J. F. Sequeda and O. Corcho. Linked stream data: A position paper. In 2ndInternational Workshop on Semantic Sensor Networks (SSN), Washington DC, US,2009.5. L. von Ahn. Human Computation. PhD thesis, Carnegie Mellon University, 2005.

Managing the Provenance of Crowdsourced Disruption Reports

https://aura.abdn.ac.uk/bitstream/2164/4060/1/ipaw2012.pdf

Managing the Provenance of Crowdsourced Disruption Reports

Abstract

Similar works

Full text

Available Versions

Crossref

Research Archive