Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions

Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar - all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.Comment: 40 pages main paper, 5 pages appendi

Exploring Societal Computing based on the Example of Privacy

Data privacy when using online systems like Facebook and Amazon has become an increasingly popular topic in the last few years. This thesis will consist of the following four projects that aim to address the issues of privacy and software engineering. First, only a little is known about how users and developers perceive privacy and which concrete measures would mitigate their privacy concerns. To investigate privacy requirements, we conducted an online survey with closed and open questions and collected 408 valid responses. Our results show that users often reduce privacy to security, with data sharing and data breaches being their biggest concerns. Users are more concerned about the content of their documents and their personal data such as location than about their interaction data. Unlike users, developers clearly prefer technical measures like data anonymization and think that privacy laws and policies are less effective. We also observed interesting differences between people from different geographies. For example, people from Europe are more concerned about data breaches than people from North America. People from Asia/Pacific and Europe believe that content and metadata are more critical for privacy than people from North America. Our results contribute to developing a user-driven privacy framework that is based on empirical evidence in addition to the legal, technical, and commercial perspectives. Second, a related challenge to above, is to make privacy more understandable in complex systems that may have a variety of user interface options, which may change often. As social network platforms have evolved, the ability for users to control how and with whom information is being shared introduces challenges concerning the configuration and comprehension of privacy settings. To address these concerns, our crowd sourced approach simplifies the understanding of privacy settings by using data collected from 512 users over a 17 month period to generate visualizations that allow users to compare their personal settings to an arbitrary subset of individuals of their choosing. To validate our approach we conducted an online survey with closed and open questions and collected 59 valid responses after which we conducted follow-up interviews with 10 respondents. Our results showed that 70% of respondents found visualizations using crowd sourced data useful for understanding privacy settings, and 80% preferred a crowd sourced tool for configuring their privacy settings over current privacy controls. Third, as software evolves over time, this might introduce bugs that breach users' privacy. Further, there might be system-wide policy changes that could change users' settings to be more or less private than before. We present a novel technique that can be used by end-users for detecting changes in privacy, i.e., regression testing for privacy. Using a social approach for detecting privacy bugs, we present two prototype tools. Our evaluation shows the feasibility and utility of our approach for detecting privacy bugs. We highlight two interesting case studies on the bugs that were discovered using our tools. To the best of our knowledge, this is the first technique that leverages regression testing for detecting privacy bugs from an end-user perspective. Fourth, approaches to addressing these privacy concerns typically require substantial extra computational resources, which might be beneficial where privacy is concerned, but may have significant negative impact with respect to Green Computing and sustainability, another major societal concern. Spending more computation time results in spending more energy and other resources that make the software system less sustainable. Ideally, what we would like are techniques for designing software systems that address these privacy concerns but which are also sustainable - systems where privacy could be achieved "for free", i.e., without having to spend extra computational effort. We describe how privacy can indeed be achieved for free an accidental and beneficial side effect of doing some existing computation - in web applications and online systems that have access to user data. We show the feasibility, sustainability, and utility of our approach and what types of privacy threats it can mitigate. Finally, we generalize the problem of privacy and its tradeoffs. As Social Computing has increasingly captivated the general public, it has become a popular research area for computer scientists. Social Computing research focuses on online social behavior and using artifacts derived from it for providing recommendations and other useful community knowledge. Unfortunately, some of that behavior and knowledge incur societal costs, particularly with regards to Privacy, which is viewed quite differently by different populations as well as regulated differently in different locales. But clever technical solutions to those challenges may impose additional societal costs, e.g., by consuming substantial resources at odds with Green Computing, another major area of societal concern. We propose a new crosscutting research area, Societal Computing, that focuses on the technical tradeoffs among computational models and application domains that raise significant societal issues. We highlight some of the relevant research topics and open problems that we foresee in Societal Computing. We feel that these topics, and Societal Computing in general, need to gain prominence as they will provide useful avenues of research leading to increasing benefits for society as a whole

An investigation into the role of crowdsourcing in generating information for flood risk management

Flooding is a major global hazard whose management relies on an accurate understanding of its risks. Crowdsourcing represents a major opportunity for supporting flood risk management as members of the public are highly capable of producing useful flood information. This thesis explores a wide range of issues related to flood crowdsourcing using an interdisciplinary approach. Through an examination of 31 different projects a flood crowdsourcing typology was developed. This identified five key types of flood crowdsourcing: i) Incident Reporting, ii) Media Engagement, iii) Collaborative Mapping, iv) Online Volunteering and v) Passive VGI. These represent a wide range of initiatives with radically different aims, objectives, datasets and relationships with volunteers. Online Volunteering was explored in greater detail using Tomnod as a case study. This is a micro-tasking platform in which volunteers analyse satellite imagery to support disaster response. Volunteer motivations for participating on Tomnod were found to be largely altruistic. Demographics of participants were significant, with retirement, disability or long-term health problems identified as major drivers for participation. Many participants emphasised that effective communication between volunteers and the site owner is strongly linked to their appreciation of the platform. In addition, the feedback on the quality and impact of their contributions was found to be crucial in maintaining interest. Through an examination of their contributions, volunteers were found to be able to ascertain with a higher degree of accuracy, many features in satellite imagery which supervised image classification struggled to identify. This was more pronounced in poorer quality imagery where image classification had a very low accuracy. However, supervised classification was found to be far more systematic and succeeded in identifying impacts in many regions which were missed by volunteers. The efficacy of using crowdsourcing for flood risk management was explored further through the iterative development of a Collaborative Mapping web-platform called Floodcrowd. Through interviews and focus groups, stakeholders from the public and private sector expressed an interest in crowdsourcing as a tool for supporting flood risk management. Types of data which stakeholders are particularly interested in with regards to crowdsourcing differ between organisations. Yet, they typically include flood depths, photos, timeframes of events and historical background information. Through engagement activities, many citizens were found to be able and motivated to share such observations. Yet, motivations were strongly affected by the level of attention their contributions receive from authorities. This presents many opportunities as well as challenges for ensuring that the future of flood crowdsourcing improves flood risk management and does not damage stakeholder relationships with participants

Customer co-creation in innovations : a protocol for innovating with end users

The transition into the information revolution or age has made it possible for consumers and users to interfere in the conceptualization, design, production and sales processes of firms. Consumers and users can express their needs in more direct way to producing firms, they have access to the way products and services are made, and last but not least, have access to information on competing products and services that even producers don’t know about. Consumers have become more knowledgeable and are therefore capable of designing and producing their own products and services. The success of innovations or new product and service development is highly dependent on whether they take in consideration the needs and demands of potential users and consumers. In other words, a market orientation is essential for the success of an innovation. Firms would therefore welcome the idea of consumers and users expressing their demands and probably appreciate consumers who want to participate in the new product or service development, because they would have immediate feedback on the potential success of the innovation. Question is, however, how to achieve this and how to successfully co-create with customers in the innovation process. This design research addresses customer co-creation in innovations for product and service industries. It addresses how firms should successfully activate customers or users and what process they should follow, i.e. the kind of customers or users to involve, the tools and techniques to apply, and procedures to be followed. It develops the appropriate interventions for this in a Customer Co-Creation in Innovations (3CI) - Protocol. The nature of this research is prescriptive, based on the Design Science principles, aiming to design a solution for firms that are interested in the co-creation role that customers can play in their organizations regarding innovations. The research results in a protocol which organizations that want to co-create with customers in their innovation process, can use or apply, to effectively co-create with these customers. Effectively in this sense means that the customer input will be of added value to the innovation, resulting in the outcome that the organization succeeds in bringing the innovation into the market or in use. This doesn’t necessarily mean that the innovation will be a commercial success, because this success depends on more and other factors than just customer co-creation. But, in this context customer co-creation gives the organization the necessary confirmation that the innovation fits needs and demands in the market, and thus leads to a higher adaptation than one should expect when not co-creating with customers. There is an abundance of literature that argue the benefits of involving customers in the innovation process, while other address the issue of which customers to involve, so, the research focuses itself on best practices, experiments, and such to develop this protocol. This has been accomplished by studying the diverse modes or appearances of customer involvement in product or service development, such as market research, empathic design, user-centered design, co-design, mass customization, user innovation, open source software development, user generated content, crowdsourcing, and customer co-creation. Although there is a lot of overlap and similarities among these modes of involvement, there are also many differences, indicating that customer co-creation in innovations is contingent on many factors and aspects. To reduce the confusion, a construct of customer co-creation in innovations has been developed, which has been defined as the process where product manufacturers and/or service providers actively engage with their end users or customers in (parts or phases of) innovation projects to jointly perform innovation activities and co-create value, with the aim of increasing effectiveness and efficiency of the innovation process. Effectiveness refers to (1) the result of meeting users’ and customers’ needs and demands in a better way; and (2) increasing customer loyalty. Efficiency refers to (1) the reduction of research and development costs; and (2) the reduction of development time. And to analyze differences and similarities so that the appropriate design propositions can stated, a 3CI framework was developed, covering the following topics: (1) how to determine whether a firm can co-create with its customers in innovations, which are the so called context conditions; (2) how to identify, select, and motivate potential customers to participate in customer-open innovations; (3) how to engage and involve these customers in the innovation process in an effective and efficient way, the process, procedures and methods one can follow, the tools one can use to accomplish this. With this framework the practice of customer co-creation was analyzed by means of five case studies, in which two of them, the author was an actor in designing and executing the process of co-creation. The cases, selected for their diversity, reveal the opportunities and challenges of customer-inclusive innovation. Customer involvement was at least a partial success in all cases. At the same time, it was never a ‘silver bullet’ to permanently transform the way the company worked. 3CI seems to be capable to support both incidental and repeating innovation initiatives of a firm. Another observation is that, whether a B2B or B2C type of firm, a manufacturer or service provider, small or large firms, all seem to be capable of and suited for 3CI. Common in all cases, however, is that the organization’s offerings and markets should be heterogeneous, thereby containing opportunities to either develop line extensions or really novel (radical) offerings. The technology base of the organization, however, does not seem to be a prerequisite. Another theme cutting across the cases is the nature of an ‘innovation community’, where users test, experiment with and modify or enhance existing prototypes and products, paving the roadway to innovation. As for the relationship between innovation type and type of customer, the cases undoubtedly demonstrate that ‘ordinary’ users can provide useful input to develop radical or novel innovations. The cases also demonstrate that nearly all innovation activities can be conducted by co-creating with customers, including needs assessment, ideation, the screening of ideas or concepts, concept testing, design and development, the commercialization of the innovation and even the re-innovation or use stage. So, although one could get the idea of 3CI being of particular interest in the front end of an innovation stage, we see that in all later stages 3CI can be beneficial as well. Typical across all cases is also the contingency of the channel of involvement (online versus face-to-face) with the amount of customers involved, which we have typed as the degree of openness. The more people are involved, the more open (no secrecy) the co-creation is and the more the involvement is obtained through the online channel, either with communities or on an open call. Conversely, the less participants, the more secrecy is needed and the sooner the physical presence or offline participation seems to be imminent in participation. Finally, regarding the use of tools it can be concluded that sophisticated methods for customer co-creation are a complement rather than the sole source of user information. More important seems to be the occurrence of a dialogue between firm and participating customers, implying that the quality of the interaction depends on mutual trust, appreciation, commitment and equality. Tools that support this dialogue, such as the ZMET¿, OBR, or similar techniques, seem to be important to assure effective and efficient contribution from customers. Subsequently, the design process was conducted, first by defining 16 design requirements for the protocol – subdivided in functional and use requirements, and design restrictions and boundary conditions – followed by the development of the design propositions. A grand total of 28 design propositions have been identified, regarding the context of 3CI (10 propositions), the customer requirements (10 propositions) and process (8 propositions). The context propositions reflected the context decisions to be made, i.e. the appropriate strategy, the suitability of the firm’s market, the initiator for the co-creation (firm or customer), and the type of innovation (incremental vs. radical, open vs. closed mode). Wherever appropriate we have also reviewed the appropriate methods, tools and techniques for the best implementation of the interventions. These are the first decisions the firm has to make when undertaking the 3CI Journey. Only when these decisions are made a next step, i.e. determining which customer requirements are appropriate, can be made. It has been argued that any organization can co-create with its customers in innovations, provided that they adopt and maintain a market oriented strategy, along with the necessary tools, space, freedom and transparency for customers to participate. Customer co-creation leads to at least effective incremental innovations, but when the organization applies Customer Knowledge Methods it increases the chance for an effective radical innovation. If secrecy is required, a closed mode approach of co-creation can be followed, entailing that a minimum amount and diversity of external participants are involved, provided that there is a clear scope of innovation objectives and the market it is intended for. Finally, organizations can either rely on customer-initiated ideas or initiate an innovation itself. In the first approach the organization is recommended to create and maintain a customer community, which can be observed and interacted with to elicit the customers’ ideas. The 10 customer design propositions deal with the type of customers to co-create with in innovations and the available interventions to engage with and maintain involvement from the selected participants. We have argued that all (potential) customers are eligible to participate, as long as they have a certain use experience with the product, service or category of innovation. Only in the case of a radical innovation, the company can choose to add some lead users in order to increase the chance of generating really novel ideas or concepts. To find these lead users, the company can make an appeal on the customer community, since lead users are usually known in communities. In order to benefit in the best way from the participating ordinary and lead users, the company should select them on the basis of their willingness to participate. On top of that, participants should be trained or educated in the tools, techniques and methods that are applied during their involvement. To prevent a decrease of intrinsic motivation with participants, companies have to be very prudent with the promise and administering of financial rewards. Rewards can be given, but preferably unexpected and contingent on task complexity and performance demonstrated by the participant. Depending on the channel of involvement, a minimum of 15 to an undetermined maximum of participants is possible, provided that the company reserves sufficient resources to handle the amount of participants. To our initial 20 design propositions we have added an additional 8 design propositions regarding the process of co-creation. We have seen that all innovation stages are suited to co-create with customers. For the appropriate activities in which these customers can contribute we have developed a table depicting activities and contributions per innovation stage. Co-creation can take place in one, more or all stages; to receive the most benefit, customers should be involved as early as possible in the innovation process. To prevent loss of attention, de-motivation and premature abandonment, we have proposed to change participants with ongoing activities; relying on the same customers in all stages can result in ‘myopic’ results. Both online and offline co-creation are possible, depending on openness, amount of participants and available resources. If participation is online, we recommend applying crowdsourcing methods and techniques, preferably within the customer community. To support an effective communication, we finally proposed to use metaphor or analogy based ‘language’ and to treat the participants as if they were team members. Through scrutinizing and analyzing the 28 design propositions in relation to one another and some pre-defined design requirements, we have identified four main routes – metaphorically named the dreamcatcher, contest, touchstone and employment route – that a company can follow when aiming to co-create with customers in the innovation process. The dreamcatcher route appeals on a user community – existing or yet to be created, preferably online, but with a physical possibility – where existing products, services or platforms are used, reviewed and discussed by customers. The company observes and participates in this discussion through a dialogue, possibly also moderating the community. Opportunities are identified by the company and translated into innovation projects by the company, in which customers again can participate. In the contest route the company can pose users with a specific question or request, a challenge, for which they are expected to think of a solution, of which typically one, or a limited amount of solutions are eligible. The intention is to specifically involve the customer in the front end of the innovation, because the company does not know or is not aware yet of customer needs and wants, or the intended product or service requirements. Customer input is then required in the first stage (Conception), but is not necessary excluded in later stages, where customers can test prototypes, assist in the commercialization and the re-innovation. In the touchstone route the company can decide to co-create with customers in any, arbitrary stage or activity of the innovation process, a sort of a one off. In such a case, the company usually has already identified the opportunities, the innovation project and its goals. Customer co-creation is opportune to verify assumptions, fill in details, and provide additional, not thought of product or service requirements. Of course it is possible to co-create with the customer in more than one activity, but this approach is seen as discrete co-creation activities to support just that particular and specific stage, in which the co-creation is required, usually in the implementation stage and thereafter. Finally, in the employment route the company can integrate one or more (limited amount of) customers in the innovation project, e.g. by temporarily employing them. This approach is of particular interest in idea generation, design and development activities, i.e. the Conception and Implementation stage, but later stages aren’t excluded. We can see this approach applied in customized projects, where it is the intention to create something for a specific set of customers or segment. This can be on request by the customer or because the company has discovered an unfulfilled or unattended set of needs with these customers, e.g. through dreamcatching. To decide which route(s) is or are appropriate we have discussed some premises and considerations – objectives for co-creation, stages and contributions for co-creation, type and openness of innovation – that a company has to assess systematically. Each route was elaborated on, providing preparation steps and do’s and don’ts for an effective and efficient contribution from customers. The four routes are also interrelated and do not exclude one another, but nevertheless provide a company with the optimal approach for 3CI. The 3CI-protocol is therefore a robust, handy guideline for companies to co-create with their customers in innovations. Because of the systematic and rigorous analysis and synthesis of theory and practice, the protocol can be applied in most situations. To test and prove the correctness of this last assertion we validated the design by having it reviewed by some potential users, some experts and some scholars, and to base the conclusion of its validity on the opinions of these reviewers. A total of 25 potential reviewers, both national and international, consisting of product/service developers, co-creation intermediaries, consultants and scholars were approached independently from and ‘blind’ to each other to conduct this review. Ten of them consented in participation; three abandoned the review process prematurely for personal reasons, leaving a total of 7 reviewers that have submitted comments. It was agreed on to enhance the review with a Delphi if responses were very divergent. All reviewers found the protocol useful and helpful for guiding the process of customer co-creation. Comments or critique referred mainly to the readability of the protocol, with the remark that users might lose attention because of the academic reasoning. Some of them provided useful additions to the protocol in order to enhance the readability. Also, suggestions were made to promote the protocol to practice, for instance by publishing it via a community and a management book. The comments did not contain divergent viewpoints on the subject, the design and its content, so the Delphi was left out. Based on these comments and suggestions by the reviewers, we have redesigned the protocol into the 3CI Protocol version 1.0, which can be published as a separate document, detached from this thesis, which all potential users can get hold of and apply without having to acquire a copy of the thesis. We propose to use this protocol to further validate it in practice and giving us feedback on its effectiveness. Our main contribution to research in management and organization has been to develop a comprehensive how-to guideline for practitioners, based on and grounded in a diversity of theory. Therefore, we believe that we have contributed with a design that is applicable in all kind of business and organizational contexts where the interaction with end users is aimed at developing new offerings. However, modesty is also in place, when we observe that this has to be proven, yet. Further research can be aimed at obtaining this proof, while other research could focus on the underlying assumptions, which we named generative mechanisms, of the design

CrowdSourcing como estrategia de Defensa

Crowdsourcing has taken root in the business world, but applications in the fields of science and pop culture pre-date this boom. By now it’s clear that the catalyst for this Cambrian explosion of collaborative projects has been the Internet. Crowdsourcing efforts come in many different forms, but the same characteristics appear over and over in the most successful examples. Crowdsourcing has had a profound impact wherever an organization’s needs intersect the crowd’s desires to accomplish something. Entire sections of industries have been decimated by crowdsourcing. Government organizations are not completely immune to the effect. A public good such as defense is safe from obsolescence, but there is strong evidence to indicate that there’s nothing preventing defense from reaping rewards from crowdsourcing. Cost savings, innovation, manning issues, and even success on the battlefield might all benefit from applying the lessons learned from crowdsourcing. Other industries have carved a roadmap for defense to follow. It needs to start experimenting with these methods now before rivals learn to perfect crowdsourcing’s varied and proven techniques