Structuring, Aggregating, and Evaluating Crowdsourced Design Critique

Feedback is an important component of the design process, but gaining access to high-quality critique outside a class-room or firm is challenging. We present CrowdCrit, a web-based system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer’s experience and bene-fits of the critiques. In the first study, we compared crowd and expert critiques and found evidence that aggregated crowd critique approaches expert critique. In a second study, we found that designers who got crowd feedback perceived that it improved their design process. The third study showed that designers were enthusiastic about crowd critiques and used them to change their designs. We conclude with implications for the design of crowd feedback services. Author Keywords Design; critique; feedback; social computing; crowdsourcin

Supporting Answerers with Feedback in Social Q&A

Prior research has examined the use of Social Question and Answer (Q&A) websites for answer and help seeking. However, the potential for these websites to support domain learning has not yet been realized. Helping users write effective answers can be beneficial for subject area learning for both answerers and the recipients of answers. In this study, we examine the utility of crowdsourced, criteria-based feedback for answerers on a student-centered Q&A website, Brainly.com. In an experiment with 55 users, we compared perceptions of the current rating system against two feedback designs with explicit criteria (Appropriate, Understandable, and Generalizable). Contrary to our hypotheses, answerers disagreed with and rejected the criteria-based feedback. Although the criteria aligned with answerers' goals, and crowdsourced ratings were found to be objectively accurate, the norms and expectations for answers on Brainly conflicted with our design. We conclude with implications for the design of feedback in social Q&A.Comment: Published in Proceedings of the Fifth Annual ACM Conference on Learning at Scale, Article No. 10, London, United Kingdom. June 26 - 28, 201

Considering Human Aspects on Strategies for Designing and Managing Distributed Human Computation

A human computation system can be viewed as a distributed system in which the processors are humans, called workers. Such systems harness the cognitive power of a group of workers connected to the Internet to execute relatively simple tasks, whose solutions, once grouped, solve a problem that systems equipped with only machines could not solve satisfactorily. Examples of such systems are Amazon Mechanical Turk and the Zooniverse platform. A human computation application comprises a group of tasks, each of them can be performed by one worker. Tasks might have dependencies among each other. In this study, we propose a theoretical framework to analyze such type of application from a distributed systems point of view. Our framework is established on three dimensions that represent different perspectives in which human computation applications can be approached: quality-of-service requirements, design and management strategies, and human aspects. By using this framework, we review human computation in the perspective of programmers seeking to improve the design of human computation applications and managers seeking to increase the effectiveness of human computation infrastructures in running such applications. In doing so, besides integrating and organizing what has been done in this direction, we also put into perspective the fact that the human aspects of the workers in such systems introduce new challenges in terms of, for example, task assignment, dependency management, and fault prevention and tolerance. We discuss how they are related to distributed systems and other areas of knowledge.Comment: 3 figures, 1 tabl

Feedback aggregation in crowd feedback systems

Abstract. The purpose of this literature review is to research the way different crowd feedback systems aggregate and visualize their data for the user. First the concept of crowdsourcing for design purposes is introduced as well as four different crowd feedback systems, which are Voyant, CrowdCrit, Decipher and Paragon. Crowdsourcing means giving a task for a crowd of people to perform, usually online. Crowdsourcing is often used when there is a need for a large amount of responses because of its low cost compared to other methods. Crowd feedback systems use crowdsourcing to achieve their goal that is collecting feedback from a crowd. For a crowd feedback system to provide value into the design process, they should not only collect feedback but also convey the collected data to the designer in an informative but also easily understandable manner. This requires that the system provides support for non-experts for them to give feedback in a professional manner. The results of this thesis give an insight into how crowd feedback systems differ from each other. The results showed that different crowd feedback systems collect and present their feedback in very different ways. Voyant and CrowdCrit both visualize feedback using visual markers and stacked bar charts, but Voyant also uses word clouds for this purpose. Decipher shows whether the feedback is considered negative or positive and what the feedback provider had to say about the design. Paragon presents collected feedback with the help of examples that the feedback provider has chosen to help describe their feelings about the design. Voyant and CrowdCrit were eventually considered to be the most visually pleasing of these four crowd feedback systems. The way Voyant aggregated its feedback was seen more versatile but CrowdCrit collected feedback in a way that provided more useful feedback from non-experts.Palautteen koostaminen joukkoistamisen palautejärjestelmissä. Tiivistelmä. Tämän kirjallisuuskatsauksen tarkoitus on tutkia, millä tavoin erilaiset joukkoistavat palautejärjestelmät koostavat ja visualisoivat keräämänsä palautteen käyttäjälle. Ensin esitellään joukkoistamisen rooli suunnittelussa ja sen myötä myös neljä palautejärjestelmää, jotka ovat Voyant, CrowdCrit, Decipher ja Paragon. Joukkoistamisella tarkoitetaan tehtävien antamista joukolle ihmisiä, yleensä verkossa. Joukkoistamista käytetään usein, kun tarvitaan iso määrä palautetta, johtuen sen käytön edullisuudesta verrattuna muihin metodeihin. Joukkoistamisen palautejärjestelmät hyödyntävät joukkoistamista saavuttaakseen tavoitteensa, joka on palautetteen kerääminen joukolta ihmisiä. Jotta palautejärjestelmä voisi tuoda lisäarvoa suunnitteluprosessiin, täytyy sen palautteen keräämisen lisäksi myös esittää saatu data käyttäjälleen informatiivisessa, mutta myös helposti ymmärrettävässä muodossa. Tämä vaatii, että palautejärjestelmä tukee jollain tavalla ei-asiantuntijoita, jotta he voisivat antaa palautetta asiantuntevalla tavalla. Tämän kandidaatintyön tulokset antavat käsityksen siitä, miten joukkoistamisen palautejärjestelmät eroavat toisistaan. Tulokset osoittivat, että eri joukkoistamisen palautejärjestelmät keräävät ja esittävät keräämänsä palautteen hyvin eri tavoilla. Voyant ja CrowdCrit visualisoivat palautteen visuaalisten markkereiden ja pinottujen pylväsdiagrammien avulla, mutta Voyant käyttää myös sanapilviä tähän tarkoitukseen. Decipher ilmoittaa, onko palaute nähty positiivisena, negatiivisena vai neutraalina ja mitä mieltä palautteen antaja on ollut designista. Paragon esittää keräämänsä palautteen esimerkkikuvien avulla, jotka palautteenantaja on valinnut kuvaamaan tuntemuksiaan. Lopulta Voyant ja CrowdCrit nähtiin visuaalisesti miellittävimpinä näistä neljästä palautejärjestelmästä. Voyantin tapa koostaa palaute koettiin monipuolisempana, mutta CrowdCrit keräsi palautetta tavalla, joka tuotti hyödyllisempää palautetta ei-asiantuntijoilta

Text and visual annotation tools for scalable design feedback

Designers who wish to solicit feedback online have access to a variety of tools. Yet when selecting one tool over another for feedback collection, there is little empirical evidence to guide a designer's decision. We conducted an online study (N=360) where participants provided design feedback using two representative classes of feedback collection interfaces: spatial and nonspatial. For each interface, we also manipulated access to history feedback. Our results showed that the presence of history introduced a fixation effect where providers entered feedback that was more similar to the feedback they reviewed. Providers in the non-spatial condition entered feedback that was 24% longer than the spatial condition; whereas providers in the spatial condition left more investigation feedback. There was no difference in specificity between conditions. Results suggest that the more important choice designers must make is not the class of tool they use but whether history feedback is included

Spam elimination and bias correction : ensuring label quality in crowdsourced tasks.

Crowdsourcing is proposed as a powerful mechanism for accomplishing large scale tasks via anonymous workers online. It has been demonstrated as an effective and important approach for collecting labeled data in application domains which require human intelligence, such as image labeling, video annotation, natural language processing, etc. Despite the promises, one big challenge still exists in crowdsourcing systems: the difficulty of controlling the quality of crowds. The workers usually have diverse education levels, personal preferences, and motivations, leading to unknown work performance while completing a crowdsourced task. Among them, some are reliable, and some might provide noisy feedback. It is intrinsic to apply worker filtering approach to crowdsourcing applications, which recognizes and tackles noisy workers, in order to obtain high-quality labels. The presented work in this dissertation provides discussions in this area of research, and proposes efficient probabilistic based worker filtering models to distinguish varied types of poor quality workers. Most of the existing work in literature in the field of worker filtering either only concentrates on binary labeling tasks, or fails to separate the low quality workers whose label errors can be corrected from the other spam workers (with label errors which cannot be corrected). As such, we first propose a Spam Removing and De-biasing Framework (SRDF), to deal with the worker filtering procedure in labeling tasks with numerical label scales. The developed framework can detect spam workers and biased workers separately. The biased workers are defined as those who show tendencies of providing higher (or lower) labels than truths, and their errors are able to be corrected. To tackle the biasing problem, an iterative bias detection approach is introduced to recognize the biased workers. The spam filtering algorithm proposes to eliminate three types of spam workers, including random spammers who provide random labels, uniform spammers who give same labels for most of the items, and sloppy workers who offer low accuracy labels. Integrating the spam filtering and bias detection approaches into aggregating algorithms, which infer truths from labels obtained from crowds, can lead to high quality consensus results. The common characteristic of random spammers and uniform spammers is that they provide useless feedback without making efforts for a labeling task. Thus, it is not necessary to distinguish them separately. In addition, the removal of sloppy workers has great impact on the detection of biased workers, with the SRDF framework. To combat these problems, a different way of worker classification is presented in this dissertation. In particular, the biased workers are classified as a subcategory of sloppy workers. Finally, an ITerative Self Correcting - Truth Discovery (ITSC-TD) framework is then proposed, which can reliably recognize biased workers in ordinal labeling tasks, based on a probabilistic based bias detection model. ITSC-TD estimates true labels through applying an optimization based truth discovery method, which minimizes overall label errors by assigning different weights to workers. The typical tasks posted on popular crowdsourcing platforms, such as MTurk, are simple tasks, which are low in complexity, independent, and require little time to complete. Complex tasks, however, in many cases require the crowd workers to possess specialized skills in task domains. As a result, this type of task is more inclined to have the problem of poor quality of feedback from crowds, compared to simple tasks. As such, we propose a multiple views approach, for the purpose of obtaining high quality consensus labels in complex labeling tasks. In this approach, each view is defined as a labeling critique or rubric, which aims to guide the workers to become aware of the desirable work characteristics or goals. Combining the view labels results in the overall estimated labels for each item. The multiple views approach is developed under the hypothesis that workers\u27 performance might differ from one view to another. Varied weights are then assigned to different views for each worker. Additionally, the ITSC-TD framework is integrated into the multiple views model to achieve high quality estimated truths for each view. Next, we propose a Semi-supervised Worker Filtering (SWF) model to eliminate spam workers, who assign random labels for each item. The SWF approach conducts worker filtering with a limited set of gold truths available as priori. Each worker is associated with a spammer score, which is estimated via the developed semi-supervised model, and low quality workers are efficiently detected by comparing the spammer score with a predefined threshold value. The efficiency of all the developed frameworks and models are demonstrated on simulated and real-world data sets. By comparing the proposed frameworks to a set of state-of-art methodologies, such as expectation maximization based aggregating algorithm, GLAD and optimization based truth discovery approach, in the domain of crowdsourcing, up to 28.0% improvement can be obtained for the accuracy of true label estimation

The analysis and presentation of patents to support engineering design

This paper explores the role of patents in engineering design, and how the extraction and presentation of patent data could be improved for designers. We propose the use of crowdsourcing as a means to post tasks online for a crowd of people to participate and complete. The is-sues of assessment, searching, clustering and knowledge transfer are evaluated with respect to the literature. Opportunities for potential crowd intervention are then discussed, before the presentation of two initial studies. These related to the categorization and interpretation of patents respectively using an online platform. The initial results establish basic crowd capabilities in understanding patent text and interpreting patent drawings. This has shown that reasonable results can be achieved if tasks of appropriate duration and complexity are set, and if test questions are incorporated to ensure a basic level of understanding exists in the workers

Aligning Crowdworker Perspectives and Feedback Outcomes in Crowd-Feedback System Design

Leveraging crowdsourcing in software development has received growing attention in research and practice. Crowd feedback offers a scalable and flexible way to evaluate software design solutions and the potential of crowd-feedback systems has been demonstrated in different contexts by existing research studies. However, previous research lacks a deep understanding of the effects of individual design features of crowd-feedback systems on feedback quality and quantity. Additionally, existing studies primarily focused on understanding the requirements of feedback requesters but have not fully explored the qualitative perspectives of crowd-based feedback providers. In this paper, we address these research gaps with two research studies. In study 1, we conducted a feature analysis (N=10) and concluded that from a user perspective, a crowd-feedback system should have five core features (scenario, speech-to-text, markers, categories, and star rating). In the second study, we analyzed the effects of the design features on crowdworkers’ perceptions and feedback outcomes (N=210). We learned that offering feedback providers scenarios as the context of use is perceived as most important. Regarding the resulting feedback quality, we discovered that more features are not always better as overwhelming feedback providers might decrease feedback quality. Offering feedback providers categories as inspiration can increase the feedback quantity. With our work, we contribute to research on crowd-feedback systems by aligning crowdworker perspectives and feedback outcomes and thereby making the software evaluation not only more scalable but also more human-centered