Crowdsourcing creative work

Abstract Creative work is launched on paid crowdsourcing platforms, yet we lack an in-depth understanding of how the two key stakeholders of crowdsourcing platforms (crowd workers and requesters) perceive and experience creative work. Creativity is a human characteristic that is difficult to automate by machines, and supplying requesters with crowdsourced human insights and complex creative work is, therefore, a timely topic for research. According to value-sensitive design, the integration of human insight into complex socio-technical systems will need to consider the perspectives of the two key stakeholders. This article-based doctoral thesis explores the stakeholder perspectives and experiences of crowdsourced creative work on two of the leading crowdsourcing platforms. The thesis has two parts. In the first part, we explore creative work from the perspective of the crowd worker. In the second part, we explore and study the requester’s perspective in different contexts and several case studies. The research is exploratory and we contribute empirical insights using survey-based and artefact-based approaches common in the field of Human-Computer Interaction (HCI). In the former approach, we explore the key issues that may limit creative work on paid crowdsourcing platforms. In the latter approach, we create computational artefacts to elicit authentic experiences from both crowd workers and requesters of crowdsourced creative work. The thesis contributes a classification of crowd workers into five archetypal profiles, based on the crowd workers’ demographics, disposition, and preferences for creative work. We propose a three-part classification of creative work on crowdsourcing platforms: creative tasks, creativity tests, and creativity judgements (also referred to as creative feedback). The thesis further investigates the emerging research topic of how requesters can be supported in interpreting and evaluating complex creative work. Last, we discuss the design implications for research and practice and contribute a vision of creative work on future crowdsourcing platforms with the aim of empowering crowd workers and fostering an ecosystem around tailored platforms for creative microwork.Tiivistelmä Luovassa työssä käytetään maksullisia joukkoistusalustoja, mutta meiltä puuttuu kuitenkin vielä syvällinen käsitys siitä, miten kaksi avainasemassa olevaa joukkoistusalustojen sidosryhmää (joukkotyöntekijät ja toimeksiantajat) ymmärtävät ja kokevat luovan työn. Luovuus on ihmisen ominaisuus, jota on vaikea automatisoida, ja joukkoistettujen inhimillisten näkemysten ja kompleksisen luovan työn välittäminen toimeksiantajille ovat siitä syystä ajankohtainen tutkimuskohde. Arvosensitiivisen suunnittelun mukaan inhimillisen ymmärryksen integroinnissa kompleksisiin sosioteknisiin järjestelmiin on otettava huomioon kahden avainasemassa olevan sidosryhmän näkökulmat. Tässä artikkeliväitöskirjassa tutkitaan sidosryhmien näkökulmia ja kokemuksia joukkoistetusta luovasta työstä kahdella joukkoistusalustalla. Väitöskirja koostuu kahdesta osasta. Ensimmäisessä osassa tarkastellaan luovaa työtä joukkotyöntekijän näkökulmasta. Toisessa osassa tarkastellaan toimeksiantajan näkökulmaa useissa tapaustutkimuksissa. Tällä tutkimuksella halutaan syventää empiiristä ymmärrystä hyödyntämällä kyselytutkimuksiin perustuvia lähestymistapoja ja ihmisen ja tietokoneen välisessä vuorovaikutuksessa (Human-Computer Interaction, HCI) yleisiä lähestymistapoja. Ensimmäisessä lähestymistavassa tarkastellaan keskeisiä seikkoja, jotka voivat rajoittaa maksullisilla joukkoistusalustoilla tehtävää luovaa työtä. Jälkimmäisessä lähestymistavassa luodaan laskennallisia artefakteja, joilla halutaan tuoda esiin joukkotyöntekijöiden ja joukkoistetun luovan työn toimeksiantajien aitoja kokemuksia. Väitöskirjassa joukkotyöntekijät luokitellaan viiteen arkkityyppiprofiiliin, jotka perustuvat joukkotyöntekijöiden demografisiin tietoihin, ajattelumalleihin ja luovaa työtä koskeviin mieltymyksiin. Väitöskirjassa ehdotetaan kolmiosaista luokittelua joukkoistusalustoilla tehtävälle luovalle työlle: luovuutta edellyttävät tehtävät, luovuustestit ja luovuuden arvioinnit (joita kutsutaan myös luovuuspalautteeksi). Lisäksi väitöskirjassa tutkitaan uutta tutkimusaihetta eli sitä, miten toimeksiantajia voidaan tukea monimutkaisen luovan työn tulkinnassa ja arvioinnissa. Lopuksi tutkimuksessa tarkastellaan mallin vaikutusta tutkimukseen ja käytäntöön, ja siinä esitetään tulevaisuuden joukkoistusalustoilla tehtävästä luovasta työstä visio, jonka päämääränä on parantaa joukkotyöntekijöiden valmiuksia ja tukea luovan mikrotyön räätälöityjen ympäristöjen ympärille rakentuvaa ekosysteemiä

Towards eliciting feedback for artworks on public displays

Abstract The internet and digital technologies have had a strong influence on how art is created, distributed and perceived. Museums and art galleries, however, are still predominantly places of passive consumption of art. In our project, we explore new forms of communication and feedback between artists and their audience, mediated by public displays. In this work-in-progress paper, we present a situated feedback system for giving feedback on artworks in a public setting. We present a preliminary evaluation of the system with artists and potential audience on their reactions to the system and eight different types of feedback

Experizone:integrating situated scientific experimentation with teaching of the scientific method

Abstract Citizen Science projects ask their participants to contribute work to pre-defined topics, thereby typically rendering the participants as mere consumers of often narrowly defined tasks. In this work-in-progress paper, we present our work on an interactive experimentation platform that allows anybody — researchers as well as members of the crowd — to run experiments and test scientific hypotheses with a local crowd of volunteers. The platform also enforces a lightweight review process for teaching its users how to formulate valid scientific hypotheses and experimental designs

Design recommendations for augmenting creative tasks with computational priming

Abstract Supporting creativity is a grand challenge in HCI. A critical component of creativity is the ability for divergent thinking, and divergent thinking can be fostered through looking at the problem through the lens of a different person, by assuming a role. Prior work found that assuming a role and affective stimulation with images may lead individuals to be more creative. In this work, we investigate the use of roles in stimulating the creativity of individuals in two complementary studies. In the first study, we implemented an online instrument for augmenting creativity with roles and images, and recruited crowd workers (n = 60) to complete a divergent thinking task while assuming a role. Interestingly, and in contrast to earlier findings, our analysis could not confirm the computational priming having an effect on the outcome of a small batch of creative tasks. In the second study, we observed the effect of roles on the ideation process of individuals when they reach an impasse in the flow of ideas. Our complementary studies highlight that adopting roles can help when one runs out of ideas, but this is not a silver bullet for improving divergent thinking, especially in online crowd-sourcing environments that are increasingly being used for experiments and data collection in behavioural science. Our work informs the design of future crowd-powered creativity support tools and contributes a timely case study to the body of literature in the growing field of creativity support online

Towards metaphors for cascading AI

Abstract In the future, more and more systems will be powered by AI. This may exacerbate existing blind spots in explainability research, such as focusing on outputs of an individual AI pipeline as opposed to a holistic and integrative view on the system dynamics of data, algorithms, stakeholders, context and their respective interactions. AI systems will increasingly rely on patterns and models of other AI systems. This will likely introduce a major shift in the desiderata of interpretability, explainability and transparency. In this world of Cascading AI (CAI), AI systems will use the output of other AI systems as their inputs. The typical formulations of desiderata for explaining AI decision-making, such as post-hoc interpretability or model-agnostic explanations, may simply not hold in a world of cascading AI. In this paper, we propose two metaphors which may help designers to frame their efforts when designing Cascading AI systems

CrowdUI:supporting web design with the crowd

Abstract Web design is a complex and challenging task. It involves making many design decisions that materialise preconceived notions of user needs that may or may not be true. In this paper, we investigate supporting the co-design of a website with visual feedback elicited from the website’s community of users. Website users can express their needs by re-arranging and modifying the website’s layout and design. To explore and validate this idea, we present CrowdUI, a web-based tool that enables members of the community of a website to visually express their design improvement ideas, frustrations and needs, and to send this feedback to the person in charge of designing or maintaining the website. CrowdUI is validated in a study with 45 users of a popular social media and networking website. Second, our qualitative evaluation with 60 experienced web developers shows that CrowdUI is able to elicit diverse and meaningful feedback. Put together, our results suggest that CrowdUI’s approach constitutes a productive setting for eliciting visual feedback from the user community as a complement to traditional ways of eliciting feedback and participatory design. Finally, based on our experiences, we discuss a design space for crowdsourced web design and provide design recommendations for similar future tools

CampusTracker:assessing mobile workers' momentary willingness to work on paid crowdsourcing tasks

Abstract In mobile crowdsourcing, labour can be opportunistically elicited by sending notifications to workers who complete tasks on-the-go. While much work has focused on optimizing the work quality and quantity in mobile crowdsourcing, surprisingly few studies have explored the type of tasks that might be suitable for different user contexts. This paper presents results from a proof-of-concept user study that aimed to uncover where, when and what type of tasks mobile workers are willing to complete. We find that different contexts do affect the type of work users are willing to complete. Finally, we lay out a complete design, key challenges and opportunities for a longer field trial that we hope to conduct in the near-future

Search support for exploratory writing

Abstract Writing articles involves searching, exploring, evaluating and reflecting upon different perspectives. To this end, online search engines are commonly used tools to support writing. However, online search engines, such as Google, fall short in supporting complex queries that satisfy multiple criteria simultaneously. In this paper, we present our studies with GAS, a crowd-powered tool that allows writers to discover viewpoints, solutions and ideas that best fulfil multiple criteria simultaneously. Our user studies validate GAS as a beneficial companion to online search engines in supporting writing. We found that GAS helps people come up with ideas and write with more confidence, resulting in a higher self-reported article quality and accuracy when compared to only using an online search engine. Through our experiments, we also develop an understanding of the distinct process that people employ when searching for and exploring open-ended, subjective information to support exploratory writing

Hardhats and bungaloos:comparing crowdsourced design feedback with peer design feedback in the classroom

Abstract Feedback is an important aspect of design education, and crowdsourcing has emerged as a convenient way to obtain feedback at scale. In this paper, we investigate how crowdsourced design feedback compares to peer design feedback within a design-oriented HCI class and across two metrics: perceived quality and perceived fairness. We also examine the perceived monetary value of crowdsourced feedback, which provides an interesting contrast to the typical requester-centric view of the value of labor on crowdsourcing platforms. Our results reveal that the students (N = 106) perceived the crowdsourced design feedback as inferior to peer design feedback in multiple ways. However, they also identified various positive aspects of the online crowds that peers cannot provide. We discuss the meaning of the findings and provide suggestions for teachers in HCI and other researchers interested in crowd feedback systems on using crowds as a potential complement to peers

VR ethnography:a pilot study on the use of virtual reality ’go-along’ interviews in Google street view

Abstract Go-along interviewing is an emerging qualitative research method for eliciting contextualised perspectives in which informants and observers conduct mobile interviews while navigating in real or imagined sites. This paper describes results of a pilot study that use virtual reality (VR) go-along interviews to explore university community members’ (N=6) contextualized perceptions of urban habitat fragmentation due to new transportation infrastructure. Participants were immersed into the popular Google Street View and asked to navigate from the University campus to the city center. Along that route, construction sites featured in 360° images acted as prompts for discussing ecological change. Preliminary results indicate that VR go-along interviews are able to evoke emotions and inform a broad range of research questions with regards to both verbal and non-verbal feedback received from the informants