10 research outputs found

    In What Mood Are You Today?

    Get PDF
    The mood of individuals in the workplace has been well-studied due to its influence on task performance, and work engagement. However, the effect of mood has not been studied in detail in the context of microtask crowdsourcing. In this paper, we investigate the influence of one's mood, a fundamental psychosomatic dimension of a worker's behaviour, on their interaction with tasks, task performance and perceived engagement. To this end, we conducted two comprehensive studies; (i) a survey exploring the perception of crowd workers regarding the role of mood in shaping their work, and (ii) an experimental study to measure and analyze the actual impact of workers' moods in information findings microtasks. We found evidence of the impact of mood on a worker's perceived engagement through the feeling of reward or accomplishment, and we argue as to why the same impact is not perceived in the evaluation of task performance. Our findings have broad implications on the design and workflow of crowdsourcing systems

    Novel Methods for Designing Tasks in Crowdsourcing

    Get PDF
    Crowdsourcing is becoming more popular as a means for scalable data processing that requires human intelligence. The involvement of groups of people to accomplish tasks could be an effective success factor for data-driven businesses. Unlike in other technical systems, the quality of the results depends on human factors and how well crowd workers understand the requirements of the task, to produce high-quality results. Looking at previous studies in this area, we found that one of the main factors that affect workers’ performance is the design of the crowdsourcing tasks. Previous studies of crowdsourcing task design covered a limited set of factors. The main contribution of this research is the focus on some of the less-studied technical factors, such as examining the effect of task ordering and class balance and measuring the consistency of the same task design over time and on different crowdsourcing platforms. Furthermore, this study ambitiously extends work towards understanding workers’ point of view in terms of the quality of the task and the payment aspect by performing a qualitative study with crowd workers and shedding light on some of the ethical issues around payments for crowdsourcing tasks. To achieve our goal, we performed several crowdsourcing experiments on specific platforms and measured the factors that influenced the quality of the overall result

    Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions

    Get PDF
    Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar - all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.Comment: 40 pages main paper, 5 pages appendi

    Systems for Managing Work-Related Transitions

    Get PDF
    Peoples' work lives have become ever-populated with transitions across tasks, devices, and environments. Despite their ubiquitous nature, managing transitions across these three domains has remained a significant challenge. Current systems and interfaces for managing transitions have explored approaches that allow users to track work-related information or automatically capture or infer context, but do little to support user autonomy at its fullest. In this dissertation, we present three studies that support the goal of designing and understanding systems for managing work-related transitions. Our inquiry is motivated by the notion that people lack the ability to continue or discontinue their work at the level they wish to do so. We scope our research to information work settings, and we use our three studies to generate novel insights about how empowering peoples' ability to engage with their work can mitigate the challenges of managing work-related transitions. We first introduce and study Mercury, a system that mitigates programmers' challenges in transitioning across devices and environments by enabling their ability to continue work on-the-go. Mercury orchestrates programmers' work practices by providing them with a series of auto-generated microtasks on their mobile device based on the current state of their source code. Tasks in Mercury are designed so that they can be completed quickly without the need for additional context, making them suitable to address during brief moments of downtime. When users complete microtasks on-the-go, Mercury calculates file changes and integrates them into the user's codebase to support task resumption. We then introduce SwitchBot, a conversational system that mitigates the challenges in discontinuing work during the transition between home and the workplace. SwitchBot's design philosophy is centered on assisting information workers in detaching from and reattaching with their work through brief conversations before the start and end of the workday. By design, SwitchBot's detachment and reattachment dialogues inquire about users' task-related goals or user's emotion-related goals. We evaluated SwitchBot with an emphasis on understanding how the system and its two dialogues uniquely affected information workers' ability to detach from and later reattach with their work. Following our study of Mercury and SwitchBot, we present findings from an interview study with crowdworkers aimed at understanding the work-related transitions they experience in their work practice from the perspective of tools. We characterize the tooling observed in crowdworkers' work practices and identified three types of "fragmentation" that are motivated by tooling in the practice. Our study highlights several distinctions between traditional and contemporary information work settings and lays a foundation for future systems that aid next-generation information workers in managing work-related transitions. We conclude by outlining this dissertation's contributions and future research directions

    Designing for quality in real-world mobile crowdsourcing systems

    Get PDF
    PhD ThesisCrowdsourcing has emerged as a popular means to collect and analyse data on a scale for problems that require human intelligence to resolve. Its prompt response and low cost have made it attractive to businesses and academic institutions. In response, various online crowdsourcing platforms, such as Amazon MTurk, Figure Eight and Prolific have successfully emerged to facilitate the entire crowdsourcing process. However, the quality of results has been a major concern in crowdsourcing literature. Previous work has identified various key factors that contribute to issues of quality and need to be addressed in order to produce high quality results. Crowd tasks design, in particular, is a major key factor that impacts the efficiency and effectiveness of crowd workers as well as the entire crowdsourcing process. This research investigates crowdsourcing task designs to collect and analyse two distinct types of data, and examines the value of creating high-quality crowdwork activities on new crowdsource enabled systems for end-users. The main contribution of this research includes 1) a set of guidelines for designing crowdsourcing tasks that support quality collection, analysis and translation of speech and eye tracking data in real-world scenarios; and 2) Crowdsourcing applications that capture real-world data and coordinate the entire crowdsourcing process to analyse and feed quality results back. Furthermore, this research proposes a new quality control method based on workers trust and self-verification. To achieve this, the research follows the case study approach with a focus on two real-world data collection and analysis case studies. The first case study, Speeching, explores real-world speech data collection, analysis, and feedback for people with speech disorder, particularly with Parkinson’s. The second case study, CrowdEyes, examines the development and use of a hybrid system combined of crowdsourcing and low-cost DIY mobile eye trackers for real-world visual data collection, analysis, and feedback. Both case studies have established the capability of crowdsourcing to obtain high quality responses comparable to that of an expert. The Speeching app, and the provision of feedback in particular were well perceived by the participants. This opens up new opportunities in digital health and wellbeing. Besides, the proposed crowd-powered eye tracker is fully functional under real-world settings. The results showed how this approach outperforms all current state-of-the-art algorithms under all conditions, which opens up the technology for wide variety of eye tracking applications in real-world settings

    Supporting users' influence in gamification settings and game live-streams

    Get PDF
    Playing games has long been important to mankind. One reason for this is the associated autonomy, as players can decide on many aspects on their own and can shape the experience. Game-related sub-fields have appeared in Human-Computer Interaction where this autonomy is questionable: in this thesis, we consider gamification and game live-streams and here, we support the users' influence at runtime. We hypothesize that this should affect the perception of autonomy and should lead to positive effects overall. Our contribution is three-fold: first, we investigate crowd-based, self-sustaining systems in which the user's influence directly impacts the outcome of the system's service. We show that users are willing to expend effort in such systems even without additional motivation, but that gamification is still beneficial here. Second, we introduce "bottom-up" gamification, i.e., the idea of self-tailored gamification. Here, users have full control over the gamification used in a system, i.e., they can set it up as they see fit at the system's runtime. Through user studies, we show that this has positive behavioral effects and thus adds to the ongoing efforts to move away from "one-size-fits-all" solutions. Third, we investigate how to make gaming live-streams more interactive, and how viewers perceive this. We also consider shared game control settings in live-streams, in which viewers have full control, and we contribute options to support viewers' self-administration here.Seit jeher nehmen Spiele im Leben der Menschen eine wichtige Rolle ein. Ein Grund hierfür ist die damit einhergehende Autonomie, mit der Spielende Aspekte des Spielerlebnisses gestalten können. Spiele-bezogene Teilbereiche werden innerhalb der Mensch-Maschine-Interaktion untersucht, bei denen dieser Aspekt jedoch diskutabel ist: In dieser Arbeit betrachten wir Gamification und Spiele Live-Streams und geben Anwendern mehr Einfluss. Wir stellen die Hypothese auf, dass sich dies auf die Autonomie auswirkt und zu positiven Effekten führt. Der Beitrag dieser Dissertation ist dreistufig: Wir untersuchen crowdbasierte, selbsterhaltende Systeme, bei denen die Einflussnahme des Einzelnen sich auf das Systemergebnis auswirkt. Wir zeigen, dass Nutzer aus eigenem Antrieb bereit sind, sich hier einzubringen, der Einfluss von Gamification sich aber förderlich auswirkt. Im zweiten Schritt führen wir "bottom-up" Gamification ein. Hier hat der Nutzer die volle Kontrolle über die Gamification und kann sie nach eigenem Ermessen zur Laufzeit einstellen. An Hand von Nutzerstudien belegen wir daraus resultierende positive Verhaltenseffekte, was die anhaltenden Bemühungen bestärkt, individuelle Gamification-Konzepte anzubieten. Im dritten Schritt untersuchen wir, wie typische Spiele Live-Streams für Zuschauer interaktiver gestaltet werden können. Zudem betrachten wir Fälle, in denen Zuschauer die gemeinsame Kontrolle über ein Spiel ausüben und wie dies technologisch unterstützt werden kann

    Scalable and Quality-Aware Training Data Acquisition for Conversational Cognitive Services

    Full text link
    Dialog Systems (or simply bots) have recently become a popular human-computer interface for performing user's tasks, by invoking the appropriate back-end APIs (Application Programming Interfaces) based on the user's request in natural language. Building task-oriented bots, which aim at performing real-world tasks (e.g., booking flights), has become feasible with the continuous advances in Natural Language Processing (NLP), Artificial Intelligence (AI), and the countless number of devices which allow third-party software systems to invoke their back-end APIs. Nonetheless, bot development technologies are still in their preliminary stages, with several unsolved theoretical and technical challenges stemming from the ambiguous nature of human languages. Given the richness of natural language, supervised models require a large number of user utterances paired with their corresponding tasks -- called intents. To build a bot, developers need to manually translate APIs to utterances (called canonical utterances) and paraphrase them to obtain a diverse set of utterances. Crowdsourcing has been widely used to obtain such datasets, by paraphrasing the initial utterances generated by the bot developers for each task. However, there are several unsolved issues. First, generating canonical utterances requires manual efforts, making bot development both expensive and hard to scale. Second, since crowd workers may be anonymous and are asked to provide open-ended text (paraphrases), crowdsourced paraphrases may be noisy and incorrect (not conveying the same intent as the given task). This thesis first surveys the state-of-the-art approaches for collecting large training utterances for task-oriented bots. Next, we conduct an empirical study to identify quality issues of crowdsourced utterances (e.g., grammatical errors, semantic completeness). Moreover, we propose novel approaches for identifying unqualified crowd workers and eliminating malicious workers from crowdsourcing tasks. Particularly, we propose a novel technique to promote the diversity of crowdsourced paraphrases by dynamically generating word suggestions while crowd workers are paraphrasing a particular utterance. Moreover, we propose a novel technique to automatically translate APIs to canonical utterances. Finally, we present our platform to automatically generate bots out of API specifications. We also conduct thorough experiments to validate the proposed techniques and models

    Communicating, Networking: Interacting: The International Year of Global Understanding - IYGU

    Get PDF
    Communication Studies; Sustainable Development; Communications Engineering, Networks; Computer Systems Organization and Communication Network

    Communicating, Networking: Interacting: The International Year of Global Understanding - IYGU

    Get PDF
    Communication Studies; Sustainable Development; Communications Engineering, Networks; Computer Systems Organization and Communication Network
    corecore