Leveraging Crowdsourcing Data For Deep Active Learning - An Application: Learning Intents in Alexa

This paper presents a generic Bayesian framework that enables any deep learning model to actively learn from targeted crowds. Our framework inherits from recent advances in Bayesian deep learning, and extends existing work by considering the targeted crowdsourcing approach, where multiple annotators with unknown expertise contribute an uncontrolled amount (often limited) of annotations. Our framework leverages the low-rank structure in annotations to learn individual annotator expertise, which then helps to infer the true labels from noisy and sparse annotations. It provides a unified Bayesian model to simultaneously infer the true labels and train the deep learning model in order to reach an optimal learning efficacy. Finally, our framework exploits the uncertainty of the deep learning model during prediction as well as the annotators' estimated expertise to minimize the number of required annotations and annotators for optimally training the deep learning model. We evaluate the effectiveness of our framework for intent classification in Alexa (Amazon's personal assistant), using both synthetic and real-world datasets. Experiments show that our framework can accurately learn annotator expertise, infer true labels, and effectively reduce the amount of annotations in model training as compared to state-of-the-art approaches. We further discuss the potential of our proposed framework in bridging machine learning and crowdsourcing towards improved human-in-the-loop systems

Competing or aiming to be average?: Normification as a means of engaging digital volunteers

Engagement, motivation and active contribution by digital volunteers are key requirements for crowdsourcing and citizen science projects. Many systems use competitive elements, for example point scoring and leaderboards, to achieve these ends. However, while competition may motivate some people, it can have a neutral or demotivating effect on others. In this paper we explore theories of personal and social norms and investigate normification as an alternative approach to engagement, to be used alongside or instead of competitive strategies. We provide a systematic review of existing crowdsourcing and citizen science literature and categorise the ways that theories of norms have been incorporated to date. We then present qualitative interview data from a pro-environmental crowdsourcing study, Close the Door, which reveals normalising attitudes in certain participants. We assess how this links with competitive behaviour and participant performance. Based on our findings and analysis of norm theories, we consider the implications for designers wishing to use normification as an engagement strategy in crowdsourcing and citizen science systems

A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality

Microtask crowdsourcing is increasingly critical to the creation of extremely large datasets. As a result, crowd workers spend weeks or months repeating the exact same tasks, making it necessary to understand their behavior over these long periods of time. We utilize three large, longitudinal datasets of nine million annotations collected from Amazon Mechanical Turk to examine claims that workers fatigue or satisfice over these long periods, producing lower quality work. We find that, contrary to these claims, workers are extremely stable in their quality over the entire period. To understand whether workers set their quality based on the task's requirements for acceptance, we then perform an experiment where we vary the required quality for a large crowdsourcing task. Workers did not adjust their quality based on the acceptance threshold: workers who were above the threshold continued working at their usual quality level, and workers below the threshold self-selected themselves out of the task. Capitalizing on this consistency, we demonstrate that it is possible to predict workers' long-term quality using just a glimpse of their quality on the first five tasks.Comment: 10 pages, 11 figures, accepted CSCW 201

ENHANCING USERS’ EXPERIENCE WITH SMART MOBILE TECHNOLOGY

The aim of this thesis is to investigate mobile guides for use with smartphones. Mobile guides have been successfully used to provide information, personalisation and navigation for the user. The researcher also wanted to ascertain how and in what ways mobile guides can enhance users' experience. This research involved designing and developing web based applications to run on smartphones. Four studies were conducted, two of which involved testing of the particular application. The applications tested were a museum mobile guide application and a university mobile guide mapping application. Initial testing examined the prototype work for the ‘Chronology of His Majesty Sultan Haji Hassanal Bolkiah’ application. The results were used to assess the potential of using similar mobile guides in Brunei Darussalam’s museums. The second study involved testing of the ‘Kent LiveMap’ application for use at the University of Kent. Students at the university tested this mapping application, which uses crowdsourcing of information to provide live data. The results were promising and indicate that users' experience was enhanced when using the application. Overall results from testing and using the two applications that were developed as part of this thesis show that mobile guides have the potential to be implemented in Brunei Darussalam’s museums and on campus at the University of Kent. However, modifications to both applications are required to fulfil their potential and take them beyond the prototype stage in order to be fully functioning and commercially viable

A theoretical model for the application of Web 2.0 in e-Government

Government organisations in many countries have started embracing modern technologies such as second generation web (Web 2.0) in an attempt to maximize on the benefits of these technologies as well as keeping up with the current trend. Nevertheless, the advancement and the adoption of these of technologies is in its initial stages in the public sector. Therefore, the research problem is that the literature surrounding the application of Web 2.0 is still highly tentative and exploratory. In particular, there is a lack of research exploring the application of Web 2.0 technologies in the context of local e-Government. This study aims to address this research problem by presenting a comprehensive decision-making tool to aid the effective application of Web 2.0 technologies amongst local government authorities (LGAs). In doing so, resulting in the development of a theoretical model that is underpinned by information systems evaluation criteria and impact factors of Web 2.0 from an internal organizational perspective. By addressing the research problem, this study will make a significant contribution to the normative literature by providing new insights of Web 2.0 technologies within the public sector. This will be of specific relevance to scholars, policy makers, LGAs and practitioners who are interested in the adoption of Web 2.0 technologies in an e-Government context. This paper presents the proposed theoretical model and is largely devoted to an explanation on the development of the model

Dynamic Changes in Organizational Motivations to Crowdsourcing for GLAMs

Crowdsourcing has gained popularity as a form of outsourcing. Outsourcing researchers have extensively studied the motivations to outsource IT, but very few have studied the motivations of organizations to crowdsourcing, in particular for GLAMs (galleries, libraries, archives, museums). GLAM institutions are increasingly adopting crowdsourcing technologies due to budgetary constraints and to stay relevant. In this study, findings from an examination of the organizational motivations for crowdsourcing by the National Library of Australia (NLA) are examined for its part in the Australian Newspapers Digitization Program (ANDP). The study found that the NLA was motivated by a set of goals that dynamically changed throughout implementation of the crowdsourcing project ranging from cost reduction to access to external expertise through to social engagement. Identification and recognition of the dynamic nature of organizational motivation demonstrates the long-term value for GLAMs and have implications for other forms of non-profit collaboration aimed at the common good