Worker Retention, Response Quality, and Diversity in Microtask Crowdsourcing: An Experimental Investigation of the Potential for Priming Effects to Promote Project Goals

Online microtask crowdsourcing platforms act as efficient resources for delegating small units of work, gathering data, generating ideas, and more. Members of research and business communities have incorporated crowdsourcing into problem-solving processes. When human workers contribute to a crowdsourcing task, they are subject to various stimuli as a result of task design. Inter-task priming effects - through which work is nonconsciously, yet significantly, influenced by exposure to certain stimuli - have been shown to affect microtask crowdsourcing responses in a variety of ways. Instead of simply being wary of the potential for priming effects to skew results, task administrators can utilize proven priming procedures in order to promote project goals. In a series of three experiments conducted on Amazon’s Mechanical Turk, we investigated the effects of proposed priming treatments on worker retention, response quality, and response diversity. In our first two experiments, we studied the effect of initial response freedom on sustained worker participation and response quality. We expected that workers who were granted greater levels of freedom in an initial response would be stimulated to complete more work and deliver higher quality work than workers originally constrained in their initial response possibilities. We found no significant relationship between the initial response freedom granted to workers and the amount of optional work they completed. The degree of initial response freedom also did not have a significant impact on subsequent response quality. However, the influence of inter-task effects were evident based on response tendencies for different question types. We found evidence that consistency in task structure may play a stronger role in promoting response quality than proposed priming procedures. In our final experiment, we studied the influence of a group-level priming treatment on response diversity. Instead of varying task structure for different workers, we varied the degree of overlap in question content distributed to different workers in a group. We expected groups of workers that were exposed to more diverse preliminary question sets to offer greater diversity in response to a subsequent question. Although differences in response diversity were revealed, no consistent trend between question content overlap and response diversity prevailed. Nevertheless, combining consistent task structure with crowd-level priming procedures - to encourage diversity in inter-task effects across the crowd - offers an exciting path for future study

The Crowd in Requirements Engineering: The Landscape and Challenges

Crowd-based requirements engineering (CrowdRE) could significantly change RE. Performing RE activities such as elicitation with the crowd of stakeholders turns RE into a participatory effort, leads to more accurate requirements, and ultimately boosts software quality. Although any stakeholder in the crowd can contribute, CrowdRE emphasizes one stakeholder group whose role is often trivialized: users. CrowdRE empowers the management of requirements, such as their prioritization and segmentation, in a dynamic, evolved style through collecting and harnessing a continuous flow of user feedback and monitoring data on the usage context. To analyze the large amount of data obtained from the crowd, automated approaches are key. This article presents current research topics in CrowdRE; discusses the benefits, challenges, and lessons learned from projects and experiments; and assesses how to apply the methods and tools in industrial contexts. This article is part of a special issue on Crowdsourcing for Software Engineering

Locational wireless and social media-based surveillance

The number of smartphones and tablets as well as the volume of traffic generated by these devices has been growing constantly over the past decade and this growth is predicted to continue at an increasing rate over the next five years. Numerous native features built into contemporary smart devices enable highly accurate digital fingerprinting techniques. Furthermore, software developers have been taking advantage of locational capabilities of these devices by building applications and social media services that enable convenient sharing of information tied to geographical locations. Mass online sharing resulted in a large volume of locational and personal data being publicly available for extraction. A number of researchers have used this opportunity to design and build tools for a variety of uses – both respectable and nefarious. Furthermore, due to the peculiarities of the IEEE 802.11 specification, wireless-enabled smart devices disclose a number of attributes, which can be observed via passive monitoring. These attributes coupled with the information that can be extracted using social media APIs present an opportunity for research into locational surveillance, device fingerprinting and device user identification techniques. This paper presents an in-progress research study and details the findings to date

End-User Programming of Mobile Services: Empowering Domain Experts to Implement Mobile Data Collection Applications

The widespread use of smart mobile devices (e.g., in clinical trials or online surveys) offers promising perspectives with respect to the controlled collection of high-quality data. The design, implementation and deployment of such mobile data collection applications, however, is challenging in several respects. First, various mobile operating systems need to be supported, taking the short release cycles of vendors into account as well. Second, domain-specific requirements need to be flexibly aligned with mobile application development. Third, usability styleguides need to be obeyed. Altogether, this turns both programming and maintaining mobile applications into a costly, time-consuming, and error-prone endeavor. To remedy these drawbacks, a model-driven framework empowering domain experts to implement robust mobile data collection applications in an intuitive way was realized. The design of this end-user programming framework is based on experiences gathered in real-life mobile data collection projects. Facets of various stakeholders involved in such projects are discussed and an overall architecture as well as its components are presented. In particular, it is shown how the framework enables domain experts (i.e., end users) to flexibly implement mobile data collection applications on their own. Overall, the framework allows for the effective support of mobile services in a multitude of application domains

Beyond Traditional Feedback Channels: Extracting Requirements-Relevant Feedback from TikTok and YouTube

The increasing importance of videos as a medium for engagement, communication, and content creation makes them critical for organizations to consider for user feedback. However, sifting through vast amounts of video content on social media platforms to extract requirements-relevant feedback is challenging. This study delves into the potential of TikTok and YouTube, two widely used social media platforms that focus on video content, in identifying relevant user feedback that may be further refined into requirements using subsequent requirement generation steps. We evaluated the prospect of videos as a source of user feedback by analyzing audio and visual text, and metadata (i.e., description/title) from 6276 videos of 20 popular products across various industries. We employed state-of-the-art deep learning transformer-based models, and classified 3097 videos consisting of requirements relevant information. We then clustered relevant videos and found multiple requirements relevant feedback themes for each of the 20 products. This feedback can later be refined into requirements artifacts. We found that product ratings (feature, design, performance), bug reports, and usage tutorial are persistent themes from the videos. Video-based social media such as TikTok and YouTube can provide valuable user insights, making them a powerful and novel resource for companies to improve customer-centric development

Increasing the credibility of scientific dissemination using crowdsourcing

Abstract. This thesis introduces Article Enhancer, a semi-automated web application that utilizes crowdsourcing services, specifically Amazon’s Mechanical Turk platform, for augmenting articles with various referencing content gathered from the crowd-workers, on demand. The main goal of Article Enhancer is to address the question of how scientific articles can be made more credible, before dissemination to the public. This application serves as a tool in helping users find suitable supporting content for their articles in a novel way, removing all the manual work of doing it themselves. Media literacy, social media, fake news and crowdsourcing are discussed as part of related work. Also, tools that offer a similar functionality are reviewed. Furthermore, system design and implementation for Article Enhancer is presented. It is important to mention that the referencing content provided through Article Enhancer comes from already existing online content. Although Article Enhancer is semi-automated system, its strongest point compared to the other systems, is that it doesn’t require extra human effort to enrich articles especially with visualization content, and providing already existing content on the web avoiding the process of creating new content, making it a fresh approach in this line of software service. To evaluate Article Enhancer, we deployed the web app in a real-life setting, a space oriented towards students known as Tellus, at the University of Oulu. This testing proceedings helped in determining that the system appears alluring and attractive to new users. Article Enhancer proved to be unique and thrilling after the first encounter for many of the users. Feedback also shows that adding and embedding content is an innovative way to make articles become more credible in the eye of the reader