Crowdsourcing in Software Development: A State-of-the-Art Analysis

As software development cycles become shorter and shorter, while software complexity increases and IT budgets stagnate, many companies are looking for new ways of acquiring and sourcing knowledge outside their boundaries. One promising solution to aggregate know-how and manage large distributed teams in software development is crowdsourcing. This paper analyzes the existing body of knowledge regarding crowdsourcing in software development. As a result, we propose a fundamental framework with five dimensions to structure the existing insights of crowdsourcing in the context of software development and to derive a research agenda to guide further research

CAN LAYMEN OUTPERFORM EXPERTS? THE EFFECTS OF USER EXPERTISE AND TASK DESIGN IN CROWDSOURCED SOFTWARE TESTING

In recent years, crowdsourcing has increasingly gained attention as a powerful sourcing mechanism for problem-solving in organizations. Depending on the type of activity addressed by crowdsourcing, the complexity of the tasks and the role of the crowdworkers may differ substantially. It is crucial that the tasks are designed and allocated according to the capabilities of the targeted crowds. In this pa-per, we outline our research in progress which is concerned with the effects of task complexity and user expertise on performance in crowdsourced software testing. We conduct an experiment and gath-er empirical data from expert and novice crowds that perform different software testing tasks of vary-ing degrees of complexity. Our expected contribution is twofold. For crowdsourcing in general, we aim at providing valuable insights for the process of framing and allocating tasks to crowds in ways that increase the crowdworkers’ performance. Secondly, we intend to improve the configuration of crowdsourced software testing initiatives. More precisely, the results are expected to show practition-ers what types of testing tasks should be assigned to which group of dedicated crowdworkers. In this vein, we deliver valuable decision support for both crowdsourcers and intermediaries to enhance the performance of their crowdsourcing initiatives

Given Enough Eyeballs, all Bugs are Shallow - A Literature Review for the Use of Crowdsourcing in Software Testing

Over the last years, the use of crowdsourcing has gained a lot of attention in the domain of software engineering. One key aspect of software development is the testing of software. Literature suggests that crowdsourced software testing (CST) is a reliable and feasible tool for manifold kinds of testing. Research in CST made great strides; however, it is mostly unstructured and not linked to traditional software testing practice and terminology. By conducting a literature review of traditional and crowdsourced software testing literature, this paper delivers two major contributions. First, it synthesizes the fields of crowdsourcing research and traditional software testing. Second, the paper gives a comprehensive overview over findings in CST-research and provides a classification into different software testing types

A Systematic Mapping Study of Empirical Studies on Software Cloud Testing Methods

Context: Software has become more complicated, dynamic, and asynchronous than ever, making testing more challenging. With the increasing interest in the development of cloud computing, and increasing demand for cloud-based services, it has become essential to systematically review the research in the area of software testing in the context of cloud environments. Objective: The purpose of this systematic mapping study is to provide an overview of the empirical research in the area of software cloud-based testing, in order to build a classification scheme. We investigate functional and non-functional testing methods, the application of these methods, and the purpose of testing using these methods. Method: We searched for electronically available papers in order to find relevant literature and to extract and analyze data about the methods used. Result: We identified 69 primary studies reported in 75 research papers published in academic journals, conferences, and edited books. Conclusion: We found that only a minority of the studies combine rigorous statistical analysis with quantitative results. The majority of the considered studies present early results, using a single experiment to evaluate their proposed solution

Data-Driven Usability Refactoring: Tools and Challenges

Usability has long been recognized as an important software quality attribute and it has become essential in web application development and maintenance. However, it is still hard to integrate usability evaluation and improvement practices in the software development process. Moreover, these practices are usually unaffordable for small to medium-sized companies. In this position paper we propose an approach and tools to allow the crowd of web users participate in the process of usability evaluation and repair. Since we use the refactoring technique for usability improvement, we introduce the notion of “data-driven refactoring”: use data from the mass of users to learn about refactoring opportunities, plus also about refactoring effectiveness. This creates an improvement cycle where some refactorings may be discarded while others introduced, depending on their evaluated success. The paper also discusses some of the challenges that we foresee ahead

Joukkoistetun kotoistamisen luonteesta kääntämisenä

Web 2.0 has led to proliferation of crowdsourcing in many areas, not the least of which is translation and particularly localization. Another, simultaneous but unconnected phenomenon has been the proliferation of free/libre open source software (FLOSS). We use examples from a case of crowdsourced localization of FLOSS, the Finnish localization of KDE Plasma desktop environment. We discuss the role of the translator in FLOSS, the nature of crowdsourced localization as translation, and the ways in which crowdsourced localization may change the way we see some basic concepts of translation such as the source and target texts.Web 2.0 has led to proliferation of crowdsourcing in many areas, not the least of which is translation and particularly localization. Another, simultaneous but unconnected phenomenon has been the proliferation of free/libre open source software (FLOSS). We use examples from a case of crowdsourced localization of FLOSS, the Finnish localization of KDE Plasma desktop environment. We discuss the role of the translator in FLOSS, the nature of crowdsourced localization as translation, and the ways in which crowdsourced localization may change the way we see some basic concepts of translation such as the source and target texts

Crowdfunding To Generate Crowdsourced R&D: The Alternative Paradigm Of Societal Problem Solving Offered By Second Generation Innovation And R&D

In a global context of resource scarcity few incentives exist for firms to pursue innovations that provide social externalities if these are not inherently profitable. The purpose of this article is to present an alternative paradigm of societal problem solving entirely premised on ‘second generation innovation’ processes. Further, a theoretical model of multidimensional, or three dimensional, knowledge creation is offered, together with the notion of a ‘multiplier effect’ that relates to how knowledge creation can increase exponentially when knowledge is not constrained by proprietary requirements. Second generation innovation is based on probabilistic processes that utilize and maximize economies of scale in pursuit of problem solving. Two processes that contribute to the potential of second generation innovation to solve societal problems are crowdfunding and crowdsourcing. It is argued that the processes required to enable a new paradigm in societal problem solving already exist. A further model is developed based on potential synergies between crowdfunding and crowdsourced research and development. This theoretical model predicts that R&D productivity can be accelerated significantly, and if applied in fields such as proteomics or medical research in general can accelerated increases in research output and therefore benefits to society

Quality Assessment Methods for Textual Conversational Interfaces: A Multivocal Literature Review

The evaluation and assessment of conversational interfaces is a complex task since such software products are challenging to validate through traditional testing approaches. We conducted a systematic Multivocal Literature Review (MLR), on five different literature sources, to provide a view on quality attributes, evaluation frameworks, and evaluation datasets proposed to provide aid to the researchers and practitioners of the field. We came up with a final pool of 118 contributions, including grey (35) and white literature (83). We categorized 123 different quality attributes and metrics under ten different categories and four macro-categories: Relational, Conversational, User-Centered and Quantitative attributes. While Relational and Conversational attributes are most commonly explored by the scientific literature, we testified a predominance of User-Centered Attributes in industrial literature. We also identified five different academic frameworks/tools to automatically compute sets of metrics, and 28 datasets (subdivided into seven different categories based on the type of data contained) that can produce conversations for the evaluation of conversational interfaces. Our analysis of literature highlights that a high number of qualitative and quantitative attributes are available in the literature to evaluate the performance of conversational interfaces. Our categorization can serve as a valid entry point for researchers and practitioners to select the proper functional and non-functional aspects to be evaluated for their products

Usabilidad comunitaria : Un sistema colaborativo para la mejora de la experiencia de usuario

Las aplicaciones web tienen un papel cada vez más importante en nuestras vidas. La usabilidad y la experiencia de usuario se han convertido en un importante campo de investigación. En este trabajo nos proponemos desarrollar un sistema que permita la mejora de la experiencia de navegación web haciendo uso de un mecanismo de inyección de scripts de usuario basado en una extensión al navegador web, permitiendo de esta manera afectar a cualquier sitio web, sin necesidad de colaboración de los dueños de dicho sitio en todo el proceso. Considerando el gran volumen de trabajo que implicaría la generación de estas soluciones proponemos construir una comunidad donde los usuarios reporten los problemas detectados, manifiesten su interés en obtener una solución a problemas previamente reportados, propongan las soluciones que resolverían dichos problemas y las evalúen dando feedback sobre las mismas para que, luego de concluido el período de evaluación, pueda determinarse la mejor solución. Dado que a lo largo de este trabajo se detectaron múltiples posibilidades de profundización decidimos promover la extensibilidad del sistema exponiendo sus funcionalidades a través de una API, facilitando de este modo su interacción con otras aplicaciones.Facultad de Informátic