Crowdsourcing in Software Development: A State-of-the-Art Analysis

As software development cycles become shorter and shorter, while software complexity increases and IT budgets stagnate, many companies are looking for new ways of acquiring and sourcing knowledge outside their boundaries. One promising solution to aggregate know-how and manage large distributed teams in software development is crowdsourcing. This paper analyzes the existing body of knowledge regarding crowdsourcing in software development. As a result, we propose a fundamental framework with five dimensions to structure the existing insights of crowdsourcing in the context of software development and to derive a research agenda to guide further research

CAN LAYMEN OUTPERFORM EXPERTS? THE EFFECTS OF USER EXPERTISE AND TASK DESIGN IN CROWDSOURCED SOFTWARE TESTING

In recent years, crowdsourcing has increasingly gained attention as a powerful sourcing mechanism for problem-solving in organizations. Depending on the type of activity addressed by crowdsourcing, the complexity of the tasks and the role of the crowdworkers may differ substantially. It is crucial that the tasks are designed and allocated according to the capabilities of the targeted crowds. In this pa-per, we outline our research in progress which is concerned with the effects of task complexity and user expertise on performance in crowdsourced software testing. We conduct an experiment and gath-er empirical data from expert and novice crowds that perform different software testing tasks of vary-ing degrees of complexity. Our expected contribution is twofold. For crowdsourcing in general, we aim at providing valuable insights for the process of framing and allocating tasks to crowds in ways that increase the crowdworkers’ performance. Secondly, we intend to improve the configuration of crowdsourced software testing initiatives. More precisely, the results are expected to show practition-ers what types of testing tasks should be assigned to which group of dedicated crowdworkers. In this vein, we deliver valuable decision support for both crowdsourcers and intermediaries to enhance the performance of their crowdsourcing initiatives

Given Enough Eyeballs, all Bugs are Shallow - A Literature Review for the Use of Crowdsourcing in Software Testing

Over the last years, the use of crowdsourcing has gained a lot of attention in the domain of software engineering. One key aspect of software development is the testing of software. Literature suggests that crowdsourced software testing (CST) is a reliable and feasible tool for manifold kinds of testing. Research in CST made great strides; however, it is mostly unstructured and not linked to traditional software testing practice and terminology. By conducting a literature review of traditional and crowdsourced software testing literature, this paper delivers two major contributions. First, it synthesizes the fields of crowdsourcing research and traditional software testing. Second, the paper gives a comprehensive overview over findings in CST-research and provides a classification into different software testing types

The Role of Time Pressure in Software Projects: A Literature Review and Research Agenda

The finding that deadlines affect work in organizational settings holds particularly true for software projects, which are usually conducted under time pressure. While the role of time pressure in software projects has been extensively studied, the findings yielded are diverse. Some authors report a positive relationship between time pressure and software project outcomes, while others find it to be negative or to follow an inverted U-shape pattern. Since many aspects concerning time pressure remain unexplained and its relationship with project outcomes is more complex than it might seem at first glance, we synthesize pertinent research to develop a research agenda aimed at improving the understanding in this domain. Our literature review shows a variety of time pressure conceptualizations, research approaches, and research contexts. The results reveal an inconsistent picture of time pressure’s impact on software projects. Our research agenda includes five themes we deem beneficial to consider in future research

WHEN IS CROWDSOURCING ADVANTAGEOUS? THE CASE OF CROWDSOURCED SOFTWARE TESTING

Crowdsourcing describes a novel mode of value creation in which organizations broadcast tasks that have been previously performed in-house to a large magnitude of Internet users that perform these tasks. Although the concept has gained maturity and has proven to be an alternative way of problem-solving, an organizational cost-benefit perspective has largely been neglected by existing research. More specifically, it remains unclear when crowdsourcing is advantageous in comparison to alterna-tive governance structures such as in-house production. Drawing on crowdsourcing literature and transaction action cost theory, we present two case studies from the domain of crowdsourced software testing. We systematically analyze two organizations that applied crowdtesting to test a mobile appli-cation. As both organizations tested the application via crowdtesting and their traditional in-house testing, we are able to relate the effectiveness of crowdtesting and the associated costs to the effective-ness and costs of in-house testing. We find that crowdtesting is comparable in terms of testing quality and costs, but provides large advantages in terms of speed, heterogeneity of testers and user feedback as added value. We contribute to the crowdsourcing literature by providing first empirical evidence about the instances in which crowdsourcing is an advantageous way of problem solving

入出力データの順序情報に基づくブラックボックステスト手法に関する研究

筑波大学 (University of Tsukuba)201

More testers - The effect of crowd size and time restriction in software testing

Context: The questions of how many individuals and how much time to use for a single testing task are critical in software verification and validation. In software review and usability evaluation contexts, positive effects of using multiple individuals for a task have been found, but software testing has not been studied from this viewpoint. Objective: We study how adding individuals and imposing time pressure affects the effectiveness and efficiency of manual testing tasks. We applied the group productivity theory from social psychology to characterize the type of software testing tasks. Method: We conducted an experiment where 130 students performed manual testing under two conditions, one with a time restriction and pressure, i.e., a 2-h fixed slot, and another where the individuals could use as much time as they needed. Results: We found evidence that manual software testing is an additive task with a ceiling effect, like software reviews and usability inspections. Our results show that a crowd of five time-restricted testers using 10 h in total detected 71% more defects than a single non-time-restricted tester using 9.9 h. Furthermore, we use F-score measure from the information retrieval domain to analyze the optimal number of testers in terms of both effectiveness and validity of testing results. We suggest that future studies on verification and validation practices use F-score to provide a more transparent view of the results. Conclusions: The results seem promising for the time-pressured crowds by indicating that multiple time-pressured individuals deliver superior defect detection effectiveness in comparison to non-time-pressured individuals. However, caution is needed, as the limitations of this study need to be addressed in future works. Finally, we suggest that the size of the crowd used in software testing tasks should be determined based on the share of duplicate and invalid reports produced by the crowd and by the effectiveness of the duplicate handling mechanisms. (C) 2012 Elsevier B.V. All rights reserved

Gamifying Software Testing – A Focus on Strategy & Tools Development

This study aims to introduce new software testing strategies and tools with the aim of creating a more engaging and rewarding environment for software testers. For this purpose, gamification has been selected as a potential solution to raise the performances of testers. Empirical experiments were conducted to validate key factors and metrics influencing the design and development of a gamified software testing system

Testitapausten testikattavuuden riittämättömyys

Tiivistelmä. Testitapausten luonti on keskeisessä roolissa ohjelmistotestauksessa. Tämä pätee niin testauksen toteutukselle, tutkimukselle kuin myös muulle alan teoreettiselle pohjalle. Testaus on hyvin pitkälti rakentunut testitapausten ympärille. Aiempi tutkimus on havainnut testitapauksilla testattaessa myös löytyvän sellaisia ohjelmistovirheitä, jotka eivät ole testitapausten määräämällä ohjelman alueella. Testitapauksia käytettäessä testauksessa, ne määräävät testattavasta ohjelmasta toiminnallisuuksien ja ominaisuuksien alueen, jota testit koskevat. Silloin on myös oltava ohjelman alue, joita testitapaukset eivät koske, ts., testitapausten ulkopuolinen alue. Testitapausten käsikirjoituksen määräämä testikattavuus ei siis välttämättä täysin vastaa toteutunutta testauksen laajuutta, kun testitapausten joukon testit ajetaan manuaalisesti. Tämän opinnäytetyön tarkoituksena on tutkia tätä testitapausten testikattavuuden ulkopuolisten virheiden ilmiötä. Näin kaksi tutkimuskysymystä on esitetty: ”Kuinka suuri osa käsikirjoitetun testauksen löytämistä ohjelmistovirheistä ovat käsikirjoitettujen testitapausten kattamia?” sekä ”Vaikuttaako testitapausten testausohjeistuksen rakenne ulkopuolisten virheiden esiintymiseen?” Näihin kysymyksiin vastaamiseksi järjestettiin aikarajoitettu opiskelijoilla tehtävä koe manuaalisesta funktionaalisesti käsikirjoitetusta testauksesta, joka pohjautui ennalta luotuihin testitapauksiin. Testikattavuuden yksiköitä usein luonnehditaan riittämättömäksi täysin kattavalle testaukselle, mutta tämän kokeen tulokset tuo tarkastelun alle vielä toisenlaisen ongelman testikattavuudesta: vaikka testitapausten määräämä testikattavuus olisi riittävä, se ei kuitenkaan välttämättä realisoidu riittävinä testauksen tuloksina. Vain osa virheistä löytyivät, jotka olisi pitänyt testitapauksia testaamalla löytää. Tämän lisäksi merkittävä osuus löytyneistä virheistä eivät olleet testitapausten kattamalla alueella. Testitapausten ulkopuolinen olisi tiedostettava ja huomioitava paremmin. Sen huomiotta jättäminen ja testitapauksiin sokea luottaminen saattaa aiheuttaa ongelmia ja ylimääräisiä kustannuksia ohjelmistoalalla.The insufficiency of test case test coverage. Abstract. The creation of test cases is in a central role in software testing. This applies to software testing in the software industry as well as its’ research and other theoretical basis. Testing is largely based on test cases. Prior research conducted on test case-based testing has noted that the testing process discovers defects from the tested software which are not covered by the test cases’ script, i.e., defects external to the test cases’ coverage. When test cases are used in testing, they determine an area from the functionalities and other features of the software which the tests cover. This also means that there should exist an area outside this coverage: the external area of test coverage. The purpose of this thesis is to study the phenomenon of the discovery of external defects of test case test coverage. Thus, two research questions have been formed as follows: “How large part of the defects found by scripted testing are covered by the test coverage of the script?” and “Does the structure of the test script affect the emergence of external defects of the test case test coverage?” To answer these questions an experiment of scripted testing was conducted with students as participants. While the metrics of test coverage are often described as insufficient for complete testing, yet, as a result of this thesis, another issue of test coverage is brought upon inspection: even if the test coverage was sufficient; the testing conducted to achieve the coverage does not necessarily actualize as sufficient. Only a part of the defects were found with the tests which cover the defects and, in addition to this, a portion of the defects found by testing the test script are not actually covered by the script. The external of the test case should be acknowledged and noted in greater detail. The ignorance of the external and blind reliance on a test script may cause troubles and extra costs in the field of software testing