Leveraging Human Computation for Quality Assurance in Open Source Communities

Software developed under the open source development model (OSSD) has risen to significant importance over the recent decades. With more and more critical components being developed under the OSSD, the need for extensive quality assurance (QA) increases. This thesis investigates any potential for conducting formalized user testing through inexperienced volunteer community members under the OSSD. A human computation platform to aggregate such test results was designed and named open crowdsourced user-testing suite (OPEN-CUTS). A usability study of a prototype of OPEN-CUTS confirms the viability of this approach and points to potential future research questions.Software, die unter dem quelloffenen Entwicklungsmodell (Open Source Development Model, OSSD) entwickelt wird, hat in den letzten Jahrzehnten massiv an Bedeutung gewonnen. Immer mehr kritische Komponenten werden im Rahmen des OSSD entwickelt und der Bedarf für tiefgehende Qualitätssicherung (Quality Assurance, QA) steigt. Diese Arbeit untersucht, ob das OSSD Potenzial für formalisierte Nutzertests durch unerfahrene freiwillige Community-Mitglieder bietet. Eine Human Computation Plattform zur Sammlung solcher Testergebnisse wurde entworfen und trägt den Namen open crowdsourced user-testing suite (OPEN-CUTS). Eine mit einem Prototyp von OPEN-CUTS durchgeführte Benutzbarkeitsstudie bestätigt diesen Ansatz und zeigt mögliche zukünftige Forschungsfragen auf

Crowdsourced User-Testing

The presented thesis investigates facilitating software quality assurance in open source communities through a human computation platform. Inexperienced community members can contribute formalized user testing data, which is then aggregated and presented to the developers. The implemented prototype, named open crowdsourced user-testing suite (OPEN-CUTS), was evaluated in a usability study in the UBports Community. The viability of this approach has been demonstrated, and further goals for research and development are proposed

Engineering Crowdsourced Stream Processing Systems

A crowdsourced stream processing system (CSP) is a system that incorporates crowdsourced tasks in the processing of a data stream. This can be seen as enabling crowdsourcing work to be applied on a sample of large-scale data at high speed, or equivalently, enabling stream processing to employ human intelligence. It also leads to a substantial expansion of the capabilities of data processing systems. Engineering a CSP system requires the combination of human and machine computation elements. From a general systems theory perspective, this means taking into account inherited as well as emerging properties from both these elements. In this paper, we position CSP systems within a broader taxonomy, outline a series of design principles and evaluation metrics, present an extensible framework for their design, and describe several design patterns. We showcase the capabilities of CSP systems by performing a case study that applies our proposed framework to the design and analysis of a real system (AIDR) that classifies social media messages during time-critical crisis events. Results show that compared to a pure stream processing system, AIDR can achieve a higher data classification accuracy, while compared to a pure crowdsourcing solution, the system makes better use of human workers by requiring much less manual work effort

Leveraging Social Media and Web of Data for Crisis Response Coordination

There is an ever increasing number of users in social media (1B+ Facebook users, 500M+ Twitter users) and ubiquitous mobile access (6B+ mobile phone subscribers) who share their observations and opinions. In addition, the Web of Data and existing knowledge bases keep on growing at a rapid pace. In this scenario, we have unprecedented opportunities to improve crisis response by extracting social signals, creating spatio-temporal mappings, performing analytics on social and Web of Data, and supporting a variety of applications. Such applications can help provide situational awareness during an emergency, improve preparedness, and assist during the rebuilding/recovery phase of a disaster. Data mining can provide valuable insights to support emergency responders and other stakeholders during crisis. However, there are a number of challenges and existing computing technology may not work in all cases. Therefore, our objective here is to present the characterization of such data mining tasks, and challenges that need further research attention

A survey of the use of crowdsourcing in software engineering

The term 'crowdsourcing' was initially introduced in 2006 to describe an emerging distributed problem-solving model by online workers. Since then it has been widely studied and practiced to support software engineering. In this paper we provide a comprehensive survey of the use of crowdsourcing in software engineering, seeking to cover all literature on this topic. We first review the definitions of crowdsourcing and derive our definition of Crowdsourcing Software Engineering together with its taxonomy. Then we summarise industrial crowdsourcing practice in software engineering and corresponding case studies. We further analyse the software engineering domains, tasks and applications for crowdsourcing and the platforms and stakeholders involved in realising Crowdsourced Software Engineering solutions. We conclude by exposing trends, open issues and opportunities for future research on Crowdsourced Software Engineering