A systematic review on cloud testing

A systematic literature review is presented that surveyed the topic of cloud testing over the period (2012-2017). Cloud testing can refer either to testing cloud-based systems (testing of the cloud), or to leveraging the cloud for testing purposes (testing in the cloud): both approaches (and their combination into testing of the cloud in the cloud) have drawn research interest. An extensive paper search was conducted by both automated query of popular digital libraries and snowballing, which resulted into the final selection of 147 primary studies. Along the survey a framework has been incrementally derived that classifies cloud testing research along six main areas and their topics. The paper includes a detailed analysis of the selected primary studies to identify trends and gaps, as well as an extensive report of the state of art as it emerges by answering the identified Research Questions. We find that cloud testing is an active research field, although not all topics have received so far enough attention, and conclude by presenting the most relevant open research challenges for each area of the classification framework.This paper describes research work mostly undertaken in the context of the European Project H2020 731535: ElasTest. This work has also been partially supported by: the Italian MIUR PRIN 2015 Project: GAUSS; the Regional Government of Madrid (CM) under project Cloud4BigData (S2013/ICE-2894) cofunded by FSE & FEDER; and the Spanish Government under project LERNIM (RTC-2016-4674-7) cofunded by the Ministry of Economy and Competitiveness, FEDER & AEI

Test oracle assessment and improvement

We introduce a technique for assessing and improving test oracles by reducing the incidence of both false positives and false negatives. We prove that our approach can always result in an increase in the mutual information between the actual and perfect oracles. Our technique combines test case generation to reveal false positives and mutation testing to reveal false negatives. We applied the decision support tool that implements our oracle improvement technique to five real-world subjects. The experimental results show that the fault detection rate of the oracles after improvement increases, on average, by 48.6% (86% over the implicit oracle). Three actual, exposed faults in the studied systems were subsequently confirmed and fixed by the developers

Problems and Solutions in Mobile Application Testing

Mobiilirakenduste testimise alaste teadusartiklite arv on viimastel aastatel visalt suurenenud. Samas testivad vähesed mobiilirakendustega tegelevad teadlased oma oletusi ja lahendusi firmades. Selle lõputöö eesmärgiks on pakkuda ülevaade teaduskirjanduses mainitud mobiilirakenduste testimisega seotud probleemidest ja potentsiaalsetest lahendustest ning kõrvutada seda alal igapäevaselt tegutsevate professionaalide arvamusega. Kõigepealt viiakse selle töö käigus läbi teaduskirjanduse uuring probleemide ja potentsiaalsete lahenduste väljaselgitamiseks, misjärel intervjueeritakse kuue mobiilirakenduste testimisega tegeleva firma esindajaid, et välja selgitada, kas kirjelduses esile toodud probleemid on olulised ka tööstuses. Intervjuude tulemusena selgus, et kuigi firmad hindavad probleemide tähtsust väga erinevalt, on siiski olemas mõned võtmeprobleemid, mida peetakse oluliseks nii teaduses kui ka tööstuses. Samas on teaduskirjanduses pakutud lahendused tihti liiga teoreetilised, üldised või vananenud, et firmade esindajatele huvi pakkuda.In recent years the amount of scientific papers published on the topic of mobile applications has significantly increased. However, few researchers test their assumptions and solutions in industry. This thesis aims to provide an overview of what current scientific literature considers problems and potential solutions in mobile application testing, and compare it to opinions of industry professionals. A literature review is performed to extract the list of problems and potential solutions, after which representatives of six Estonian companies involved in the field are interviewed to verify whether the problems and solutions proposed in the literature are relevant for industry. The study reveals that while the relevance of each problem is highly variable from one company to another, there are some key problems that are generally considered vital both by research and industry. However, the solution concepts proposed by scientific literature are often too theoretical, general or outdated to be of much interest to industry professionals

Generierung und Optimierung von Testzeitplänen im Rahmen des SOA Change Managements

Tester einer dienstorientieren Architektur stehen, anders als beim traditionellen Software-Testing, enormen Herausforderungen gegenüber. Heterogene Systemlandschaften, über Unternehmensgrenzen hinweg verteilte Akteure und die dynamische Natur einer solchen Architektur erfordern neue Ansätze beim Testing. Die Fehler-Ursachen-Analyse wird bei zusammengesetzten Services zum großen Problem, da bei der Integration von vielen Komponenten unklar ist, wo die Ursache eines Fehlers zu suchen ist. Das entwickelte Konzept nutzt im Rahmen der SOA Governance Abhängigkeiten aus, um einen geordneten Testzeitplan zu generieren. Seine Ausführung stellt sicher, dass während einer Testperiode nur eine ungetestete Service-Version beteiligt ist, was die möglichen Fehlerursachen stark einschränkt. Darüber hinaus wird die Nebenläufigkeit bei der Testausführung gefördert, indem die zuständigen Tester parallel an unterschiedlichen Testperioden arbeiten. Ein Prototyp des Konzepts wird als Teil eines SOA Governance Repositories implementiert. Er implementiert die Verwaltung von Releases, die Durchführung der Testzeitplan-Generierung und eine Testzeitplan-Visualisierung. Der Generierungsprozess selbst nutzt eine topologische Sortierung des umgekehrten Abhängigkeitsgraphen, um die Testperioden zu erstellen. Vier implementierte Optimierungen können den generierten Testzeitplan gezielt verbessern, um beispielsweise eine kurze Gesamt-Testdauer zu erzielen. Unter Verwendung der SOA-Daten eines großen Automobilherstellers wird die Implementierung unter realitätsnahen Bedingungen ausgeführt. Dabei zeigt die Fallstudie, dass das automatisierte Verfahren performant arbeitet und einen praxistauglichen Testzeitplan generiert

Oracle Assessment, Improvement and Placement

The oracle problem remains one of the key challenges in software testing, for which little automated support has been developed so far. This thesis analyses the prevalence of failed error propagation in programs with real faults to address the oracle placement problem and introduces an approach for iterative assessment and improvement of the oracles. To analyse failed error propagation in programs with real faults, we have conducted an empirical study, considering Defects4J, a benchmark of Java programs, of which we used all 6 projects available, 384 real bugs and 528 methods fixed to correct such bugs. The results indicate that the prevalence of failed error propagation is negligible. Moreover, the results on real faults differ from the results on mutants, indicating that if failed error propagation is taken into account, mutants are not a good surrogate of real faults. When measuring failed error propagation, for each method we use the strongest possible oracle as postcondition, which checks all externally observable program variables. The low prevalence of failed error propagation is caused by the presence of such a strong oracle, which usually is not available in practice. Therefore, there is a need for a technique to assess and improve existing weaker oracles. We propose a technique for assessing and improving test oracles, which necessarily places the human tester in the loop and is based on reducing the incidence of both false positives and false negatives. A proof showing that this approach results in an increase in the mutual information between the actual and perfect oracles is provided. The application of the approach to five real-world subjects shows that the fault detection rate of the oracles after improvement increases, on average, by 48.6%. The further evaluation with 39 participants assessed the ability of humans to detect false positives and false negatives manually, without any tool support. The correct classification rate achieved by humans in this case is poor (29%) indicating how helpful our automated approach can be for developers. The comparison of humans’ ability to improve oracles with and without the tool in a study with 29 other participants also empirically validates the effectiveness of the approach