SmartUnit: Empirical Evaluations for Automated Unit Testing of Embedded Software in Industry

In this paper, we aim at the automated unit coverage-based testing for embedded software. To achieve the goal, by analyzing the industrial requirements and our previous work on automated unit testing tool CAUT, we rebuild a new tool, SmartUnit, to solve the engineering requirements that take place in our partner companies. SmartUnit is a dynamic symbolic execution implementation, which supports statement, branch, boundary value and MC/DC coverage. SmartUnit has been used to test more than one million lines of code in real projects. For confidentiality motives, we select three in-house real projects for the empirical evaluations. We also carry out our evaluations on two open source database projects, SQLite and PostgreSQL, to test the scalability of our tool since the scale of the embedded software project is mostly not large, 5K-50K lines of code on average. From our experimental results, in general, more than 90% of functions in commercial embedded software achieve 100% statement, branch, MC/DC coverage, more than 80% of functions in SQLite achieve 100% MC/DC coverage, and more than 60% of functions in PostgreSQL achieve 100% MC/DC coverage. Moreover, SmartUnit is able to find the runtime exceptions at the unit testing level. We also have reported exceptions like array index out of bounds and divided-by-zero in SQLite. Furthermore, we analyze the reasons of low coverage in automated unit testing in our setting and give a survey on the situation of manual unit testing with respect to automated unit testing in industry.Comment: In Proceedings of 40th International Conference on Software Engineering: Software Engineering in Practice Track, Gothenburg, Sweden, May 27-June 3, 2018 (ICSE-SEIP '18), 10 page

Random Unit-Test Generation with MUT-aware Sequence Recommendation

A key component of automated object-oriented unit-test generation is to find method-call sequences that generate desired inputs of a method under test (MUT). Previous work cannot find desired sequences effectively due to the large search space of possible sequences. To address this issue, we present a MUT-aware sequence recommendation approach called RecGen to improve the effectiveness of random object-oriented unit-test generation. Unlike existing random testing approaches that select sequences without considering how a MUT may use inputs generated from sequences, Rec-Gen analyzes object fields accessed by a MUT and recommends a short sequence that mutates these fields. In addition, for MUTs whose test generation keeps failing, RecGen recommends a set of sequences to cover all the methods that mutate object fields accessed by the MUT. This technique further improves the chance of generating desired inputs. We have implemented RecGen and evaluated it on three libraries. Evaluation results show that RecGen improves code coverage over previous random testing tools

Testigenerointityökalut Java-sovelluskehityksen tukena

Testien rooli sovelluskehityksessä on suuri, mutta eri syistä sovelluskehittäjien voi olla vaikeaa kirjoittaa relevantteja ja hyödyllisiä testejä. Testigenerointia voidaan käyttää sovelluskehityksen tukena, joko etsimässä aikaisemmin paljastumattomia virheitä tai kuvaamassa kohteena olevan ohjelmiston toimintaa tarkemmin. Testigeneroinnille on Java-kehityksen kontekstissa erilaisia työkaluja, joista tässä tutkielmassa arvioidaan kaupallisia Agitar Technologies-yrityksen AgitarOnea ja Parasoftin Jtestiä ja avoimia työkaluja, CodePro Analytixiä, Palusta, Randoopia ja EvoSuitea. Tutkielman tavoitteena on vastata kysymykseen, saavutetaanko kaupallisilla työkaluilla merkittävää etua verrattuna avoimien työkalujen käyttöön. Tutkielmassa osoitetaan, että AgitarOnella ja Jtestillä on tiettyjä osa-alueita, joissa ne ovat selkeästi parempia kuin avoimet työkalut, mutta myös, että niiden generoimia testejä on rajoitettu, eikä niitä voi hyödyntää yhtä vapaasti kuin avoimien työkalujen generoimia testejä. Tutkielman lopuksi todetaan myös, miten arvioidut sovelluskehittimet tukevat virheiden etsintää myös muilla merkittävillä tavoilla, eikä pelkkä testigenerointi anna koko kuvaa niiden tarjoamasta hyödystä

Evaluating Software Testing Techniques: A Systematic Mapping Study

Software testing techniques are crucial for detecting faults in software and reducing the risk of using it. As such, it is important that we have a good understanding of how to evaluate these techniques for their efficiency, scalability, applicability, and effectiveness at finding faults. This thesis enhances our understanding of testing technique evaluations by providing an overview of the state of the art in research. To accomplish this we utilize a systematic mapping study; structuring the field and identifying research gaps and publication trends. We then present a small case study demonstrating how our mapping study can be used to assist researchers in evaluating their own software testing techniques. We find that a majority of evaluations are empirical evaluations in the form of case studies and experiments, most of these evaluations are of low quality based on proper methodology guidelines, and that relatively few papers in the field discuss how testing techniques should be evaluated

複数バージョンのあるソフトウェアの自動検証・検査 : 複数バージョン管理時代のソフトウェアの品質向上

学位の種別: 課程博士審査委員会委員 : (主査)東京大学准教授 佐藤 周行, 東京大学教授 相田 仁, 東京大学教授 峯松 信明, 東京大学准教授 小川 剛史, 東京大学准教授 鶴岡 慶雅, 東京大学准教授 近山 隆, 日本IBMシニアリサーチャー 河内谷 清久仁University of Tokyo(東京大学