SmartUnit: Empirical Evaluations for Automated Unit Testing of Embedded Software in Industry

In this paper, we aim at the automated unit coverage-based testing for embedded software. To achieve the goal, by analyzing the industrial requirements and our previous work on automated unit testing tool CAUT, we rebuild a new tool, SmartUnit, to solve the engineering requirements that take place in our partner companies. SmartUnit is a dynamic symbolic execution implementation, which supports statement, branch, boundary value and MC/DC coverage. SmartUnit has been used to test more than one million lines of code in real projects. For confidentiality motives, we select three in-house real projects for the empirical evaluations. We also carry out our evaluations on two open source database projects, SQLite and PostgreSQL, to test the scalability of our tool since the scale of the embedded software project is mostly not large, 5K-50K lines of code on average. From our experimental results, in general, more than 90% of functions in commercial embedded software achieve 100% statement, branch, MC/DC coverage, more than 80% of functions in SQLite achieve 100% MC/DC coverage, and more than 60% of functions in PostgreSQL achieve 100% MC/DC coverage. Moreover, SmartUnit is able to find the runtime exceptions at the unit testing level. We also have reported exceptions like array index out of bounds and divided-by-zero in SQLite. Furthermore, we analyze the reasons of low coverage in automated unit testing in our setting and give a survey on the situation of manual unit testing with respect to automated unit testing in industry.Comment: In Proceedings of 40th International Conference on Software Engineering: Software Engineering in Practice Track, Gothenburg, Sweden, May 27-June 3, 2018 (ICSE-SEIP '18), 10 page

The Practice of Unit Testing by Five Software Development Companies in Malaysia

Software testing is regarded as the most important and the most crucial of the final stages in software development. Profound knowledge and experience in software testing is often an asset in a software company to ensure the reliability and quality of any developed project. The core of software testing strategies mainly lies with unit testing, which is a type of testing carried out on single subroutine or module, almost throughout the development process, depending on the development model used by the developer. There are some issues raised in software testing literature, be it in textbooks, journals, or articles, regarding the process of unit testing, its importance, and the problems faced by developers in performing unit tests. However, through the research and readings of the author on the literature, these issues have not been formally discussed in a Malaysian perspective. Hence, this study of the unit testing practice of software development companies operating in Malaysia is carried out. After reviewing the current available literature on unit testing with no specific geographical or time frame limitation, a study is carried out on five software development companies, within the geographical limit of Malaysia and as of the current time, on their engaged practice of unit testing. It is meant to gauge the importance they place in unit testing, as well as to find out their methods, their processes, and the problems they encounter in unit testing. These findings are then analyzed and compared to the current literature. It is hoped that these case studies will shed more light on the real-world practice of software development, especially of unit testing, in a Malaysian context

The State of Practice for Security Unit Testing: Towards Data Driven Strategies to Shift Security into Developer\u27s Automated Testing Workflows

The pressing need to “shift security left” in the software development lifecycle has motivated efforts to adapt the iterative and continuous process models used in practice today. Security unit testing is praised by practitioners and recommended by expert groups, usually in the context of DevSecOps and achieving “continuous security”. In addition to vulnerability testing and standards adherence, this technique can help developers verify that security controls are implemented correctly, i.e. functional security testing. Further, the means by which security unit testing can be integrated into developer workflows is unique from other standalone tools as it is an adaptation of practices and infrastructure developers are already familiar with. Yet, software engineering researchers have so far failed to include this technique in their empirical studies on secure development and little is known about the state of practice for security unit testing. This dissertation is motivated by the disconnect between promotion of security unit testing and the lack of empirical evidence on how it is and can be applied. The goal of this work was to address the disconnect towards identifying actionable strategies to promote wider adoption and mitigate observed challenges. Three mixed-method empirical studies were conducted wherein practitioner-authored unit test code, Q&A posts, and grey literature were analyzed through three lenses: Practices (what they do), Perspectives and Guidelines (what and how they think it should be done), and Pain Points (what challenges they face) to incorporate both technical and human factors of this phenomena. Accordingly, this work contributes novel and important insights into how developers write functional unit tests for at least nine security controls, including a taxonomy of 53 authentication unit test cases derived from real code and a detailed analysis of seven unique pain points that developers seek help with from peers on Q&A sites. Recommendations given herein for conducting and adopting security unit testing, including mitigating challenges and addressing gaps between available and needed support, are grounded in the guidelines and perspectives on the benefits, limitations, use cases, and integration strategies shared in grey literature authored by practitioners

AUnit - a testing framework for alloy

textWriting declarative models of software designs and analyzing them to detect defects is an effective methodology for developing more dependable software systems. However, writing such models correctly can be challenging for practitioners who may not be proficient in declarative programming, and their models themselves may be buggy. We introduce the foundations of a novel test automation framework, AUnit, which we envision for testing declarative models written in Alloy -- a first-order, relational language that is supported by its SAT-based analyzer. We take inspiration from the success of the family of xUnit frameworks that are used widely in practice for test automation, albeit for imperative or object-oriented programs. The key novelty of our work is to define a basis for unit testing for Alloy, specifically, to define the concepts of test case and test coverage as well as coverage criteria for declarative models. We reduce the problems of declarative test execution and coverage computation to partial evaluation without requiring SAT solving. Our vision is to blend how developers write unit tests in commonly used programming languages with how Alloy users formulate their models in Alloy, thereby facilitating the development and testing of Alloy models for both new Alloy users as well as experts. We illustrate our ideas using a small but complex Alloy model. While we focus on Alloy, our ideas generalize to other declarative languages (such as Z, B, ASM).Electrical and Computer Engineerin

Multiple unit root tests under uncertainty over the initial condition:some powerful modifications

We modify the union-of-rejection unit root test of Harvey et al. "Unit Root Testing in Practice: Dealing with Uncertainty over the Trend and Initial Condition" (Harvey, Econom Theory 25:587-636, 2009). This test rejects if either of two different unit root tests rejects but controls the inherent multiple testing issue by suitably modifying the critical values to ensure the desired null rejection probability. We evaluate the new tests' power relative to existing ones' and to the Gaussian asymptotic power envelope. An empirical application illustrates the usefulness of the new statistics

Unit Testing Challenges with Automated Marking

Teaching software testing presents difficulties due to its abstract and conceptual nature. The lack of tangible outcomes and limited emphasis on hands-on experience further compound the challenge, often leading to difficulties in comprehension for students. This can result in waning engagement and diminishing motivation over time. In this paper, we introduce online unit testing challenges with automated marking as a learning tool via the EdStem platform to enhance students' software testing skills and understanding of software testing concepts. Then, we conducted a survey to investigate the impact of the unit testing challenges with automated marking on student learning. The results from 92 participants showed that our unit testing challenges have kept students more engaged and motivated, fostering deeper understanding and learning, while the automated marking mechanism enhanced students' learning progress, helping them to understand their mistakes and misconceptions quicker than traditional-style human-written manual feedback. Consequently, these results inform educators that the online unit testing challenges with automated marking improve overall student learning experience, and are an effective pedagogical practice in software testing.Comment: 5 pages, accepted at the 30th Asia-Pacific Software Engineering Conference (APSEC 2023

Lag Length Selection for Unit Root Tests in the Presence of Nonstationary Volatility

A number of recently published papers have focused on the problem of testing for a unit root inthe case where the driving shocks may be unconditionally heteroskedastic. These papers have,however, assumed that the lag length in the unit root test regression is a deterministic functionof the sample size, rather than data-determined, the latter being standard empirical practice. Inthis paper we investigate the finite sample impact of unconditional heteroskedasticity onconventional data-dependent methods of lag selection in augmented Dickey-Fuller type unit roottest regressions and propose new lag selection criteria which allow for the presence ofheteroskedasticity in the shocks. We show that standard lag selection methods show a tendency toover-fit the lag order under heteroskedasticity, which results in significant power losses in the(wild bootstrap implementation of the) augmented Dickey-Fuller tests under the alternative. Thenew lag selection criteria we propose are shown to avoid this problem yet deliver unit root testswith almost identical finite sample size and power properties as the corresponding tests based onconventional lag selection methods when the shocks are homoskedastic.econometrics;