Achievements, open problems and challenges for search based software testing

Search Based Software Testing (SBST) formulates testing as an optimisation problem, which can be attacked using computational search techniques from the field of Search Based Software Engineering (SBSE). We present an analysis of the SBST research agenda, focusing on the open problems and challenges of testing non-functional properties, in particular a topic we call 'Search Based Energy Testing' (SBET), Multi-objective SBST and SBST for Test Strategy Identification. We conclude with a vision of FIFIVERIFY tools, which would automatically find faults, fix them and verify the fixes. We explain why we think such FIFIVERIFY tools constitute an exciting challenge for the SBSE community that already could be within its reach

Software-Based Self-Test of Set-Associative Cache Memories

Embedded microprocessor cache memories suffer from limited observability and controllability creating problems during in-system tests. This paper presents a procedure to transform traditional march tests into software-based self-test programs for set-associative cache memories with LRU replacement. Among all the different cache blocks in a microprocessor, testing instruction caches represents a major challenge due to limitations in two areas: 1) test patterns which must be composed of valid instruction opcodes and 2) test result observability: the results can only be observed through the results of executed instructions. For these reasons, the proposed methodology will concentrate on the implementation of test programs for instruction caches. The main contribution of this work lies in the possibility of applying state-of-the-art memory test algorithms to embedded cache memories without introducing any hardware or performance overheads and guaranteeing the detection of typical faults arising in nanometer CMOS technologie

Automatic Performance Testing using Input-Sensitive Profiling

During performance testing, software engineers commonly perform application profiling to analyze an application\u27s traces with different inputs to understand performance behaviors, such as time and space consumption. However, a non-trivial application commonly has a large number of inputs, and it is mostly manual to identify the specific inputs leading to performance bottlenecks. Thus, it is challenge is to automate profiling and find these specific inputs. To solve these problems, we propose novel approaches, FOREPOST, GA-Prof and PerfImpact, which automatically profile applications for finding the specific combinations of inputs triggering performance bottlenecks, and further analyze the corresponding traces to identify problematic methods. Specially, our approaches work in two different types of real-world scenarios of performance testing: i) a single-version scenario, in which performance bottlenecks are detected in a single software release, and ii) a two-version scenario, in which code changes responsible for performance regressions are detected by considering two consecutive software releases

Developmental Flight Test Lessons Learned from Open Architecture Software in the Mission Computer of the U.S. Navy E-2C Group II Aircraft

The Naval Air Systems Command commissioned the E-2C Hawkeye Group II Mission Computer Replacement Program and tasked Air Test and Evaluation Squadron Two-Zero and the E-2C Integrated Test Team to evaluate the integration of the form, fit, and function of the OL-698/ASQ Mission Computer Replacement (MCR) for replacement of the Litton L-304 Mission Computer in the E-2C Group II configured aircraft. As part of the life cycle support of the E-2C aircraft, the MCR configuration fields a new, more reliable Commercial-off-the-Shelf (COTS) hardware system and preserves the original software investment by emulating the existing Litton Instructional Set Architecture (LISA) legacy code. Incorporating Northrop Grumman Space Technology’s Reconfigurable Processor for Legacy Applications Code Execution (RePLACE) software re-hosting technique, the investment in the LISA software is maintained. Conducting developmental test of robust software systems, such as the MCR and its associated software, provided dramatically different challenges than traditional developmental testing. A series of lessons were learned through particular discrepancies and deficiencies discovered through the six month flight test period. Specific deficiencies illustrate where proper planning could ease the difficulties encountered in software testing. Keys to successful developmental software tests include having the appropriate personnel on the test team with the proper equipment and capability. Equally important, inadequate configuration management creates more problems than fixes. Software re-programming can provide faster fixes than traditional developmental test. The flexibility of software programming makes configuration management a challenge as multiple versions become available in a short amount of time. Multiple versions of software heighten the risk of configuration management breakdown during limited amount of available flight tests. Each re-programmed version potentially fixes targeted deficiencies, but can also cause new issues in functional areas already tested. Inherently, regression testing impacts the schedule. Software testing requires a realistic schedule that the author believes should compensate for anticipated problems. Data collection, reduction, and analysis always prove to be valuable in developmental testing. A solid instrumentation plan for data collection from all parties involved in flight tests, especially data link network tests, are critical for trouble shooting discovered deficiencies. Software testing is relatively new to the developmental test world and can be seen as the way of the future. Software upgrades lure program managers into a potentially cost effective option in the face of aging avionics systems. With realistic planning and configuration management, the cost and performance effectiveness of software upgrades and development is more likely to become realized

Estimating Sample Size for Usability Testing

One strategy used to assure that an interface meets user requirements is to conduct usability testing. When conducting such testing one of the unknowns is sample size. Since extensive testing is costly, minimizing the number of participants can contribute greatly to successful resource management of a project. Even though a significant number of models have been proposed to estimate sample size in usability testing, there is still not consensus on the optimal size. Several studies claim that 3 to 5 users suffice to uncover 80% of problems in a software interface. However, many other studies challenge this assertion. This study analyzed data collected from the user testing of a web application to verify the rule of thumb, commonly known as the “magic number 5”. The outcomes of the analysis showed that the 5-user rule significantly underestimates the required sample size to achieve reasonable levels of problem detection

Estimating Sample Size for Usability Testing

One strategy used to assure that an interface meets user requirements is to conduct usability testing. When conducting such testing one of the unknowns is sample size. Since extensive testing is costly, minimizing the number of participants can contribute greatly to successful resource management of a project. Even though a significant number of models have been proposed to estimate sample size in usability testing, there is still not consensus on the optimal size. Several studies claim that 3 to 5 users suffice to uncover 80% of problems in a software interface. However, many other studies challenge this assertion. This study analyzed data collected from the user testing of a web application to verify the rule of thumb, commonly known as the “magic number 5”. The outcomes of the analysis showed that the 5-user rule significantly underestimates the required sample size to achieve reasonable levels of problem detection

Ensuring interoperability between network elements in next generation networks

Next Generation Networks (NGNs), based on the Internet Protocol (IP), implement several services such as IP-based telephony and are beginning to replace the classic telephony systems. Due to the development and implementation of new powerful services these systems are becoming increasingly complex. Implementing these new services (typically software-based network elements) is often accompanied by unexpected and erratic behaviours which can manifest as interoperability problems. The reason for this caused by insufficient testing at the developing companies. The testing of such products is by nature a costly and time-consuming exercise and therefore cut down to what is considered the maximum acceptable level. Ensuring the interoperability between network elements is a known challenge. However, there exists no concept of which testing methods should be utilised to achieve an acceptable level of quality. The objective of this thesis was to improve the interoperability between network elements in NGNs by creating a testing scheme comprising of three diverse testing methods: conformance testing, interoperability testing and posthoc analysis. In the first project a novel conformance testing methodology for developing sets of conformance test cases for service specifications in NGNs was proposed. This methodology significantly improves the chance of interoperability and provides a considerable enhancement to the currently used interoperability tests. It was evaluated by successfully applying it to the Presence Service. The second report proposed a post-hoc methodology which enables the identification of the ultimate causes for interoperability problems in a NGN in daily operation. The new methods were implemented in the tool IMPACT (IP-Based Multi Protocol Posthoc Analyzer and Conformance Tester), which stores all exchanged messages between network elements in a database. Using SQL queries, the causes for errors can be found efficiently. Overall the presented testing scheme improves significantly the chance that network elements interoperate successfully by providing new methods. Beyond that, the quality of the software product is raised by mapping these methods to phases in a process model and providing well defined steps on which test method is the best suited at a certain stage

Toward Scalable Docker-Based Emulations of Blockchain Networks

Blockchain, like any other technology, needs a strong testing methodology to support its evolution. Setting up meaningful blockchain tests is a notoriously complex task for several reasons: software is complex, large number of nodes are involved, network is non ideal, etc. Developers usually adopts small virtual laboratories or costly real devnets, based on real software. Researchers usually prefer simulations of a large number of nodes, based on simplified models. In this paper, we aim to obtain the advantages of both approaches, i.e., performing large, realistic, unexpensive, and flexible experiments, using real blockchain software within a virtual environment. To do that, we tackle the challenge of running large blockchain networks in a single physical machine, leveraging Linux and Docker. We analyze a number of problems that arise when large blockchain networks are emulated and we provide technical solutions for all of them. Finally, we describe our experience of emulating a fairly large blockchain network, comprising more than 3000 containers, for research purposes