Visualizing test diversity to support test optimisation

Diversity has been used as an effective criteria to optimise test suites for cost-effective testing. Particularly, diversity-based (alternatively referred to as similarity-based) techniques have the benefit of being generic and applicable across different Systems Under Test (SUT), and have been used to automatically select or prioritise large sets of test cases. However, it is a challenge to feedback diversity information to developers and testers since results are typically many-dimensional. Furthermore, the generality of diversity-based approaches makes it harder to choose when and where to apply them. In this paper we address these challenges by investigating: i) what are the trade-off in using different sources of diversity (e.g., diversity of test requirements or test scripts) to optimise large test suites, and ii) how visualisation of test diversity data can assist testers for test optimisation and improvement. We perform a case study on three industrial projects and present quantitative results on the fault detection capabilities and redundancy levels of different sets of test cases. Our key result is that test similarity maps, based on pair-wise diversity calculations, helped industrial practitioners identify issues with their test repositories and decide on actions to improve. We conclude that the visualisation of diversity information can assist testers in their maintenance and optimisation activities

Test Set Diameter: Quantifying the Diversity of Sets of Test Cases

A common and natural intuition among software testers is that test cases need to differ if a software system is to be tested properly and its quality ensured. Consequently, much research has gone into formulating distance measures for how test cases, their inputs and/or their outputs differ. However, common to these proposals is that they are data type specific and/or calculate the diversity only between pairs of test inputs, traces or outputs. We propose a new metric to measure the diversity of sets of tests: the test set diameter (TSDm). It extends our earlier, pairwise test diversity metrics based on recent advances in information theory regarding the calculation of the normalized compression distance (NCD) for multisets. An advantage is that TSDm can be applied regardless of data type and on any test-related information, not only the test inputs. A downside is the increased computational time compared to competing approaches. Our experiments on four different systems show that the test set diameter can help select test sets with higher structural and fault coverage than random selection even when only applied to test inputs. This can enable early test design and selection, prior to even having a software system to test, and complement other types of test automation and analysis. We argue that this quantification of test set diversity creates a number of opportunities to better understand software quality and provides practical ways to increase it.Comment: In submissio

Searching for test data with feature diversity

There is an implicit assumption in software testing that more diverse and varied test data is needed for effective testing and to achieve different types and levels of coverage. Generic approaches based on information theory to measure and thus, implicitly, to create diverse data have also been proposed. However, if the tester is able to identify features of the test data that are important for the particular domain or context in which the testing is being performed, the use of generic diversity measures such as this may not be sufficient nor efficient for creating test inputs that show diversity in terms of these features. Here we investigate different approaches to find data that are diverse according to a specific set of features, such as length, depth of recursion etc. Even though these features will be less general than measures based on information theory, their use may provide a tester with more direct control over the type of diversity that is present in the test data. Our experiments are carried out in the context of a general test data generation framework that can generate both numerical and highly structured data. We compare random sampling for feature-diversity to different approaches based on search and find a hill climbing search to be efficient. The experiments highlight many trade-offs that needs to be taken into account when searching for diversity. We argue that recurrent test data generation motivates building statistical models that can then help to more quickly achieve feature diversity.Comment: This version was submitted on April 14th 201

DSpot: Test Amplification for Automatic Assessment of Computational Diversity

Context: Computational diversity, i.e., the presence of a set of programs that all perform compatible services but that exhibit behavioral differences under certain conditions, is essential for fault tolerance and security. Objective: We aim at proposing an approach for automatically assessing the presence of computational diversity. In this work, computationally diverse variants are defined as (i) sharing the same API, (ii) behaving the same according to an input-output based specification (a test-suite) and (iii) exhibiting observable differences when they run outside the specified input space. Method: Our technique relies on test amplification. We propose source code transformations on test cases to explore the input domain and systematically sense the observation domain. We quantify computational diversity as the dissimilarity between observations on inputs that are outside the specified domain. Results: We run our experiments on 472 variants of 7 classes from open-source, large and thoroughly tested Java classes. Our test amplification multiplies by ten the number of input points in the test suite and is effective at detecting software diversity. Conclusion: The key insights of this study are: the systematic exploration of the observable output space of a class provides new insights about its degree of encapsulation; the behavioral diversity that we observe originates from areas of the code that are characterized by their flexibility (caching, checking, formatting, etc.).Comment: 12 page

Test-bed development & measurement plan for evaluating transmit diversity in DVB networks

This paper presents a test-bed development and measurement plan for evaluating transmit diversity in the DVB network. Transmit diversity reduces the complexity and improves the power consumption of the personal receiving devices by improving the transmission of signals in NLOS cluttered environments. Also, it is more practical than receive diversity due to the difficulty of locating two receive antennas far enough apart in a small mobile device. Test service scenarios were developed to illustrate the benefits of such technologies so that effectiveness can be researched in a variety of service and terrain scenarios using purpose built test systems. The laboratory tests were designed to validate the theoretical measurements from the theoretical analysis and these results will be verified by a field measurement campaign in short and long time spans

Gender Diversity in the Boardroom and Financial Performance of Commercial Banks: Evidence from Bangladesh

In today’s corporate world, board diversity is a much talked-about topic and gender diversity is an important aspect of board diversity. Gender diversity refers to the presence of women on corporate boards of directors. In this paper, an effort has been made to examine whether an association exists between the financial performance of commercial banks in Bangladesh and presence of women on the boards of directors of these banks and in order to examine the existence of this association, a non-parametric test, namely Kruskal-Wallis H test has been conducted. But the test has yielded conflicting results at different significance levels.Gender diversity, Board of Directors, Financial Performance and Commercial Banks

Creating a test collection to evaluate diversity in image retrieval

This paper describes the adaptation of an existing test collection for image retrieval to enable diversity in the results set to be measured. Previous research has shown that a more diverse set of results often satisfies the needs of more users better than standard document rankings. To enable diversity to be quantified, it is necessary to classify images relevant to a given theme to one or more sub-topics or clusters. We describe the challenges in building (as far as we are aware) the first test collection for evaluating diversity in image retrieval. This includes selecting appropriate topics, creating sub-topics, and quantifying the overall effectiveness of a retrieval system. A total of 39 topics were augmented for cluster-based relevance and we also provide an initial analysis of assessor agreement for grouping relevant images into sub-topics or clusters

Examining the Link Between Diversity and Firm Performance: The Effects of Diversity Reputation and Leader Racial Diversity

Given the scarcity of empirical research on the impact of diversity on organizational performance, we used longitudinal data for 100 firms to test hypotheses related to the effects of diversity reputation and leader racial diversity on firm financial outcomes. The results showed a positive relationship between diversity reputation and book-to-market equity, and a curvilinear U-shaped relationship between leader diversity and revenues, net income and book-to-market equity. Our analyses suggest that economic benefits generated from diversity reputation may primarily derive from capital rather than product markets. Further, firm performance declines with increases in the representation of racial minorities in leadership up to a point, beyond which further increases in diversity are associated with increases in performance

Models suggesting field experiments to test two hypotheses explaining successional diversity

A simple mathematical model of competition is developed that includes two alternative mechanisms promoting successional diversity. The first underpins the competition-colonization hypothesis in which early successional species are able to persist because they colonize disturbed habitats before the arrival of late successional dominant competitors. The second underpins the niche hypothesis, in which early successional species are able to persist, even with unlimited colonization by late successional dominants, because they specialize on the resource-rich conditions typical of recently disturbed sites. We modify the widely studied competition-colonization model so that it also includes the mechanism behind the niche hypothesis. Analysis of this model suggests simple experiments that determine whether the successional diversity of a field system is maintained primarily by the competition-colonization mechanism, primarily by the niche mechanism, by neither, or by both. We develop quantitative metrics of the relative importance of the two mechanisms. We also discuss the implications for the management of biodiversity in communities structured by the two mechanisms