Scalable diversified antirandom test pattern generation with improved fault coverage for black-box circuit testing

Pseudorandom testing is incapable of utilizing the success rate of preceding test patterns while generating subsequent test patterns. Many redundant test patterns have been generated that increase the test length without any significant increase in the fault coverage. An extension to pseudorandom testing is Antirandom that induces divergent patterns by maximizing the Total Hamming Distance (THD) and Total Cartesian Distance (TCD) of every subsequent test pattern. However, the Antirandom test sequence generation algorithm is prone to unsystematic selection when more than one patterns possess maximum THD and TCD. As a result, diversity among test sequences is compromised, lowering the fault coverage. Therefore, this thesis analyses the effect of Hamming distance in vertical as well as horizontal dimension to enhance diversity among test patterns. First contribution of this thesis is the proposal of a Diverse Antirandom (DAR) test pattern generation algorithm. DAR employs Horizontal Total Hamming Distance (HTHD) along with THD and TCD for diversity enhancement among test patterns as maximum distance test pattern generation. The HTHD and TCD are used as distance metrics that increase computational complexity in divergent test sequence generation. Therefore, the second contribution of this thesis is the proposal of tree traversal search method to maximize diversity among test patterns. The proposed method uses bits mutation of a temporary test pattern following a path leading towards maximization of TCD. Results of fault simulations on benchmark circuits have shown that DAR significantly improves the fault coverage up to 18.3% as compared to Antirandom. Moreover, the computational complexity of Antirandom is reduced from exponential O(2n) to linear O(n). Next, the DARalgorithm is modified to ease hardware implementation for on-chip test generation. Therefore, the third contribution of this thesis is the design of a hardware-oriented DAR (HODA) test pattern generator architecture as an alternative to linear feedback shift register (LFSR) that consists of large number of memory elements. Parallel concatenation of the HODA architecture is designed to reduce the number of memory elements by implementing bit slicing architecture. It has been proven through simulation that the proposed architecture has increased fault coverage up to 66% and a reduction of 46.59% gate count compared to the LFSR. Consequently, this thesis presents uniform and scalable test pattern generator architecture for built-in self-test (BIST) applications and solution to maximum distance test pattern generation for high fault coverage in black-box environment

Test vectors reductoin for integrated circuit testing using horizontal hamming distance

In testing digital combinational logic for stuck-at faults, it is required to determine the most appropriate test sequence needed to detect the required number of possible faults. The exhaustive test pattern generation method is the simplest approach to implement as it produces test patterns consisting of all possible input combinations of the circuit under test. However, a consequence of this approach is that it results in a large test set when the number of circuit inputs is large. This can take an unnecessarily long time to apply on the circuit under test as during the test process, only a small fraction of all possible test vectors is actually required to produce high percentage of fault coverage. As an alternative, random test pattern generation applies a random set of test patterns which can be used to reduce the number of test patterns compared to exhaustive test. However, both test pattern generation approaches generate unnecessary test vectors to apply to the circuit as multiple patterns typically detect the same fault. Antirandom testing on the other hand ensures that the identified test vectors to use do not detect the same fault by introducing the concept of Hamming distance between test vectors and this distance is be maximized. This results in a reduction in the number of required test vectors when compared to an exhaustive test. However, the algorithm for Antirandom test vector generation is computation intensive and vague in its definition when there are more than one possible next test vectors. In this study, efficient calculation of Hamming distance has been proposed, moreover the choice of the next test vector is addressed by using the proposed Horizontal Hamming distance method which has not yet been explored. The approach effectively detects faults at a much faster rate and produces a much higher fault coverage than the existing Antirandom method

Searching for test data with feature diversity

There is an implicit assumption in software testing that more diverse and varied test data is needed for effective testing and to achieve different types and levels of coverage. Generic approaches based on information theory to measure and thus, implicitly, to create diverse data have also been proposed. However, if the tester is able to identify features of the test data that are important for the particular domain or context in which the testing is being performed, the use of generic diversity measures such as this may not be sufficient nor efficient for creating test inputs that show diversity in terms of these features. Here we investigate different approaches to find data that are diverse according to a specific set of features, such as length, depth of recursion etc. Even though these features will be less general than measures based on information theory, their use may provide a tester with more direct control over the type of diversity that is present in the test data. Our experiments are carried out in the context of a general test data generation framework that can generate both numerical and highly structured data. We compare random sampling for feature-diversity to different approaches based on search and find a hill climbing search to be efficient. The experiments highlight many trade-offs that needs to be taken into account when searching for diversity. We argue that recurrent test data generation motivates building statistical models that can then help to more quickly achieve feature diversity.Comment: This version was submitted on April 14th 201

Test Set Diameter: Quantifying the Diversity of Sets of Test Cases

A common and natural intuition among software testers is that test cases need to differ if a software system is to be tested properly and its quality ensured. Consequently, much research has gone into formulating distance measures for how test cases, their inputs and/or their outputs differ. However, common to these proposals is that they are data type specific and/or calculate the diversity only between pairs of test inputs, traces or outputs. We propose a new metric to measure the diversity of sets of tests: the test set diameter (TSDm). It extends our earlier, pairwise test diversity metrics based on recent advances in information theory regarding the calculation of the normalized compression distance (NCD) for multisets. An advantage is that TSDm can be applied regardless of data type and on any test-related information, not only the test inputs. A downside is the increased computational time compared to competing approaches. Our experiments on four different systems show that the test set diameter can help select test sets with higher structural and fault coverage than random selection even when only applied to test inputs. This can enable early test design and selection, prior to even having a software system to test, and complement other types of test automation and analysis. We argue that this quantification of test set diversity creates a number of opportunities to better understand software quality and provides practical ways to increase it.Comment: In submissio

УПРАВЛЯЕМОЕ СЛУЧАЙНОЕ ТЕСТИРОВАНИЕ

Анализируются управляемые случайные тесты и методы их генерирования. Показываетсяобщность процедур генерирования тестовых векторов управляемых случайных тестов, использующих жадный оптимизационный алгоритм и метрики расстояния между тестовыми наборами. Предлагается метод построения оптимальных управляемых случайных тестов, характеризующихся максимальной полнотой покрытия в сравнении со случайными и управляемыми случайными тестами в силу максимального отличия тестовых наборов. Оптимальные управляемые случайные тесты характеризуются минимальной вычислительной сложностью их генерирования

КВАЗИСЛУЧАЙНОЕ ТЕСТИРОВАНИЕ ВЫЧИСЛИТЕЛЬНЫХ СИСТЕМ

Various modified random testing approaches have been proposed for computer system testing in the black box environment. Their effectiveness has been evaluated on the typical failure patterns by employing three measures, namely, P-measure, E-measure and F-measure. A quasi-random testing, being a modified version of the random testing, has been proposed and analyzed. The quasi-random Sobol sequences and modified Sobol sequences are used as the test patterns. Some new methods for Sobol sequence generation have been proposed and analyzed.Анализируются причинно-следственные связи при возникновении неисправностей вычисли-тельных систем. Даются определения понятий «неисправность», «ошибка» и «неисправное поведение вычислительных систем», показывается их общность для программной и аппаратной частей вычислительных систем. Рассматривается классификация обобщенных входных тестовых воздей-ствий на три категории: точечные тестовые наборы, узкополосные тестовые наборы и блочные тестовые наборы. Приводится анализ методов тестирования вычислительных систем по методике черного ящика, показывается эффективность использования квазислучайного тестирования. Анали-зируются и предлагаются методы формирования квазислучайных тестовых воздействий

МНОГОКРАТНЫЕ УПРАВЛЯЕМЫЕ ВЕРОЯТНОСТНЫЕ ТЕСТЫ

Controlled Random Tests and methods for their generation have been analyzed and investigated. The similarities of all known controlled random testing approaches are shown. A new method and algorithm for Multiple Controlled Random Tests have been proposed and analyzed.Рассматриваются однократные управляемые вероятностные тесты, методы их формирования, а также их применение для тестирования средств вычислительных систем. Показываются основные недостатки построения однократных вероятностных тестов. Предлагается метод построения многократных управляемых вероятностных тестов на базе исходного однократного теста. Анализируются различные численные метрики для построения как однократных, так и многократных управляемых вероятностных тестов

Методы генерирования квазислучайных тестовых последовательностей

This thesis deals with quasi-random test sequences and methods of their generation. The advantages of quasi-random sequences over pseudo-random sequences are also shown

The Effectiveness of <i>t</i>-Way Test Data Generation

Modern society is increasingly dependent on the correct functioning of software and increasingly so in areas that are considered safety related or safety critical. Therefore, there is an increasing need to be able to verify and validate that the software is in fact correct and will perform its intended function. Many approaches to this problem have been proposed; however, none seems likely to supplant the role of testing in the near future. If we accept that there is, and will be, a continuing need to be able to test software then the question becomes one of how can this be done effectively, both in terms of ability to detect errors and in terms of cost. One avenue of research that offers prospects of improving both of these aspects is the automatic generation of test data. There has recently been a large amount of work conducted in this area. One particularly promising direction has been the application of ideas from the field of experimental design and in particular, the field of t-way adequate factorial designs. The area however, is not without issues; there is evidence that the technique is capable of detecting errors but that evidence is not unequivocal. Moreover, as with almost all work in the area of automatic test generation, there has been very little comparative work comparing the technique with other test data generation techniques. Worse, there has been effectively no work done that compares any automatic test data generation technique with the effectiveness of tests generated by humans. Another major issue with the technique is the number of tests that applying the technique can result in. This implies that there is a need for an automated oracle if the technique is to be successfully applied. The flaw with this is of course that in most situations the oracle is the human that is conducting the tests, a point often ignored in testing research. The work presented here addresses both of these points. To do this I have used a code base taken from an industrial engine control system that has an existing set of high quality unit tests developed by hand. To complement this, several other techniques for automatically generating test data have been applied, namely random testing, random experimental designs and a technique for generating single factor experiments. To address the issue of being able to compare the error detection ability of all of the sets of test vectors, rather than the usual effectiveness surrogates of code coverage I have used mutation analysis on the code base to directly measure the ability of each set of test vectors to discover common coding errors. The results presented here show that test data generation techniques based on t-way factorial designs are at least as effective as handgenerated tests and superior to random testing and the factor experimental technique. The oracle problem associated with the factorial design techniques was addressed using a test set minimisation approach. The mutation tool monitored which vectors could “kill” which code mutants. After a subset of the test vectors had been run, the most effective vectors were retained and the rest discarded. Likewise, mutants that were killed were removed from further consideration and the process repeated. Experimental results show that this minimisation procedure is effective at reducing computational overhead and is capable of producing final sets of test vectors that are comparable in size with the sets of hand-generated tests and so amenable to final hand checking