Symposium Presentation (for Acquisition Research Program)Symposium PresentationNaval Postgraduate School Acquisition Research ProgramApproved for public release; distribution is unlimited

Housel, Thomas J.

Kanevsky, Valery A.

Pfeiffer, Karl D.

Symposium Presentation (for Acquisition Research Program)Symposium PresentationApproved for public release; distribution is unlimited.Naval Postgraduate School Acquisition Research Progra

Calhoun, Institutional Archive of the Naval Postgraduate School

Calhoun: The NPS Institutional ArchiveReports and Technical Reports All Technical Reports Collection2010-05-12An information-theoretic approach tosoftware test-retest problemsKarl D. Pfeifferhttp://hdl.handle.net/10945/33535An information-theoretic approach to software test-retest problemsKarl D. PfeifferValery A. Kanevsky Thomas J. HouselMay 13, 20102Overview• Open architectures (OA) and reusable software components offer the promise of more rapid fielding of increments in systems development• Testing and re-testing these components requires a significant level of effort as new systems are developed and old systems are upgraded• How much testing is enough?  When can we stop testing?3MotivationHighHighLowLowKnowledge of or confidence in system operation under  load Cost of testing in budget and scheduleGood testing strategies offer the most information per unit costPoor strategies return less information for the investment in testingEven good strategies may reach a point where continued testing yields no new information4Model fundamentals• We can identify good testing strategies by constructing a simplemodel of the system, its components, and its attendant test suite• This model requires – Estimates of a prior probability of failure for modules within the system– Estimates of the coverage for each test in the suite over these modules• These estimates need not be precise to make the model useful– Monte Carlo simulation can be used to sample around the estimates as means, offering some insight into model sensitivity5Model fundamentals• This model should help answer questions like:– Given a desired level of confidence in system operation, how much testing should we accomplish?  How much will this cost?– Given a fixed budget for testing, how much confidence in system performance can we achieve through testing?– Given a particular test suite, how much information is attainable given infinite resources?6Model fundamentals• A module Mi is modeled as a unit circle with probability of being defective bi• Test Tx exercises region Aixin module Mi• In general we assume that Txmay exercise several regions across several modules• A test has two possible outcomes:‒ PASS indicates that the test did not detect a defect in any of the exercised regions within the modules tested ‒ FAIL indicates that at least one module exercised is defective, though we may not know which one7Model fundamentals• These ambiguities offer a rich framework for modeling realistic system testing scenarios– We need not execute (and pay for) Tx to forecast information returned– Within this language of expression we can formulate a quantitativeassessment of the information returned by a test sequence• Across the system of modules Mi we can measure the information returned by a test using the classic residual entropy for a distribution of probabilities:2 2log (1 ) log (1 )i i i i iiib b bh bH = − − − −=∑ ∑8Model fundamentalsAt maximum entropy we have a 50/50 chance that ourmodule is good or bad—we might as well flip a coinTesting increases the displacement from maximum entropy at h(bi=1/2) by nudging bi closer to 0 or 1, and this means increased certainty in the state of module Mi• From entropy, we derive the forecast measure:• Let cx be the cost of executing test Tx in appropriate units of time or money (or both)  A good strategy will sequence the suite of tests such that:• These ratios represent  information per unit cost  9Model fundamentals[1] [2] [ ][1] [2] [ ]) )( ( )( mmQ T Q T Q Tc c c≥ ≥L( )max( ,1 ) ( fails) + max( ,1 ) (  passes)( ) fail failx i i x i pass pa xissib b P T b b PQ TT = − −∑• Within the decision aid, for simple investigations, a fully randomized system can be created with only a few user specified constraints • If the user has a few system details but only vague insight about others, these aspects can be augmented with randomized parameters (e.g. sizes and number of coverages)• A system with well-documented interdependencies can be completely specified by the user in terms of modules, tests and coverages10Model implementation11Model implementation12Model implementation13Model implementation14Model implementation15Model implementation16Model implementation17Model implementation18Model implementation19Model implementation20Model implementationBut, what does it all mean?• Effective, cost-efficient testing is critical to the long-term success of open architecture programs• This model and prototype decision aid provide a rigorous yet tractable way ahead to improve system testing• Using this framework we can build the tools to:– Lower the testing costs for a given level of system reliability – Improve the use of existing suites for a given budget or schedule – Design better, more targeted test suites to minimize redundancy– Provide insight into the power or sensitivity of current test suites21

An information-theoretic approach to software test-retest problems

http://calhoun.nps.edu/bitstream/handle/10945/33535/NPS-AM-10-104.pdf?sequence=3&isAllowed=y

An information-theoretic approach to software test-retest problems

Abstract

Similar works

Full text

Available Versions

Calhoun, Institutional Archive of the Naval Postgraduate School