Recent mandates require that rigorous statistical and mathematical approached be applied to all tests that fall under developmental and operational test and evaluation (T&SE).  On October 19, 2010, J. Michael Gilmore, director of Operational Test and Evaluation, released a memorandum to the T&E community within DoD that describes an initiative designed to increase the use of scientific and statistical methods to develop rigorous methods for test and data analysis.  Dr. Gilmore's memo specifies the need for using rigorous statistical  based testing methods in order to ensure that proper and sufficient data is collected to answer the question of interest.Approved for public release; distribution is unlimited

Parker, William J, III

Sammito, Ginger

Silvestrini, Rachel T.

Calhoun, Institutional Archive of the Naval Postgraduate School

Calhoun: The NPS Institutional ArchiveFaculty and Researcher Publications Faculty and Researcher Publications Collection2012-10Design of experiments for information technology systemsSilvestrini, Rachel T.Defense Acquisition UniversityDefense AT&L, September/October 2012http://hdl.handle.net/10945/47553Design of ExInformation TecDeriments forinology SystemsAbout the Plan and Design PhasesRachel T. Silvestrini, Ph.D. m Maj. William J. Parker III m Ginger Sammitoecent mandates require thatrigorous statistical and math-ematical approaches be ap-plied to all tests that fall underevelopmental and operationalest and evaluation (T&E). OnOctober 19, 2010, J. MichaelGilmore, director of OperationalTest and Evaluation, released amemorandum to the T&E com-munity within the DoD that de-scribes an initiative designedto increase the use of scientificSilvestrini is an assistant professor in the Operations Research Department at the NavalPostgraduate School. Parker is a C4ISR systems operational test director and operationsresearch system analyst for the Homeland Security/Information Assurance Portfolio at theJoint Interoperability Test Command. Sammito is a principai operations research systemanalyst for the Force Application/Force Protection Portfolio at the Joint Interoperability TestCommand.31 Defense AT&L: September-October 2012The use of DoE ensures thaexperiment is planned in such a way thatminimizes the resources spent, whilefnaximizing the information obtained.and statistical methods to develop rigorous methods for testand data analysis. Dr. Gilmore's memo specifies the need forusing rigorous statistical based testing methods in order to en-sure that proper and sufficient data is collected to answer thequestion of interest. In addition, Edward R. Greer, the directorof Developmental Test and Evaluation, has championed theskillsets of design of experiments (DoE), statistics, and test de-sign principles in the rejuvenation and development of the T&Eworkforce as one of his top initiatives to the practice of T&E.The framework that encompasses the statistical and math-ematical approaches for T&E is called scientific based testdesign (SBTD). SBTD can be applied to all fields and applica-tion areas within the T&E realm. There is no set of T&E experi-ments in which SBTD does not apply. For example, considerthe program manager (PM) who is involved with IT systemsand feels that SBTD cannot be applied to his/her respectivesystem because the variable measures of interest in the ex-periment results in a binary outcome. In other words, did thesystem work (yes or no)? Although this is a formidable chal-lenge that must be considered prior to running the experiment,it is not a showstopper.SBTD is a framework that includes statistical based methodsfor T&E such as DoE and regression analysis. DoE is a for-mal approach for the development of a set of teststo be carried out in an experiment. An experiment isa large number of individual tests (also called trialsor runs) where variables are manipulated and datais collected.There are abundant sources of literature on DoE thatdescribe the mathematical and statistical based tac-tics for designing and analyzing the results of an ex-periment that can meet the needs of any experimen-tal goals. These methods ensure that valid, objective,and scientific conclusions are reached. Additionally,the use of DoE ensures that the experiment is plannedin such a way that minimizes the resources spent,while maximizing the information obtained. Figure 1highlights the four phases of the DoE approach: Plan,Design, Execute, and Analyze.Unlike the T&E of traditional weapons systems suchas aircraft, tanks, artillery, maritime vessels, etc.,the PM involved with IT systems testing may expe-rience slightly different challenges associated with the T&Eprocesses. However, the phases of DoE process do not changefor anyone. While this article is primarily aimed at the PMwithin T&E of IT systems, it is intended to be beneficial readingfor any PM involved with T&E in the DoD. The remainder ofthis article will briefly cover how to apply the first two phasesof DoE through an example application to an IT system. Whenappropriate, specific challenges one might encounter will behighlighted.Applying Science Based Testing DesignsThe DoE approach to the experiments conducted during theT&E process is displayed in Figure 1. The first two phases ofthis process (Plan and Design) will be discussed through anexample application to an IT system.Suppose that a PM is in charge of oversight for a new soft-ware application being developed as a test tool. The experi-ment used to test the software is called Bravo Test. DuringBravo Test different message types for multiple platformswith an Identification Friend or Foe (IFF) system are bothtransmitted and received. A DoD architecture frameworkis illustrated in Figure 2. Bravo Test will take place at thesystems level (middle view).Figure 1. Design oí Experiments (DoE) ProcessSequentially for Discoverys, Responses and Levelsstatistically to ModelPerformanceModel, Predictions, BoundsDoEProcessDesign ':Confidencein the InteroperabilityPower, Test Matrices^ ^ % Execute ^ \to Control UncertaintyRandomize, Block, ReplicateCompliments: 46th Test Wing OADefense AT&L: September-October 2012 32Phase 1: PlanThe first phase in the DoE process is Plan. This phase includesstatement of the goal of the experiment as well as the develop-ment of a list of variables involved in the experiment. Thereare three types of variables important to list:• variables that will be manipulated or controlled during theexperiment• variables that cannot be controlled, but may change dur-ing the experiment• variables used to measure the system (outcomes)The goal of Bravo Test is to test the accuracy and timelinessof messages transmitted and received. The first objective ofBravo Test is to determine whether or not each of four differ-ent platforms transmits or receives messages with accuracyrate above 99 percent. The second objective is to model theexpected time to transmit and receive a message as a func-tion of the different platforms, identification systems, and typeof message. The PM should be aware that the recognition ofthe goal and objectives in a test often aid in identifying thevariables present in the experiment.Table 1 illustrates the three different controllable variables thatwill be manipulated (changed) over the course of Bravo Test.Remember; variables that can be controlled as well as thosethat cannot be controlled should be identified. For example,during Bravo Test the average system load during the trans-mission of a message may be measurable, but it may not be avariable that is directly controllable. The PM should be eagerto identify all uncontrollable variables possible and additionallykeep in mind that it is possible that a few variables may notbe known initially, but will emerge later. This should not be afigure 2. DoD Architecture Frameworkwith Systems View in Centerstumbling point, but an opportunity for the PM to refine thetest during the next cycle with more information. This involvesgoing back to the planning phase and proceeding from there.Example Factors to be variedduring Bravo TestControllable VariablesIFF (identification. Friend,or Foe)Message typesProducing or ConsumingPlatformsSettings During TestRange 0 - 5UTF-8, UTF-16, UTF-32 (UTE =Unicode Transformation Format)A, B, C, DIn Bravo Test, there are two outcome variables: (1) accuracyof message and (2) time to transmit/receive message. Ac-curacy is a binary variable: if the message is 100 percent cor-rect, the data point will be considered 1 (accurate); otherwise0 (not accurate). In IT systems testing, a binary response isa common metric of interest. Also, many outcome variablesmay be collected for a single test within the experiment; thisis important to note and is used when assessing the quantityof tests required for the experiment.Without proper care in the Plan phase of the experiment, thedirection of the experiment may become unclear. This leads tothe collection of erroneous or incomplete information, whichwill prevent the experimental goals from being met. Often,determining the variables of interest in an experiment can bea difficult task that should be undertaken with caution. Fish-bone diagrams as well as other brainstorming techniques oftenwork well during subject matter expert meetings to discussvariable selection.The Operational View describesand interrelates the operationalelements, tasks and activities, andinformation flows required toaccomplish mission operations.The Systems View describes andinterrelates the existing or postu-lated technologies, systems, andother resources intended to supportthe operational requirements.The Technical View describes theprofile of rules, standards, andconventions governing systemsimplementation and forecasts theirfuture direction.DoD Architectural Framework (DoDAF)Phase 2: DesignThe Design Phase involves map-ping out the sets of tests that willbe conducted during the experi-ment. Specifically, this phase in-volves the selection of the designtype and the determination of thenumber of tests to be conductedin the experiment (also known assample size). Each test involvesthe control and manipulation ofvariables identified in the PlanPhase. There are a number ofdifferent experimental designtechniques found in various text-books, journal articles, technicalreports, and case studies.Examples of design selectionsinclude factorial design, frac-tional factorial design, centralcomposite design, coveringarray, and optimal design. While33 Defense AT&L: September-October 2012Figure 3. JMP—User Interface for theDevelopment oí Full Factorial Designs DOE - Full Factorial Design - JMP£ile £dit labte ßows £ol$ ßO£ ôn»lyie fireph Tflols \fiew üfmdow Help¡ " Full Factorial Design¿1 ResponsesResponse NameTmeinessLowerUn« UpptrUmI knportsncc4 FactorsConinuous v j IcatigooNameC F F¿ Message TypeC PlatformOH^II Ramovt ]RoleCategoricalCategoricalCategoricalValues0 hUTF-8A2 3UTF-16IB c4 IsUTF-32ID6x3x4 FactorialOutput OpijonsRun OrderNumber of Runs:Number of Center Points:Number of Repicates:I Make TableRandomiz« •03288Figure 4. JMP—Full Factorial 'o 6x3x4 Factorial - JMPFile Edit lables Rows ¿ols DOE Analyze Graph Tfiols[§] ? * à* -96x3x4 Fadnial >Design 6x3x4 Factorial¡Vi Screening:l5^l Model» Cokimns(6il)iL Pattern (Q¿ I F F *¿ Message Type ^A Platfonn:|C^|I8!B +• RowsAll rows 26SSelected 0exduded 0Hidden 0> y A - r l A J ^ L.1234S67gg1011121314151617161«2021222324252627262930PMtani324134S11414533S24411421622132322412224112123124434213412413521632622434513622512323611S14311122Hosaageff \ Type2 tJTF-160 UTF-324 UTF-Ä3 UTF-64 UTF-324 UTF-163 UTF-83 UTF-165 UTF-160 UTF-322 UTF-163 UTF-81 ÜTF-160 UTF-80 UTF-160 UTF-163 (nF-321 U1T-83 UTF-63 UTF-84 tnF-165 UTF-325 UTF-163 UTF-324 irTF-85 UTF-164 UTF-82 UTF-165 UTF-84 UTF-60 UTF-16raJbJe DesignView Window Help ^ ^ • • • • l 'PMform ^ ^ ^ ^ ^ ^ B TImeUnett | 1DD •A •DC •D •A "A •B •BBBDBCDDCBCABBDCBBCADE- 4 - 4LT ^•1 :•*- 1- - 1.__._. i 111•1•1' ^ ^ i^ ' ¡•1 "  1'• ' 11•! 1• *•1B • •' ^ ' lî •? a» Aa PM does not necessarily need to knoweach different design, they should recognizethat different designs are appropriate fordifferent experimental goals. For example, afractional factorial design is an appropriatedesign choice when the experimental goalinvolves finding the subset of factors thatinfluence the outcome variable of interest.This is a goal typically encountered in theearly phase of testing. For situations involv-ing multiple responses with overlapping orconflicting goals, a hybrid design approach,in which different design choices are com-bined, can be used to satisfy all objectivesof the experiment.In addition to design choice, the number oftests to run (or the sample size) of an experi-ment must be determined during this phase.Given the opportunity, a PM might prefer tochoose an unlimited sample size. However,cost, time, and resource constraints oftendrive sample size choices.For Bravo Test, a full factorial design withfour replicates is selected to support thegoals of testing the accuracy and timelinessof messages transmitted and received. Astatistical software package, such as JMP(illustrated), can be used to create the de-sign. Snapshots of the design creation areshown in Figure 3 and Figure 4. Figure 3 il-lustrates the user interface that guides theinputs to the development of the design.Figure 4 contains the design. The designdictates the running of every experimentaltest. For example, the first experimentaltest will be conducted with IFF = 2, Mes-sage Type = ÜTF-16, and Platform = D.A full factorial design is appropriate forthe needs of Bravo Test. In Bravo Test,simple relationships between IFF, Mes-sage Type and Platform will be inves-tigated. In other situations, differentdesigns may be more apt. The factorialdesign dictates a baseline number of runsin the experiment. That number can bealtered by repetition of the experiment(as seen in one of the selection tabs inFigure 3). It is important for the PM torealize that within a resource-constrainedenvironment, a single experiment cannotprovide unlimited answers. Both designchoice and sample size restrictions trans-late to restrictions on what informationcan be obtained. Statistical and math-Defense AT&L: September-October 2012 34ematical analysis can greatly help overcome sample sizedilemma by focusing on answering the following:• Given a fixed sample size, what information can be mea-sured and modeled?• Given measurement or modeling requirements, whatsample size is required?Approach (1) involves identifying risks in the constrained envi-ronment and approach (2) involves determining requirementsof sample size based on the risks the experimenter is will-ing to accept. Risks can be discussed in terms of confidencelevel and/or power of mathematical estimation. These aretwo terms related to statistical analysis that PMs should beor become familiar with.During the Design Phase, the PM should encourage documen-tation of the methodology that includes rationale for selectinga design, sample size, and lessons learned from the process.Clear documentation will help the PM face the challenges ofthe iterative DoE process and development stages as the soft-ware moves towards maturity.ConclusionSBTD methods, specifically DoE, can and should be appliedto T&E of IT systems. There are many case studies that docu-ment the success of the DoE approach for both IT and non-IT systems. This article covered the Plan and Design phasesin the DoE approach. It is believed that the Plan and Designphases are of utmost importance because an inadequatelydesigned experiment will result in poor results and possiblyincorrect conclusions, thus making the Execute and Analyzephases meaningless.The Execute Phase refers to the running of each test in theexperiment. For Bravo Test, the experiment to be run is illus-trated in Figure 4. During this phase, it is imperative that eachtest is run to specification. This involves ensuring that properblocking, randomization, and replication are carried out asspecified by the design. The Analyze Phase encompasses amathematical study of the resulting data to obtain valid andobjective conclusions.Sometimes the challenges and decisions in the creation ofan experimental design approach appear endless for the PM,especially as requirements shift from traditional testing to rig-orous SBTD for IT systems. The PM must ensure compliancewith applicable policies. The PM is also responsible for thequality and consistency to those standards while developingtest reports based on a sound, scientific rigor that have notformally been a part of any IT system/program. The PM needsto look beyond the present in facing these SBTD challengesin IT systems and focus on the valid, objective, and measure-able approach that ultimately saves time and money over thedevelopment cycle of the IT system. ^The authors can be reached at rtsilvescSnps.edu, william.j.parker60@mail.mil, and ginger.j.sainmito.civ@)mail.mil.@ DAU Alumni AssociationJOIN THE SUCCESS NETWORKThe DAU Alumni Association operis the door to a worldwide network of DefenseAcquisition University graduates, faculty, staff members, and defense industryrepresentatives—all ready to share their expertise with you and benefit from yours.Be part of a two-way exchange of information with other acquisitionprofessionals.• Stay connected to DAU and link to other professional organizations.• Keep up to date on evolving defense acquisition policies and developmentsthrough DAUAA newsletters and symposium papers.• Attend the DAUAA Annual Acquisition Community Conference/ Symposiumand earn Continuous Learning Points (CLPs) toward DoD continuing educationrequirements.Membership is open to all DAU graduates, faculty, staff, and defense industrymembers. It's easy to join, right from the DAUAA website at ivivw.dauaa.org.For more information,call 703-960-6802 or 800-755-8805, or e-mail dauaa2(at)aol.com.35 Defense AT&L: September-October 2012This content is in the Public Domain.

Design of experiments for information technology systems

http://calhoun.nps.edu/bitstream/handle/10945/47553/Silvestrini-design-c.pdf?sequence=3&isAllowed=y

Design of experiments for information technology systems

Abstract

Similar works

Full text

Available Versions

Calhoun, Institutional Archive of the Naval Postgraduate School